# Test brief: machine learning model service

Product type: machine learning model service. A model behind an endpoint that returns predictions or scores.
Risk map and evidence: https://howtotestaproduct.com/how-to-test/ml-model-service/
Printable checklist: https://howtotestaproduct.com/checklists/ml-model-service/

Run these 9 mandatory tests in this order. A test is done only when its done condition is true on the build you will release.

1. release smoke tests (risks: wrong-decisions). Tool class: API client. Usually run by: developer.
   Done when: After each model or code deploy, a fixed set of reference inputs returns the stored expected scores and labels from the live endpoint.
2. data integrity and migration testing (risks: wrong-decisions, user-harm, regulatory-penalty). Tool class: data validation. Usually run by: QA engineer.
   Done when: On a held-out set labelled by people, accuracy and false-positive results are recorded per group and per input quality, and the release record shows them.
3. exploratory testing (risks: user-harm, wrong-decisions). Tool class: test management. Usually run by: QA engineer.
   Done when: A session log covers low-quality, edge and adversarial inputs, records each false match found and links it to a filed issue.
4. analytics and event tracking tests (risks: wrong-decisions, user-harm). Tool class: data validation. Usually run by: product manager.
   Done when: Production logs record each score, threshold and the action taken, and a weekly review compares a reviewed sample against the model output.
5. unit tests (risks: wrong-decisions, regulatory-penalty). Tool class: unit framework. Usually run by: developer.
   Done when: In the test report, every missing, out-of-range and low-quality sample input gets its documented value, and every sample score comes back with at least one reason.
6. regression suite (risks: wrong-decisions, regulatory-penalty). Tool class: test management. Usually run by: QA engineer.
   Done when: Each new model version is scored on the same held-out set as the live one, and a worse result for any group blocks the release.
7. compliance checks (risks: regulatory-penalty, data-breach). Tool class: test management. Usually run by: product manager.
   Done when: Each score returned for a credit or screening use carries the top reasons in plain words, the model record lists data sources and tested groups, and a sample of prediction logs holds no raw personal fields.
8. security scanning (risks: data-breach). Tool class: DAST scanner. Usually run by: security specialist.
   Done when: The scan of the endpoint, logs and training buckets shows no public access and no raw personal fields outside approved stores.
9. penetration testing (risks: data-breach). Tool class: proxy. Usually run by: security specialist.
   Done when: A tester with one client key cannot read another client's predictions, logs or training data through the API.

Acceptance criteria wording:
A release of the machine learning model service is accepted when the reference inputs return the stored expected scores from the live endpoint and the new model version scores no worse than the live one for any group on the held-out set. Every score for a credit or screening use carries its top reasons in plain words, and a tester with one client key cannot read another client's predictions or logs.

Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file).

Licence: CC BY 4.0.