Machine learning model service testing checklist

Download the checklistMarkdownCSVBrief (Markdown)

1. 1.release smoke testsDone when: After each model or code deploy, a fixed set of reference inputs returns the stored expected scores and labels from the live endpoint.Risks: wrong-decisions. Tool class: API client. Usually run by: developer.
2. 2.data integrity and migration testingDone when: On a held-out set labelled by people, accuracy and false-positive results are recorded per group and per input quality, and the release record shows them.Risks: wrong-decisions, user-harm, regulatory-penalty. Tool class: data validation. Usually run by: QA engineer.
3. 3.exploratory testingDone when: A session log covers low-quality, edge and adversarial inputs, records each false match found and links it to a filed issue.Risks: user-harm, wrong-decisions. Tool class: test management. Usually run by: QA engineer.
4. 4.analytics and event tracking testsDone when: Production logs record each score, threshold and the action taken, and a weekly review compares a reviewed sample against the model output.Risks: wrong-decisions, user-harm. Tool class: data validation. Usually run by: product manager.
5. 5.unit testsDone when: In the test report, every missing, out-of-range and low-quality sample input gets its documented value, and every sample score comes back with at least one reason.Risks: wrong-decisions, regulatory-penalty. Tool class: unit framework. Usually run by: developer.
6. 6.regression suiteDone when: Each new model version is scored on the same held-out set as the live one, and a worse result for any group blocks the release.Risks: wrong-decisions, regulatory-penalty. Tool class: test management. Usually run by: QA engineer.
7. 7.compliance checksDone when: Each score returned for a credit or screening use carries the top reasons in plain words, the model record lists data sources and tested groups, and a sample of prediction logs holds no raw personal fields.Risks: regulatory-penalty, data-breach. Tool class: test management. Usually run by: product manager.
8. 8.security scanningDone when: The scan of the endpoint, logs and training buckets shows no public access and no raw personal fields outside approved stores.Risks: data-breach. Tool class: DAST scanner. Usually run by: security specialist.
9. 9.penetration testingDone when: A tester with one client key cannot read another client's predictions, logs or training data through the API.Risks: data-breach. Tool class: proxy. Usually run by: security specialist.