How to test a machine learning model service
Direct answer
Shipped untested, a machine learning model service returns a false match that labels a real person as a suspect, so staff confront or remove them on the alert, and gives credit or screening clients scores with no specific reasons and no accuracy test across groups. 9 tests are mandatory here, each tied to a risk in the map below. After the release smoke check, start the risk tests with data integrity and migration testing: accuracy and false-positive results per group and per input quality on a held-out set labelled by people are the record every later release is compared against.
9 mandatory testsfirst: data integrity and migration testing
What counts as a machine learning model service
A machine learning model service here is a model behind an API endpoint that returns predictions or scores to client systems, which use them to act on people. It counts when requests carry personal data, when several clients share the endpoint, or when scores feed credit, hiring, housing or identification decisions. A chat assistant that answers in free text is out of scope, and so is a model that runs only on a user's device.
Adjacent product types not covered here
- LLM-based assistant: An LLM-based assistant answers people in free text and calls tools, so prompt injection and harm to teens in companion chats lead instead of false matches and missing reason codes.
- data pipeline: A data pipeline moves and reconciles records on a schedule, while a model service turns one request into a score that someone acts on.
- lending platform: A lending platform makes and services the credit decision itself, while a model service supplies the score and must carry the reasons a lender has to give.
Risk map
| Risk | Grade | Note | Evidence | |
|---|---|---|---|---|
| Physical or psychological harm to a user user-harm | harm | primary | A false match from the model labels a real person as a suspect, and staff confront or remove them on the alert. | |
| Fine or enforcement by a regulator regulatory-penalty | legal | primary | A credit or screening client cannot give the specific reasons behind a model score, or the model was never tested for accuracy across groups. | |
| Personal or sensitive data exposed data-breach | data | secondary by design analysis | Prediction logs keep raw request payloads with personal fields, and they are copied into training sets and debugging dashboards without access limits. | |
| Users or operators act on wrong numbers wrong-decisions | trust | primary | The model flags more false matches for some groups, nobody tracks the false-positive rate after launch, and operators act on every alert. |
Mandatory tests (9)
Step 1: release smoke tests
Why:wrong-decisions
Done when: After each model or code deploy, a fixed set of reference inputs returns the stored expected scores and labels from the live endpoint.
Tool class: API client. Who usually runs it: developer.
Step 2: data integrity and migration testing
Why:wrong-decisionsuser-harmregulatory-penalty
Done when: On a held-out set labelled by people, accuracy and false-positive results are recorded per group and per input quality, and the release record shows them.
Tool class: data validation. Who usually runs it: QA engineer.
Step 3: exploratory testing
Done when: A session log covers low-quality, edge and adversarial inputs, records each false match found and links it to a filed issue.
Tool class: test management. Who usually runs it: QA engineer.
Step 4: analytics and event tracking tests
Done when: Production logs record each score, threshold and the action taken, and a weekly review compares a reviewed sample against the model output.
Tool class: data validation. Who usually runs it: product manager.
Step 5: unit tests
Why:wrong-decisionsregulatory-penalty
Done when: In the test report, every missing, out-of-range and low-quality sample input gets its documented value, and every sample score comes back with at least one reason.
Tool class: unit framework. Who usually runs it: developer.
Step 6: regression suite
Why:wrong-decisionsregulatory-penalty
Done when: Each new model version is scored on the same held-out set as the live one, and a worse result for any group blocks the release.
Tool class: test management. Who usually runs it: QA engineer.
Step 7: compliance checks
Why:regulatory-penaltydata-breach
Done when: Each score returned for a credit or screening use carries the top reasons in plain words, the model record lists data sources and tested groups, and a sample of prediction logs holds no raw personal fields.
Tool class: test management. Who usually runs it: product manager.
Step 8: security scanning
Why:data-breach
Done when: The scan of the endpoint, logs and training buckets shows no public access and no raw personal fields outside approved stores.
Tool class: DAST scanner. Who usually runs it: security specialist.
Step 9: penetration testing
Why:data-breach
Done when: A tester with one client key cannot read another client's predictions, logs or training data through the API.
Tool class: proxy. Who usually runs it: security specialist.
Recommended and optional tests
contract testsrecommended
Becomes mandatory when outside clients integrate with the endpoint.
performance and load testingrecommended
Becomes mandatory when outage-under-load evidence is attached or clients call the endpoint in real time.
resilience and failure testingrecommended
Becomes mandatory when dependency-failure evidence is attached for the feature store or model host.
integration testsoptional
Optional: catches wiring faults between feature store and model.
Not needed and why
Not needed: the service has no user interface; the release smoke covers the endpoint. Becomes recommended when a review screen for operators ships.
Not needed: no attached risk comes from users misreading screens. Becomes recommended when operators review alerts in a console.
Not needed: the service has no user interface. Becomes recommended when an operator console ships.
device and browser compatibility
Not needed: clients call one HTTP endpoint. Becomes recommended when an on-device model ships.
Order of work
First release
- release smoke tests: After the model or code deploy, send the fixed set of reference inputs to the live endpoint and compare each score and label with the stored expected one.
- data integrity and migration testing: Score the held-out set labelled by people and record accuracy and false-positive results per group and per input quality in the release record.
- exploratory testing: Probe the endpoint with low-quality, edge and adversarial inputs, log each false match found and link it to a filed issue.
- analytics and event tracking tests: Check production logs record each score, the threshold and the action taken, and compare a reviewed sample against the model output.
- unit tests: Run the unit suite on missing, out-of-range and low-quality sample inputs and confirm in the test report that each gets its documented value and every score has at least one reason.
- regression suite: Score the new model version on the same held-out set as the live one and block the release on a worse result for any group.
- compliance checks: Confirm each score returned for a credit or screening use carries its top reasons in plain words, the model record lists data sources and tested groups, and a sample of prediction logs holds no raw personal fields.
- security scanning: Scan the endpoint, logs and training buckets for public access and for raw personal fields outside approved stores.
- penetration testing: Have a tester with one client key try to read another client's predictions, logs or training data through the API.
Ongoing cadence
- release smoke tests: After each model or code deploy.
- regression suite: Each time a model is retrained or replaced.
- data integrity and migration testing: When the held-out set is relabelled or a new group or input source is added.
- analytics and event tracking tests: At each scheduled review of reviewed alerts against model output, and when the alert threshold changes.
- compliance checks: When a client starts using scores for a new credit, hiring, housing or identification use.
- penetration testing: When a new client is given a key to the shared endpoint.
- contract tests: When an outside client integrates with the endpoint.
Tools
Listed alphabetically inside each test type, filtered to the tests above and to the platforms of machine learning model service (api). Not recommendations.
release smoke tests
data integrity and migration testing
- dbt testsTests on warehouse models inside dbt. apiopen-source
- Great ExpectationsData validation with declarative expectations. apiopen-source
unit tests
regression suite
security scanning
- Burp SuiteProxy and scanner for web application security testing. webapifreemium
- GarakProbing LLM applications for prompt injection and leakage. apiopen-source
- mitmproxyInteractive HTTPS proxy. iosandroidwebapiopen-source
- OWASP ZAPDynamic application security scanner. webapiopen-source
- SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source
- SemgrepStatic analysis with custom rules. apiwebiosandroidfreemium
- SnykDependency and code vulnerability scanning. apiwebiosandroidfreemium
- TrivyScanner for containers, dependencies and configuration. apiopen-source
penetration testing
- Burp SuiteProxy and scanner for web application security testing. webapifreemium
contract tests
- PactConsumer-driven contract testing with a broker. apiopen-source
- PostmanAPI client with collections and scripted assertions. apifreemium
- SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source
performance and load testing
- ArtilleryLoad testing with YAML scenarios. apiwebfreemium
- GatlingLoad testing with Scala, Java or Kotlin scenarios. apiwebfreemium
- JMeterLoad testing tool from the Apache Foundation. apiwebopen-source
- k6Load testing scripted in JavaScript. apiwebopen-source
- LocustLoad testing scripted in Python. apiwebopen-source
resilience and failure testing
- Chaos MeshFault injection for Kubernetes. apiopen-source
- GremlinManaged fault injection. apicommercial
- ToxiproxyTCP proxy for simulating network failures. apiopen-source
What to send to the person who will test
Send the checklist and the brief below. The printable checklist for machine learning model service has the same 9 items with checkboxes.
Download the checklistMarkdownCSVBrief (Markdown)
Acceptance criteria wording
A release of the machine learning model service is accepted when the reference inputs return the stored expected scores from the live endpoint and the new model version scores no worse than the live one for any group on the held-out set. Every score for a credit or screening use carries its top reasons in plain words, and a tester with one client key cannot read another client's predictions or logs.
Brief
# Test brief: machine learning model service Product type: machine learning model service. A model behind an endpoint that returns predictions or scores. Risk map and evidence: https://howtotestaproduct.com/how-to-test/ml-model-service/ Printable checklist: https://howtotestaproduct.com/checklists/ml-model-service/ Run these 9 mandatory tests in this order. A test is done only when its done condition is true on the build you will release. 1. release smoke tests (risks: wrong-decisions). Tool class: API client. Usually run by: developer. Done when: After each model or code deploy, a fixed set of reference inputs returns the stored expected scores and labels from the live endpoint. 2. data integrity and migration testing (risks: wrong-decisions, user-harm, regulatory-penalty). Tool class: data validation. Usually run by: QA engineer. Done when: On a held-out set labelled by people, accuracy and false-positive results are recorded per group and per input quality, and the release record shows them. 3. exploratory testing (risks: user-harm, wrong-decisions). Tool class: test management. Usually run by: QA engineer. Done when: A session log covers low-quality, edge and adversarial inputs, records each false match found and links it to a filed issue. 4. analytics and event tracking tests (risks: wrong-decisions, user-harm). Tool class: data validation. Usually run by: product manager. Done when: Production logs record each score, threshold and the action taken, and a weekly review compares a reviewed sample against the model output. 5. unit tests (risks: wrong-decisions, regulatory-penalty). Tool class: unit framework. Usually run by: developer. Done when: In the test report, every missing, out-of-range and low-quality sample input gets its documented value, and every sample score comes back with at least one reason. 6. regression suite (risks: wrong-decisions, regulatory-penalty). Tool class: test management. Usually run by: QA engineer. Done when: Each new model version is scored on the same held-out set as the live one, and a worse result for any group blocks the release. 7. compliance checks (risks: regulatory-penalty, data-breach). Tool class: test management. Usually run by: product manager. Done when: Each score returned for a credit or screening use carries the top reasons in plain words, the model record lists data sources and tested groups, and a sample of prediction logs holds no raw personal fields. 8. security scanning (risks: data-breach). Tool class: DAST scanner. Usually run by: security specialist. Done when: The scan of the endpoint, logs and training buckets shows no public access and no raw personal fields outside approved stores. 9. penetration testing (risks: data-breach). Tool class: proxy. Usually run by: security specialist. Done when: A tester with one client key cannot read another client's predictions, logs or training data through the API. Acceptance criteria wording: A release of the machine learning model service is accepted when the reference inputs return the stored expected scores from the live endpoint and the new model version scores no worse than the live one for any group on the held-out set. Every score for a credit or screening use carries its top reasons in plain words, and a tester with one client key cannot read another client's predictions or logs. Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file). Licence: CC BY 4.0.
For founders: what to forward and what to ask back. For product managers: turning this page into tickets.
Questions about testing a machine learning model service
What testing does a machine learning model service need?
9 tests are mandatory, starting with a reference-input smoke run after each deploy and per-group accuracy and false-positive results on a held-out set labelled by people. Each is tied to one of the 4 risks in the map: false matches that harm a person, more false matches for some groups with no tracking after launch, scores with no reasons or no group testing, and personal fields kept in prediction logs. Contract, performance and resilience tests are recommended, each with the event that makes it mandatory.
How are the answers of a machine learning model service evaluated?
Each model version is scored on a held-out set labelled by people, with accuracy and false-positive results recorded per group and per input quality, and a worse result for any group blocks the release. Exploratory sessions add low-quality, edge and adversarial inputs and file each false match. In production, logs record each score, threshold and action so a reviewed sample can be compared against the model output.
What does GDPR mean for testing a machine learning model service?
The map cites the General Data Protection Regulation for the by design analysis data breach risk, where prediction logs keep raw payloads with personal fields and are copied into training sets without access limits. Compliance checks a sample of prediction logs for raw personal fields, security scanning checks the endpoint, logs and training buckets, and penetration testing checks one client cannot read another's logs.
Which tests in a vendor test plan for a machine learning model service are not needed?
End-to-end UI, usability and accessibility testing are not needed while the service has no user interface, until an operator review console ships. Compatibility testing is not needed while clients call one HTTP endpoint, until an on-device model ships.
What goes on an API testing checklist for a machine learning model service?
Send the fixed reference inputs to the live endpoint after each deploy and compare scores and labels with stored values. Check each credit or screening score carries its top reasons, and have a tester with one client key try to read another client's predictions. Request and response schema checks become mandatory when outside clients integrate.
Evidence
ev-0288regulatorprimary
Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards, Federal Trade Commission
Test, assess, measure, document, or inquire about the accuracy of its facial recognition technology before deploying it
Regulator order: a face-matching model flagged shoppers as suspects with more false positives in some communities, nobody tracked the false-positive rate, and employees acting on false-positive alerts followed, searched or removed shoppers. Supports wrong-decisions, user-harm and regulatory-penalty for model services.
published 2023-12-19 · verified 2026-10-07
ev-0030regulatorprimary
Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex, European Union
The EU AI Act sets obligations by risk level, including accuracy, resilience and transparency duties for high-risk AI systems and disclosure duties for chatbots.
published undated · verified 2026-10-07
ev-0289regulatorprimary
Consumer Financial Protection Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms, Consumer Financial Protection Bureau
Creditors who use complex algorithms, including artificial intelligence or machine learning, in any aspect of their credit decisions must still provide a notice that discloses the specific principal reasons for taking an adverse action.
Regulator rule interpretation: a lender using a model score must still give specific reasons for each denial. Supports regulatory-penalty for model services whose scores feed credit decisions.
published 2022-05-26 · verified 2026-10-07
ev-0025regulatorprimary
Regulation (EU) 2016/679 (General Data Protection Regulation), EUR-Lex, European Union
The GDPR sets lawful basis, consent, data subject rights, security of processing (Article 32) and breach notification duties for anyone processing personal data of people in the EU.
published undated · verified 2026-10-07
ev-0029standardprimary
AI Risk Management Framework, National Institute of Standards and Technology
The NIST AI RMF describes how to map, measure and manage risks of AI systems, including validity, reliability and harmful outputs.
published undated · verified 2026-10-07
Disclaimer
This page lists tests and risks for a product category, not for your product. Risks are attached only with the evidence shown in the Evidence section. Tools are listed alphabetically and are not recommendations. Nothing here is legal, regulatory or security advice; where a regulator or a store sets a requirement, their page is linked and takes precedence.