How to test an LLM-based assistant

Direct answer

Shipped untested, an LLM-based assistant follows instructions hidden in an uploaded file or retrieved web page and calls a tool the user never asked for, such as sending an email, and gives a child or teenager who treats it as a companion answers unsuitable for their age that the operator never tested. 9 tests are mandatory here, each tied to a risk in the map below. After the release smoke check, start the risk tests with exploratory testing: policy questions, crisis phrases and off-topic requests show what the assistant says in free text before any fixed question set exists.

9 mandatory testsfirst: exploratory testing

What counts as an LLM-based assistant

An LLM-based assistant here is a chat or agent product built on a language model with tools or retrieval, offered on web, iOS, Android and through an API, that stores chat history per user. It counts when it answers the public in free text, reads uploaded files or retrieved pages, calls tools, or runs on a hosted model provider. A model endpoint that returns only a score is out of scope, and so is a scripted chatbot with fixed answer trees and no language model.

Adjacent product types not covered here

  • machine learning model service: A machine learning model service returns a score to a client system, so per-group accuracy and reason codes lead instead of prompt injection and harm to teens in companion chats.
  • data pipeline: A data pipeline moves records with no user conversation, so dropped rows and reconciliation lead instead of answers that bind the business.
  • messaging app: A messaging app carries chats between people, while an assistant writes the replies itself and can act through tools on the user's behalf.

Risk map

7 risks attached to LLM-based assistant with evidence. Sorted by severity.
RiskGradeNoteEvidence
Physical or psychological harm to a user
user-harm
harmprimaryA child or teenager treats the assistant as a companion and gets answers unsuitable for their age, and the operator has never tested what it says in those chats.
Fraud or abuse by users
fraud-abuse
moneyprimaryInstructions hidden in an uploaded file or retrieved web page make the assistant call a tool the user never asked for, such as sending an email.
Fine or enforcement by a regulator
regulatory-penalty
legalprimaryThe assistant answers children with no age gate, and prompts are kept for model training without a stated legal basis; or the assistant does not tell users they are talking to an AI, or is sold with quality claims nobody tested.
Personal or sensitive data exposed
data-breach
dataprimaryA fault in the chat history layer exposes users' conversations and subscribers' payment details.
Third-party dependency fails
dependency-failure
outageprimaryThe model provider's API times out or silently returns degraded answers, and the assistant shows a spinner or a confident wrong reply with no fallback message.
Outage during a demand peak
outage-under-load
outageprimaryOne account scripts long prompts in a loop and uses up the shared token budget and rate limit, so other users' chats stop answering.
Users or operators act on wrong numbers
wrong-decisions
trustprimaryThe support assistant states a refund rule that the policy page contradicts, and the company is held to the chatbot's answer.

Mandatory tests (9)

  1. Step 1: release smoke tests

    Why:dependency-failurewrong-decisions

    Done when: On the release build, a signed-in user sends a prompt, gets an answer, triggers one tool call, finds the chat saved in history, and one fixed policy question returns its approved answer.

    Tool class: browser automation. Who usually runs it: QA engineer.

  2. Step 2: exploratory testing

    Why:wrong-decisionsuser-harm

    Done when: A session log covers policy questions, crisis phrases and off-topic requests, and records each answer next to the approved policy text.

    Tool class: test management. Who usually runs it: QA engineer.

  3. Step 3: compliance checks

    Why:regulatory-penaltyuser-harm

    Done when: A signed review shows the age gate, the training-data opt-out and the AI disclosure, with a screenshot of each from the release build.

    Tool class: test management. Who usually runs it: product manager.

  4. Step 4: end-to-end functional tests

    Why:user-harm

    Done when: A user who types a self-harm phrase gets the crisis message and helpline link in the same turn, on web and both mobile apps.

    Tool class: browser automation. Who usually runs it: QA engineer.

  5. Step 5: penetration testing

    Why:fraud-abusedata-breach

    Done when: A tester planting instructions in an uploaded file or a retrieved page cannot make the assistant call a tool, reveal its system prompt or show another user's data.

    Tool class: proxy. Who usually runs it: security specialist.

  6. Step 6: security scanning

    Why:data-breach

    Done when: The scan of the chat API and history store shows no open high-severity finding, and one account's conversation IDs return nothing from another account's session.

    Tool class: DAST scanner. Who usually runs it: security specialist.

  7. Step 7: performance and load testing

    Why:outage-under-load

    Done when: One account sending prompts in a loop hits its own limit, and other accounts keep getting answers at their usual speed.

    Tool class: load generator. Who usually runs it: developer.

  8. Step 8: resilience and failure testing

    Why:dependency-failure

    Done when: With the model API returning errors or timeouts, the user sees a retry message in the chat and no half-written answer is saved as final.

    Tool class: chaos tool. Who usually runs it: developer.

  9. Step 9: regression suite

    Why:wrong-decisions

    Done when: A fixed question set with approved answers is rerun after every prompt, model or retrieval change, and a named owner reviews every changed answer.

    Tool class: test management. Who usually runs it: QA engineer.

Recommended and optional tests

  • integration testsrecommended

    Becomes mandatory when the assistant can call tools that write data or send messages.

  • analytics and event tracking testsrecommended

    Becomes mandatory when a third-party analytics SDK runs on the chat screen.

  • contract testsrecommended

    Becomes mandatory when more than one model provider or model version sits behind the assistant.

  • accessibility testingrecommended

    Becomes mandatory when the assistant replaces a public service or support channel covered by accessibility law.

  • unit testsoptional

    May help: catches template and parsing breaks without calling the model.

  • usability testingoptional

    May help: users who cannot find the report button leave wrong answers unflagged.

  • device and browser compatibilityoptional

    May help: streaming output and file upload behave differently across mobile browsers.

Not needed and why

  • localization testing

    Not needed: no evidence of harm tied to UI language here. Becomes recommended when the assistant is offered in a second language with its own approved answers.

  • offline and poor-network testing

    Not needed: the assistant cannot answer without a network, and no lost-message evidence is recorded. Becomes recommended when drafts or voice notes are queued offline.

Order of work

First release

  1. release smoke tests: On the release build, sign in, send a prompt, get an answer, trigger one tool call, find the chat saved in history, and ask one fixed policy question to confirm it returns its approved answer.
  2. exploratory testing: Ask policy questions, type crisis phrases and send off-topic requests, and log each answer next to the approved policy text.
  3. compliance checks: Screenshot the age gate, the training-data opt-out and the AI disclosure from the release build and attach them to a signed review.
  4. end-to-end functional tests: Type a self-harm phrase on web and both mobile apps and confirm the crisis message and helpline link appear in the same turn.
  5. penetration testing: Have a tester plant instructions in an uploaded file and a retrieved page and try to make the assistant call a tool, reveal its system prompt or show another user's data.
  6. security scanning: Scan the chat API and history store, and request one account's conversation IDs from another account's session.
  7. performance and load testing: Send prompts in a loop from one account until it hits its own limit, and confirm other accounts keep getting answers at their usual speed.
  8. resilience and failure testing: Make the model API return errors and timeouts and confirm the user sees a retry message and no half-written answer is saved as final.
  9. regression suite: Rerun the fixed question set with approved answers and have a named owner review every changed answer.

Ongoing cadence

Tools

Listed alphabetically inside each test type, filtered to the tests above and to the platforms of LLM-based assistant (web, api, ios, android). Not recommendations.

48 tools shown

release smoke tests

  • Firebase Test LabRuns app tests on hosted devices. iosandroidfreemium
  • TestRailTest case and run management. webiosandroidapicommercial
  • XrayTest management inside Jira. webiosandroidapicommercial

end-to-end functional tests

  • AppiumCross-platform mobile UI automation over WebDriver. iosandroidopen-source
  • ApplitoolsVisual comparison of screens across builds. webiosandroidcommercial
  • BrowserStackCloud of real devices and browsers. webiosandroidcommercial
  • CypressBrowser-based end-to-end testing with a visual runner. webfreemium
  • DeepEvalUnit-style evaluation for LLM applications. apiopen-source
  • DetoxGray-box end-to-end testing for React Native apps. iosandroidopen-source
  • EspressoAndroid UI testing framework from Google. androidopen-source
  • LambdaTestCloud browser and device grid. webiosandroidcommercial
  • MaestroMobile UI flows written in YAML. iosandroidfreemium
  • PercyVisual review of UI changes. webcommercial
  • PlaywrightBrowser automation for Chromium, Firefox and WebKit. webopen-source
  • PostmanAPI client with collections and scripted assertions. apifreemium
  • PromptfooEvaluation and regression tests for LLM prompts and outputs. apiwebopen-source
  • Sauce LabsCloud test infrastructure for web and mobile. webiosandroidcommercial
  • SeleniumWebDriver-based browser automation. webopen-source
  • XCTestApple's framework for unit and UI tests. iosopen-source

penetration testing

  • Burp SuiteProxy and scanner for web application security testing. webapifreemium

security scanning

  • Burp SuiteProxy and scanner for web application security testing. webapifreemium
  • GarakProbing LLM applications for prompt injection and leakage. apiopen-source
  • mitmproxyInteractive HTTPS proxy. iosandroidwebapiopen-source
  • MobSFStatic and dynamic analysis for mobile binaries. iosandroidopen-source
  • OWASP ZAPDynamic application security scanner. webapiopen-source
  • SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source
  • SemgrepStatic analysis with custom rules. apiwebiosandroidfreemium
  • SnykDependency and code vulnerability scanning. apiwebiosandroidfreemium
  • TrivyScanner for containers, dependencies and configuration. apiopen-source

performance and load testing

  • ArtilleryLoad testing with YAML scenarios. apiwebfreemium
  • GatlingLoad testing with Scala, Java or Kotlin scenarios. apiwebfreemium
  • JMeterLoad testing tool from the Apache Foundation. apiwebopen-source
  • k6Load testing scripted in JavaScript. apiwebopen-source
  • LighthouseAudits for accessibility and performance in Chrome. webopen-source
  • LocustLoad testing scripted in Python. apiwebopen-source

resilience and failure testing

  • Chaos MeshFault injection for Kubernetes. apiopen-source
  • GremlinManaged fault injection. apicommercial
  • ToxiproxyTCP proxy for simulating network failures. apiopen-source

regression suite

  • CypressBrowser-based end-to-end testing with a visual runner. webfreemium
  • DeepEvalUnit-style evaluation for LLM applications. apiopen-source
  • PlaywrightBrowser automation for Chromium, Firefox and WebKit. webopen-source
  • PromptfooEvaluation and regression tests for LLM prompts and outputs. apiwebopen-source
  • TestRailTest case and run management. webiosandroidapicommercial
  • XrayTest management inside Jira. webiosandroidapicommercial

integration tests

  • Charles ProxyHTTP proxy for inspecting and throttling traffic. iosandroidwebcommercial
  • JestJavaScript test runner with mocking and snapshots. webapiopen-source
  • JUnitTest framework for the JVM. apiandroidopen-source
  • PostmanAPI client with collections and scripted assertions. apifreemium
  • pytestTest framework for Python with fixtures and plugins. apiopen-source
  • VitestTest runner for Vite-based JavaScript and TypeScript projects. webapiopen-source
  • XCTestApple's framework for unit and UI tests. iosopen-source

analytics and event tracking tests

  • AvoTracking plan and event validation. webiosandroidfreemium
  • Segment ProtocolsTracking plan validation for Segment events. webiosandroidcommercial

contract tests

  • PactConsumer-driven contract testing with a broker. apiopen-source
  • PostmanAPI client with collections and scripted assertions. apifreemium
  • SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source

accessibility testing

  • Accessibility ScannerGoogle's on-device accessibility checker for Android. androidopen-source
  • axe-coreAccessibility rules engine for automated checks. webopen-source
  • LighthouseAudits for accessibility and performance in Chrome. webopen-source
  • Pa11yCommand-line accessibility testing. webopen-source

What to send to the person who will test

Send the checklist and the brief below. The printable checklist for LLM-based assistant has the same 9 items with checkboxes.

Download the checklistMarkdownCSVBrief (Markdown)

Acceptance criteria wording

A release of the LLM-based assistant is accepted when a signed-in user can send a prompt, get an answer, trigger one tool call and find the chat in history, and the fixed question set returns its approved answers with every changed answer reviewed by a named owner. A self-harm phrase brings the crisis message and helpline link in the same turn, and instructions planted in an uploaded file cannot make the assistant call a tool or show another user's data.

Brief

# Test brief: LLM-based assistant

Product type: LLM-based assistant. A chat or agent product built on a language model with tools or retrieval.
Risk map and evidence: https://howtotestaproduct.com/how-to-test/llm-assistant/
Printable checklist: https://howtotestaproduct.com/checklists/llm-assistant/

Run these 9 mandatory tests in this order. A test is done only when its done condition is true on the build you will release.

1. release smoke tests (risks: dependency-failure, wrong-decisions). Tool class: browser automation. Usually run by: QA engineer.
   Done when: On the release build, a signed-in user sends a prompt, gets an answer, triggers one tool call, finds the chat saved in history, and one fixed policy question returns its approved answer.
2. exploratory testing (risks: wrong-decisions, user-harm). Tool class: test management. Usually run by: QA engineer.
   Done when: A session log covers policy questions, crisis phrases and off-topic requests, and records each answer next to the approved policy text.
3. compliance checks (risks: regulatory-penalty, user-harm). Tool class: test management. Usually run by: product manager.
   Done when: A signed review shows the age gate, the training-data opt-out and the AI disclosure, with a screenshot of each from the release build.
4. end-to-end functional tests (risks: user-harm). Tool class: browser automation. Usually run by: QA engineer.
   Done when: A user who types a self-harm phrase gets the crisis message and helpline link in the same turn, on web and both mobile apps.
5. penetration testing (risks: fraud-abuse, data-breach). Tool class: proxy. Usually run by: security specialist.
   Done when: A tester planting instructions in an uploaded file or a retrieved page cannot make the assistant call a tool, reveal its system prompt or show another user's data.
6. security scanning (risks: data-breach). Tool class: DAST scanner. Usually run by: security specialist.
   Done when: The scan of the chat API and history store shows no open high-severity finding, and one account's conversation IDs return nothing from another account's session.
7. performance and load testing (risks: outage-under-load). Tool class: load generator. Usually run by: developer.
   Done when: One account sending prompts in a loop hits its own limit, and other accounts keep getting answers at their usual speed.
8. resilience and failure testing (risks: dependency-failure). Tool class: chaos tool. Usually run by: developer.
   Done when: With the model API returning errors or timeouts, the user sees a retry message in the chat and no half-written answer is saved as final.
9. regression suite (risks: wrong-decisions). Tool class: test management. Usually run by: QA engineer.
   Done when: A fixed question set with approved answers is rerun after every prompt, model or retrieval change, and a named owner reviews every changed answer.

Acceptance criteria wording:
A release of the LLM-based assistant is accepted when a signed-in user can send a prompt, get an answer, trigger one tool call and find the chat in history, and the fixed question set returns its approved answers with every changed answer reviewed by a named owner. A self-harm phrase brings the crisis message and helpline link in the same turn, and instructions planted in an uploaded file cannot make the assistant call a tool or show another user's data.

Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file).

Licence: CC BY 4.0.

For founders: what to forward and what to ask back. For product managers: turning this page into tickets.

Questions about testing an LLM-based assistant

What testing does an LLM-based assistant need?

9 tests are mandatory, starting with a prompt, tool call and history smoke run and exploratory sessions on policy questions and crisis phrases. Each is tied to one of the 7 risks in the map, from prompt injection that triggers tools and harm to teens in companion chats to policy answers the business is held to and one account using up the shared token budget. Integration, analytics, contract and accessibility tests are recommended, each with the event that makes it mandatory.

How are the answers of an LLM-based assistant evaluated?

A fixed question set with approved answers is rerun after every prompt, model or retrieval change, and a named owner reviews every changed answer. Exploratory sessions log answers to policy questions, crisis phrases and off-topic requests next to the approved policy text. The map attaches this to the risk that the assistant states a refund rule the policy page contradicts and the company is held to it.

What is the minimum testing for an LLM-based assistant at MVP stage?

The minimum is the 9 mandatory tests once the assistant answers the public in free text, reads files or calls tools, stores chat history per user and runs on a hosted model provider. Unit, usability and compatibility testing are optional, and localization and offline tests that vendors often add are not needed until the conditions on the page change.

Which tests in a vendor test plan for an LLM-based assistant are not needed?

Localization testing is not needed while no harm is tied to UI language, until the assistant is offered in a second language with its own approved answers. Offline testing is not needed because the assistant cannot answer without a network, until drafts or voice notes are queued offline.

What goes on an API testing checklist for an LLM-based assistant?

Request one account's conversation IDs from another account's session through the chat API. Make the model API return errors and timeouts and check no half-written answer is saved as final. Loop prompts from one account until it hits its own limit while other accounts keep getting answers.

Evidence

  1. ev-0269regulatorprimary

    Artificial intelligence: stop to ChatGPT by the Italian SA, Garante per la protezione dei dati personali

    A data breach affecting ChatGPT users’ conversations and information on payments by subscribers to the service had been reported on 20 March.

    The Italian data protection authority limited a chat assistant over a breach of user conversations, no legal basis for training on personal data, inaccurate personal data in answers and no age check for children.

    published 2023-03-31 · verified 2026-10-07

  2. ev-0271regulatorprimary

    FTC Launches Inquiry into AI Chatbots Acting as Companions, U.S. Federal Trade Commission

    The FTC opened an inquiry into how chatbot operators test and monitor negative effects on children and teens who treat the assistant as a friend or confidant.

    published 2025-09-11 · verified 2026-10-07

  3. ev-0272standardprimary

    LLM01:2025 Prompt Injection, OWASP Gen AI Security Project

    The OWASP entry describes direct and indirect prompt injection, where instructions in user input or retrieved content change what the assistant does, including tool calls.

    published undated · verified 2026-10-07

  4. ev-0274standardprimary

    LLM06:2025 Excessive Agency, OWASP Gen AI Security Project

    Excessive Agency is the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction.

    Applies to assistants that call tools or act for the user: a manipulated or wrong model output becomes an action with real effect.

    published undated · verified 2026-10-07

  5. ev-0030regulatorprimary

    Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex, European Union

    The EU AI Act sets obligations by risk level, including accuracy, resilience and transparency duties for high-risk AI systems and disclosure duties for chatbots.

    published undated · verified 2026-10-07

  6. ev-0270regulatorprimary

    FTC Announces Crackdown on Deceptive AI Claims and Schemes, U.S. Federal Trade Commission

    The complaint alleges that the company did not conduct testing to determine whether its AI chatbot’s output was equal to the level of a human lawyer, and that the company itself did not hire or retain any attorneys.

    The FTC acted against a chatbot marketed as a substitute for a lawyer that had not been tested against that claim, so untested quality claims about assistant output are an enforcement risk.

    published 2024-09-25 · verified 2026-10-07

  7. ev-0273standardprimary

    LLM02:2025 Sensitive Information Disclosure, OWASP Gen AI Security Project

    The OWASP entry lists personal data, credentials and confidential business data exposed through LLM application output as a top risk for assistants.

    published undated · verified 2026-10-07

  8. ev-0277post-mortemprimary

    API, ChatGPT & Sora Facing Issues (incident of December 11, 2024), OpenAI

    The model provider reports an outage of its API caused by a telemetry deployment that overwhelmed its cluster control plane, so assistants built on that API lost service.

    published 2024-12-11 · verified 2026-10-07

  9. ev-0278post-mortemprimary

    A postmortem of three recent issues, Anthropic

    This is a technical report on three bugs that intermittently degraded responses from Claude.

    The model provider reports infrastructure bugs that degraded answer quality for some requests without an outage, so an assistant built on a hosted model can give worse answers with no error to detect.

    published 2025-09-17 · verified 2026-10-07

  10. ev-0276standardprimary

    LLM10:2025 Unbounded Consumption, OWASP Gen AI Security Project

    The OWASP entry describes high-volume and oversized requests that degrade service availability and run up per-token cost (denial of wallet) for the operator of an LLM application.

    published undated · verified 2026-10-07

  11. ev-0268court-or-fineprimary

    Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia

    It makes no difference whether the information comes from a static page or a chatbot.

    The tribunal held an airline liable for a bereavement-fare rule its website chatbot stated wrongly, so answers an LLM assistant gives customers bind the business that deploys it.

    published 2024-02-14 · verified 2026-10-07

  12. ev-0275standardprimary

    LLM09:2025 Misinformation, OWASP Gen AI Security Project

    The OWASP entry treats factual inaccuracies in model output that users act on as a top risk, citing an airline chatbot that gave travellers wrong policy information.

    published undated · verified 2026-10-07

Disclaimer

This page lists tests and risks for a product category, not for your product. Risks are attached only with the evidence shown in the Evidence section. Tools are listed alphabetically and are not recommendations. Nothing here is legal, regulatory or security advice; where a regulator or a store sets a requirement, their page is linked and takes precedence.