Users or operators act on wrong numbers
Dashboards, scores or model outputs are wrong and nobody notices.
Severity: trust. Attached to 10 of 46 published product types, backed by 14 evidence records.
Product types that carry it
| Product type | Grade | Note for this type | Evidence |
|---|---|---|---|
| lending platform | primary | The underwriting model declines an applicant for one reason while the adverse action notice lists different reasons drawn from a static template. | |
| stock trading app | practitioner single incident, no second source | The portfolio screen shows customers a negative cash balance or buying power that is wrong. | |
| insurance app | primary | A claims triage or pricing model drifts after a data change, and adjusters keep approving or denying claims on its scores without noticing. | |
| electronic health record system | secondary | After go-live the quality and patient safety reports that accreditation depends on cannot be built from the new record data, and managers run the ward on gaps. | |
| ride-hailing app | primary | A fraud score or a run of low rider ratings deactivates a driver automatically, cutting their income with no human review of the flag. | |
| ERP system | secondary | Finance cannot produce auditable statements from the new ledger, and its reports rest on inaccurate postings. | |
| data pipeline | primary | A dashboard fed by the pipeline publishes a daily count missing every row the load step dropped, or reports rest on data whose known quality problems were never remediated. | |
| machine learning model service | primary | The model flags more false matches for some groups, nobody tracks the false-positive rate after launch, and operators act on every alert. | |
| LLM-based assistant | primary | The support assistant states a refund rule that the policy page contradicts, and the company is held to the chatbot's answer. | |
| EV charging app | primary | The map shows a charger as available while it reports out of order, and a driver low on range detours to a dead bay. |
Tests that address it
- end-to-end functional tests: mandatory for this risk on 2 product types.
- unit tests: mandatory for this risk on 2 product types.
- data integrity and migration testing: mandatory for this risk on 7 product types.
- compliance checks: mandatory for this risk on 2 product types.
- release smoke tests: mandatory for this risk on 3 product types.
- regression suite: mandatory for this risk on 3 product types.
- analytics and event tracking tests: mandatory for this risk on 2 product types.
- exploratory testing: mandatory for this risk on 2 product types.
Evidence
ev-0029standardprimary
AI Risk Management Framework, National Institute of Standards and Technology
The NIST AI RMF describes how to map, measure and manage risks of AI systems, including validity, reliability and harmful outputs.
published undated · verified 2026-10-07
ev-0079regulatorsecondary
Deficits with Metrics Following Implementation of the New Electronic Health Record at the Mann-Grandstaff VA Medical Center in Spokane, Washington, U.S. Department of Veterans Affairs, Office of Inspector General
One year after go-live, gaps existed between required and available metrics using new EHR data.
Government inspection report: after an EHR go-live, quality, patient safety and accreditation metrics could not be produced from the new data. Supports wrong-decisions for EHR systems whose reports feed quality and safety oversight.
published 2022-06-01 · verified 2026-10-07
ev-0097standardprimary
NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers, National Association of Insurance Commissioners
These standards require, at a minimum, that decisions made by Insurers are not inaccurate, arbitrary, capricious, or unfairly discriminatory.
Supports wrong-decisions for an insurance app: insurers using AI or predictive models in underwriting, pricing or claims are expected to test for errors, bias and model drift, and adopting states hold them to it.
published 2023-12-04 · verified 2026-10-07
ev-0100regulatorprimary
CFPB Acts to Protect the Public from Black-Box Credit Models Using Complex Algorithms, Consumer Financial Protection Bureau
Supports wrong-decisions and regulatory-penalty for a lending platform: creditors using algorithmic credit decisions must still give declined applicants the specific, accurate reasons, so the decision engine and the notice must agree.
published 2022-05-26 · verified 2026-10-07
ev-0159regulatorprimary
Uber fined nearly [amount] for automated driver blocking, Autoriteit Persoonsgegevens
If that software detected a suspicion of fraud or customer reviews were too low, the accounts of the drivers concerned were automatically deactivated.
GDPR fine because fraud signals and low ratings deactivated driver accounts with no human review. Supports wrong-decisions and regulatory-penalty for ride-hailing apps that automate driver deactivation.
published 2026-08-21 · verified 2026-10-07
ev-0181court-or-fineprimary
Robinhood Financial LLC Letter of Acceptance, Waiver and Consent, FINRA
During the March outage, all of Robinhood's customers were unable to trade.
Supports outage-under-load, wrong-decisions and regulatory-penalty for a stock trading app: the regulator found a retail brokerage app went down during the March 2020 market volatility, showed customers inflated negative cash balances and approved options accounts through automated approval bots with limited oversight.
published 2021-06-30 · verified 2026-10-07
ev-0193post-mortemprimary
PHE statement on delayed reporting of COVID-19 cases, Public Health England (GOV.UK)
Every one of these cases received their COVID-19 test result as normal and all those who tested positive were advised to self-isolate.
Official government statement: result files exceeded a size limit in the data load process, so positive cases were left out of the daily reported figures. Supports data-corruption and wrong-decisions for analytics dashboards and BI tools fed by file loads, since both report from the loaded data.
published 2020-10-04 · verified 2026-10-07
ev-0215regulatorprimary
Federal Reserve Board fines Citigroup for violating the Board’s 2020 enforcement action, Board of Governors of the Federal Reserve System
Citigroup has made insufficient progress remediating its problems with data quality management and failed to implement compensating controls to manage its ongoing risk.
Regulator fine for unremediated data quality management problems. Supports regulatory-penalty for data pipelines that feed regulated reporting.
published 2024-07-10 · verified 2026-10-07
ev-0234regulatorsecondary
DOD Financial Management: Implementation Weaknesses in Army and Air Force Business Systems Could Jeopardize DOD's Auditability Goals, U.S. Government Accountability Office
Government audit report on ERP rollouts: it records deficiencies in data accuracy, an inability to produce auditable financial reports, manual workarounds, and an interface specification with less functionality than the legacy one. Supports data-corruption and wrong-decisions for ERP systems.
published 2012-02-28 · verified 2026-10-07
ev-0235regulatorprimary
23 CFR Part 680: National Electric Vehicle Infrastructure Standards and Requirements, U.S. Federal Highway Administration (eCFR)
The price for charging must be displayed prior to initiating a charging transaction and be based on the price for electricity to charge in $/kWh.
The US rule for federally funded chargers requires price shown before a session starts, payment methods accessible to people with disabilities, OCPP between charger and network, network-to-network roaming and real-time port status for third-party apps.
published undated · verified 2026-10-07
ev-0238regulatorprimary
The Public Charge Point Regulations 2023, regulation 7: Reliability of rapid charge points, UK Government (legislation.gov.uk)
The charge point operator must publish information on its compliance with the reliability requirement on its website.
Reliability is measured from each charger's reported status (inoperative or out of order), so the status shown and published by the operator is a regulated figure, not a convenience.
published undated · verified 2026-10-07
ev-0268court-or-fineprimary
Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia
It makes no difference whether the information comes from a static page or a chatbot.
The tribunal held an airline liable for a bereavement-fare rule its website chatbot stated wrongly, so answers an LLM assistant gives customers bind the business that deploys it.
published 2024-02-14 · verified 2026-10-07
ev-0275standardprimary
LLM09:2025 Misinformation, OWASP Gen AI Security Project
The OWASP entry treats factual inaccuracies in model output that users act on as a top risk, citing an airline chatbot that gave travellers wrong policy information.
published undated · verified 2026-10-07
ev-0288regulatorprimary
Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards, Federal Trade Commission
Test, assess, measure, document, or inquire about the accuracy of its facial recognition technology before deploying it
Regulator order: a face-matching model flagged shoppers as suspects with more false positives in some communities, nobody tracked the false-positive rate, and employees acting on false-positive alerts followed, searched or removed shoppers. Supports wrong-decisions, user-harm and regulatory-penalty for model services.
published 2023-12-19 · verified 2026-10-07