Resilience and failure testing

Injecting failures (timeouts, dependency down, partial outage) to see recovery.

Mandatory for 20 of 46 product types. Usually run by a developer.

What it catches

• cascading failures
• missing retries
• stuck states

What it misses

• normal-path bugs

Which product types need it and why

Done conditions used on playbook pages

Each mandatory cell has its own done condition, written for that product type.

• accounting and invoicing SaaS (Fintech): When the bank feed provider is cut off in a test environment, the account shows a visible stale-feed warning, and the catch-up import after recovery creates no duplicates.
• clinical trial management software (Healthcare): With the randomisation service switched off in staging, the site follows the documented fallback, and entries made during the outage appear with correct timestamps after recovery.
• crypto exchange (Fintech): With one matching engine node or database replica stopped during a load test, orders keep matching and the status page reports degraded service.
• data pipeline (Infrastructure and developer products): With the source stream or warehouse paused mid-run, the job retries, resumes from its last checkpoint and loads each record once.
• ecommerce store (Commerce): With the CDN switched off in staging, the storefront serves from origin or a status page, and no order is charged without being recorded.
• electronic health record system (Healthcare): With the main system taken offline in a test site, clinicians open the downtime view and read current medications, allergies and active orders for a test patient.
• EV charging app (Infrastructure and developer products): With the roaming network or charger link cut mid-session, the app shows the session as interrupted, billing stops, and the receipt matches the energy delivered before the cut.
• government services portal (Public sector): When the identity or case backend is slowed in a test, applicants see a queue or retry message and no submitted form is lost.
• identity and access management service (B2B SaaS): With the session store or quota service behind token validation cut in staging, existing sessions stay valid until expiry, new sign-ins show a clear error, and service returns without a manual restart.
• smart home or IoT app (Infrastructure and developer products): With the vendor cloud blocked, the app marks each device's last known state as stale, and queued schedules run once when the cloud returns.
• LLM-based assistant (Infrastructure and developer products): With the model API returning errors or timeouts, the user sees a retry message in the chat and no half-written answer is saved as final.
• messaging app (Consumer apps): With the network gateway or cache tier throttled, clients back off, connected users keep sending messages, and the service recovers without manual restarts.
• mobile banking app (Fintech): With the core banking test service switched off, the app shows a service message, and a transfer submitted during the outage posts once after recovery.
• mobile game (Consumer apps): When the service-discovery cluster or session store degrades, players see a queue screen and keep their inventory and progress after recovery.
• neobank (Fintech): With the partner bank feed stopped, the app marks balances as delayed instead of showing stale figures as current, and blocks withdrawals it cannot confirm.
• software as a medical device (Healthcare): When the sensor or transmitter is made to fail on the test rig, the app raises the failure alarm, and during a forced app crash loop the paired device's battery level drops no faster than with the app closed.
• SDK or client library (Infrastructure and developer products): With the SDK backend returning errors, timeouts or malformed config, the sample host app keeps running and its own features work.
• event ticketing platform (Commerce): When one seat-map server is shut down mid-onsale, buyers in the queue keep their place and held seats stay held.
• stock trading app (Fintech): When the market data feed or the order router is cut in a test environment, the app shows an outage banner and blocks new orders instead of accepting them silently.
• wearable companion app (Healthcare): With the cloud API blocked, the app still shows readings stored on the band and phone, and queued data uploads once when the cloud returns.

Tools

Alphabetical. Not recommendations.