How to read a vendor test plan line by line

A vendor plan is checked in two directions: lines with no risk behind them, and mandatory tests that are missing. The 46 published test plans give both lists. Across them, 198 test entries are marked not needed, and 123 of those are tests that vendors often add for that product type. Those 123 are the entries shown in section 6 of each playbook.

Lines that most often have no risk behind them

Counted over the 123 entries marked "vendors often add":

• Localization testing: 31 product types. Mandatory on 0 of 46.
• Usability testing: 22 product types. Mandatory on 0 of 46.
• Analytics and event tracking tests: 17 product types. Mandatory on 11.
• Performance and load testing: 9 product types. Mandatory on 9.
• Contract tests: 7 product types. Mandatory on 3.
• Device and browser compatibility: 7 product types. Mandatory on 15.

The last four are mandatory on some types and padding on others, so the playbook for your type decides, not the test name. Performance testing is not needed on the lending platform, insurance app, accounting and invoicing SaaS, expense management tool and 5 other types, and none of those 9 risk maps records an outage under load.

Lines that must be there

• Release smoke tests and compliance checks: mandatory on all 46. Release smoke is order 1 on all 46.
• End-to-end functional tests: mandatory on 43.
• Security scanning: mandatory on 42.
• Penetration testing: mandatory on 39, run by a security specialist on 30 and an external lab on 9.

Procedure

1. Map each line of the vendor plan to one of the 20 test types on the tests index. A line that fits none of them goes back to the vendor for a definition.
2. Open your playbook from the how-to-test index and mark each mapped line with its status there: mandatory (section 4), recommended or optional (section 5), or not needed and often added by vendors (section 6).
3. For each line in section 6, read the reason. Each of the 123 reasons names the change that would reopen the test, for example a second language or market shipping on the mobile banking app. If that change has not happened, ask the vendor which risk the line addresses.
4. For each recommended line, check the condition in section 5. All 211 recommended entries start "Becomes mandatory when". If the condition is false for your product, the line can wait.
5. List the mandatory tests the plan leaves out. Start with the five above, then the rest of section 4. A plan without a release smoke test has no first step.
6. Check every mandatory line for a done condition. Replace "tested" or "verified" with the done condition from section 4, word for word.
7. Check the role for each line. Penetration testing is run by a security specialist or an external lab on all 39 plans where it is mandatory, and accessibility testing by an accessibility specialist on all 17.
8. Send the vendor the marked plan with the checklist for your type. Name the removed lines and the added ones, each with its section number.

Plans to compare

The SDK or client library marks usability testing not needed because the SDK has no end-user interface. The crypto exchange marks it not needed until order-entry errors drive support tickets. The dating app marks contract tests not needed while one team owns the app and its API.