Method
How we test a kit
Each kit is built, then attacked on purpose, on made-up data. This page says what that means and where it stops.
Made-up data, every time
Each kit carries its own made-up data. Where a generator makes it, the seed is fixed (for example seed 2026), so the same run gives the same numbers. Nothing in our results was measured on real exports.
The tests make no network calls to outside services.
Mock services instead of real accounts
Stores, the CRM and the mail service are mocks we wrote. No real store, vendor or sheet account is used.
Where it matters, the same sync also runs over real HTTP against the mocks on the same machine, with one 429 'too many requests' answer injected.
Deliberate breakages
We change one small thing in a kit's code on purpose, for example allowing one retry too many, and run the tests again. If at least one test fails, the planted bug is caught. If none fails, a test is missing.
Tests and planted bugs, per kit
| Kit | Tests passed | Planted bugs caught |
|---|---|---|
| Two-store inventory sync | 24 | 28 of 28 |
| Store product file checker | 66 | 20 of 20 |
| Orders-to-CRM sync | 34 | 34 of 34 |
| Messy sales export cleaner | 28 | 25 of 25 |
| Support-ticket script rescue | 18 | 19 of 20 |
| Spreadsheet sales tracker | 84 | 46 of 46 |
| PDF statement and invoice reader | 105 | 18 of 18 |
| All seven | 359 | 190 of 191 |
The one that got through
In the support-ticket script rescue, 1 of 20 planted bugs survived. It treats a 403 answer like any other error, and our mock service never sends a 403, so no test reaches that line. We show it instead of hiding it.
What we do not test
- Live accounts. The spreadsheet sales tracker has not been run on a live account. A difference between the mock and the real service would not show up.
- Real imports. No live import was run with the store product file checker. The checks stop at the file.
- Scanned pages. The PDF statement and invoice reader takes text-based PDFs only; no OCR.
- Real documents and other layouts. Its accuracy numbers say nothing about them.
- Other export layouts. The messy sales export cleaner's rules fit three synthetic layouts.
- Actions in your other apps. The two-store inventory sync lists an oversold order; it does not cancel orders.
Who does the work
We use an AI coding assistant to move faster. A person reviews and tests every kit before it is shown.
Each kit then gets a separate verification pass that re-checks its numbers against its own files.