The Taxing Master

The platform budget is due and Finance sends back one line against one row: Data quality — what does this buy? The answer is the answer everybody gives: 391 automated tests over a 48-model warehouse, nightly and on every pull request, green for weeks; a data-observability licence at €36,000 a year over 60 tables; an on-call rota. The line totals €68,367.31.

That is a description of what the money is spent on, not of what it buys. A test that passes is evidence about the data; it is not evidence about the test.

You find out what a suite is worth by breaking the thing on purpose. Make one small wrong edit, rebuild, run every test, record which fail, undo, repeat. The share of deliberate defects caught is the mutation score — a measurement of the tests rather than of the data.

Two populations were injected into a real DuckDB build of the project:

• 126 data faults — one value nulled, one row duplicated, an orphan foreign key, a value outside its domain, 1% of a table missing, a sign flip, a ×1000, a timestamp in 2031.
• 163 code defects — one regex edit to one model’s SQL: inner join → left join, sum() → avg(), a where deleted, coalesce(x, 0) → x, decimal(18,2) → decimal(18,0), a minus that became a plus.

The suite is good at what it was written for and poor at what a person types. Weighting every operator equally instead of by catalogue mix gives 40.99% for the code population — both are published, neither is near what a green board implies. The layer with the best coverage is the marts (60%); the reports people read score 28.85%.

In stg_order_items, cast(item_price_cents / 100.0 as decimal(18,2)). Change 100.0 to 1.0.

• 22 of 48 models change, all 9 served money objects change
• monthly GMV rises 12,287%; the tier-1 headline moves €825,055,143.66
• 0 of 391 tests fail

Why none could: every assertion in the suite is a relationship between numbers that all scaled together. Commission is a fixed share of price. Order totals still sum to their items. Monthly still reconciles to daily. Internal consistency is preserved by almost every systematic error, and internal consistency is all a test suite checks.

The multiplier is not 100 but 123.867, because refunds come from a different staging model and did not scale — so the refund rate fell by a factor of a hundred and the one test that might have noticed was made less likely to fire by the bug.

Twenty hand-written assertions out-catch two hundred and eighty-seven generated ones on logic defects, 49 to 43 — 16.35× per test, at €9.08 a catch against €148.46. unique and accepted_values caught none at all — checked, and structural: 29 code defects did change the row count of a model carrying a unique test; a code defect changes which rows exist, it does not make a key repeat. Against data faults those two kinds earn their keep (81 and 8 catches).

The licence computes a nightly profile — row count, null counts, column means — and complains when it moves more than a set amount. It catches 77 of 131 code defects, twice the suite, including the two-character edit (one mean moves 12,299%). Of the 131, 17 are caught only by the tests, 53 only by the monitor, 24 by both, 37 by neither.

Nobody ever heard from it because of one number. It was configured at 0.25%. Measured against 29 tables’ own overnight movement over 365 nights, that raises 6,885 alerts a year with nothing wrong (18.9 a day) and all 29 tables exceed any human tolerance. So it was muted.

The canyon: at 1% it raises 4 alerts a year and still catches 77 code defects (against 83 at the noisy setting). Not a tuning — a gap in the data with nothing in it. Triaging the 6,885 at twelve minutes each would cost €107,406/yr — 2.98× the licence and 1.57× the whole quality budget.

• The compute is free. 391 tests × 1013 runs = 3.89 TiB, inside the 1 TiB/month free tier: €0.00 (€22.35 at list). 100% of table references are billed at the 10 MB minimum, so the bill for running your tests is not a function of your data volume at all.
• A test costs €22.24 a year to own and nothing to run (€8,697 across the suite).
• 19.9% of the bill is people, 0.03% is machines (175 hours = 0.11 FTE), and 80.0% is two contracts nobody re-reads.
• Whole apparatus per number anyone reads: €0.37.

Over 549 days, 114 tests failed at least once and 277 never failed at all. Retiring the silent ones deletes 276 tests, of which 155 demonstrably catch something — code coverage falls 41 → 35 kills, data 99 → 71.

The rule that works needs the experiment: retire a test that has never failed in production and killed nothing in either population. That is 121 tests (90 not_null, 14 unique, 10 relationships, 6 accepted_values, one hand-written) and deleting all of them changes no kill in either population — asserted in the generator.

And the saving is €2,691.40, or 3.94% of the year. The cut everybody reaches for first is real, correct, worth doing — and worth four per cent. Separately, 29 tests are the only thing catching some specific fault, and none of them is identifiable without an experiment.

112 incidents in 549 days; 278,892 reads of the twelve served objects; each incident replayed through every arrangement of detectors.

Today scores 97.6573%. Moving the monitor’s threshold from 0.25% to 1% and routing it to the rota scores 99.6479% — it removes 84.97% of the remaining wrong answers, buys no software, writes no tests and hires nobody.

The recommendation that lost. Ten hand-written range and reconciliation assertions were measured against the same 131 defects: they caught 12, only 3 of them new. The reason is exact — the monitor’s threshold is one per cent of the value the table actually has, while a human-written range is a guess: the median of the ten allowed the number to grow 202% before complaining, one of them 982%. You cannot write the number down. You have to compute it from history.

Running the profile diff inside the pull request doubles the defects that never reach production (20 → 41) and is worth only 148 wrong answers: prevention beats detection by exactly the detection lag, and no more.

Every number above is an upper bound. A mutation score measures the faults you were able to inject. Of the 20 failure shapes this series has catalogued across Lessons 06–30, 3 can be injected as a fault in the data or the code, two more in part, and 15 cannot be injected at all — they are not a property of any row. 1 of 20 is expressible as a predicate over the rows of a table.

• A test suite is a list of predicates. It grows by accretion, it is counted, its health is a pass rate.
• An assurance argument starts from the other end: here are the ways this system can be wrong; here is the evidence each is covered; here is what is knowingly uncovered and why we accept it.

391 is a property of the first kind of document. Nothing in it belongs in the second.

1. What is our mutation score, and when did we last measure it? A test count is not an answer.
2. Which tests have never failed — and of those, which ones can? The first half is a query; the second is an experiment, and only the second licences a deletion.
3. What would happen if a number were a hundred times too big? If the answer is “a test would catch it”, ask which one, by name.
4. What is the threshold on our anomaly monitors, and what is the natural overnight movement of the tables they watch? If nobody has both numbers, the monitor is muted or ignored.
5. How many alerts last month, and how many were real? Multiply by twelve minutes; read it as money.
6. Does the suite run before the merge or after it? The same assertions are worth different amounts.

1. On a throwaway branch, make one wrong edit to each of your three most-read models (sum→avg, delete a where, drop a divisor).
2. Run the full suite against each. Note which tests fail. Most likely: none.
3. Run select count(*), avg(<money column>) from <model> on the branch and on production; note the percentage difference. That is a profile diff, by hand.
4. For one table, compute those two numbers per day for a year and take the largest overnight change. That is the smallest threshold you can set without lying to yourself.
5. Schedule one deliberate defect a week into a sandbox, scored by whether CI goes red. Here that costs €609.46/yr — 0.89% of the bill — and it is the only line item that produces evidence.

A green test suite is evidence that your data matched your assertions last night. It is not evidence that your assertions would notice if the data stopped being right, and those are different claims with different prices. The first is free and automatic; the second costs one afternoon a quarter and almost nothing in money — and until somebody pays it, the number of tests you have is a fact about your repository, not about your warehouse.

The cheapest thing on the table was never a tool. It was a threshold, set once by somebody being careful, never checked against the noise it was set against, and left to make the most capable detector in the building unreadable. 85% of the residual damage, for the price of editing one line of YAML.

Vocabulary

A 48-model warehouse over ten source tables, generated deterministically (splitmix64 finalizer, no RNG): 150,008 orders, 185,930 order items, 240,000 listings, 36,680 refunds over 549 days. Real SQL, built in DuckDB; 391 real predicates; baseline 390 green and one long-standing muted failure. Both campaigns executed, not modelled, with exact decimal sums and an order-independent row hash for the diff (never a float sum — order-dependent, and it would report every rebuild as a change).

Observations used and not derived: the incident log (112 incidents), the BI audit log (508 reads/day), the licence, the rota, the loaded engineer rate (€78/h), 209 merges/yr at 3.1 pushes, 96 new tests and 143 edits a year. The one modelled assumption is that each incident is drawn uniformly from the measured fault catalogue.

Materiality sweep (an answer counts as wrong at this error on the object’s headline): 0.1% → 48.59%, 0.5% → 94.40%, 1% → 97.66%, 2% → 97.90%, 5% → 98.33%. The ladder’s ordering is unaffected; its level is not.

Verification. 31 queries and structural checks re-run in DuckDB against figures.json: 31 matched, 0 failed — including three written only to attack round numbers. Round trip: the 14 source files were extracted from the rendered page, run in an empty directory, and the resulting figures.json diffed key by key — 141 top-level keys, 13,613 scalar values, 0 differences, byte-identical by md5, no container path leaked.

What is not here. The profile monitor is modelled as a whole-table profile against a perfect baseline, which is generous (a real product compares against a forecast and will do worse); against that, its false-alarm rate is measured on real overnight movement. And the largest caveat is §10: 15 of 20 catalogued failure shapes cannot be injected as a fault in a table, so they appear in no coverage number on this page — or in any that anyone will show you.