The Schedule of Exclusions

Second week of July. Slide four is one number in a green box: 99.6348% of scheduled tasks finished successfully last quarter, against an objective of 99.5%. Third green quarter running.

You ask the other question: of the numbers this company looked at last quarter and acted on, how many were right? Nobody has ever computed it. The green number is a ratio over 29,302 job runs, and nobody in the business reads a job run.

Over the same 91 days there were 47,278 reads — a person opening a dashboard, a scheduled report landing in an inbox, the repricing service pulling the markdown list at 04:00 — against 24 served objects. 6,232 of them returned an answer more than 1% away from the truth.

All four are correct ratios. They disagree because they are ratios over four different populations. The second one is the tell: it is what the quarter would have scored if the only things that could go wrong were the things the monitoring stack is built to see. The instrumented world was healthy — by selection.

• SLI — “a carefully defined quantitative measure of some aspect of the level of service that is provided.”
• SLO — “a target value or range of values for a service level that is measured by an SLI.”
• Error budget — “the difference between these two numbers is the ‘budget’ of how much ‘unreliability’ is remaining.” (All three, Google SRE.)

A ratio has a denominator, and choosing it is the whole design. The team chose job runs because the orchestrator already emits them: no new instrumentation, no judgement, no second opinion. It is free. That is exactly why it was chosen, and exactly why it measures the wrong thing.

Correlation between daily availability and daily correctness across the quarter: r = +0.0648, and 0 of the ten worst days for availability is among the ten worst for correctness. A task that fails at 03:00 and succeeds on retry at 03:12 costs the availability ratio a whole run and costs the 09:00 reader nothing.

What the objective actually allows. 0.5% of 47,278 reads is 236.39 wrong answers for the quarter — 2.60 a day for the whole company. At 99.9% you are allowed 0.52 a day. Nobody who has said “let’s aim for three nines on data quality” has multiplied it out. Google: “don’t pick a target based on current performance.”

The threshold nobody wrote down. “Materially wrong” = off by more than 1%. That was a choice, and it sets the score:

The curve is flat below 1.18% because the smallest error in the catalogue is Lesson 24’s extract. Set the line at 2% and that event stops counting on 306 days a year and counts only on Mondays. One event — the frozen currency feed, 0.4% wrong for 13 h — paged, was fixed the same day, and under the SLO’s own definition was never a bad answer at all. Lesson 20’s plural failure in a new hat.

The same ambiguity is already inside the availability figure: counting tasks after retries the quarter scored 99.6348%; counting attempts, 98.0455% — four times the failure rate from the same log. And 28 of the 91 days were individually below 99.5% in a quarter that was comfortably green.

Every data-observability product sells the same four perils, because they are the four things a tool can check without knowing your business: exit status, freshness, row-count volume, declared tests.

Each event’s signature below is what the earlier lesson on it actually measured (Lesson 22: every test passes by construction; Lesson 24: freshness green 358/358; Lesson 21: 0 of 43 tests failing). Coverage is therefore computed, not asserted.

14 of 26 events trip a peril — 53.85%. Score the same schedule by the thing the business suffers and it admits 7.57% of the wrong answers.

The four perils detect events — something changed, suddenly, visibly in a count or a timestamp. Twelve of twenty-six are not that shape. A constant is not an event. A drift is not an event. A row that never arrives leaves no trace among the rows that did. An extract a day behind by design has a perfectly fresh timestamp of its own. Three served objects — the exec weekly GMV tile, the Monday trading pack and sell-through by cohort — were materially wrong on every single read of the quarter, with no alert on any of them.

The cleanest catch of the 91 days points the same way from the other side: the skewed join that ran 4 h 12 late tripped freshness at 05:18, paged, and was diagnosed by breakfast. It produced zero wrong answers.

“You stacked the deck”

Each excluded event alone, with the other twenty-five deleted:

Seven of nine miss a 99.5% correctness objective on their own.

Cost = read rate × duration, and duration splits into before-anyone-knew and after. Of the 6,232 wrong answers, 5,521 (88.59%) were served before anybody knew there was anything to fix.

Every one of the nine excluded events burns the correctness budget fast enough that Google’s published burn-rate rule (14.4× over 1 h, 6× over 6 h) would have paged within hours of onset — seven of the nine in under seven hours, against actual detection of 9–22 days or never. Nothing was missing from the alerting design. What was missing is a column saying whether an answer was right.

The counterfactual ladder

Halving MTTD buys only 21.41% because the expensive events have no detection time to halve. The prize is going from never to daily, not from daily to hourly (worth a further 3.56%).

An error budget makes reliability negotiable instead of moral: you are not asking anyone to be careful, you are handing them an allowance.

• Availability budget: 146.51 runs → ended the quarter 73.03% spent.
• Correctness budget: 236.39 answers → gone by 09:00 on 3 April, day 3 of 91, and closed the quarter 26.4× over.

Burn rate = how fast the budget is going relative to the objective. Google’s published configuration for a 99.9% SLO (SRE Workbook, table 5-8, read from source):

One honest limit. Switch the correctness SLI on tomorrow and the 6 h / 6× rule fires on 84.8% of the hours in the quarter (median burn 22.2×). A burn-rate alert on a system permanently outside its objective is a lamp that is always on — which is the SLO doing its job, telling you the objective is wrong for the scope.

248 pages in 91 days — 2.73 a day — of which 22 were a real event: 8.87% precision and 49.6 hours of triage at 12 min each (the one assumption in this lesson).

The best-precision channel in the stack is the smallest, and it sits next to 1,147 warn lines nobody has read since week two (Lesson 15 measured a real failure arriving into that stream at z = +0.44).

Route by consequence, not by channel. Page only on a wrong answer in a tier-1 object:

Three tiers, written down, is the whole on-call design: page (a tier-1 answer is wrong now) · ticket (wrong, not in front of a decision today, named owner + due date) · dashboard (everything else, reviewed weekly, never routed to a phone).

An availability SLI is free, because the system emits its own exit codes. A correctness SLI is never free, because being right is not a property a system can observe about itself. To know whether an answer was right you need a second path to the same fact, and a second path costs money to build, run and keep honest.

That is why data SLOs are hard — and it is not what the vendor slides say. The objective is easy. The budget is easy. The alerting policy is solved, published and free to copy. The indicator is the whole problem. Which makes it a budgeting question, and therefore answerable. Four second opinions this series has already built and measured:

None of these knows the truth. Each asks a structural yes/no question — no threshold, no ground truth.

Three events survive the entire portfolio, and the residue is the honest part:

• One person, several customer rows (L19) — every row correct; the error is in an assumed correspondence stored nowhere. Needs a bridge table.
• A cleaning rule applied twice, differently (L11) — a constant, not an event; there is no before to reconcile against. Needs one cleaning layer.
• The 9.67-second rewrite window (L17) — expected cost 0.45 answers a quarter, of which one landed, on the repricing service, which wrote what it read onto live listings. Needs an atomic write.

Architecture, not instrumentation. Knowing which of your problems is which is most of what an SLO review is for.

And the sampling audit you were going to do instead. Re-computing 5 tier-1 answers a week has a 2.48% chance of catching a problem running at the rate the objective allows. Policing 99.5% by sampling needs 321 re-computations a week; at k = 5 the smallest rate you can reliably see is 27.52%. Sample to calibrate structural checks; never to replace them.

slo: tier1_answer_correctness
  indicator:
    population:  every read of a tier-1 served object
    good:        the object's answer is within 1% of the same figure
                 recomputed from source by an independent path
    measured_by: four structural checks (C1-C4), not a sample
    materiality: 1.0%          # changing this changes the score. owner: CTO
  objective:     99.0% over a rolling 28 days
  budget:        1.0% = 27.6 answers per 28 days
  policy:
    budget > 25% remaining:  ship normally
    budget < 25% remaining:  no new served objects until it recovers
    budget exhausted:        the next sprint is reliability work, by default
  alerting:
    page:   burn rate >= 6 over 6h, confirmed over 30m   # tier-1 only
    ticket: burn rate >= 1 over 3d
    review: everything else, weekly, by a person
  review:
    every incident: how long was it wrong before anything noticed?
    every quarter:  which check would have caught it, and what does it cost?

Two details do most of the work. The objective is 99.0% over seven objects, not 99.5% over twenty-four — it is the only number the measured evidence supports (98.0250% with the portfolio), and an objective you miss every week is indistinguishable from no objective. The budget policy is the part with teeth — an error budget with no consequence attached is a chart; the consequence must be agreed in advance, in writing, by the person who will later want to ship instead.

The incident review question

Replace “what broke and how did we fix it” (repair is 5.31% of the problem) with:

1. How long was it wrong before anything noticed? If more than a day, the finding is the gap, not the bug.
2. What is the cheapest check that would have made that number smaller, and what does it cost per year? If the answer is under a dollar — four times out of six here, it was — the reason it does not exist is not budget. Nobody has been asked to name one.

1. “Our reliability number — what is the denominator?” Jobs, runs, tasks or pipelines means you are being shown the health of the instrument, not the service.
2. “Name the five numbers we would most hate to be wrong. What checks whether they are right?” The useful answer is a query, not a person’s name.
3. “Last time a number was wrong, how long was it wrong before we knew?” If nobody can say, that is the finding — start recording it in every postmortem this week.
4. “How many pages did on-call get last month, and how many were real?” Under 20% precision, the rota is being trained to ignore the pager.
5. “What does ‘wrong’ mean — by how much?” No materiality threshold means several undocumented ones.

The generator in the artifact writes read_log.csv and job_run.csv beside itself.

1. Compute both SLIs — you should get 99.6348% and 86.8184% exactly.
2. Move materiality from 1% to 2% and watch 1,312 wrong answers become right ones with nothing changing in the data. Decide what you would defend, and why.
3. Find the hour the budget ran out. Then set the objective to 99.9% and find it again.
4. Build the burn-rate alert and count the hours it would be firing (ours: 84.8%). Ask whether the answer is a different threshold or a different scope.
5. List the ten served objects your company actually decides with. For each, write the one query that would tell you it is wrong without knowing the right answer. The ones where you cannot write that query are your tier-1 risks — and you have just written your first schedule of exclusions.

-- the SLI the team reports
select count(*) as task_runs,
       count(*) filter (where not succeeded) as failed,
       1 - count(*) filter (where not succeeded)*1.0/count(*) as sli_availability
from job_run;

-- the SLI nobody reports
select count(*) as reads,
       count(*) filter (where rel_error > 0.01) as bad,
       1 - count(*) filter (where rel_error > 0.01)*1.0/count(*) as sli_correctness
from read_log;

-- the hour the quarter's error budget ran out
with hourly as (
  select date_trunc('hour', read_ts) as h, count(*) as reads,
         count(*) filter (where rel_error > 0.01) as bad
  from read_log group by 1)
select h, sum(bad) over (order by h) as cum_bad
from hourly qualify cum_bad >= 0.005 * (select count(*) from read_log)
order by h limit 1;

-- burn rate on a six-hour sliding window, against a 99.5% objective
with hourly as (
  select date_trunc('hour', read_ts) as h, count(*) as reads,
         count(*) filter (where rel_error > 0.01) as bad
  from read_log group by 1)
select h, sum(bad) over w * 1.0 / nullif(sum(reads) over w, 0) / 0.005 as burn_6h
from hourly
window w as (order by h rows between 5 preceding and current row)
qualify burn_6h >= 6.0
order by h;

An SLI is a ratio, and whoever picks the denominator picks the answer. Pick job runs and the instrument reports on itself — 99.6348%, green, three quarters running. Pick the answers people act on and the same 91 days score 86.8184%. Availability telemetry is free because a system can observe its own exit codes; correctness telemetry is never free, because being right is not a property a system can observe about itself. Everything else — the objective, the budget, the burn-rate alerting — is solved and copyable. The error budget for data is, in practice, a budget for how much second opinion you buy: this quarter’s was $0.97 a year and would have taken tier-1 from 79.1899% to 98.0250%.

Vocabulary

One deterministic generator, no random seed (splitmix64 over integer keys), simulating 91 days of 29,302 task runs and 47,278 reads against 24 served objects, scored against 26 incidents. Three things worth stating plainly:

• The signatures are not free parameters. Whether an incident trips freshness, volume, exit status or a test is taken from what the earlier lesson on it measured. Coverage in CL. 25.03 is computed from those signatures.
• The error magnitudes are the earlier lessons’ own numbers (1.18% and 16.89% from L24, 10.71% from L22, 31.17% from L21, 34.06% from L11…). What is constructed is the portfolio — all fifteen failure shapes in one quarter — which is why CL. 25.03 also reports each alone.
• The SQL printed here was run. Both logs loaded into DuckDB, published queries executed: 20 checks, all matching the Python exactly.
• Appendix reproduction check: generator extracted from the rendered HTML, run in a clean directory — 22 top-level keys, 1,521 scalar values, 0 differences, no container path in the shipped code.

One quantity is an assumption rather than a measurement and is flagged as such: 12 minutes to triage a page, used only for the 49.6-hour figure.

Series links: builds directly on L06 (the six tests, alert fatigue), L11 (constants), L12 (drifts), L13 (unwatched, and the detection ladder), L14 (reversible), L15 (unreproduced, the muted WARN), L16 (bundled), L17 (transient), L18 (extremal), L19 (referential), L20 (plural), L21 (remote, the lineage gate), L22 (absent, the two-sided count), L23 (permitted), L24 (detached, the served-window check).