The Clause Nobody Wrote

Wednesday 19 August, 14:07 CEST. The backend team ships PR #4471 — grade taxonomy cleanup against shop.listings, the table our warehouse copies hourly. Four column changes; two matter.

Change 1 broke six dashboards at 14:11 — four minutes after deploy. Slack 14:19, patch merged 14:52. Nothing was ever wrong; things were absent. Cost: an annoying afternoon, €0.

Change 2 broke nothing. fct_price.sql, written eleven months earlier:

CASE item_condition
  WHEN 'A' THEN 1.00   -- as new: full price
  WHEN 'B' THEN 0.90   -- light wear: -10%
  WHEN 'C' THEN 0.80   -- visible wear: -20%
  ELSE 0.80            <-- the branch that ate the money
END AS markdown_multiplier

From 14:07 the column held 'excellent', 'good', 'fair'. No WHEN matched. Every one of 8,000 listings fell to ELSE 0.80 — a 20% markdown on everything, including the 2,080 grade-A pieces that should have sold at full price. Nine days, 2,285 orders, €29,289.95 under-charged (10.91% of intended GMV, €12.82/order). Every job green throughout.

A rename is a broken bone: it hurts immediately and you set it the same day. A re-coding is a slow bleed: nothing hurts, and you find it from the blood on the floor.

Drift sorts on two axes; only one is expensive.

The instinct after an outage is to harden against the right-hand column, because that’s what woke people up. The money is in the left-hand column. An outage is a change that announced itself.

Two signals existed for nine days. The series habit is to compute before claiming invisibility — here the answer is half:

• Revenue: genuinely ambiguous. Median daily GMV fell 15.9% (z = −2.18), but 3 of the 9 post-release days land inside the pre-release min–max band [€27,186 – €36,784]. A 16% dip in late August is an argument, not an alarm.
• Shape: unmissable. Markdown as a share of list price sat inside 9.99%–10.31% for 18 days, then hit exactly 20.00% and stayed there for nine days, to the second decimal. Blended rates over a real mix of goods do not land on a round number and hold.

Two more one-line checks, either of which ends it on day one:

Revenue is an aggregate, and aggregates have weather. The shape of the data — distinct values, branch mix, blended rates — has no weather.

A machine-checkable agreement at the boundary between producer and consumers. Not a wiki page — a file in the producer’s repo, checked in CI, versioned like code.

# contracts/shop.listings.yaml
version:   1.4.0
owner:     team-marketplace-backend
consumers: [data-platform, pricing-service, finance-reporting]
freshness: {max_lag: 1h}          # lesson 06's threshold, written down
volume:    {min_rows: 6500, max_rows: 9500}

fields:
  - name: item_condition
    type: string
    enum: [A, B, C]               # <-- the clause nobody wrote
    meaning: >
      Grader's condition band, assigned at intake.
      A = as new · B = light wear · C = visible wear.
    required: true

  - name: price_cents
    type: integer
    unit: EUR_cents               # units are a promise, not a comment

evolution:
  additive: allowed               # new fields, widened types: minor bump
  breaking: expand_and_contract   # renames, re-codings: never in one release
  notice:   14d

Six clauses, each buying something:

With the enum clause, PR #4471 never merges: the contract test sees 'excellent' outside the declared domain and fails the build, naming the column, the offending values and the three consuming teams. Cost of the incident: one conversation.

1. Expand — producer emits both: item_condition keeps A/B/C, new condition_grade carries excellent/good/fair. Contract 1.4.0 → 1.5.0, additive, nothing can break.
2. Migrate — consumers move on their own schedule inside the notice window. The contract’s consumer list plus lineage tells you who is left. (Impossible without knowing who reads you.)
3. Contract — when nobody reads the old field, drop it as a major bump 1.5.0 → 2.0.0.

Same end state as PR #4471, reached with nobody’s prices wrong for nine days. Cost: one extra column for a fortnight. For event streams the same discipline is packaged as a schema registry (Avro/Protobuf + Confluent Schema Registry), enforcing BACKWARD/FORWARD/FULL compatibility before a producer may publish.

And one habit worth more than any tool: never SELECT * across a team boundary. A wildcard forwards whatever arrives — which is exactly how a structural change becomes a silent one. An explicit column list turns a rename into a rung-1 build failure, for free.

1. For our three most important upstream tables — who owns them, and do they know we read them? If a backend engineer renamed a column tomorrow, would anything stop the merge?
2. Where are the enum domains and units written down? Not the types — the allowed values, and whether a column is cents or euros, local time or UTC.
3. Do we have a CASE … ELSE in pricing, categorisation or reporting where the ELSE is a silent default rather than a loud NULL? What’s the blast radius if every row lands there?
4. Which of our models SELECT * from a source another team owns?
5. Last time we changed a column another team consumed — expand-and-contract, or just changed it?
6. Do we monitor any shape metric — distinct values, branch mix, blended rate — or only volume, freshness and revenue?

1. Reproduce. Run seed.py (below). Confirm 8,000 listings, 2,080/3,680/2,240 grade mix, UNDER-CHARGED EUR 29,289.95. No RNG — your figures match to the cent.
2. Write the contract. Save the YAML as contracts/listings.yaml. Write validate(rows, contract) returning violations: missing required field, wrong type, value outside enum, null rate over threshold.
3. Fail the build. Pre-release rows → clean. Post-release rows → one violation naming item_condition and the three offending values. That function in CI is the lesson.
4. Add the two shape monitors. Emit COUNT(DISTINCT markdown_multiplier) and the blended markdown rate daily; alert when the distinct count drops or the rate moves >1pp off its 14-day median. Show both fire on 19 Aug while a GMV-band alert stays quiet on 3 of 9 days.
5. Expand and contract. Emit both item_condition and condition_grade; show an old and a new consumer both passing against 1.5.0, then drop the old field as 2.0.0.

Push as de-practice/08-data-contracts. README: name one real upstream table we depend on, its consumers, and the one enum or unit clause that is true today but written down nowhere.

Structure breaks. Semantics lie. A pipeline detects the first on its own — a missing column is an error. It can never detect the second, because 'excellent' is a perfectly valid string. The only thing that catches a meaning change is a written-down promise about what the values may be, checked by a machine, in the producer’s build, before the change ships.

Smallest useful action: pick the one upstream column whose values your business logic branches on, and add an accepted_values test. One clause, five minutes, and the expensive half of schema drift becomes the cheap half.

• Schema drift — any unannounced change to a source you don’t own: added, renamed, dropped, retyped or (worst) re-coded.
• Structural vs semantic change — names and types vs units, allowed values and meaning. Structural breaks loudly; semantic doesn’t break at all.
• Data contract — versioned, machine-checkable agreement between a producing team and its consumers: schema, semantics, guarantees, evolution rules, owner.
• Enum domain / accepted values — the closed set a column may contain. The single clause that would have prevented this incident.
• Expand and contract — three-step migration for breaking changes: emit both, let consumers move, remove the old one in a major version.
• Schema registry — the streaming equivalent; rejects producer schema versions that break declared compatibility.
• Shape monitoring — tracking distinct values, branch mix and blended rates over time; the signals that have no weather.
• Quarantine table — where contract-violating rows go, so they are neither silently accepted nor silently dropped.