Roadmap To Be A Data Engineer / Lesson 26

Lesson 26 Late data Fundamentals §4 Modelling / §7 Serving (+§6) About 25 min read

The Late Final

The same query gives a different number on Monday and Friday.

Late-arriving data, restatements and as-of reporting


Stop press. Add up the twenty-four monthly figures this company has published to its board and you get €56,346,864.98. Add up the same twenty-four months today and you get €51,035,018.72. Nothing was wrong with either total on the day it was computed. The difference is €5,311,846.26 — 10.41% of two years of trading.

1First edition

Monday 7 September, 09:00, the weekly trading meeting. Week 35 — Monday 24 to Sunday 30 August — closed at €652,642.39 of net GMV. Best week since May.

Friday 11 September, 14:12. The CFO runs the same query, for the same week, against the same table. €624,898.87. Down €27,743.52 in four days — −4.25%.

Nothing was deployed. No backfill ran. No row was amended in place. Both numbers are correct totals of the rows that existed when they were computed — and neither is the answer. Week 35 settles at €536,668.11: 21.61% below Monday’s figure and 16.44% below Friday’s.

One grain up: the August board pack was written on Wednesday 2 September and reported €2,669,189.17. The identical query today returns €2,517,617.22. August settles at €2,316,475.40. The figure the board approved was 15.23% high.

The curve is not monotone: it jumps up on D+2 (a Tuesday, when the wholesale partner’s weekly file lands) and then falls smoothly for six weeks with no event in it anywhere.

2Two kinds of late

August moved by €352,713.77 between the board pack and settlement:

  • €5,253.87 of gross sales arrived after the pack — orders that had already happened but had not reached the warehouse (partner file: 4.51% of orders, median lag 5 days, longest 8). This is late arrival, and it has an engineering fix (L22’s ingestion sequence, L12’s log-based CDC).
  • €357,967.64 of refunds posted after the pack. Every one of those rows reached the warehouse the same day it was created. Arrival lag: zero. This is late emergence — on 2 September the event had not happened yet.

1.45% of the restatement was the pipeline. 98.55% of it was the calendar.

And the calendar is not negotiable. Under § 356(2)(1)(a) BGB — Article 9(2)(b) of Directive 2011/83/EU — the consumer’s fourteen days begin “on the day on which the consumer … acquires physical possession of the goods”, not on the day they order. Article 14(1) gives fourteen more days to send it back; Article 13(1) gives the trader fourteen days from being told to refund. With delivery, return transit and grading, the measured median lag from sale to refund here is 15 days before 1 June and 19 after.

Nothing a data team can build makes a return arrive sooner. Late arrival is a defect. Late emergence is a property of the business, and the only thing engineering can do about it is represent it.

3Every month has an age

Age Net GMV as a share of final (median of 24 months)
D+2 — the board pack 109.89%
D+7 106.64%
D+14 102.42%
D+30 100.01%
D+31 every one of the 24 months is within 1% of final

At D+2 the twenty-four months run from 108.98% to 115.23% of their final value — a 6.25 pp span — so even two months read at the same age are only comparable enough to argue about.

Why: net GMV is a difference between two quantities that finish at different speeds. At D+2 gross sales are 99.57% complete and refunds are 53.90% complete. The month is wrong because one half is done and the other is barely started.

4The run-off triangle

Origin month down the side, age across the top, the future missing from the corner — general insurance has drawn this since before computers: “the incremental or cumulative loss of accident year i and development year k is observable if and only if i + k < n.”

Net GMV as a percentage of its final value:

Origin month D+2 D+7 D+14 D+21 D+30 D+45 D+60 D+90 settles at
Sep 2024 110.3 106.4 102.2 100.4 100.0 100.0 100.0 100.0 €1,926,845
Oct 2024 109.7 106.7 102.4 100.5 100.0 100.0 100.0 100.0 €2,027,244
Nov 2024 109.3 106.5 102.4 100.6 100.0 100.0 100.0 100.0 €2,116,986
Dec 2024 110.2 106.7 102.3 100.5 100.0 100.0 100.0 100.0 €1,998,344
Jan 2025 109.4 106.6 102.3 100.5 100.0 100.0 100.0 100.0 €2,059,788
Feb 2025 110.3 107.3 102.8 100.6 100.0 100.0 100.0 100.0 €1,784,704
Mar 2025 110.1 106.5 102.5 100.5 100.0 100.0 100.0 100.0 €2,033,473
Apr 2025 110.3 107.0 102.5 100.5 100.0 100.0 100.0 100.0 €1,913,834
May 2025 109.4 106.8 102.4 100.4 100.0 100.0 100.0 100.0 €1,920,863
Jun 2025 110.2 106.7 102.5 100.5 100.0 100.0 100.0 100.0 €1,826,246
Jul 2025 109.3 106.3 102.2 100.4 100.0 100.0 100.0 100.0 €1,811,902
Aug 2025 110.1 106.6 102.3 100.5 100.0 100.0 100.0 100.0 €1,984,260
Sep 2025 109.8 106.5 102.3 100.4 100.0 100.0 100.0 100.0 €2,232,236
Oct 2025 109.5 106.4 102.3 100.4 100.0 100.0 100.0 100.0 €2,425,109
Nov 2025 110.1 106.5 102.3 100.4 100.0 100.0 100.0 100.0 €2,534,919
Dec 2025 109.9 106.8 102.4 100.5 100.0 100.0 100.0 100.0 €2,310,242
Jan 2026 109.0 106.5 102.4 100.5 100.0 100.0 100.0 100.0 €2,462,964
Feb 2026 109.7 107.0 102.5 100.5 100.0 100.0 100.0 100.0 €2,115,913
Mar 2026 109.8 106.5 102.5 100.5 100.0 100.0 100.0 100.0 €2,448,649
Apr 2026 109.8 106.8 102.4 100.5 100.0 100.0 100.0 100.0 €2,250,970
May 2026 109.9 106.3 102.2 100.4 100.0 100.0 100.0 100.0 €2,324,250
Jun 2026 114.6 111.5 106.5 103.2 101.0 100.0 100.0 — €2,112,625
Jul 2026 113.9 111.2 106.3 103.4 101.0 — — — €2,096,177
Aug 2026 115.2 111.5 — — — — — — €2,316,475

Read down a column and months are comparable, because they are the same age. Read across a row and you are watching one month find out what it was. The board pack reads the D+2 column; “let me re-run it” reads the diagonal; the annual accounts read the far right. The most recent month — the one with the meeting attached — is the last row, and it is the only row with no column it can honestly be compared against.

5What it decided

Plan attainment

12 of 12 months beat plan on the day the pack was written. 2 of 12 beat it once settled. 10 flips. Mean immaturity bias +10.73% against a mean plan gap of 1.86% — the bias is 5.8× the size of the thing it is supposed to measure.

Month Plan As reported (D+2) vs plan Settled vs plan Verdict
Sep 2025 €2,312,000 €2,450,568 +5.99% €2,232,236 −3.45% beat → missed
Oct 2025 €2,433,000 €2,654,576 +9.11% €2,425,109 −0.32% beat → missed
Nov 2025 €2,540,000 €2,792,128 +9.93% €2,534,919 −0.20% beat → missed
Dec 2025 €2,398,000 €2,539,333 +5.89% €2,310,242 −3.66% beat → missed
Jan 2026 €2,472,000 €2,684,163 +8.58% €2,462,964 −0.37% beat → missed
Feb 2026 €2,142,000 €2,320,912 +8.35% €2,115,913 −1.22% beat → missed
Mar 2026 €2,440,000 €2,689,413 +10.22% €2,448,649 +0.35% beat → beat
Apr 2026 €2,297,000 €2,471,883 +7.61% €2,250,970 −2.00% beat → missed
May 2026 €2,305,000 €2,553,313 +10.77% €2,324,250 +0.84% beat → beat
Jun 2026 €2,191,000 €2,420,134 +10.46% €2,112,625 −3.58% beat → missed
Jul 2026 €2,174,000 €2,387,304 +9.81% €2,096,177 −3.58% beat → missed
Aug 2026 €2,381,000 €2,669,189 +12.10% €2,316,475 −2.71% beat → missed

(If the plan itself had been built from immature actuals, part of the bias would cancel — and nobody knows which it was, because the plan does not record the age of the numbers it was built from.)

Category growth — and the difference from Lesson 24

Lesson 24’s dashboard was uniformly one day behind, and a uniform bias divides out of every ratio: category ranking there moved zero positions. Here the bias is proportional to each category’s return rate, so at D+2 the six categories sit anywhere from 104.64% to 132.04% of final — a 27.40 pp spread.

Category Return rate YoY at D+2 YoY settled Rank Maturity at D+2
Dresses 29.0% +51.6% +18.5% #1 → #1 127.92%
Shoes 33.0% +51.3% +14.6% #2 → #6 132.04%
Knitwear 18.0% +34.4% +17.6% #3 → #3 114.29%
Outerwear 20.0% +34.2% +14.6% #4 → #5 117.15%
Bags 7.5% +24.9% +18.0% #5 → #2 105.78%
Accessories 5.5% +20.1% +14.8% #6 → #4 104.64%

Read on 2 September, Shoes were the second-fastest-growing category in the business. Settled, Shoes are last of six. Bags go fifth to second.

Company total: August 2026 against August 2025 reads +34.52% as published and +16.74% settled. And the obvious fix does not work: compare the two Augusts at the same age, D+2 against D+2, and you still get +22.18%. The extra 5.44 points is the returns window going from 14 days to 30 on 1 June, which made August 2026 a less mature two-day-old month than August 2025 was.

Retention, in the other direction

Settled, the 60-day repeat rate of twenty-four cohorts sits between 16.95% and 21.89% with no trend worth a slide. Measured today, the August cohort reads 13.59% against a settled 19.21% — -5.62 pp, and 4.75 pp below the trailing ten cohorts.

A systematic bias divides out of every ratio (Lesson 24). A bias that is proportional to something divides out of nothing: it moves the ranking, the mix, the growth rate and the cohort chart, each by a different amount, and it moves two of them in opposite directions.

6You do not have to wait. You have to adjust.

The chain ladder: take the months that are already mature, add up what they were worth at age a, add up what they were worth at the end, and the ratio turns any month’s age-a figure into an estimate of its ultimate.

Age As published Worst month Chain ladder (12-month factors) Same, after the policy change
D+0 10.93% 15.92% 0.442% 4.17%
D+2 — board pack 9.81% 15.23% 0.296% 4.40%
D+3 9.28% 14.52% 0.291% 4.25%
D+5 8.02% 13.05% 0.223% 4.40%
D+7 6.65% 11.49% 0.199% 4.48%
D+10 4.57% 9.50% 0.133% 4.46%
D+14 2.39% 6.54% 0.108% 3.95%
D+21 0.48% 3.37% 0.033% 2.79%
D+31 0.00% 0.86% 0.002% 0.81%

Mean absolute error over the 21 months whose returns policy never changed. At D+2 the adjustment takes the board number from 9.81% wrong to 0.296% wrong, using nothing but the company’s own history.

Two things matter more than the estimate:

It stops helping when you stop needing it. From D+31 the raw number is already better than the adjusted one — and D+31 is the age at which every month here is within 1% of final anyway. An adjustment is a device for the first month, not a permanent correction.

It breaks precisely when it matters. On 1 June 2026 the voluntary returns window went from the statutory 14 days to 30 — a growth experiment with no data consequence written on the ticket. For June, July and August the chain-ladder error is 4.40% against 0.296% in the stable months: 14.9× worse, and wrong in the same direction as the raw number, so it does not even reduce the damage.

Corrections & clarifications. This section was planned around a tidier claim: develop gross and refunds separately rather than the net figure, because they finish at different speeds. It is what an actuary would do, and measured here it is simply worse — 0.476% against 0.296% for the net basis, at every age (D+0, D+2, D+7, D+14, D+21) and every fit window (6, 12, 18 months) tested. The arithmetic: the net development factor is 0.91069 with a cross-month spread of 0.343%; the refund factor is 1.85831 with a spread of 2.357%, and refunds are 22.50% of net, so the refund factor alone injects 0.530% of error — 1.55× the net basis’s total. Predicted 1.55×; measured 1.61×. Splitting a quantity into a nearly-finished part and a half-finished part hands the whole estimate to the noisy half.

7You cannot re-read what you did not keep

Mechanism Cost What it answers Verdict
A transaction-time column (available_at, then where available_at <= '2026-09-02') 9.31 MB for 24 months of both tables any as-of, for any question — but only back to the day the column was added, and only for appended facts the one to buy
Snapshot the published answer (daily copy of the month × category aggregate) 61,320 rows, 2.81 MB a year “what did we say”, at the grain you thought to keep buy it as well
Warehouse time travel — BigQuery’s window “covers the past seven days by default”, settable “from a minimum of two days to a maximum of seven” free, already on any query over any table, for seven days reaches back to 4 Sep; the pack was written 2 Sep. Missed by 2 days; 0 of 24 board figures reproducible
Daily full snapshots of the fact tables 37.34 GB a year — 4106× the column the same as the column strictly worse, for 4106× the bytes

The cheapest form of time travel is a timestamp column, and it is the only one that answers questions you had not thought of yet.

Two limits: it reconstructs appended facts perfectly and amended ones not at all (a row corrected in place needs a version — L09 — or a change stream — L12); and it can never reach back before the day it was added, which is the only reason to add it this week rather than next quarter.

8There is no alarm for “the number changed”

Over the last 365 days the current month’s net GMV differed from its own previous-day value on 365 of 365 days. August’s closed figure was revised on all 11 days since the month ended, by between +0.52% and −0.76% a day. The daily net series crosses its own trailing 90-day envelope on 23 of 651 days (3.53%) — exactly the noise it has always had.

So a check on “yesterday’s number moved” fires every single day and is worth nothing, and a trailing band sees a drift (L12) rather than an event. That is not an instrumentation gap. There is nothing wrong to detect.

What is detectable, cheaply and with no ground truth, is a change in the shape of the delay — the thing that quietly invalidates every completion factor and every fixed-age comparison in the building:

Months Share of refunds posting > 30 days after the order Median lag
21 months before 1 June 2026 mean 0.51%, maximum 0.62% 15.1 days
June 2026 19.14% — 31.0× the largest value in twenty-one months 18 days

One group by, fires in the first month of the change, needs no threshold anybody has to defend — the gap between the old regime and the new one is a canyon rather than a tuning. Be exact about what it proves: it detects that the immaturity changed. It cannot detect the immaturity itself.

The sixteenth failure axis — the unsettled

Every row is true. Every test passes. The query is unchanged, the warehouse is fresh, nothing is stale, nothing is missing and nobody has made a mistake — and the answer is different every day, because the quantity being measured had not finished happening when it was measured.

It is not Lesson 24’s detached copy: there is no copy, and freshness is perfect. It is not Lesson 22’s absent rows: no row is missing that exists anywhere — the rows do not exist yet. Every other axis in this series is about a test failing to notice something; this one has no test to write, because there is nothing wrong. The only defence is representational: publish the age of the measurement next to its value, and never compare two numbers of different ages without saying so.

9What to print instead

Policy Restatement What it costs you
As reported — freeze the figure on publication day none, by construction the series records what you believed, not what happened: €5,311,846.26 (10.41%) of published net GMV never existed
Restated — re-run everything, every time total, every day (365/365) no two runs agree, and the newest period is always the immature one
Fixed maturity — always read at the same age (here weekly at D+8) none the level is still 17.53% high (sd 1.38 pp, range 15.56%–21.61%), but week-on-week growth differs from settled by only 0.60 pp on average, 3.33 pp at worst, with 6 sign flips in 103 weeks
~~Book refunds in the month they post~~ — the dodge −0.22% — settled on day two only 38.04% of the refunds posted in August belong to August’s orders: no cohort, no category return rate, no campaign payback

Fixed maturity makes a series comparable with itself, not correct: the 1 June change moved the level bias from 17.06% to 20.84%.

The rule worth writing down. Publish two series and one date. The as reported series, frozen, answers “what did we decide on” and is the only defensible thing to put in a minute. The restated series, live, answers “what happened”. Every tile carries the age of its newest period in days, and the current period carries a completion estimate rather than a bare number. And one measured threshold: no period enters a comparison until it is 31 days old.

10Ask the team

  1. “How old is this number?” If nobody can answer in days, nothing downstream can be trusted to a percent.
  2. “Can we reproduce last month’s board figure exactly as it was printed?” If not a flat yes, the fact table has no transaction-time column and your published history is unrecoverable past the time-travel window.
  3. “Which of our metrics keep moving after the period closes, and for how long?” A curve, not an opinion. Anyone can compute it this afternoon; everything else follows from having it.
  4. “When we compare this month to last year, are the two months the same age?” Usually one is two days old and the other four hundred.
  5. “What changed about our returns policy, our delivery times or our partner reporting this year?” Every one silently invalidates every completion factor in the building, and none arrives as a data ticket.

11Hands on (45 minutes, no warehouse)

The generator in the artifact writes orders.csv and refunds.csv beside itself (1,239,840 orders, 242,836 refunds).

  1. Run it; August’s three figures should come out to the cent.
  2. Build the maturity curve. Find the age at which your own worst month is within 1%.
  3. Re-run the board table with available_date <= the pack date and again with today’s; the difference is your restatement.
  4. Fit chain-ladder factors on twelve months, score the thirteenth, then score June, July and August 2026 and work out from the data alone what changed.
  5. Open one fact table in your own warehouse and answer whether it carries a column saying when each row became visible. If it does not, that is the ticket.
-- the as-of query: what the August pack said, and what the same query says now
select round(
         (select sum(gross_eur) from orders
           where order_date >= date '2026-08-01' and order_date < date '2026-09-01'
             and available_date <= date '2026-09-02')          -- <-- the whole idea
       - (select coalesce(sum(refund_eur),0) from refunds
           where order_date >= date '2026-08-01' and order_date < date '2026-09-01'
             and available_date <= date '2026-09-02'), 2) as net_gmv_as_of;
--  2026-09-02 -> 2669189.17     2026-09-11 -> 2517617.22     2026-12-31 -> 2316475.40

-- the monitor that works: the SHAPE of the delay, not the value
select date_trunc('month', order_date) as order_month,
       count(*)                                              as refunds,
       median(date_diff('day', order_date, refund_date))     as p50_lag_days,
       round(100.0 * count(*) filter (where date_diff('day', order_date, refund_date) > 30)
             / count(*), 4)                                  as share_over_30d
from refunds
where order_date < date '2026-09-01'
group by 1 order by 1;
--  2026-05: 15 days, 0.5767%   2026-06: 18 days, 19.1408%   <- fires in month one

(The triangle query and the as-of cohort query are in the artifact.)

12Takeaway

A figure is a measurement of a period taken at a moment, so it carries two dates. Almost every warehouse stores one of them. Once the second date is missing, “what was August” has no single answer — €2,669,189.17 on the day the board approved it, €2,517,617.22 today, €2,316,475.40 in the end, all three arithmetically correct — and the disagreement is invisible to every test you own, because nothing is wrong.

So stop trying to detect it and start representing it: put a transaction-time column on the fact table (9.31 MB bought this quarter’s history back), freeze what you publish, keep the restated series beside it, and stamp the age on every number that goes in front of a decision. The one number to get from your own data this week is the age at which your months stop moving. Here it is 31 days, and the board pack is written on day two.

Vocabulary

Term What it means, and the part people get wrong
Event time / processing time When something happened, versus when your system found out.
Valid time / transaction time The two clocks of a bitemporal table. Storing both makes as-of reporting possible; storing one makes it impossible, quietly.
As-of reporting Answering “what did this look like on date D”. Needs transaction time, a snapshot, or a time-travel window — none of which appears after the fact.
Restatement The same query returning a different, also-correct answer later. Not a bug, and not detectable as one.
Late-arriving fact The event happened, the row was slow. An engineering problem with an engineering fix.
Late-emerging fact The row was instant, the event was slow — a return, a chargeback, a churn confirmation. No pipeline fixes this; 98.55% of the restatement here.
Development / run-off triangle Origin period down, age across, the future missing from the corner. Borrowed from insurance reserving.
Development factor (chain ladder) Sum of the mature periods’ ultimates over the sum of the same periods at age a. A weighted average of your own history, and a bet that the history still applies.
Maturity / completion How much of a period’s final value is in yet. The one number worth computing for your own business today.
IBNR “Incurred but not reported” — the insurance name for the part of the period that exists in the world and not in your table. Your returns queue is IBNR.
Fixed-maturity reporting Always read at the same age. Comparable with itself; not correct.
Time travel The warehouse’s short memory of previous table states. BigQuery: seven days by default, two to seven configurable. Never long enough for a board pack.
Late-arriving dimension The other half: a fact whose dimension row does not exist yet. Handled with an inferred member (L05’s unknown member), never by dropping the fact.

13How these numbers were made

One deterministic generator, no random seed (splitmix64 over integer keys), simulating 1,239,840 orders and 242,836 refunds across 710,093 customers over 853 days of a second-hand fashion marketplace (AOV €61.61, 18.27% of gross value returned), with a returns-window change on 1 June 2026 and a weekly wholesale partner file. The last 111 days are generated but never published: they are the future the warehouse cannot see on 11 September, and the only source of the word “settled”.

  • The SQL printed in the artifact was run. Both CSVs loaded into DuckDB, the published queries executed and compared to the Python: 259 checks, 259 matched, 0 failed — including all 156 observable cells of the triangle, every category growth rate at both ages, every month’s refund-lag median and the sum of the twenty-four board figures.
  • The §06 claim was tested until it broke. Developing the components separately was the planned recommendation; it lost at every age and every fit window, so the recommendation changed.
  • One artefact was found and fixed rather than published. The first run had a 60-day repeat rate of 38.22% in the opening cohort, decaying for five months — an artefact of starting the customer pool at zero. The pool is now seeded with 180,000 customers acquired before the window, and the settled series is flat between 16.95% and 21.89%, which is what made the immature cohort visible as the anomaly it is.
  • Nothing here is a wall-clock measurement. Every figure is a count, a sum or a ratio.
  • Appendix reproduction check: generator extracted from the rendered HTML, run in a clean directory — 18 top-level keys, 2,977 scalar values, 0 differences, byte-identical to source, no container path leaked.

Sources for the non-simulated facts: § 356 BGB and Directive 2011/83/EU Articles 9, 13 and 14 (gesetze-im-internet.de and EUR-Lex); Google Cloud’s BigQuery time-travel documentation; and the Casualty Actuarial Society’s Methods and Models of Loss Reserving Based on Run-Off Triangles.


Series links: builds on L06 (the band that sees nothing here), L09 (validity intervals), L12 (drifts, and CDC for amended facts), L20 (one word, many correct answers), L22 (the ingestion-sequence watermark), L24 (a systematic bias divides out of a ratio — and this one does not) and L25 (a correctness SLI needs a second opinion; here it needs a second date).

Back to top