Tomorrow's Almanac

Paulina has the model review at ten. The return-risk score went live on 15 June 2026; it is now 17 September and there are thirteen weeks of it making real decisions. In May the model scored 0.8782 on a held-out fifth of the training data — comfortably over the 0.85 the team agreed was worth shipping. On the 77,158 orders it scored between launch and 9 August (the ones whose returns have settled) it scores 0.6895.

Three explanations get offered, in this order: the model is stale, so retrain weekly; the returns window changed from fourteen days to thirty on 1 June so the label means something different; the online feature store is serving slightly different numbers than training used. All three are testable and all three are wrong.

Two of the model’s sixteen inputs are aggregates over an account’s own history: seller_return_rate and buyer_return_rate. The training pipeline computes them with one GROUP BY over the whole table and joins on seller_id alone — no time key. A row from October 2025 is handed an aggregate computed in June 2026, which contains the outcome of the very order being predicted plus every order the account placed afterwards.

The decisive distinction is between two timestamps an order has:

• event time — when the thing happened;
• knowledge time — when the warehouse could first see it.

For an order row these are minutes apart. For a refund row they are 13.1 days apart on average and up to 39 — and the refund is what the feature counts.

One real training row (seller 14181, order placed 7 October 2025) under the three builds:

Build ② is what most people mean by “point-in-time”. It still reads a refund filed five days after the checkout it is being used to predict, because the filter is on the order’s date and the refund inherits it.

• buyer_return_rate as the pipeline computes it: 0.8101 AUC on its own.
• The same feature computed as of the decision: 0.5370.
• The buyer’s true latent propensity, which the feature is an estimate of: 0.5976.

An estimator cannot beat its own estimand. That one comparison is a proof of leakage that needs no lineage tool: if a derived feature outscores the thing it is derived from, it is not estimating, it is reading the answer.

The size is arithmetic: if an account has n orders in the table and one of them is this one, its all-time rate contains this order’s label at weight 1/n.

At n = 1 the feature is the label. Buyers are small — the median has 7 lifetime orders — so 44.0% of training rows sit in the two smallest buckets. Sellers are large (median 233) and the same bug is worth a quarter as much (0.6889 against 0.6286).

This is not a marketplace pathology. Any feature of the form “this entity’s historical rate of the thing we are predicting” has the structure, and the smaller the entity the larger the leak: account, device, postcode, listing, session.

“Was this order returned?” is not a fact about an order; it is a fact about an order and a date. On build day, 11.2% of training rows were younger than the longest possible refund lag.

The newest week shows 2.1% of its own returns. Across the whole year-long window the damage looks trivial — 3,204 rows (0.67% of the table), a headline return rate of 18.75% against a true 19.42%. An aggregate cannot see a defect concentrated in 11% of the rows, and the model has no column telling it which 11%. The fix costs patience only: hold the training window back by the longest lag the policy allows (here 53,310 rows), which costs 0.0000 AUC and removes the bias in the predicted rate.

The shipped model loses 0.1887 AUC between the holdout and the live period. The point-in-time build moves 0.0032 — very slightly up. The honest model looks 0.141 worse in the review and is 0.051 better in production, and reaches 96.2% of the signal there is.

Where the money is

Same number of interstitials, 5,931 more returns reached a year and a flag 21.2% more likely to be right. At €11.40 per return, every percentage point of the interstitial’s effectiveness is worth €676 a year. The finance accrual has no such escape: it is out by €1,053,829, 15.2% of the real number. A leak does not only cost accuracy; it biases the level, and a level is what finance books.

Retrain weekly. Eight rebuilds on eight consecutive Mondays: each reports between 0.8733 and 0.8774 on its own holdout and delivers between 0.6801 and 0.6939 on the week it was built for. The point-in-time rebuild averages 0.7406. Freshness is orthogonal to correctness: a pipeline that reads the future reads a more recent future.

The returns-policy change. It added 6,585 returns to the live period (11.01% → 19.54%). Re-scored against labels defined under the old window the model gets 0.6730 — 0.0166 worse. It moves the number the wrong way, by a twelfth of the gap.

A stale online store. Scoring with values as of the previous midnight instead of the instant of checkout: 0.6895 against 0.6895. A day of staleness is worth 0.00001 AUC; the join key is worth 0.0510, about a thousand times more. The thing with a dashboard is not the thing with the damage.

The importance check passes with the sign reversed: deleting the leaky feature costs 0.1085 AUC offline and gains 0.0460 in production.

The parity check is the interesting failure. “Log a day of live feature values, recompute them with the training pipeline, compare” is the standard defence against training/serving skew. On live traffic: 16.1% of rows differ by more than 1 pp, r = 0.982 — it passes, because a build dated June cannot contain the future of a July order. On a training row: 82.3% differ, r = 0.680. Across the whole training set: 79.9%, r = 0.728. Training/serving skew is measured on live traffic by every tool that measures it. The skew lives in the training set. On both days the two means differ by hundredths of a point, so a check on distributions rather than rows passes everywhere.

Read from the shipped Feast 0.66.0 wheel, not the documentation:

# feast/infra/offline_stores/bigquery.py — the point-in-time join template
    AND subquery.event_timestamp <= entity_dataframe.entity_timestamp
    {% if filter_by_created_timestamp and featureview.created_timestamp_column %}
    AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
    {% endif %}

# feast/feature_store.py
def get_historical_features(..., filter_by_created_timestamp: bool = False, ...)

# feast/data_source.py
self.created_timestamp_column = created_timestamp_column if created_timestamp_column else ""

The join always filters on the event timestamp. The knowledge-time cutoff needs two conditions: the source must declare a created_timestamp_column (defaults to the empty string) and the caller must pass a flag (defaults to False). Out of the box you get build ②. With the flag off, created_timestamp is used to take MAX(created_timestamp) per event timestamp and as an ORDER BY created_timestamp DESC tiebreak — that is, to prefer the most recently written version of a row, the exact opposite of point-in-time. Snowflake’s offline store leaves the flag off entirely and says why: supports_filter_by_created_timestamp stays False: the ASOF JOIN cannot express a created_timestamp cutoff.

How much does build ② cost here? Almost nothing: 0.7400 against 0.7405. The two differ only over refunds filed between the decision and the build, which against a lifetime denominator is a small correction. The error is governed by the ratio of the knowledge lag to the feature’s window, and short windows are exactly the ones people add for freshness:

The two means never differ by more than 0.70 pp at any window length.

-- the running record of what was knowable, per seller
create table seller_hist as
  select seller_id, ts,
         sum(dn) over w as n_orders,
         sum(dk) over w as n_returns
  from (select seller_id, order_ts        as ts, 1 as dn, 0 as dk from fct_orders
        union all
        select seller_id, refund_posted_at as ts, 0 as dn, 1 as dk from fct_orders
        where returned)            -- the refund enters when it POSTS
  window w as (partition by seller_id order by ts
               rows between unbounded preceding and current row);

-- one aggregate per row, as of that row's own instant
select o.order_id, o.returned,
       (coalesce(h.n_returns,0) + 4*0.194326)
         / (coalesce(h.n_orders,0) + 4) as seller_return_rate
from fct_orders o
asof left join seller_hist h
  on h.seller_id = o.seller_id
 and h.ts < o.order_ts;

And the audit that would have stopped the first build from being approved — no model, no holdout, only the two timestamps:

select avg(case when n_all > n_prior then 1 else 0 end) as rows_reading_the_future,
       median(n_all)           as inputs_read,
       median(n_all - n_prior) as inputs_that_had_not_happened
from feature_audit
where order_day between 273 and 637;
-- 1.0    233    53          (sellers)
-- 1.0      7     3          (buyers)

Every one of the 476,233 training rows fails it. On average 31.8% of what a feature reads is the future.

The acceptance test for a model is not a holdout, it is a replay. Pick a Monday in the past, rebuild using only rows whose knowledge time is before it, train, score the week that follows. Eight replays cost minutes of compute and would have reported the whole problem: claimed 0.8754, delivered 0.6897, eight weeks out of eight.

• “Which column says when we found out?” Not when it happened — when the row appeared. If it is overwritten on every refresh, there is no knowledge time and no point-in-time join is possible, only one that looks like it.
• “What is the lag between the two timestamps for each source, and how long is the shortest feature window?” The ratio is the size of the problem.
• “Show me the replay, not the holdout.” One past Monday, one week forward.
• “Does any feature score better than the thing it estimates?” Where no ground truth exists, the proxy is: does the feature score better on small accounts than on large ones? It should not.
• “How old is the youngest row we trained on, and how long does the label take?”
• “Is filter_by_created_timestamp on?” Ask for the line of code, not the intention.

Run the generator embedded in the artifact (numpy only; it writes figures.json).

1. Open leak_by_buyer_size. Confirm the first bucket reads exactly 1.000 and work out why that is arithmetic rather than a bug; then why the point-in-time column reads exactly 0.500 there.
2. Change prior_w in rate() from 4.0 to 40.0 — heavy smoothing, the usual advice for noisy rates. The leak shrinks and does not go away: smoothing compresses the signal, it does not remove the label from the numerator.
3. Set the refund posting lag to zero. Builds ② and ③ become the same number — the whole event-time/knowledge-time distinction is proportional to how long the truth takes to arrive.
4. Add a fifth model: point-in-time features, settled labels, trained on six months instead of twelve. Compare its production AUC to build D and decide whether the extra half-year of leaky data was ever worth anything.

Eighteen lessons of failures, and every one made something worse: a wrong row, a missing row, a stale copy, a number that changed. This one makes everything better. The offline AUC is higher, the feature importance stronger, the holdout cleaner.

The anachronistic. Every row is true, every join key matches, every filter does what it says — and the value attributed to a moment could not have been known at that moment. The unit of failure is a timestamp that is not in the table. Its signature is inversion: for the first eighteen shapes the defect degrades the measurements, here it flatters them, so every quality check improves as the failure worsens. It is the only shape with no wrong value anywhere to find, and therefore the only one whose sole detector is a replay that deliberately withholds the future.

It generalises past machine learning: any backtest of a pricing rule, any “what would this alert have caught” exercise, any cohort analysis joining today’s customer segment onto last year’s orders, any capacity plan validated against a history that has since been restated. All the same join, all flattering by default.

It also completes an argument running since Lesson 26. A warehouse that stores only when things happened can answer what is true now. A warehouse that also stores when things became known can answer what was true then — what a restatement, an audit, a backtest and a training set all need. Lesson 26 priced the column: 9.31 MB a year against 37.34 GB for snapshotting the table.

Vocabulary

Event time / valid time — when the thing happened. Knowledge time / transaction time / created timestamp — when your system could first have known it; a bitemporal table stores both. Point-in-time join / as-of join — attaching to each row the aggregate as it stood at that row’s own instant; correct only when the cutoff is on knowledge time. Target leakage — a feature containing, directly or through an aggregate, the outcome it predicts; self-inclusion is the common form and its size is 1/n. Label maturity — how much of an outcome has finished happening when you write it down. Training/serving skew — offline and online computations of a feature disagreeing. Offline store / online store — the historical feature table used for training and the low-latency one read at inference. Replay / walk-forward validation — rebuilding the pipeline as of a past instant and scoring forward; the only evaluation a leak cannot pass. Backfill — writing rows dated in the past; every backfill creates a gap between the two timestamps.

How these numbers were made

One deterministic generator: 952,370 orders over 746 days, 24,000 sellers and 300,000 buyers, latent return propensity per account, refund posting lag drawn from the returns policy in force. No random seed — every draw is a splitmix64 finalizer over (row index, salt). Logistic regressions fitted by IRLS from a zero start; AUC is the exact Mann-Whitney rank statistic with average ranks for ties.

Two things were checked rather than asserted. The published SQL — the running-history tables, the ASOF JOIN, the timestamp audit, the label-maturity buckets, and AUC and recall as rank statistics — was re-run in DuckDB over the generator’s own tables: 74 checks, 74 matched, 0 failed. The generator was extracted back out of the rendered artifact, run in a clean directory and diffed key by key: 29 top-level keys, 3,572 scalar values, 0 differences, byte-identical to source. The Feast quotations are from the shipped 0.66.0 wheel.

Three claims are the generator’s design rather than its findings, and are labelled as such in the artifact: the interstitial’s effectiveness is never assumed, only broken even against; the ceiling is computable only because the generator knows each order’s true probability; and the €11.40 per-return cost is quoted from operations and used in exactly one multiplication.