Feature Pipelines and Point-in-Time Correctness

A model is trained on rows, and each row makes an unstated claim: at the moment this prediction would have been made, the system could see these input values. If the claim is false, the training set describes a world the production system never had. The model can look excellent offline and disappoint in service, and nothing in the training code will say why.

This article is about keeping that claim true. A feature pipeline is the set of jobs that turn raw events and reference tables into model inputs, store them, and make them available both for building training sets and for answering live prediction requests. Point-in-time correctness is the property that every feature value attached to a training example is the value the production system would have had at that example’s prediction time, no earlier knowledge and no later. The article covers four things in order: the clocks that a feature value carries, the consistency problem between training and serving, the leakage paths that break the claim, and the backfills and tests that keep it true as pipelines change.

The reader is assumed to know SQL joins, batch jobs, and basic supervised learning vocabulary (features, labels, train and test splits). From the earlier articles in this series, the discussion leans on two ideas named by metadata, lineage, and impact analysis and access, retention, and erasure: feature tables are data assets that need owners and traceable inputs, and stored feature history is subject to retention and deletion duties. Nothing below depends on the details of those articles. The worked example uses invented data and a small SQLite program written for this article.

A feature value is a claim about time

Start with a concrete, hypothetical setting. A retailer scores each customer every afternoon with a model that predicts whether the customer will place an order in the next seven days. The label for a training example is the observed outcome in the seven days after the prediction time. The inputs include the customer’s loyalty tier and the number of orders in the last seven days.

A feature has a definition (how the value is obtained) and a value (the result for one example). The definition for the second input is: count the customer’s orders whose event time falls in the half-open interval (t − 7 days, t], where t is the prediction time. That definition looks complete, and it is not. It does not say which orders are visible. Suppose a store’s point-of-sale system uploads its transactions in a nightly sync. An order placed on Monday afternoon exists in the real world on Monday, but the feature service learns about it on Tuesday morning. A prediction made on Monday evening could not have counted it, yet a training query that runs next week sees it in the table and counts it.

So every feature value has to be reasoned about with three questions:

1. What does the value describe, and when was that true in the real world? This is the value’s event time or effective time.
2. When did the system learn the value, in a form that serving could use? This is its availability time, also called record time or knowledge time.
3. For which prediction time is it being selected? This is the as-of time.

A training row is point-in-time correct when, for each feature, the selected value satisfies both clock conditions relative to the as-of time: it was effective at or before that time, and the system had it by then. The rest of the article elaborates what that rule requires from storage, joins, serving, and tests.

The clocks: event time, effective time, and availability

Several timestamps appear in a feature pipeline, and mixing them up is the most common root cause of silent errors. The table lists the ones that matter and what each answers.

Clock	Meaning	Typical column	Question it answers
Event time	When the modeled occurrence happened	event_time	When did this order happen?
Effective (valid) time	When a fact starts to hold in the business world	effective_from	Since when has this customer been gold tier?
Ingestion or record time	When a system recorded the fact	ingested_at, recorded_at	When could a consumer first see it?
Materialization time	When a pipeline wrote a computed feature value to a store	created_at	When did this feature row come into existence?
Prediction (as-of) time	When the model is asked to decide	prediction_time	What should the model have seen?
Label time	When the outcome becomes known	label_available_at	When can this example be used for training or evaluation?

The event time versus processing time distinction is familiar from stream processing: the time an event happened and the time an operator handled it can differ, and replaying the stream does not change the first. Effective time and record time are the same idea applied to slowly changing facts. Valid time and system time name the two axes, and Martin Fowler’s description of bitemporal history uses the labels actual and record. His payroll example makes the point cleanly: on one date the company paid an employee at the old rate, and a letter received weeks later said the raise had been effective earlier. Asked what the salary was on the payroll date, “what we now know” and “what we thought then” are both legitimate answers to different questions. A feature pipeline that trains a model to behave as the production system would have behaved needs the second answer.

The as-of selection rule

For one entity, one feature, and one as-of time t, the eligible values are those that satisfy all of the following:

• Effective time is at or before t.
• Record time is at or before t.
• If the feature has a freshness limit (a time-to-live, TTL), t minus the effective time does not exceed it.

Among the eligible values, choose the one with the latest effective time. When two eligible values share an effective time, which happens when a correction restates the same moment, choose the one with the latest record time. If nothing is eligible, the result is missing, and the pipeline must say whether missing means null, a default, or an excluded example, consistently in training and serving. This is the rule behind a point-in-time join with one extra condition (the record-time filter) that many implementations leave out by default.

Windowed aggregates follow the same rule with a different shape. An order contributes to the seven-day count for as-of time t when its event time is inside (t − 7 days, t] and its ingestion time is at or before t. The half-open interval matters: an order at exactly t − 7 days is counted by the prediction made seven days earlier and not by this one, so no instant belongs to two adjacent windows or to none.

What a feature store does and does not do here

Feast, an open-source feature store, shows how one system exposes these clocks. According to its documentation on point-in-time joins, historical retrieval takes an entity dataframe whose timestamps are the moments at which the user wants to reproduce the state of the world, and for each row it scans backward in time from that timestamp up to the feature view’s TTL. The TTL is measured relative to each row’s own timestamp, not to the moment the query runs. The retrieval documentation describes the entity timestamp as an inclusive upper bound: the join takes the latest value at or before it.

The same page then states the caveat that matters for this article. By default the join constrains only the feature’s event timestamp. If the data source also has a created-timestamp column, that column is used to break ties between rows with the same event timestamp, the highest created timestamp winning, but it is not otherwise filtered. A value that was backfilled or corrected after the entity timestamp can therefore still be returned for that entity row. The documentation describes an opt-in flag, filter_by_created_timestamp=True, that adds a created-timestamp condition so each row sees only values created at or before its own timestamp, and says it is meaningful only if the created timestamp reflects when the value became available online. It also notes that rows with a null created timestamp are excluded when the flag is on, and that not every offline store supports the flag, in which case an error is raised rather than the flag being ignored. Check the behavior of the version and offline store you actually run; the point for design purposes is that availability filtering is something you must ask for and must supply timestamps for. A feature store gives you the mechanism, not the guarantee. See the glossary entry on a feature store for the broader trade-offs.

Two paths, one definition: training and serving consistency

A deployed model needs its inputs in two situations that behave differently. Training reads history in bulk and can afford joins and scans. Serving answers one request at a time under a latency budget, reading precomputed values or computing them from recent data. Google’s MLOps architecture guidance describes a feature store as a component that standardizes the definition, storage, and access of features for training and serving and offers both a high-throughput batch interface and a low-latency one. It lists avoiding training-serving skew as one purpose: the same store is the data source for experimentation, continuous training, and online prediction.

The same guidance, in its description of a process where data scientists hand a model to engineers, points at the origin of the problem. The engineers must make the required features available for low-latency serving, and that step can introduce training-serving skew. Google’s Rules of Machine Learning separates the skew into three observable gaps: between training and holdout performance (always present, not always bad), between holdout and next-day data (large drops suggest time-sensitive features), and between next-day data and live data, where the same example run through the training path and the serving path should produce exactly the same result, so a discrepancy there is “probably an engineering error.”

A map of the two paths

The following text diagram shows where the two paths share work and where they diverge. It is a generic architecture, not any one product.

raw events, reference tables
        |
        v
 feature transformations (one definition, versioned)
        |
        +--> offline store: full history, with effective + record time
        |         |
        |         +--> as-of join per (entity, prediction_time) --> training set
        |
        +--> materialization job (latest value per entity, plus freshness stamp)
                  |
                  v
            online store --> serving lookup --> model input
                                   |
                                   +--> serving log (values actually used) --> parity checks

Skew enters at each arrow that is implemented twice. The table lists the typical sources, with the check that exposes each.

Source of skew	How it happens	What exposes it
Different logic	The batch job and the service reimplement the same formula, possibly in different languages	Run both on the same raw records and compare feature vectors
Different data	Training reads a table corrected after the fact; serving read the pre-correction values	Compare served values against reconstruction from record-time-filtered history
Different time semantics	Training selects by event time only; serving can only use what had arrived	The future-perturbation test described below
Different freshness	Online store updated hourly; training assumes values are current to the minute	Compare value age at serving to the TTL assumed in training
Different defaults	A missing value is null in training and zero in serving	Compare null and default rates per feature across the paths
Mutable joins	Training joins a table that changes between training and serving time	Snapshot the table or log served values

Rules of Machine Learning addresses the last row directly: if you join a table at training and at serving time, its contents may change between the two, and the simplest remedy is to log features at serving time. It also gives two broader remedies. Rule 32 recommends reusing code between training and serving where possible, for example by gathering all inputs into one human-readable object and passing it through a single function that produces the model’s format, and it warns against using different programming languages for the two paths. Rule 29 recommends saving the features used at serving time and piping them to a log for use at training time, and notes that even a small logged fraction lets a team measure consistency.

Log-and-replay versus reconstruct

Two strategies produce training features. Log-and-replay trains on the values the service actually used. It is exact for those requests, because the log is the truth about what the model saw. Its limits are structural. It can only supply features that existed in production when logging began, so it cannot evaluate a new feature, and it inherits the sampling and logging gaps of the serving system. Reconstruction recomputes feature values for historical prediction times from stored raw history using the as-of rule. It can create a feature that never ran in production, and it can regenerate a training set after a bug fix. Its limit is fidelity: it is only as faithful as the recorded timestamps, and the serving system’s caching, expiry, and fallback behavior must also be reproduced to claim equality with what was served.

Mature setups use both. Reconstruction produces the training sets. A sample of served-feature logs becomes the reference to which reconstruction is compared, which turns the claim “our offline features match online” into a measurement with a number attached.

A worked example: three ways to build the same training rows

The following program builds the same small set of prediction rows in three ways and compares them. The data are invented for this article; nothing in them is a measurement. Python 3.12 and SQLite 3.53 were used by the author to run the program and produce the output shown after it. The code uses only the standard library.

The setup has three customers, seven orders, and six tier records. Two orders reach the system days after they happened. One tier change is recorded four days after its effective date, and one tier record is a correction that restates the same effective time as an earlier record. There are nine predictions, three customers at each of three prediction times.

"""Point-in-time worked example: illustrative data, Python standard library only (sqlite3)."""
import sqlite3
from datetime import datetime, timedelta

FMT = "%Y-%m-%d %H:%M"

SCHEMA = """
CREATE TABLE order_events (order_id TEXT, customer_id INT, event_time TEXT, ingested_at TEXT);
CREATE TABLE tier_history (customer_id INT, tier TEXT, effective_from TEXT, recorded_at TEXT);
CREATE TABLE predictions  (prediction_id INT, customer_id INT, prediction_time TEXT);
"""
ORDERS = [
    ("o1", 1, "2025-01-02 10:00", "2025-01-02 10:05"),
    ("o2", 1, "2025-01-09 18:00", "2025-01-13 08:00"),   # arrives four days late
    ("o3", 1, "2025-01-11 09:00", "2025-01-11 09:02"),
    ("o4", 2, "2025-01-05 12:00", "2025-01-05 12:01"),
    ("o5", 2, "2025-01-10 20:00", "2025-01-10 20:01"),
    ("o6", 3, "2025-01-14 11:00", "2025-01-16 09:00"),   # arrives two days late
    ("o7", 3, "2025-01-12 08:00", "2025-01-12 08:03"),
]
TIERS = [
    (1, "silver", "2025-01-01 00:00", "2025-01-01 00:00"),
    (1, "gold",   "2025-01-10 00:00", "2025-01-14 00:00"),  # effective Jan 10, recorded Jan 14
    (2, "silver", "2025-01-01 00:00", "2025-01-01 00:00"),
    (2, "gold",   "2025-01-01 00:00", "2025-01-16 00:00"),  # correction of the same effective time
    (3, "silver", "2025-01-01 00:00", "2025-01-01 00:00"),
    (3, "gold",   "2025-01-13 00:00", "2025-01-13 00:00"),
]
PRED_TIMES = ["2025-01-12 12:00", "2025-01-15 12:00", "2025-01-18 12:00"]


def load(extra_orders=()):
    db = sqlite3.connect(":memory:")
    db.executescript(SCHEMA)
    db.executemany("INSERT INTO order_events VALUES (?,?,?,?)", ORDERS + list(extra_orders))
    db.executemany("INSERT INTO tier_history VALUES (?,?,?,?)", TIERS)
    pid = 0
    for pt in PRED_TIMES:
        for cid in (1, 2, 3):
            pid += 1
            db.execute("INSERT INTO predictions VALUES (?,?,?)", (pid, cid, pt))
    return db


def minus7d(ts):
    return (datetime.strptime(ts, FMT) - timedelta(days=7)).strftime(FMT)


def build(db, mode):
    """mode 'event_time_only' ignores when data became known; 'known_at' enforces it."""
    db.create_function("minus7d", 1, minus7d)
    known_orders = "AND o.ingested_at <= p.prediction_time" if mode == "known_at" else ""
    known_tiers = "AND t.recorded_at <= p.prediction_time" if mode == "known_at" else ""
    sql = f"""
    SELECT p.prediction_id,
           (SELECT t.tier FROM tier_history t
             WHERE t.customer_id = p.customer_id
               AND t.effective_from <= p.prediction_time {known_tiers}
             ORDER BY t.effective_from DESC, t.recorded_at DESC LIMIT 1) AS tier,
           (SELECT COUNT(*) FROM order_events o
             WHERE o.customer_id = p.customer_id
               AND o.event_time >  minus7d(p.prediction_time)
               AND o.event_time <= p.prediction_time {known_orders}) AS orders_7d
    FROM predictions p ORDER BY p.prediction_id"""
    return {r[0]: (r[1], r[2]) for r in db.execute(sql)}


def replay_serving(db):
    """Independent online path: consume arrivals in arrival order, answer requests from state."""
    timeline = []
    for oid, cid, et, ing in db.execute("SELECT * FROM order_events"):
        timeline.append((ing, 0, ("order", cid, et)))
    for cid, tier, eff, rec in db.execute("SELECT * FROM tier_history"):
        timeline.append((rec, 0, ("tier", cid, tier, eff, rec)))
    for pid, cid, pt in db.execute("SELECT * FROM predictions"):
        timeline.append((pt, 1, ("predict", pid, cid, pt)))   # arrivals at the same instant win
    timeline.sort(key=lambda x: (x[0], x[1]))
    orders, tiers, served = [], [], {}
    for _, _, ev in timeline:
        if ev[0] == "order":
            orders.append(ev[1:])
        elif ev[0] == "tier":
            tiers.append(ev[1:])
        else:
            _, pid, cid, pt = ev
            lo = minus7d(pt)
            n = sum(1 for c, et in orders if c == cid and lo < et <= pt)
            cands = [(eff, rec, tier) for c, tier, eff, rec in tiers if c == cid and eff <= pt]
            served[pid] = (max(cands)[2] if cands else None, n)
    return served


def mismatches(a, b):
    return sorted(k for k in a if a[k] != b[k])


db = load()
served = replay_serving(db)
naive = build(db, "event_time_only")
known = build(db, "known_at")
print("== 1. training rows versus the replayed serving log")
print("pred cust  pred_time         served          event_time_only  known_at")
for pid, cid, pt in db.execute("SELECT * FROM predictions"):
    print(f"{pid:>4} {cid:>4}  {pt}  {str(served[pid]):<15} {str(naive[pid]):<16} {known[pid]}")
print("rows differing from serving: event_time_only =", len(mismatches(naive, served)),
      "| known_at =", len(mismatches(known, served)), "of", len(served))

print("\n== 2. future-perturbation test (append rows that arrive after every prediction)")
poison = [(f"x{c}", c, "2025-01-12 11:00", "2025-01-30 00:00") for c in (1, 2, 3)]
db2 = load(poison)
for mode, before in (("event_time_only", naive), ("known_at", known)):
    after = build(db2, mode)
    print(f"{mode:<16} features changed for {len(mismatches(before, after))} of {len(before)} predictions")

print("\n== 3. range-limited backfill versus backfill with lookback (orders_7d at 00:00)")
db.create_function("minus7d", 1, minus7d)
def daily(lo_bound):
    out = {}
    for day in ("2025-01-10 00:00", "2025-01-11 00:00", "2025-01-12 00:00"):
        for cid in (1, 2, 3):
            n = db.execute(
                "SELECT COUNT(*) FROM order_events WHERE customer_id=? AND event_time>? "
                "AND event_time<=? AND event_time>=?", (cid, minus7d(day), day, lo_bound)).fetchone()[0]
            out[(day, cid)] = n
    return out
full = daily("0000-00-00 00:00")
limited = daily("2025-01-10 00:00")
print("day               cust  with_lookback  output_range_only")
for k in full:
    print(f"{k[0]}  {k[1]:>4}  {full[k]:>13}  {limited[k]:>17}")

Several design choices in the code are worth stating before reading its output.

The build function takes one parameter, mode. The only difference between event_time_only and known_at is two extra predicates, ingested_at <= prediction_time and recorded_at <= prediction_time. Everything else is shared, including the window boundaries and the tie-break ORDER BY effective_from DESC, recorded_at DESC. That keeps the experiment honest: whatever differs in the output is attributable to the record-time filter alone.

The replay_serving function is a separate implementation of the same feature definitions. It does not run SQL. It merges three event streams (order arrivals, tier recordings, and prediction requests) into one timeline sorted by arrival time, keeps in-memory state, and answers each request from state. At equal timestamps arrivals sort before requests, an explicit tie-break that mirrors the <= in the SQL. Because it can only use what has arrived, it stands in for the serving log. In a real system the serving log would be recorded by the service itself; here it is simulated, which is acceptable for showing the mechanism and would not be acceptable as evidence about a real pipeline.

The window predicate is event_time > minus7d(t) and event_time <= t, the half-open interval from the definition above. Order o4 for customer 2 happens at exactly 2025-01-05 12:00, seven days before the first prediction, and is excluded from that prediction’s count.

Reading the output

The program prints the following. The first block is one row per prediction with the three results as (tier, orders in the last seven days).

== 1. training rows versus the replayed serving log
pred cust  pred_time         served          event_time_only  known_at
   1    1  2025-01-12 12:00  ('silver', 1)   ('gold', 2)      ('silver', 1)
   2    2  2025-01-12 12:00  ('silver', 1)   ('gold', 1)      ('silver', 1)
   3    3  2025-01-12 12:00  ('silver', 1)   ('silver', 1)    ('silver', 1)
   4    1  2025-01-15 12:00  ('gold', 2)     ('gold', 2)      ('gold', 2)
   5    2  2025-01-15 12:00  ('silver', 1)   ('gold', 1)      ('silver', 1)
   6    3  2025-01-15 12:00  ('gold', 1)     ('gold', 2)      ('gold', 1)
   7    1  2025-01-18 12:00  ('gold', 0)     ('gold', 0)      ('gold', 0)
   8    2  2025-01-18 12:00  ('gold', 0)     ('gold', 0)      ('gold', 0)
   9    3  2025-01-18 12:00  ('gold', 2)     ('gold', 2)      ('gold', 2)
rows differing from serving: event_time_only = 4 | known_at = 0 of 9

== 2. future-perturbation test (append rows that arrive after every prediction)
event_time_only  features changed for 9 of 9 predictions
known_at         features changed for 0 of 9 predictions

== 3. range-limited backfill versus backfill with lookback (orders_7d at 00:00)
day               cust  with_lookback  output_range_only
2025-01-10 00:00     1              1                  0
2025-01-10 00:00     2              1                  0
2025-01-10 00:00     3              0                  0
2025-01-11 00:00     1              1                  0
2025-01-11 00:00     2              2                  1
2025-01-11 00:00     3              0                  0
2025-01-12 00:00     1              2                  1
2025-01-12 00:00     2              2                  1
2025-01-12 00:00     3              0                  0

Section 1 compares the training rows to what the replayed service would have used. The known_at build matches the serving log on all nine rows. The event_time_only build differs on four rows (predictions 1, 2, 5, and 6). Each can be verified by hand from the input lists:

• Prediction 1 (customer 1, 2025-01-12 12:00). The window is (Jan 5 12:00, Jan 12 12:00]. Order o2 happened Jan 9 inside the window but arrived Jan 13, so the service saw only o3: count 1. The event-time-only query also counts o2: count 2. The tier differs too: the gold record is effective Jan 10 but was recorded Jan 14, so the service still saw silver.
• Prediction 2 (customer 2, same time). The tier correction is effective Jan 1, the same as the original silver record, but was recorded Jan 16. The tie-break prefers the later record, so an event-time-only build says gold where the service said silver.
• Prediction 5 (customer 2, Jan 15). Same correction, still not recorded at prediction time.
• Prediction 6 (customer 3, Jan 15 12:00). Order o6 happened Jan 14 but arrived Jan 16, so the count is 1 in service and 2 in the event-time-only build.

Predictions made at 2025-01-18 12:00 agree in all builds because by then every record had arrived. This is the pattern to expect in real data: disagreement concentrates in recent predictions relative to the delay in recording. A training set built long after the fact looks fine for old rows and wrong for rows close to the data delay, and a naive aggregate agreement figure (for example 5 of 9 rows equal) hides which rows are wrong.

Section 2 introduces a test that is more general than the example. After building features, the program appends rows that, by construction, could not have been known at any prediction time: an order with an event time before the first prediction but an ingestion time of Jan 30, after the last. A point-in-time-correct build must not change when such rows appear. The known_at build changed for 0 of 9 predictions. The event_time_only build changed for 9 of 9. This is a future-perturbation test, and because it does not need a serving log it can run in continuous integration on any feature definition that has both an event time and an availability time.

Section 3 turns to backfills and is discussed below.

Limits of the example

The example establishes a mechanism, not a rate. The four-of-nine mismatch is a property of the invented delays and says nothing about how often a real pipeline would disagree. The example also assumes that ingested_at and recorded_at are trustworthy. In real systems that assumption is the weak point: a column that is rewritten when a table is reloaded, or filled by the loading job’s clock instead of the source’s, silently turns availability time into something else. The glossary entry on valid time and system time makes the same point: a loaded_at column on a row that is overwritten does not preserve earlier system versions.

Leakage: the ways a feature row can know too much

Data leakage occurs when model development or evaluation uses information outside the intended prediction boundary. The scikit-learn documentation states the core definition: information that would not be available at prediction time is used when building the model, which yields overly optimistic performance estimates and worse performance on genuinely new data. In a feature pipeline the boundary is the as-of time, and there are at least five distinct ways to cross it.

1. Availability leakage. The training join selects values by effective time only. This is the case in the worked example: late-arriving orders and late-recorded tier changes enter rows that predate their arrival. It is subtle because every individual value is correct for its effective time; the error is in the claim that the system could use it. The glossary entry on a late arriving fact describes the mirror-image problem in dimensional modeling, where a late fact attached to the current dimension row uses the wrong version.
2. Current-state joins. A training query joins a table that holds only the current value, for example a customer’s current tier or current address. Every historical row then sees today’s value. The remedy is a history table with effective intervals, or snapshots taken at known times, joined by the as-of rule.
3. Label-window overlap. A feature is computed from events that fall inside the label’s observation window, such as a seven-day count that includes days after the prediction time. This happens through off-by-one window bounds or through a feature defined relative to the label date rather than to the prediction date. It often produces one dominant feature with implausibly high importance.
4. Whole-dataset statistics. A scaler, imputer, encoder, or target encoding is fitted on the entire dataset before splitting, so statistics from evaluation rows influence training features. The scikit-learn guidance is to split first and fit preprocessing on the training subset only, then apply the learned transformation to evaluation data. For time-ordered data, fit on the period before the cut.
5. Revised labels and revised history. A source restates past values, and a rebuilt training set uses the restatement for rows whose prediction time predates it. The same thing happens when a bug fix is backfilled and the corrected value is stamped as if it had always existed.

The first four can be caught by design; the fifth needs versioning discipline, covered under backfills below.

Splitting by time is necessary, and not enough

A chronological split keeps training rows before evaluation rows, but it does not repair a feature computed from later events. The glossary entry on time-based splits lists the additional conditions: a label whose outcome window crosses the boundary needs purging of training rows whose labels are not yet known at the cutoff, and a feature known only three days after the timestamp it carries is unavailable at prediction time however early its timestamp is. Rules of Machine Learning gives a related practical rule: if the model is trained on data up to a date, test on data after that date. Add a gap equal to the label delay between the last training example and the first evaluation example, measured from timestamps, not from a row count.

Detection

Suspicion signals include an unexpectedly high offline score, one feature carrying most of the weight, and a drop after deployment. None of them is proof, and each has other causes. What does give evidence is the combination of the checks in this article: the future-perturbation test for availability leakage, a serving-log comparison for skew, and boundary tests on window definitions for label-window overlap. A data versioning record that ties each training run to input versions and feature definitions lets you re-run a suspicious training set under corrected rules and compare.

Feature backfills

A backfill populates or recomputes feature values for a historical range. The glossary entry on a data backfill lists what a backfill must specify: output dates, historical input version, transformation and reference-data versions, and a publication plan. In a feature pipeline there are two quite different reasons to backfill, and they impose different rules.

• A new feature needs history. A feature that did not exist when earlier predictions were served must be computed for those past prediction times so that models can be trained with it. Nothing was ever served, so there is no log to compare to; the backfill must simulate what the pipeline would have known.
• An existing feature was wrong. A bug in the transformation or a defective input corrupted stored values. The backfill restates history. Models trained on the old values saw a different feature from the one that now exists.

Lookback: output range is not input range

A trailing-window feature at as-of time t needs input from t minus the window length. Backfilling only the output dates from only the input dates yields wrong values at the start of the range. The table is the output of section 3 of the program, which computes the seven-day order count at midnight on three days. It uses event time only, so that the lookback effect is isolated from the availability effect.

Day (00:00)	Customer	Count with lookback input	Count from output-range input only
Jan 10	1	1	0
Jan 10	2	1	0
Jan 10	3	0	0
Jan 11	1	1	0
Jan 11	2	2	1
Jan 11	3	0	0
Jan 12	1	2	1
Jan 12	2	2	1
Jan 12	3	0	0

Six of the nine cells differ, and every difference is an undercount. Customer 3 is unaffected only because that customer has no orders in the window before Jan 12 at all. The error is largest on the first day and shrinks as the range advances, because later windows contain more of the backfilled range itself. That shape is a useful diagnostic: if a backfilled feature looks wrong only at the start of the range, suspect missing lookback input before suspecting the formula.

The lesson is general. A backfill range must be extended backward by the maximum lookback of any feature it computes, and the extension is for input only: outputs are written for the requested range. Features with state, such as running totals or sessionization, need state initialized from before the range, which in practice often means the backfill starts from a snapshot of state and not from zero.

Which timestamp does a backfilled row carry?

This is the decision that most often goes unexamined. A backfilled feature row has an effective time (the historical as-of date it describes) and an availability time. There are two defensible stamps for availability, and they do opposite things in an as-of join with availability filtering.

Stamp on backfilled rows	Behavior in an as-of join that filters on availability	Appropriate use
Backfill run time	Excluded for every historical prediction time earlier than the run	Reproducing what serving could actually see; auditing
Reconstructed knowledge time (the as-of time it simulates)	Included for the prediction times it was built for	Training a new feature on history

The Feast documentation’s description of the created-timestamp filter shows the first behavior: with the filter on, a value created after an entity row’s timestamp is not visible to it. That is correct when the aim is to reproduce the online store, and it makes a naive “backfill with today’s timestamp” invisible to training. The second stamp makes backfilled history usable for training, but the rows then claim knowledge the real system never had. They are valid only if the backfill computed each value from raw inputs filtered by their own record times, which is exactly what the known_at build in the worked example does. Mark such rows as reconstructed, keep them distinguishable from served-log rows, and do not mix the two silently when computing parity. The glossary entry on the feature store makes a related warning: a temporal join alone does not establish that history matches what the service knew.

Publishing a backfill safely

Backfill output joins a table that other jobs are writing, so the usual pipeline hazards apply. The article on incremental and idempotent pipelines covers the mechanisms: write by partition so a rerun replaces a partition rather than appending to it, publish through an atomic boundary so consumers do not see half a range, and order concurrent writers so that an old backfill finishing last does not overwrite fresher data. Three feature-specific requirements are added.

• Version the feature definition. If the transformation changes, write the new values under a new feature version rather than in place. A model trained on version 1 must keep reading version 1 at serving time until it is retrained.
• Inventory the dependents. Derived features, cached training sets, and models that consumed the old values need to be listed, which is where lineage helps: a column-level path from the feature to its models turns a backfill from a guess into a bounded set of follow-ups.
• Treat restatement as an event. If the change alters figures that people or models already used, announce it, retrain or reevaluate dependent models, and compare the distribution of the old and new values before switching.

Testing feature pipelines

Passing unit tests does not establish point-in-time correctness, because most unit tests run on data that is already tidy. The table lists the tests that target the failure modes above, in roughly increasing cost.

Test	What it checks	Where it runs	Needs
Boundary cases	Window edges (an event exactly at t, at t minus the window length, one second either side), TTL expiry, tie-breaks	Unit tests	Hand-built records
Missing-history behavior	The output for an entity with no eligible value matches the stated null-or-default rule in both paths	Unit tests	Hand-built records
Future perturbation	Appending records that become known after every prediction time changes no feature	Continuous integration	Event and availability timestamps
Late-arrival fixture	A record with event time before and ingestion time after the prediction time is excluded	Continuous integration	Fixture with delays
Backfill equivalence	A backfilled range equals the same range computed incrementally, including range edges	Pre-release	Overlapping run
Lookback sufficiency	Values on the first days of a backfill match a run with a longer input range	Pre-release	Extended input
Serving parity	Sampled served values equal offline reconstruction for the same entity, time, and feature version	Scheduled in production	Serving log
Label maturity	No evaluation example has a label that was unknown at the evaluation cutoff	Training pipeline	Label availability time

The unit and integration tests glossary entry advises stating what is real and what is replaced in each test and checking behavior with a nearby counterexample, and the same applies here. A boundary test with an order one minute outside the window is worth more than a test with a typical order.

Reading parity failures

When serving parity fails, the mismatch is a symptom and not yet a diagnosis. Sort mismatches by category before investigating, since each category points to a different fix.

Observed mismatch	Likely cause	First check
Offline value higher than served, concentrated in recent rows	Late-arriving data included offline	Compare ingestion time of contributing rows to the prediction time
Served value null or default, offline populated	Online store stale or entity missing	Value age against the TTL; materialization job status
Small numeric differences everywhere	Floating-point or type differences between engines	Tolerance and type per feature
Systematic offset for one feature	Different logic or window definition	Run both implementations on identical raw records
Differences only after a definition change	Version skew between model and feature version	Feature version recorded with each logged request

Float comparisons need an explicit tolerance. Categorical features need exact equality. Report the mismatch rate per feature, not an overall rate, because a rate dominated by one noisy feature hides a defect in another.

Operating notes and limits

Point-in-time machinery has a cost, and it is worth being clear about when it pays for itself. A single batch model, one team, and a table that is rebuilt and snapshotted daily can be correct without any feature store, as long as the training query follows the as-of rule and the snapshot is retained. The case for more infrastructure grows when several models share features, when some features are served online, when training sets are assembled by hand for each project, or when teams disagree about a definition and the disagreement has already caused an error. The feature store entry lists these conditions.

Three operational responsibilities need owners whatever the tooling. First, each feature definition needs an owner who approves changes to its meaning and a recorded freshness promise. Second, the timestamps this article relies on need to be treated as data contracts: if ingested_at stops meaning “when we first saw it”, every guarantee above becomes unfounded without any job failing. Third, stored feature history and served-feature logs are data about people, so retention and deletion apply to them as to any other copy, and a long history that makes reproduction possible competes with obligations to remove data. Decide what to keep and for how long before the history becomes large enough that deleting it is expensive.

Point-in-time correctness is necessary and not sufficient. It does not make a feature’s business definition right, it does not remove distribution drift between training and serving periods, and it does not make labels trustworthy. What it provides is a precise, testable statement of which values may appear in each training row, which is the precondition for every other check on the model.

References

• Feast documentation, Point-in-time joins
• Feast documentation, Feature retrieval
• Feast documentation, Feature view
• Martin Fowler, Bitemporal History
• Google for Developers, Rules of Machine Learning: Best Practices for ML Engineering
• Google Cloud Architecture Center, MLOps: Continuous delivery and automation pipelines in machine learning
• scikit-learn documentation, Common pitfalls and recommended practices (data leakage)