Feature Stores and Point-in-Time Correctness

As-of joins prevent future leakage

Training examples must use feature values as they were at prediction time, not as they are today. Joining labels to the latest feature values leaks the future and makes offline results look better than production. Point-in-time (as-of) joins pick, for each label timestamp, the latest feature value at or before that time. Feature stores (Feast, Tecton, cloud offerings) provide this for training and serve the same features with low latency online, reducing skew and duplicated feature code.

Point-in-time versus naive latest values, run

I ran this with Python 3, MLflow 3.16.1, scikit-learn 1.9.1, scipy 1.18.1 and numpy 2.5.3, using bundled or seeded synthetic data and a local SQLite tracking store. For customer c1 labelled at time 6, the point-in-time value of orders_last_30d is 4, but a naive join would use the latest value 7, which was only known at time 9. c2 shows the same problem (1 versus 3).

from bisect import bisect_right
# feature history: (timestamp, customer, orders_last_30d) - values become known at their timestamp
history = {"c1": [(1, 2), (5, 4), (9, 7)], "c2": [(2, 1), (8, 3)]}
labels = [("c1", 6, "churned=0"), ("c2", 7, "churned=1"), ("c1", 10, "churned=0")]
def as_of(cust, t):
    times = [ts for ts, _ in history[cust]]
    i = bisect_right(times, t) - 1
    return history[cust][i][1] if i >= 0 else None
def latest(cust):
    return history[cust][-1][1]
print("customer  label_time  point-in-time value  naive 'latest' value")
for cust, t, label in labels:
    print(f"{cust:<9} {t:>10}  {as_of(cust, t):>19}  {latest(cust):>20}")

Output:

customer  label_time  point-in-time value  naive 'latest' value
c1                 6                    4                     7
c2                 7                    1                     3
c1                10                    7                     7

Store feature timestamps

Keep an event or availability timestamp on every feature value; without it point-in-time joins are impossible.

Quick check: What does a point-in-time join prevent?

• Slow queries
• Using too few features
• Using feature values that were not yet known at the label's time
• Duplicate model versions