Concept Drift and Model Decay

When yesterday's patterns stop holding

Concept drift means the relationship between inputs and outcomes changes: customers respond differently to prices, fraudsters adapt, regulations change behaviour. A model trained on old data gradually loses accuracy even if input distributions look similar. Retraining on recent data restores performance, as long as recent labels exist. How fast models decay varies widely, so measure it: evaluate old models on newer data to learn how often you need to retrain.

Keep models fresh and recover fast

Models decay as the world changes; plan retraining, rollback and incident response in advance.

An old model versus regular retraining, run

I ran this with Python 3, MLflow 3.16.1, scikit-learn 1.9.1, scipy 1.18.1 and numpy 2.5.3, using bundled or seeded synthetic data and a local SQLite tracking store. On simulated data whose relationship changes a little each month, the original model's accuracy falls from 0.902 in month 1 to 0.506 in month 9, near guessing. A model retrained on the previous month stays around 0.87 to 0.92.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(0)
def month(m, n=2000):
    X = rng.normal(size=(n, 3))
    w = np.array([1.5, -1.0, 0.5]) + np.array([0, 0.25, -0.2]) * m      # relationship slowly changes
    y = (X @ w + rng.normal(scale=0.5, size=n) > 0).astype(int)
    return X, y
X0, y0 = month(0)
model = LogisticRegression().fit(X0, y0)
print("month | original model | retrained on previous month")
for m in [1, 3, 6, 9]:
    Xm, ym = month(m); Xp, yp = month(m - 1)
    print(f"{m:>5} | {model.score(Xm, ym):>14.3f} | {LogisticRegression().fit(Xp, yp).score(Xm, ym):.3f}")

Output:

month | original model | retrained on previous month
    1 |          0.902 | 0.899
    3 |          0.787 | 0.873
    6 |          0.647 | 0.901
    9 |          0.506 | 0.916

Backtest your retraining cadence

Simulate retraining monthly, quarterly and yearly on historical data to choose a schedule with evidence.

त्वरित जाँच: What is concept drift?

• A bug in the logging code
• A change in the relationship between inputs and the target
• A faster GPU
• A new model registry