Distribution shift
Also known as: distribution shift · out-of-distribution
Distribution shift is when the data a model faces in production is different from the data it was trained (or tested) on. Quality quietly degrades because the assumptions no longer hold.
Examples:
- Trained on English web text, deployed on technical French support tickets.
- Trained on 2023 data, now answering about 2026 events.
- Trained on clean text, now fed messy user input.
The silent killer: the model looked great on the benchmark, then quietly got worse in the real world. Monitoring *live* inputs/outputs is how you catch it.