How do you test a computer vision deployment for distribution shift once it goes live?
Distribution shift is when the live images drift away from what the model trained on - new lighting, cameras, seasons, or object types - and accuracy silently degrades. Since you rarely have live labels, you monitor proxies: input statistics (brightness, resolution, embeddings), the distribution of predictions and confidence over time, and a steady trickle of sampled frames sent for human review. Alert when these move, then retrain on fresh data.
The dangerous thing about drift is that nothing crashes - the model keeps returning confident answers that are increasingly wrong. Because production usually has no ground truth, you watch for symptoms rather than measuring accuracy directly.
What to monitor
- Input drift: track image statistics or feature embeddings and compare live distributions to a training baseline.
- Prediction drift: watch class balance and detection counts - a sudden change often means the world (or a camera) changed.
- Confidence trends: a slow slide in average confidence, or more borderline scores, is an early warning.
- Human-in-the-loop sampling: label a small random sample of live frames periodically to get a real accuracy read.
Set thresholds that page someone when these metrics move, and design for retraining from the start: capture representative live frames (with privacy handled), fold them into the dataset, and refresh the model on a schedule. Deployment isn't the finish line - a vision system is only as good as its last retrain against the world it actually sees.
Related questions
Stuck on this in a real project?
Book a free call and I’ll give you a straight answer on your specific case.
Book a consultation