How does DeepSORT improve over SORT using deep visual appearance descriptors?
SORT matches objects using motion and box overlap alone, so it loses track when objects occlude or cross. DeepSORT adds a deep appearance descriptor - a learned embedding of each object's look, from a small re-identification network - and matches on appearance as well as motion. That memory of how an object looks lets it re-attach the correct ID after an occlusion, cutting ID switches dramatically.
SORT is fast but forgetful: because it only knows where things are and how they're moving, two people passing each other can swap IDs. DeepSORT gives the tracker a sense of what each object looks like.
What DeepSORT adds
- An appearance embedding: a small CNN turns each detection into a feature vector describing its look.
- A gallery: each track remembers recent embeddings, so it can recognize an object that reappears.
- Combined matching: association uses both motion (Kalman + Mahalanobis distance) and appearance (cosine distance between embeddings).
The result is far fewer identity switches through occlusions and crossings, at the cost of running that extra embedding network per detection. It's the same idea behind modern trackers like BoT-SORT, which pair strong motion models with appearance re-ID. If ID stability matters for your use case (counting, analytics), an appearance-aware tracker is worth the extra compute.
Related questions
Stuck on this in a real project?
Book a free call and I’ll give you a straight answer on your specific case.
Book a consultation