ByteTrack vs DeepSORT: which tracker should I use?
Use ByteTrack when you want speed and simplicity: it tracks on motion alone, needs no appearance model, and still scores at or above DeepSORT on the MOT benchmarks, which is why it is the common default. Choose DeepSORT (or a modern appearance tracker like BoT-SORT) when objects are occluded for a long time or cross heavily and you need to re-identify them by how they look.
Per-GPU FPS, ID fragmentation and cost per 1,000 frames for ByteTrack, BoT-SORT, OC-SORT, Deep OC-SORT, FastTrack and TrackTrack.
Both are multi-object trackers you bolt onto a detector like YOLO, but they solve the association problem differently, and that difference decides which one fits your video.
The core difference
- DeepSORT matches objects using motion (a Kalman filter) plus a learned appearance embedding, so it can recognize an object by its look after it reappears. That costs an extra re-identification network run on every detection.
- ByteTrack matches on motion and box overlap only, with no appearance model, and recovers dim objects by also associating low-confidence boxes. Lighter and faster, and it often beats DeepSORT on MOTA/IDF1 anyway.
Side by side
| ByteTrack | DeepSORT | |
|---|---|---|
| Association | Motion + box overlap (IoU) | Motion + learned appearance embedding |
| Extra model | None | A re-ID network, run on every detection |
| Recovers after long occlusion | Weak - relies on motion continuing | Strong - matches on how the object looks |
| Cost per frame | Low | Detection cost + one re-ID pass per box |
| In Ultralytics | Yes, tracker="bytetrack.yaml" | No - use BoT-SORT, its modern successor |
| Released | 2021 (Zhang et al.) | 2017 (Wojke et al.) |
What the appearance model actually costs
The usual advice stops at "appearance matching is slower". We measured how much slower. Running YOLO26n over the same 200-frame clip on ten different GPUs, comparing ByteTrack against BoT-SORT - the appearance-based tracker in DeepSORT's lineage - gives median figures across the whole hardware set:
| Tracker | Median FPS | Fragmentations | Unique IDs |
|---|---|---|---|
| ByteTrack (motion only) | 73.1 | 17 | 14 |
| BoT-SORT (appearance) | 9.2 | 16 | 14 |
| Deep OC-SORT (appearance) | 77.3 | 42 | 12 |
The result is worth pausing on: on this footage BoT-SORT's appearance model cost roughly 8x the throughput and returned 16 fragmentations against ByteTrack's 17. That is not a measurable identity benefit for an 8x speed penalty. Appearance matching earns its cost on hard footage - long occlusions, crowds, many similar-looking objects - and on easy footage it is pure overhead.
Which to pick
- Want the fast, well-supported default? ByteTrack. It is built into Ultralytics (tracker="bytetrack.yaml") and needs no extra weights.
- Objects vanish behind things for seconds, or many similar objects cross? An appearance-aware tracker earns its cost - prefer BoT-SORT (also built into Ultralytics, tracker="botsort.yaml"), the modern successor to DeepSORT's idea, which adds re-ID plus camera-motion compensation.
- On a tight compute budget (edge/CPU)? ByteTrack, because it skips the per-detection embedding network.
- Already committed to appearance matching and need the speed back? Deep OC-SORT held 77.3 median FPS in the same run - faster than ByteTrack - though it fragmented tracks more than twice as often on this clip.
In practice most people start with ByteTrack, then switch only if they measure too many ID switches on their own footage. The detector matters more than the tracker for overall quality, so get detections solid first. Full per-GPU numbers for all six trackers, including cost per 1,000 frames, are on the tracker benchmarks page linked below.