YOLO & computer vision, answered
Straight answers to the questions engineers actually paste into Google, from someone who builds this every day. Each one gives you the short answer first, then the detail and code.
Tracking & counting
How do I count objects in a video with YOLO?
To count objects in a video with YOLO, run detection with tracking enabled so every object keeps a stable ID across frames, then count each ID once as it crosses a counting line or enters a region. Ultralytics ships an ObjectCounter solution that does this out of the box.
Read the answerWhat is multi-object tracking, and how does the SORT algorithm use Kalman filters?
Multi-object tracking assigns each detected object a consistent ID as it moves through a video, so 'car #7' stays #7 frame to frame. SORT does this cheaply: a Kalman filter predicts where each existing track should appear in the next frame, those predictions are matched to the new detections by IoU using the Hungarian algorithm, and the filter updates. It's fast because it uses only motion and box geometry - no appearance model.
Read the answerHow does DeepSORT improve over SORT using deep visual appearance descriptors?
SORT matches objects using motion and box overlap alone, so it loses track when objects occlude or cross. DeepSORT adds a deep appearance descriptor - a learned embedding of each object's look, from a small re-identification network - and matches on appearance as well as motion. That memory of how an object looks lets it re-attach the correct ID after an occlusion, cutting ID switches dramatically.
Read the answerAccuracy & training
Why is my YOLO model not accurate?
In most cases a YOLO model is inaccurate because of the data, not the model. The usual causes are too few labelled examples, wrong or inconsistent labels, class imbalance, a mismatch between training images and real-world conditions, or training at the wrong image size. Fix the data before changing the architecture.
Read the answerHow many images do I need to train YOLO?
As a rough guide, Ultralytics recommends around 1,500 images and 10,000 labelled instances per class for a robust production model. You can prototype with a few hundred well-labelled images using pretrained weights and augmentation, but variety matters more than raw count.
Read the answerHow do I train YOLO on my own custom dataset?
To train YOLO on a custom dataset: label your images in YOLO format, organise them into train/val folders, write a data.yaml describing the paths and class names, then call model.train() starting from a pretrained checkpoint. Validate and export when the metrics look right.
Read the answerHow does Intersection over Union (IoU) evaluate bounding box accuracy?
IoU scores how well two boxes match by dividing the area where they overlap by the total area they cover together. It ranges from 0 (no overlap) to 1 (identical). A prediction is usually counted as correct if its IoU with the ground-truth box clears a threshold like 0.5. IoU is the building block for both NMS and mAP.
Read the answerWhat is Mean Average Precision (mAP), and how is it calculated for detection?
mAP summarizes detection quality in one number. For each class you rank detections by confidence, use an IoU threshold to label each as correct or not, trace the precision-recall curve, and take its area (the Average Precision). Averaging AP over all classes gives the mAP. 'mAP@0.5' uses a 0.5 IoU cutoff; 'mAP@0.5:0.95' averages over cutoffs from 0.5 to 0.95 for a stricter score.
Read the answerWhat is Focal Loss, and how does it solve extreme background class imbalance?
Focal Loss is a modified cross-entropy that fixes the extreme foreground-background imbalance in one-stage detectors, where the vast majority of the tens of thousands of candidate boxes are easy background. It multiplies the loss by a factor that shrinks as a prediction gets more confident and correct, so easy negatives contribute almost nothing and the model spends its effort on the rare, hard objects.
Read the answerSpeed & deployment
How do I speed up YOLO inference and get more FPS?
To speed up YOLO, export the model to ONNX or TensorRT, use a smaller variant (n or s), lower the image size, and enable half precision (FP16) or INT8 on supported hardware. Each of these trades a little accuracy for a large jump in FPS, so measure both.
Read the answerHow do I deploy YOLO on an NVIDIA Jetson?
To deploy YOLO on a Jetson, flash JetPack, install Ultralytics, then export the model to TensorRT on the device itself with FP16 (or INT8) precision. Set the Jetson to its maximum power mode and measure real FPS on your footage before relying on the numbers.
Read the answerObject detection concepts
What is the difference between image classification, object detection, and localization?
Image classification answers 'what is in this image?' with a single label. Localization goes one step further and draws a box around the main object. Object detection is the full job: it finds every object, draws a box around each one, and labels them all in a single image.
Read the answerWhat is a bounding box, and how is it mathematically represented?
A bounding box is the axis-aligned rectangle a detector draws around an object. It's represented by four numbers, in one of two common formats: corner form (x1, y1, x2, y2) or center form (cx, cy, width, height). YOLO uses center form, usually normalized to 0-1 so the values don't depend on image size.
Read the answerWhat is the core architectural difference between one-stage and two-stage detectors?
Two-stage detectors split the work in two: a first network proposes candidate regions, then a second classifies and refines each one. One-stage detectors skip the proposal step and predict all boxes and classes in a single forward pass. Two-stage tends to be more accurate on small or dense objects; one-stage is far faster and is what you use for real-time video.
Read the answerHow does a Region Proposal Network (RPN) work in Faster R-CNN?
The RPN is a small network that slides over the backbone's feature map. At every location it looks at a fixed set of anchor boxes and outputs two things per anchor: an 'objectness' score (is there an object here or just background?) and four offsets that nudge the anchor toward the real object. The high-scoring, refined anchors become the region proposals passed to the second stage.
Read the answerHow does the YOLO (You Only Look Once) algorithm predict boxes in a single pass?
YOLO runs the whole image through one convolutional network a single time. The network outputs a grid, and every grid cell predicts a few bounding boxes, an objectness score for each, and class probabilities. Everything is produced at once - that's the 'you only look once' part - then non-max suppression removes duplicate boxes for the same object.
Read the answerWhat are anchor boxes, and how are their aspect ratios determined?
Anchor boxes are a fixed set of reference rectangles, in a few sizes and aspect ratios, tiled across the image. Instead of predicting box coordinates from scratch, the detector predicts small offsets from the nearest anchor, which is much easier to learn. The best aspect ratios are typically found by running k-means clustering on the width/height of the boxes in your training set.
Read the answerWhat is Non-Max Suppression (NMS), and why is it crucial in object detection?
A detector predicts many overlapping boxes for a single object. Non-max suppression cleans that up: for each class, it keeps the highest-confidence box, removes every other box that overlaps it by more than an IoU threshold, and repeats. Without NMS you'd get a dozen stacked boxes on every object instead of one.
Read the answerWhat is the difference between anchor-based and anchor-free object detectors?
Anchor-based detectors place a fixed set of reference boxes everywhere and predict adjustments to them. Anchor-free detectors drop those references and predict objects directly - usually as a center point plus size, or as keypoints. Anchor-free removes the dataset-specific anchor tuning, has fewer hyperparameters, and is what most recent YOLO versions use.
Read the answerHow does the Feature Pyramid Network (FPN) handle multi-scale object detection?
Objects come in wildly different sizes, and a single feature map can't handle them all. An FPN builds a pyramid of maps at different resolutions and adds a top-down path with lateral connections, so rich, high-level meaning from deep layers is merged back into shallow, high-resolution layers. Small objects get detected on the fine maps, large objects on the coarse ones.
Read the answerWhat is Single Shot MultiBox Detector (SSD), and how does it differ from YOLO?
SSD (Single Shot MultiBox Detector) is, like YOLO, a one-stage detector that predicts all boxes in a single pass. Its distinguishing idea is making predictions from several feature maps of decreasing resolution, so different layers naturally handle different object sizes, using a set of default boxes per location. Early YOLO predicted from a single scale; both families have since converged on multi-scale prediction.
Read the answerVision transformers
What is a Vision Transformer (ViT), and how does it process an image without convolutions?
A Vision Transformer treats an image like a sentence. It cuts the image into a grid of fixed-size patches (say 16x16), flattens and linearly projects each patch into a token, adds a position embedding, and feeds the whole sequence to a standard transformer encoder. Self-attention then lets every patch relate to every other patch, so the model learns global relationships without a single convolution.
Read the answerHow does self-attention differ from a traditional convolutional layer operation?
A convolution combines a pixel with its immediate neighbors inside a small fixed window, using the same learned filter everywhere (local and static). Self-attention lets every token compare itself to every other token and weight them by relevance, with those weights computed on the fly from the data (global and dynamic). Convolutions bake in locality; attention learns which relationships matter.
Read the answerWhat are position embeddings, and why are they necessary in Vision Transformers?
Self-attention treats its input as an unordered set - shuffle the patches and the raw attention output is unchanged. That's a problem for images, where position is everything. Position embeddings fix it by adding a location-specific vector to each patch token, so the model can tell a patch in the corner from one in the center. Without them, a ViT sees a bag of patches with no spatial layout.
Read the answerWhy do Vision Transformers require larger training datasets than CNNs to generalize?
CNNs come with strong inductive biases baked into their architecture: nearby pixels are related (locality) and a pattern means the same thing wherever it appears (translation equivariance). Those assumptions are 'free' knowledge that fits images well. Vision Transformers have very weak built-in biases, so they have to learn that spatial structure from scratch - which only works well with large datasets or large-scale pre-training.
Read the answerWhat is a hybrid vision architecture (e.g., ConvNeXt, Swin Transformer)?
Hybrid vision architectures combine the best of CNNs and transformers rather than picking a side. Swin Transformer brings convolution-like locality and a hierarchical, multi-scale structure to attention. ConvNeXt goes the other way - a pure CNN redesigned with transformer-inspired choices. Both target strong accuracy with better data-efficiency and compute than a plain ViT.
Read the answerHow does the Swin Transformer utilize shifted windows to compute localized attention?
Global attention over every patch is too expensive at high resolution, so Swin computes attention only within small non-overlapping windows, which makes cost grow linearly with image size instead of quadratically. To stop windows from being isolated, the next layer shifts the window grid by half a window, so patches that were on opposite sides of a boundary now share a window. Alternating regular and shifted windows spreads information globally over depth.
Read the answerWhat is Multi-Head Attention, and how does it benefit complex visual scene analysis?
Multi-head attention runs several attention mechanisms in parallel, each projecting the tokens into its own smaller subspace before attending. Because each head focuses on a different kind of relationship - one might track spatial layout, another texture, another object parts - the model can attend to several patterns at once rather than being forced to average them into a single attention map. For busy scenes, that multi-angle reading is a real advantage.
Read the answerWhat is the Masked Autoencoder (MAE) approach to self-supervised vision learning?
A Masked Autoencoder hides a large fraction of an image's patches - often around 75% - and trains a transformer to reconstruct the missing pixels from the visible ones. To do that well, the model has to learn what objects and scenes actually look like, so it produces strong features with no labels at all. It's efficient too: the heavy encoder only sees the visible patches, and a light decoder fills in the rest.
Read the answerHow do you measure the computational complexity (FLOPs) of a Vision Transformer?
FLOPs (floating-point operations) estimate a model's compute per image. In a ViT the cost splits into two parts: the linear projections and MLP blocks, which scale linearly with the number of tokens, and self-attention, which scales with the square of the number of tokens because every token attends to every other. Rather than derive it by hand, measure it with a profiler such as fvcore, thop, or ptflops.
Read the answerWhat is the tokenization process for an image in a transformer network?
Tokenizing an image means converting it into a sequence of vectors, the way text is split into word tokens. You divide the image into fixed-size patches (e.g. 16x16 pixels), flatten each patch, and linearly project it to a fixed-length embedding. Those patch embeddings, plus a position embedding each, are the tokens the transformer processes. In code, a single strided convolution does the patch-and-project step at once.
Read the answerEvaluation & deployment
What is a confusion matrix, and how do precision, recall, and F1-score evaluate a vision model?
A confusion matrix is a table of predictions against ground truth, splitting results into true positives, false positives, true negatives, and false negatives. From it: precision = TP / (TP + FP), 'of what I flagged, how much was right'; recall = TP / (TP + FN), 'of what was really there, how much I caught'; and F1 is their harmonic mean, a single balanced score. Which one matters most depends on whether false alarms or misses are costlier.
Read the answerHow do you detect and handle severe class imbalance in a custom image dataset?
First, detect it: count labelled instances per class and look at per-class recall - a rare class with low recall is the tell. Then handle it with a mix of collecting more examples of the rare class, oversampling those images (or undersampling common ones), targeted augmentation, and weighting the loss toward rare classes. Finally, judge the model with per-class metrics and F1, not overall accuracy, which the majority class will flatter.
Read the answerWhat is model quantization, and how does it help deploy vision models to edge devices?
Quantization represents a model's weights and activations in lower precision - typically 8-bit integers instead of 32-bit floats. That cuts model size by around 4x and lets edge hardware run the math far faster and with less power, usually for only a small accuracy drop. Post-training quantization is the quick path; quantization-aware training recovers accuracy when the drop is too big.
Read the answerWhat is structural pruning, and how does it optimize a neural network for fast inference?
Structural pruning removes entire structural units - channels, filters, or even layers - that contribute little, yielding a physically smaller network. Because it deletes whole components rather than scattering zeros, the pruned model is dense and actually runs faster on standard CPUs and GPUs. You typically rank units by importance, remove the weakest, then fine-tune to recover accuracy.
Read the answerWhat are the key performance bottlenecks when streaming live RTSP camera feeds into a vision pipeline?
In a live RTSP pipeline the model is often not the bottleneck. The usual culprits are video decoding (H.264/H.265 on the CPU is expensive), frame buffering and latency (frames pile up if inference can't keep up), CPU-to-GPU memory transfers, and the per-frame pre/post-processing. The fixes: decode on the GPU, drop stale frames instead of queueing them, batch streams, and keep data on the GPU.
Read the answerHow do explainability tools like Grad-CAM generate visual attention maps for debugging?
Grad-CAM shows which parts of an image drove a prediction. It takes the gradient of the target class score with respect to the feature maps of the last convolutional layer, averages those gradients to get an importance weight per feature map, combines the maps with those weights, and applies a ReLU. The result is a coarse heatmap you overlay on the image to see where the model was looking.
Read the answerHow do you test a computer vision deployment for distribution shift once it goes live?
Distribution shift is when the live images drift away from what the model trained on - new lighting, cameras, seasons, or object types - and accuracy silently degrades. Since you rarely have live labels, you monitor proxies: input statistics (brightness, resolution, embeddings), the distribution of predictions and confidence over time, and a steady trickle of sampled frames sent for human review. Alert when these move, then retrain on fresh data.
Read the answerHave a question these don’t cover?
If you’re stuck on a real computer vision problem, book a free call and I’ll give you a straight answer on whether it’s doable and how to approach it.