How do explainability tools like Grad-CAM generate visual attention maps for debugging?
Grad-CAM shows which parts of an image drove a prediction. It takes the gradient of the target class score with respect to the feature maps of the last convolutional layer, averages those gradients to get an importance weight per feature map, combines the maps with those weights, and applies a ReLU. The result is a coarse heatmap you overlay on the image to see where the model was looking.
When a model is right for the wrong reasons - keying off a watermark, the background, or a spurious cue - accuracy numbers won't tell you. Grad-CAM makes the model's focus visible so you can catch it.
How it builds the heatmap
- Pick a target (e.g. the predicted class) and backprop to the last conv layer's feature maps.
- Average each map's gradients into a single importance weight (how much that map mattered).
- Take the weighted sum of the feature maps, then ReLU to keep only positive, class-supporting evidence.
- Upsample the result to image size and overlay it as a heatmap.
The last conv layer is chosen because it keeps spatial layout while holding high-level semantics. It's a debugging tool, not proof: use it to sanity-check that a detector fires on the object and not the background, to investigate confident mistakes, and to build trust with stakeholders who want to see why the model decided what it did.
Related questions
Stuck on this in a real project?
Book a free call and I’ll give you a straight answer on your specific case.
Book a consultation