What are position embeddings, and why are they necessary in Vision Transformers?
Self-attention treats its input as an unordered set - shuffle the patches and the raw attention output is unchanged. That's a problem for images, where position is everything. Position embeddings fix it by adding a location-specific vector to each patch token, so the model can tell a patch in the corner from one in the center. Without them, a ViT sees a bag of patches with no spatial layout.
Here's the subtle thing about attention: it's permutation-invariant. The mechanism itself doesn't know whether a patch came from the top-left or the bottom-right - it just sees a set of tokens. For text or images, where order and position carry meaning, that has to be repaired.
How the fix works
- Each patch position gets its own embedding vector, added to the patch's content token.
- Now a token encodes both what the patch looks like and where it is.
- Embeddings can be learned (a lookup table trained with the model) or fixed (sinusoidal functions).
One practical consequence: because standard learned position embeddings are tied to a specific grid size, running a ViT at a different input resolution than it was trained on usually requires interpolating them. It's a small detail that trips people up when fine-tuning at a new image size.
In practice
This is the exact bug I've watched cost people a day of debugging: they run or fine-tune a pretrained transformer model (a ViT or a DETR-style detector) at a different input size than it was trained on, forget to interpolate the position embeddings, and accuracy quietly tanks with no error thrown. If a transformer-based model mysteriously gets worse right after you change the input resolution, check the position embeddings before anything else.
Related questions
Stuck on this in a real project?
Book a free call and I’ll give you a straight answer on your specific case.
Book a consultation