What is a Vision Transformer (ViT), and how does it process an image without convolutions?

Convolutions build understanding locally, sliding small filters across the image. A Vision Transformer throws that out and borrows the architecture that powers language models: attention over a sequence of tokens.

The pipeline

• Patchify: split the image into non-overlapping patches (e.g. 16x16 pixels).
• Embed: flatten each patch and project it to a fixed-length vector - a 'token', like a word.
• Add position: attach a position embedding so the model knows where each patch was.
• Encode: run the token sequence through transformer blocks (self-attention + MLP).
• Read out: a special class token (or pooled tokens) feeds the final classifier.

The key difference from a CNN is receptive field. A convolution only sees its local window; self-attention lets a patch in the top-left directly attend to one in the bottom-right from the very first layer. That global view is ViT's strength - and, as the training-data question explains, also the reason it needs a lot of data.

In practice

I almost never train a ViT from scratch for client work. For detection and tracking on video I still reach for a CNN-based YOLO, because it hits real-time latency on a single GPU or a Jetson, where a plain ViT usually can't. Where transformers have genuinely earned a place in my projects is large-scale image understanding and multimodal search (CLIP-style), not the tight-latency edge jobs most computer vision work actually is.

Hands-on tutorials