Why do Vision Transformers require larger training datasets than CNNs to generalize?

The short version: a CNN is told how images work; a ViT has to figure it out. On small datasets, being told wins.

What CNNs get for free

• Locality: convolutions only look at neighborhoods, matching how images are structured.
• Translation equivariance: shared weights mean a cat is a cat wherever it sits.
• These priors mean a CNN generalizes from relatively little data.

A ViT's self-attention can connect any patch to any other and applies no locality assumption, so it must learn even basic spatial relationships from examples. With enough data - the original ViT was pre-trained on hundreds of millions of images - it learns these and can surpass CNNs. On a small dataset from scratch, it tends to underperform.

The practical takeaway: if you have a modest dataset, either use a pre-trained ViT and fine-tune, or reach for a CNN or hybrid model that brings those priors with it. Don't train a large ViT from scratch on a few thousand images and expect magic.

In practice

I've seen teams burn weeks training a ViT from scratch on a few thousand images, only to be beaten by a YOLO fine-tuned in an afternoon. On the datasets most real projects have - hundreds to low-thousands of labelled images - the CNN's inductive bias wins almost every time. I save transformers for cases where there's either a huge dataset or a strong pretrained checkpoint to fine-tune from.