How do you measure the computational complexity (FLOPs) of a Vision Transformer?

FLOPs matter because they roughly predict latency and energy, which is what you actually care about when deploying. For a ViT, two terms dominate:

• Linear/MLP terms: patch embedding and the feed-forward blocks scale as O(N x d^2) - linear in the number of tokens N.
• Attention term: query-key-value interactions scale as O(N^2 x d) - quadratic in N.

That quadratic attention term is why halving the patch size (which roughly quadruples N) explodes the cost, and why efficient variants like Swin restrict attention to windows. Don't estimate this by hand for a real model - measure it:

import torch
from fvcore.nn import FlopCountAnalysis
from torchvision.models import vit_b_16

model = vit_b_16().eval()
dummy = torch.randn(1, 3, 224, 224)

flops = FlopCountAnalysis(model, dummy)
print(f"{flops.total() / 1e9:.2f} GFLOPs")  # per image

Report FLOPs alongside parameter count and, most importantly, measured latency on your target hardware - FLOPs are a proxy, not the last word, since memory access patterns also drive real speed.

In practice

I've been burned by trusting FLOPs. I've benchmarked 'cheaper' models that ran slower than higher-FLOP ones because they were memory-bound or hit an unoptimized op in the target runtime. FLOPs are a rough sketch; the number that decides whether you ship is measured latency on the actual device and export format (ONNX, TensorRT) you'll deploy. Always confirm with a real timing loop, not the spec sheet.

Hands-on tutorials