What is the tokenization process for an image in a transformer network?

Question

Muhammad Rizwan Munawar · Accepted Answer

Tokenizing an image means converting it into a sequence of vectors, the way text is split into word tokens. You divide the image into fixed-size patches (e.g. 16x16 pixels), flatten each patch, and linearly project it to a fixed-length embedding. Those patch embeddings, plus a position embedding each, are the tokens the transformer processes. In code, a single strided convolution does the patch-and-project step at once. A transformer consumes a sequence of vectors. For text those are word/subword tokens; for images we manufacture tokens from patches. The elegant part is that 'cut into patches and project each' is exactly what a convolution with matching kernel and stride does. import torch
import torch.nn as nn

# 224x224 image, 16x16 patches -> 14x14 = 196 patch tokens of dim 768
patch_embed = nn.Conv2d(in_channels=3, out_channels=768,
                        kernel_size=16, stride=16)

img = torch.randn(1, 3, 224, 224)
x = patch_embed(img)          # (1, 768, 14, 14)
tokens = x.flatten(2).transpose(1, 2)  # (1, 196, 768) -> sequence of tokens
print(tokens.shape) After this you prepend a learnable class token (used for the final classification) and add position embeddings so the model knows where each patch came from. The patch size is a direct speed/detail dial: smaller patches mean more tokens - finer detail but quadratically more attention compute. So 'tokenization' for images is really just patch embedding: turn a grid of pixels into a short sequence of vectors the transformer can reason over. In practice Patch size is the one knob here that decides your speed/accuracy trade-off, and it bites hard: dropping from 16x16 to 8x8 patches quadruples the token count and roughly quadruples attention cost. So if a transformer model is too slow, a larger patch size (or a smaller input resolution) is usually the first lever I pull - before reaching for anything fancier like pruning or a different architecture.

What is the tokenization process for an image in a transformer network?

In practice

Related questions

Hands-on tutorials

Stuck on this in a real project?