How to Read: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [Reading Guide]

This guide will help you effectively read "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," published at ICLR 2021.

First pass (5–10 min)

What to skim in this paper:

• Title: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" – Immediately tells you the core idea: applying NLP-style Transformers to images by treating image patches as "words."
• Abstract: Provides a concise summary of the problem (Transformers limited in CV, reliance on CNNs), the solution (pure Transformer on image patches), and the main finding (excellent results with large-scale pre-training, fewer resources).
• Introduction (Section 1): Expands on the abstract, setting the context of Transformers in NLP and the dominance of CNNs in CV. It highlights the paper's main hypothesis: a pure Transformer can work for images if trained at scale.
• Conclusion (Section 5): Summarizes the key achievements (direct Transformer application, competitive with SOTA CNNs, efficient pre-training) and outlines future work (other CV tasks, self-supervision, further scaling).

What to notice:

• Core Contribution: The paper proposes Vision Transformer (ViT), a novel architecture that applies a standard Transformer directly to sequences of image patches for image classification, without relying on convolutional networks.
• Problem Addressed: Overcoming the traditional reliance on CNNs for computer vision tasks and extending the success of Transformers from NLP to CV.
• Key Setting/Condition: ViT performs exceptionally well when pre-trained on large amounts of data (e.g., JFT-300M, ImageNet-21k) and then fine-tuned on smaller downstream tasks.
• Main Finding: Large-scale pre-training allows ViT to match or exceed state-of-the-art convolutional networks (like ResNets) in accuracy while requiring substantially fewer computational resources for training.
• Inductive Bias: The paper notes that ViT lacks the inherent image-specific inductive biases (like locality and translation equivariance) found in CNNs, which makes it perform poorly on smaller datasets but excel with sufficient data.

Second pass (up to 1 hour)

Using SpatialRead in the Second Pass

• Use the Screenshot tool to capture figures, tables, diagrams, and formulae from the PDF and use AI for simple explanations.
• Ask: "Explain Figure X at a high level", "Compare Table 2 vs 3", or "What does the y-axis represent here?"
• Download SpatialRead →

What sections/figures to focus on:

• Method (Section 3): This is crucial for understanding how ViT works.
  • Figure 1 (Model overview): Study this diagram carefully. It illustrates the entire ViT pipeline: splitting an image into patches, linear embedding, adding position embeddings and a classification token, and feeding the sequence into a standard Transformer encoder.
  • Section 3.1 (Vision Transformer (ViT)): Read this section to understand the details of patch embedding (Equation 1), the role of the learnable "classification token," and how position embeddings are incorporated. Note the structure of the Transformer encoder (MSA and MLP blocks).
  • Section 3.2 (Fine-tuning and Higher Resolution): Understand how the model is adapted for transfer learning, especially the 2D interpolation of pre-trained position embeddings when fine-tuning at higher resolutions.
• Experiments (Section 4): Focus on the setup and the main comparative results.
  • Section 4.1 (Setup): Skim the datasets used (ImageNet, JFT-300M, VTAB) and the different ViT model variants (e.g., ViT-Base, ViT-Large, ViT-Huge) described in Table 1.
  • Section 4.2 (Comparison to State of the Art): This section presents the core performance claims.
  • Table 2 (Comparison with state of the art): This is a key table. Analyze the accuracy results of ViT models (especially ViT-H/14 and ViT-L/16 pre-trained on JFT-300M) against strong CNN baselines (BiT, Noisy Student) across various benchmarks. Pay attention to the "TPUv3-core-days" to understand the computational efficiency.
  • Figure 3 (Transfer to ImageNet): Observe how ViT's performance scales with the size of the pre-training dataset compared to ResNets. This visually reinforces the importance of large-scale data for ViT.
  • Figure 5 (Performance versus pre-training compute): This figure provides a high-level comparison of ViT, ResNets, and hybrids in terms of accuracy vs. computational cost, showing ViT's favorable trade-off.

Key terms/questions to note:

• Image patches, linear projection, patch embeddings.
• Classification token (similar to BERT's [CLS] token).
• Position embeddings (1D learnable).
• Transformer Encoder (Multi-Head Self-Attention, MLP).
• Pre-training on large datasets (JFT-300M, ImageNet-21k) vs. fine-tuning on smaller benchmarks.
• "Inductive bias" – why CNNs have it and ViT largely doesn't.
• How does ViT achieve competitive results despite lacking CNN's inductive biases? (Answer: large-scale data).
• What are the computational benefits of ViT?

Third pass (1–5 hours)

Using SpatialRead in the Third Pass

• Highlight complex keywords, dense paragraphs, equations, or proofs and ask for a plain-English explanation that preserves notation.
• Use AI to explain the concepts in a way that is easy to understand.
• Ask for alternative phrasings or analogies to confirm understanding without changing technical meaning.
• Connect concepts with each other using AI to build a comprehensive understanding of the paper.
• Download SpatialRead →

Deep reading guidance:

• Method (Section 3): Re-read Section 3.1 and 3.2 in detail. Understand the mathematical formulations (Equations 1-4) and the exact process of patch creation, embedding, and Transformer input. Pay close attention to the "Inductive bias" paragraph on page 4, which explains the fundamental differences between ViT and CNNs. Also, review the "Hybrid Architecture" for an alternative input strategy.
• Experiments (Section 4):
  • Section 4.3 (Pre-training Data Requirements): Dive into the detailed analysis of how dataset size impacts ViT's performance. Correlate the text with Figure 3 and Figure 4 to understand the scaling behavior and the point where ViT overtakes CNNs. Table 5 in the Appendix provides the raw numbers for Figure 3.
  • Section 4.4 (Scaling Study): Analyze the performance-compute trade-offs in detail, referring to Figure 5 and Table 6 in the Appendix. Understand the implications for model efficiency.
  • Section 4.5 (Inspecting Vision Transformer): This section offers insights into why ViT works.
    • Examine Figure 6 (Attention maps) and Figure 14 (Appendix) to see how the model attends to different parts of the image.
    • Study Figure 7 (Embedding filters, position embedding similarity, attention distance) to understand the learned representations and how attention integrates information across the image.
    • Read Appendix D.7 (Attention Distance) and D.8 (Attention Maps) for a deeper understanding of these visualizations.
• Appendix:
  • Appendix B.1 (Training & Fine-tuning): Scrutinize Table 3 and Table 4 for detailed hyperparameters (learning rates, weight decay, dropout, batch sizes, epochs) used for pre-training and fine-tuning. This is critical for understanding the experimental setup.
  • Appendix D.1 (SGD vs. Adam for ResNets): Understand the authors' choice of Adam for ResNets pre-training, which is unconventional.
  • Appendix D.2 (Transformer Shape): Analyze Figure 8 to understand the impact of scaling different Transformer dimensions (depth, width, patch size) on performance.
  • Appendix D.4 (Positional Embedding): Read the ablation study on different positional embedding strategies and refer to Table 8 and Figure 10. This explains why 1D learnable position embeddings were sufficient.

Replication pointers:

• Model Architecture: The paper states it follows the original Transformer (Vaswani et al., 2017) closely. Section 3.1 and Figure 1 provide the necessary details for implementing the ViT architecture, including patch size (e.g., 16x16), linear projection, class token, and 1D learnable position embeddings.
• Hyperparameters: Table 1 (Model variants), Table 3 (Pre-training hyperparameters), and Table 4 (Fine-tuning hyperparameters) in the Appendix are essential for replicating the training process. Pay attention to learning rates, optimizers (Adam for pre-training, SGD for fine-tuning), weight decay, and batch sizes.
• Pre-training Data: The paper emphasizes the necessity of large datasets like JFT-300M or ImageNet-21k.
• Fine-tuning Strategy: Section 3.2 details the fine-tuning process, including replacing the prediction head and 2D interpolation of position embeddings for higher resolution inputs.
• Code Availability: The footnote on page 1 mentions that "Fine-tuning code and pre-trained models are available at https://github.com/google-research/vision_transformer". This is the most direct path to replication.