How to Read: A Unified Approach to Interpreting Model Predictions [Reading Guide]

This guide will help you effectively read "A Unified Approach to Interpreting Model Predictions" (SHAP), a foundational paper from NIPS 2017, using the three-pass approach.

---

First pass (5–10 min)

What to skim in this paper: Begin by quickly reviewing the Title, Abstract, Introduction (Section 1), and Conclusion (Section 6). Pay attention to the authors and the venue (Neural Information Processing Systems, 2017), which indicates a high-impact machine learning paper.

What to notice:

• Title: "A Unified Approach to Interpreting Model Predictions" immediately signals the paper's goal: to bring together disparate methods for understanding model outputs.
• Abstract:
  • Problem: Complex models are hard to interpret, and existing interpretation methods are disconnected.
  • Contribution: Introduces SHAP (SHapley Additive exPlanations), a unified framework.
  • Key Innovations: Identifies a new class of additive feature importance measures and proves a unique solution within this class with desirable properties.
  • Impact: Unifies six existing methods and leads to new, improved methods.
• Introduction (Section 1): Reinforces the problem of the accuracy-interpretability trade-off and the lack of understanding of how existing interpretation methods relate. Note the three "surprising results" listed, which outline the paper's structure and main arguments: (1) defining additive feature attribution methods (Section 2), (2) applying game theory to this class (Section 3 & 4), and (3) proposing new estimation methods (Section 5).
• Conclusion (Section 6): Summarizes SHAP's role in unifying additive feature importance methods and its adherence to desirable properties. It also highlights the practical benefits of the new estimation methods.

Second pass (up to 1 hour)

Using SpatialRead in the Second Pass

• Use the Screenshot tool to capture figures, tables, diagrams, and formulae from the PDF and use AI for simple explanations.
• Ask: "Explain Figure X at a high level", "Compare Table 2 vs 3", or "What does the y-axis represent here?"
• Download SpatialRead →

What sections/figures to focus on: This pass aims to grasp the paper's core ideas without getting lost in every detail.

• Section 2: Additive Feature Attribution Methods:
  • Focus on Definition 1 and Equation 1. This is the fundamental definition of the class of explanation models SHAP unifies: a linear function of binary variables.
  • Skim subsections 2.1 through 2.4 (LIME, DeepLIFT, Layer-Wise Relevance Propagation, Classic Shapley Value Estimation) to understand how these diverse methods are shown to fit Definition 1. You don't need to deeply understand each method's original formulation, just recognize that they are all framed as additive feature attribution methods.
• Section 3: Simple Properties Uniquely Determine Additive Feature Attributions:
  • This section is critical. Understand the three desirable properties: Property 1 (Local accuracy), Property 2 (Missingness), and Property 3 (Consistency). These are the axioms SHAP values satisfy.
  • Note Theorem 1 and Equation 8, which state that there is a unique explanation model satisfying these properties, and that this unique solution corresponds to Shapley values.
• Section 4: SHAP (SHapley Additive exPlanation) Values:
  • Understand that SHAP values are these unique Shapley values.
  • Examine Figure 1. This diagram visually explains how SHAP values attribute the change in expected model prediction from a base value to the current output by conditioning on features.
  • Skim subsections 4.1 (Model-Agnostic Approximations) and 4.2 (Model-Specific Approximations) to get an overview of the different SHAP approximation methods (e.g., Kernel SHAP, Deep SHAP) and their connections to existing methods like LIME and DeepLIFT. Note Theorem 2 on page 6, which shows how LIME's regression can be made consistent with Shapley values.
• Section 5: Computational and User Study Experiments:
  • Skim the introduction to understand the types of experiments conducted (computational efficiency, consistency with human intuition, explaining class differences).
  • Look at Figure 3 (A) and (B) to see the comparison of Kernel SHAP, Shapley sampling, and LIME in terms of accuracy and sample efficiency. Notice Kernel SHAP's improved performance.
  • Examine Figure 4 (A) and (B), which illustrates SHAP's better alignment with human intuition compared to LIME and DeepLIFT in user studies.
  • Review Figure 5 (A) and (B), showing Deep SHAP's application to MNIST digit classification and its improved performance.

Key terms/questions to note:

• Additive feature attribution methods
• Explanation model vs. original model
• Local accuracy, Missingness, Consistency (the three desirable properties)
• Shapley values (and their connection to SHAP)
• Kernel SHAP, Deep SHAP (as specific implementations/approximations)
• How does SHAP unify existing methods?
• What are the main benefits of SHAP (e.g., theoretical guarantees, consistency with human intuition, computational performance)?

Third pass (1–5 hours)

Using SpatialRead in the Third Pass

• Highlight complex keywords, dense paragraphs, equations, or proofs and ask for a plain-English explanation that preserves notation.
• Use AI to explain the concepts in a way that is easy to understand.
• Ask for alternative phrasings or analogies to confirm understanding without changing technical meaning.
• Connect concepts with each other using AI to build a comprehensive understanding of the paper.
• Download SpatialRead →

Deep reading guidance: Now, delve into the details, focusing on understanding the derivations, proofs, and experimental setups.

• Section 2: Re-read this section carefully. Understand the mathematical form of Definition 1 (Equation 1) and how each of the six existing methods (LIME, DeepLIFT, LRP, Shapley regression values, Shapley sampling values, Quantitative Input Influence) is shown to be an instance of this definition. This is the core of the "unification."
• Section 3: Thoroughly understand the precise mathematical definitions of Property 1 (Local accuracy), Property 2 (Missingness), and Property 3 (Consistency). Grasp the significance of Theorem 1 – that these three properties uniquely define the Shapley values as the only possible additive feature attribution method. Consider the implications for methods that do not satisfy these properties.
• Section 4:
  • Understand how SHAP values are defined as the Shapley values of a conditional expectation function.
  • Kernel SHAP (page 5-6): Focus on Theorem 2 and the specific forms of the weighting kernel $\pi_x'$, loss function $L$, and regularization term $\Omega$ (Equation 6) that make LIME's regression formulation consistent with the three properties and thus recover Shapley values. This is a crucial theoretical bridge.
  • Deep SHAP (page 7): Understand the connection to DeepLIFT and how it leverages the compositional nature of deep networks to efficiently approximate SHAP values through a back-propagation-like process (Equations 13-16).
• Section 5 (Experiments): Analyze the experimental results in depth.
  • Computational Efficiency (5.1): Understand the experimental setup for Figure 3 (decision tree models, number of function evaluations) and why Kernel SHAP offers improved accuracy and sample efficiency compared to other methods.
  • Consistency with Human Intuition (5.2): Detail the user study designs (sickness score, max allocation problem) and the specific findings presented in Figure 4. Understand why SHAP aligns better with human explanations, especially for non-linear functions like max pooling.
  • Explaining Class Differences (5.3): Analyze the MNIST example in Figure 5 and how Deep SHAP improves upon DeepLIFT for convolutional networks.

Replication pointers if applicable:

• The paper provides a direct link to the codebase in footnote 1 on page 1: https://github.com/slundberg/shap. This is the primary resource for understanding the implementation details and potentially replicating results.
• To replicate the theoretical results, one would need to implement the Shapley value calculation (Equation 8) and its various approximations (Kernel SHAP, Deep SHAP) as described in Sections 4.1 and 4.2.
• For the experimental results, you would need to set up similar models (e.g., decision trees, deep neural networks for MNIST), generate synthetic data or use the MNIST dataset, and implement the different explanation methods (LIME, DeepLIFT, Shapley sampling, SHAP variants) to compare their performance as shown in Figures 3, 4, and 5.
• Replicating the user studies would involve designing similar tasks and recruiting participants, potentially using platforms like Amazon Mechanical Turk as mentioned.