SHAP (SHapley Additive exPlanations) is a unified framework for interpreting the predictions of machine learning models, grounded in cooperative game theory. It assigns each feature an importance value for a particular prediction by computing Shapley values—a solution concept from game theory that fairly distributes the total gain among players. In the context of machine learning, SHAP provides both global and local interpretability, making it a cornerstone of explainable artificial intelligence (XAI) and a standard tool for model debugging, compliance, and trust.
1 Theoretical foundations
1.1 Cooperative game theory and Shapley values
SHAP draws directly from cooperative game theory, where a coalition of players (features) collectively produces a payoff (the model’s prediction). The Shapley value is the unique method that distributes this payoff among players according to their marginal contributions, averaged over all possible coalitions. For a given prediction, each feature’s Shapley value represents the average expected marginal contribution of that feature when added to all subsets of other features.
1.1.1 Additive feature attribution methods
SHAP belongs to the class of additive feature attribution methods, which explain a prediction as a linear combination of binary variables indicating whether a feature is present or absent. Formally, the explanation model \(g(z') = \phi_0 + \sum_{i=1}^M \phi_i z'_i\) approximates the original model \(f(x)\), where \(z' \in \{0,1\}^M\) denotes feature coalitions and \(\phi_i\) are the Shapley values.
1.1.2 The Shapley value axioms (efficiency, symmetry, dummy, additivity)
The Shapley value is uniquely defined by four axioms:
- Efficiency: The sum of all Shapley values equals the total payoff minus the average prediction.
- Symmetry: Two features that contribute identically to every coalition receive equal values.
- Dummy: A feature that never changes the payoff receives a value of zero.
- Additivity: For a model that is the sum of two independent models, the Shapley value of a feature equals the sum of its values from each model.
1.2 Relationship to LIME and other interpretability methods
SHAP generalizes and unifies several existing interpretation methods, most notably LIME (Local Interpretable Model-agnostic Explanations). While LIME approximates the model locally with a sparse linear model using arbitrary weighting kernels, SHAP provides a theoretically grounded solution by satisfying the Shapley value axioms.
1.2.1 Local accuracy and consistency
SHAP guarantees local accuracy: the explanation model matches the original model’s output for the instance being explained. Consistency ensures that if a feature’s true marginal contribution increases (or stays the same) in a different model, its SHAP value cannot decrease—a property not shared by LIME.
1.2.2 Missingness property
The missingness property states that features that are not observed (absent) have zero attribution. This aligns with the dummy axiom from game theory and ensures that only present features receive credit.
2 SHAP framework and computational approaches
2.1 KernelSHAP
KernelSHAP is a model-agnostic approximation method that computes Shapley values by solving a weighted linear regression problem.
2.1.1 Linear approximation with weighted regression
KernelSHAP samples subsets of features (coalitions) and estimates the model’s prediction for each subset by integrating out missing features. It then fits a linear model with coefficients equal to the Shapley values, using a kernel weighting scheme derived from the Shapley value formula.
2.1.2 Computational complexity and approximations
Exact Shapley values require evaluating all \(2^M\) feature subsets, which is intractable for large \(M\). KernelSHAP approximates the values by sampling a fixed number of coalitions (e.g., \(2M + 2048\)) and using regularization to improve stability.
2.2 TreeSHAP
TreeSHAP is a specialized algorithm for tree-based models (e.g., decision trees, random forests, gradient boosting) that computes exact Shapley values in polynomial time.
2.2.1 Exact computation for tree-based models
TreeSHAP exploits the tree structure to estimate conditional expectations efficiently. It traverses all possible paths and adjusts for feature orderings, achieving \(O(TL2^M)\) worst-case complexity (where \(T\) is number of trees and \(L\) maximum leaves), but in practice runs much faster due to pruning.
2.2.2 Conditional expectation and feature dependence
Unlike the marginal expectation assumed by KernelSHAP, TreeSHAP uses the conditional expectation based on the empirical distribution of training data. This respects feature dependencies and yields more faithful explanations for correlated features, though it can introduce bias if the dependency structure is misspecified.
2.3 DeepSHAP
DeepSHAP adapts SHAP to deep neural networks by leveraging backpropagation and linear approximations.
2.3.1 Backpropagation-based approximation for neural networks
DeepSHAP propagates SHAP values backward through the network layers, using linear approximations at each neuron. It treats each activation as a linear function of its inputs and distributes contributions proportionally to the weights.
2.3.2 Connection to DeepLIFT
DeepSHAP is closely related to DeepLIFT (Deep Learning Important FeaTures). Both use similar backpropagation rules, but DeepSHAP reweights the multipliers to satisfy the Shapley value axioms, while DeepLIFT uses arbitrary reference values.
2.4 Other variants
2.4.1 LinearSHAP
LinearSHAP provides exact Shapley values for linear models in closed form, without requiring sampling or approximation. It computes \(\phi_j = w_j (x_j - \mathbb{E}[x_j])\), where \(w_j\) is the model coefficient.
2.4.2 SamplingSHAP
SamplingSHAP is a model-agnostic method that estimates Shapley values by Monte Carlo sampling of feature permutations. It approximates the expected marginal contribution by averaging over random orderings.
3 Interpretability outputs
3.1 Global interpretation
Global SHAP analyses summarize feature importance across an entire dataset.
3.1.1 SHAP summary plot (beeswarm)
The beeswarm plot displays each feature’s SHAP values for every instance as a swarm of points. The vertical spread shows the distribution of impact, and the color indicates the feature value (low to high). It reveals both the direction and variability of feature effects.
3.1.2 SHAP feature importance (bar plot)
The bar plot shows the mean absolute SHAP value for each feature, sorted descending. This provides a global ranking of feature importance, similar to traditional permutation importance but with a game-theoretic foundation.
3.1.3 SHAP dependence plot
A SHAP dependence plot shows the relationship between a feature’s value and its SHAP value across all instances. It can reveal nonlinear patterns and interactions by optionally coloring points with a second feature.
3.2 Local interpretation
Local SHAP explanations provide feature attributions for a single prediction.
3.2.1 Force plot
The force plot visualizes the additive contributions as arrows pushing the prediction away from a baseline (e.g., average prediction). Features that increase the prediction are shown in red, those that decrease it in blue. The magnitude of each arrow equals the SHAP value.
3.2.2 Waterfall plot
The waterfall plot builds from the base value (expected model output) and sequentially adds each feature’s SHAP value to reach the final prediction. It displays the cumulative effect, showing how each feature changes the output step by step.
3.2.3 Decision plot
The decision plot shows cumulative SHAP values as a line chart, with features ordered by their impact. Multiple instances can be overlaid for comparison, making it useful for understanding how feature contributions vary across predictions.
4 Applications and use cases
4.1 Model debugging and validation
SHAP helps identify model flaws by exposing unexpected feature behavior.
4.1.1 Detecting data leakage
If a feature has a very high mean absolute SHAP value and the model achieves suspiciously high accuracy, it may indicate that the feature indirectly contains the target information (e.g., an ID column or future data). SHAP dependence plots can reveal unnatural patterns.
4.1.2 Comparing feature contributions across models
By computing SHAP values for two models trained on the same data, practitioners can compare which features each model relies on. This aids in model selection and understanding of architectural differences.
4.2 Regulatory compliance and fairness
Many regulated industries require explainable decisions.
4.2.1 Auditing bias in credit scoring
SHAP reveals which features drive loan approval or denial. Regulators can check if protected attributes (e.g., race, gender) have unreasonably high influence, even if not included directly, through correlated proxies.
4.2.2 Explaining medical predictions
In healthcare, SHAP highlights which biomarkers or symptoms influence a diagnosis. Clinicians can verify that the model’s reasoning aligns with medical knowledge, increasing trust and enabling oversight.
4.3 Feature engineering and selection
SHAP guides the iterative improvement of feature sets.
4.3.1 Identifying redundant features
Features with consistently near-zero SHAP values across many instances are likely redundant or irrelevant. Removing them can reduce model complexity without sacrificing performance.
4.3.2 Guiding domain-specific insight
Unexpected positive or negative SHAP values for certain features can reveal hidden relationships in the data. For example, a dependence plot may show that a feature has opposite effects in different regions, prompting further investigation.
5 Limitations and considerations
5.1 Computational cost for high-dimensional data
Exact Shapley values have exponential complexity, and even approximations like KernelSHAP require many model evaluations. For datasets with thousands of features, SHAP can be prohibitively slow, especially with complex models.
5.2 Assumption of feature independence
5.2.1 Impact on conditional expectation estimation
KernelSHAP marginalizes over missing features by sampling from the training data independently. This assumes features are independent, which is often violated in real data and can produce unrealistic counterfactuals and misleading attributions.
5.3 Handling of categorical and correlated features
Categorical features with many levels increase the number of possible coalitions. Highly correlated features lead to unstable Shapley values because the marginal contribution of one feature depends heavily on the presence of another. TreeSHAP mitigates this by using conditional expectation, but the interpretation becomes dependent on the training distribution.
5.4 Adversarial misuse and overinterpretation
SHAP values can be manipulated by intentionally adding features that mimic the target or by engineering spurious correlations. Practitioners may also overinterpret small SHAP values as “no effect” when the model is uncertain, leading to false confidence. Explanations are only as reliable as the underlying model and data.
6 Software implementations and ecosystem
6.1 SHAP Python library (shap)
The shap Python library (developed by Scott Lundberg and Su-In Lee) is the primary implementation. It provides both model-agnostic and model-specific explainers.
6.1.1 Integration with scikit-learn, XGBoost, LightGBM, and deep learning frameworks
The library supports scikit-learn models (via KernelSHAP), tree‑based models (via TreeSHAP), and deep learning frameworks (TensorFlow, PyTorch) via DeepSHAP. It also offers a fast LinearExplainer for linear models.
6.1.2 Visualization API
shap includes built‑in plotting functions (shap.summary_plot, shap.force_plot, shap.waterfall_plot, etc.) that generate publication‑ready visualizations with minimal code. Integration with matplotlib allows customization.
6.2 Alternatives and complementary tools
6.2.1 LIME
LIME provides local surrogate explanations using an interpretable model (e.g., linear model) fitted around a prediction. Unlike SHAP, LIME does not satisfy consistency or the Shapley axioms, but it is often faster for high‑dimensional data.
6.2.2 Integrated gradients
Integrated Gradients is a gradient‑based attribution method for neural networks. It measures the integral of gradients along a path from a baseline to the input. It satisfies sensitivity and implementation invariance but does not provide the game‑theoretic guarantees of SHAP.
6.2.3 Partial dependence plots
Partial dependence plots (PDPs) show the average model prediction as a function of one or two features. They are a global, model‑agnostic tool complementary to SHAP, but they do not decompose individual predictions or account for feature interactions as comprehensively.