A saliency map is a computational representation that highlights the most visually or semantically prominent regions in an image or scene. Originating from neuroscience and computer vision, it mimics the human visual attention system by assigning a saliency value to each pixel or region, where higher values indicate greater importance. Saliency maps are widely used for tasks such as object detection, image segmentation, content-aware image editing, and visual question answering. They can be generated through bottom-up (data-driven) or top-down (task-driven) mechanisms, and modern deep learning approaches have significantly improved their accuracy and applicability.
1 Definition and background
A saliency map encodes the relative importance of locations in a visual input. It is typically a grayscale or heatmap image in which brighter intensities denote more salient regions. The concept draws from both biological studies of visual attention and computational efforts to replicate such selective processing.
1.1 Biological basis of saliency
In the primate visual system, attention is guided by both stimulus properties (e.g., contrast, motion) and cognitive goals. Neurons in the primary visual cortex (V1) and superior colliculus respond strongly to local features that differ from their surroundings. This bottom-up mechanism allows rapid detection of potential threats or food. Top-down influences from the prefrontal cortex modulate these responses based on task demands, such as searching for a specific object. Saliency maps model these processes by assigning higher activation to locations that would attract biological gaze.
1.2 Computational models of visual attention
Early computational models, such as Koch and Ullman's (1985) saliency-based attention system, proposed a single topographic map that integrates multiple feature channels. These models typically compute center‑surround differences across color, intensity, and orientation, then combine them into a master saliency map. The winner‑take‑all mechanism selects the most salient location, after which inhibition of return forces attention to shift to the next most salient region.
1.3 Relation to other attention mechanisms
Saliency maps belong to the broader family of attention mechanisms in machine learning. Unlike soft attention (e.g., in transformer models), which weights all input elements continuously, saliency maps often produce a spatial probability distribution. In computer vision, saliency is a form of spatial attention, whereas channel attention (e.g., squeeze‑and‑excitation networks) focuses on feature maps. Saliency maps are also related to explainability methods such as Grad‑CAM, which highlight input regions that influence a network's decision.
2 Types of saliency maps
Saliency maps are classified by their driving forces: bottom‑up, top‑down, or a combination.
2.1 Bottom-up saliency maps
Bottom‑up saliency is stimulus‑driven and independent of task or context. It relies on low‑level image properties such as color, intensity, orientation, and motion.
2.1.1 Feature integration theory
Proposed by Treisman and Gelade (1980), feature integration theory posits that early vision processes basic features in parallel, while focused attention combines them into objects. In computational terms, bottom‑up saliency maps are built by computing feature contrasts across multiple scales and then combining them into a single conspicuity map.
2.1.2 Itti–Koch model
The Itti–Koch model (1998) is a seminal bottom‑up algorithm. It decomposes an image into three feature channels (intensity, color, and orientation), computes center‑surround differences via Difference‑of‑Gaussian filters, and normalizes the maps. These are linearly summed to produce a final saliency map. The model predicts human eye fixations with reasonable accuracy and remains a benchmark.
2.2 Top-down saliency maps
Top‑down saliency is task‑ or knowledge‑driven, incorporating high‑level cues such as object categories, scene context, or learned priors.
2.2.1 Task-driven attention
When a person searches for a specific object (e.g., a red cup), regions containing features associated with that object become more salient. Computational models simulate this by modulating bottom‑up responses with target templates or by learning feature weights from task‑specific data. For instance, a model trained for pedestrian detection will assign higher saliency to regions with shape and texture typical of humans.
2.2.2 Knowledge-based priors
Prior knowledge, such as the center bias (people tend to look at image centers) or the tendency to fixate on faces, can be encoded as a top‑down saliency map. These priors are often combined with bottom‑up maps to improve fixation prediction.
2.3 Combined models
Many modern systems integrate both bottom‑up and top‑down cues into a unified saliency map. For example, a deep network might process the image through a bottom‑up stream while also receiving a task vector (e.g., "find the car") that modulates its feature maps. This hybrid approach achieves state‑of‑the‑art performance on both fixation prediction and salient object detection.
3 Algorithms and generation methods
Methods to compute saliency maps range from classical image processing to deep learning.
3.1 Classical image processing methods
These approaches rely on hand‑crafted features and heuristic rules without learning.
3.1.1 Frequency domain approaches
Spectral residual (Hou and Zhang, 2007) computes saliency by analyzing the log‑spectrum of an image. The residual between the average log‑spectrum and the image's log‑spectrum is transformed back to the spatial domain, suppressing redundant patterns (e.g., uniform textures) and highlighting novel regions. Phase‑based methods also exploit Fourier transform phase information.
3.1.2 Graph-based manifold ranking
Manifold ranking treats image pixels or superpixels as nodes in a graph, with edges weighted by feature similarity. Starting from a set of background queries (e.g., boundary nodes), the algorithm propagates saliency scores via random walks or label propagation. Regions that are structurally dissimilar from the background receive high saliency.
3.2 Deep learning methods
Deep neural networks learn saliency representations directly from data.
3.2.1 Convolutional neural networks
Fully convolutional networks (FCNs) are trained end‑to‑end on datasets with pixel‑level saliency annotations. Common architectures include a backbone (e.g., VGG, ResNet) followed by upsampling layers that produce a dense saliency map. Techniques such as multi‑scale feature fusion and dilated convolutions improve spatial resolution.
3.2.2 Attention-based transformers
Vision transformers (ViTs) and their variants apply self‑attention mechanisms across image patches. By extracting attention maps from the last layer, one can obtain saliency representations. Some models, like Saliency TRansformer (SalTR), are specifically designed to predict fixation maps directly from transformer encoders, capturing long‑range dependencies.
3.2.3 Weakly supervised methods
When pixel‑level annotations are scarce, weakly supervised learning uses image‑level labels (e.g., "contains a person") or click‑based annotations to train saliency models. Class activation maps (CAM) from image‑classifiers serve as coarse saliency maps, which can be refined with region‑growing or adversarial training.
3.3 Evaluation of saliency map quality
Quantitative evaluation requires comparing predicted maps against ground‑truth human annotations (fixations or marked salient regions).
3.3.1 Metrics: precision-recall, F-measure, AUC
For salient object detection (binary maps), precision and recall are computed over thresholded maps. The F‑measure balances them. For fixation prediction (continuous maps), the Area Under the ROC Curve (AUC) measures how well the map separates fixated from non‑fixated pixels. Other metrics include Normalized Scanpath Saliency (NSS) and Information Gain (IG).
3.3.2 Benchmark datasets (e.g., SALICON, MIT300)
SALICON contains eye‑tracking data on 20,000 images from COCO. MIT300 provides fixations from 39 observers on 300 natural images. CAT2000 includes 4,000 images across 20 categories. These datasets enable standardized comparisons among saliency models.
4 Applications
Saliency maps serve as preprocessing or auxiliary modules in many computer vision tasks.
4.1 Object detection and recognition
Saliency maps guide region proposal networks (e.g., in faster R‑CNN) to focus on promising image areas, reducing computational cost. In recognition tasks, saliency‑weighted pooling emphasizes discriminative parts, improving classification accuracy.
4.2 Image segmentation and cropping
Salient object segmentation uses saliency maps as initial seeds for graph‑cut or watershed algorithms. Content‑aware cropping automatically crops images to retain the most important region, often using saliency‑based seam carving.
4.3 Content-aware image retargeting
Image retargeting (e.g., resizing while preserving important content) uses saliency maps to assign deletion costs to pixels. Low‑saliency regions are removed or compressed, while high‑saliency regions remain undistorted.
4.4 Visual question answering
In visual question answering (VQA), saliency maps can attend to image regions relevant to a given question. Some models learn a dynamic saliency map conditioned on the question text, enabling more accurate answers.
4.5 Robotics and autonomous driving
Autonomous vehicles use saliency maps to prioritize processing of pedestrians, traffic signs, and obstacles. In robotic manipulation, saliency helps focus sensors on task‑relevant objects, reducing computational load.
5 Limitations and challenges
Despite progress, saliency maps have several inherent limitations.
5.1 Subjectivity and ground truth
Saliency is subjective: different observers may fixate differently on the same image, especially in complex scenes. Ground‑truth maps derived from aggregated fixations smooth out individual differences, potentially masking meaningful variation. There is no single "correct" saliency map for a general task.
5.2 Computational efficiency
Classical methods are lightweight, but deep learning models require significant GPU memory and time, limiting real‑time applications on mobile or embedded devices. Model compression and efficient architectures are active research areas.
5.3 Generalization across domains
A saliency model trained on natural photographs (e.g., MIT300) may perform poorly on medical images, synthetic graphics, or underwater scenes. Domain shift and dataset bias reduce transferability. Fine‑tuning on target data is often necessary but demands annotated examples.