Semantic segmentation is a computer vision task that involves partitioning an image into multiple segments and assigning a semantic label to each pixel. Unlike instance segmentation, which distinguishes individual objects, semantic segmentation classifies every pixel according to a predefined set of categories (e.g., road, car, building, sky). It serves as a fundamental building block for scene understanding, enabling applications such as autonomous driving, medical image analysis, and satellite imagery interpretation. Modern approaches predominantly rely on deep convolutional neural networks (CNNs) and transformer-based architectures to achieve high accuracy.
1 Definition and problem formulation
1.1 Pixel-level classification
Semantic segmentation is formulated as a dense prediction problem: given an input image of height \(H\) and width \(W\), the goal is to produce an output map of size \(H \times W\) where each pixel is assigned a label from a fixed set of semantic classes \(\mathcal{C} = \{1, 2, \dots, K\}\). The model learns a mapping \(f: \mathbb{R}^{H \times W \times 3} \to [1, K]^{H \times W}\) that respects the spatial structure of the scene.
1.2 Relationship to other segmentation tasks
1.2.1 Instance segmentation
Instance segmentation detects and delineates each distinct object instance (e.g., individual cars, persons) while simultaneously assigning a semantic label. It outputs per-instance masks, so each pixel of an object receives both a class and an instance ID. Semantic segmentation, in contrast, does not separate individual instances within the same class; all pixels of the same class are treated uniformly.
1.2.2 Panoptic segmentation
Panoptic segmentation unifies semantic and instance segmentation by assigning a class label to every pixel and additionally an instance ID to "thing" classes (countable objects such as people or vehicles). "Stuff" classes (amorphous regions such as sky, road) receive only a semantic label. Semantic segmentation thus forms a subset of panoptic segmentation, covering both stuff and things but without instance discrimination.
2 Historical development
2.1 Early methods: thresholding and graph-based
Before deep learning, semantic segmentation relied on low-level image features and graphical models. Basic techniques used intensity thresholding or edge detection to separate regions. More sophisticated approaches formulated segmentation as a graph partitioning or energy minimization problem.
2.1.1 Normalized cuts
Normalized cuts (Shi and Malik, 2000) treated the image as a weighted graph where nodes are pixels and edge weights reflect similarity. The segmentation was obtained by recursively partitioning the graph to minimize a normalized cut criterion, balancing the total dissimilarity between groups and the internal similarity within groups. This method produced globally consistent segmentations but was computationally expensive for large images.
2.1.2 Conditional random fields (CRFs)
Conditional random fields (CRFs) modeled the joint distribution of pixel labels given the image, incorporating both unary terms (based on pixel-wise classifiers) and pairwise potentials that enforce spatial smoothness or edge consistency. Fully connected CRFs (Krähenbühl and Koltun, 2011) allowed efficient inference with Gaussian edge potentials, significantly improving boundary localization when combined with handcrafted features or early neural networks.
2.2 Rise of deep learning
2.2.1 Fully convolutional networks (FCN)
The pioneering work of Long, Shelhamer, and Darrell (2015) introduced fully convolutional networks (FCNs), which replaced the fully connected layers of traditional classification CNNs with convolutional layers, enabling dense pixel-wise predictions from an input of arbitrary size. FCNs produced segmentation maps by upsampling coarse feature maps via transposed convolutions. This marked the beginning of end-to-end deep learning for semantic segmentation.
2.2.2 Encoder-decoder architectures
Following FCN, encoder-decoder architectures emerged to recover spatial resolution lost during downsampling. The encoder (typically a pretrained backbone such as VGG or ResNet) extracted hierarchical features, while the decoder gradually upsampled them to the original resolution. Skip connections were introduced to combine low-level spatial details with high-level semantic information, improving segmentation accuracy at object boundaries.
3 Core architectures and techniques
3.1 Fully convolutional networks (FCN)
FCN adapts a classification CNN by turning all fully connected layers into convolutional layers, resulting in a heatmap of class scores. The final map is upsampled (e.g., by factor 32) back to the input size, producing a coarse segmentation. Multiple variants (FCN-32s, FCN-16s, FCN-8s) use different upsampling strides and skip connections to refine the output.
3.1.1 Skip connections
Skip connections combine predictions from intermediate layers of the encoder with the final coarse prediction to improve detail. For example, FCN-8s adds a 2× upsampled prediction from pool4 (stride 16) to the pool3 (stride 8) features, then upscales to reach full resolution. This allows the model to recover spatial information lost in deeper layers.
3.2 U-Net
U-Net (Ronneberger, Fischer, and Brox, 2015) is a symmetric encoder-decoder designed originally for biomedical image segmentation. The encoder contracts the spatial dimensions while increasing channel depth; the decoder expands the spatial dimensions and reduces channels. Skip connections directly concatenate feature maps from each encoder level to the corresponding decoder level, preserving fine details.
3.2.1 Symmetric encoder-decoder with skip connections
The symmetry means the encoder and decoder have the same number of resolution levels (typically four). Each skip connection transmits feature maps of the same spatial resolution from the contracting path to the expanding path, allowing the decoder to use both high-level semantic features and low-level texture information. This design is especially effective when training data is limited.
3.2.2 Variants (U-Net++, Attention U-Net)
U-Net++ introduces nested dense skip connections to reduce the semantic gap between encoder and decoder features. Each decoder node receives multi-scale features from multiple encoder layers. Attention U-Net incorporates attention gates that suppress irrelevant background regions and highlight salient features, improving performance on small or ambiguous structures.
3.3 DeepLab series
The DeepLab series (v1–v3+) by Chen et al. advances semantic segmentation through atrous (dilated) convolutions and multi-scale context aggregation.
3.3.1 Atrous convolution
Atrous convolution introduces a dilation rate that expands the receptive field without increasing the number of parameters or reducing spatial resolution. By using different dilation rates in parallel or cascade, the model can capture context at multiple scales while preserving feature map resolution.
3.3.2 Atrous Spatial Pyramid Pooling (ASPP)
ASPP applies multiple atrous convolutional filters with different dilation rates to the same feature map, then concatenates the results. This captures information at several effective field-of-view sizes. DeepLabv3 added image-level features (global average pooling) to the ASPP module to incorporate global context.
3.3.3 DeepLabv3 and DeepLabv3+
DeepLabv3 uses a cascade of atrous convolutions in the backbone (e.g., ResNet) and a simplified ASPP module. DeepLabv3+ adds a decoder module that refines the segmentation by upsampling the encoder output and fusing it with low-level features from the backbone, improving boundary delineation. It achieved state-of-the-art results on multiple benchmarks.
3.4 Transformer-based models
3.4.1 Vision Transformer (ViT) adaptations
Vision Transformers process images as sequences of patches and apply self-attention to model long-range dependencies. For segmentation, ViT can be used as an encoder followed by a decoder (e.g., a series of upsampling stages or a lightweight CNN head). The patch-based representation inherently provides a coarse segmentation, which is then refined.
3.4.2 SETR (Segmentation Transformer)
SETR treats semantic segmentation as a sequence-to-sequence prediction task. It uses a pure transformer encoder (ViT) to extract features from image patches and a decoder (e.g., multiple progressive upsampling layers) to generate pixel-level predictions. It demonstrated competitive performance, especially on large-scale datasets, by capturing global context effectively.
3.4.3 Mask2Former and DETR-based approaches
Mask2Former unifies semantic, instance, and panoptic segmentation using a transformer decoder that predicts a set of binary masks and corresponding class labels. It employs masked attention to restrict cross-attention to predicted mask regions, improving efficiency. DETR-based methods (e.g., Panoptic-DETR) use a similar query-based approach, directly producing segmentation masks in an end-to-end fashion without handcrafted postprocessing.
4 Training and optimization
4.1 Loss functions
4.1.1 Cross-entropy loss
Cross-entropy loss is the most common objective for semantic segmentation. For each pixel, it measures the negative log-likelihood of the predicted probability for the true class. The total loss is the average or sum over all pixels. While simple, it treats all pixels equally and can be suboptimal when class frequencies are imbalanced.
4.1.2 Dice loss
| Dice loss is derived from the Sørensen–Dice coefficient, which measures overlap between predicted and ground-truth regions. For each class, it computes \(1 - \frac{2 | P \cap G | }{ | P | + | G | }\) where \(P\) is the predicted set and \(G\) the ground truth. Dice loss is robust to class imbalance and is widely used in medical segmentation tasks. |
|---|
4.1.3 Focal loss
Focal loss modifies cross-entropy to down-weight well-classified pixels and focus on hard, misclassified examples. It introduces a modulating factor \((1-p_t)^\gamma\) where \(p_t\) is the predicted probability for the true class and \(\gamma \geq 0\) is a focusing parameter. This helps address foreground-background imbalance in segmentation.
4.2 Data augmentation
Data augmentation increases training set diversity and improves generalization. Common techniques include random cropping, flipping, rotation, scaling, color jitter, and elastic deformations. For semantic segmentation, augmentations must preserve pixel-label correspondence. Advanced methods such as mixup (linearly blending images and labels) and CutMix (cutting and pasting patches) have also been adapted.
4.3 Class imbalance handling
Class imbalance (e.g., rare classes like pedestrian in driving scenes) can degrade segmentation performance. Strategies include: (1) reweighting the loss function (e.g., using class weights inversely proportional to frequency); (2) oversampling images containing rare classes; (3) using loss functions such as Dice or Focal that are less sensitive to imbalance; (4) designing network architectures that explicitly model minority classes via attention or multi-scale features.
5 Datasets and benchmarks
5.1 Common datasets
5.1.1 PASCAL VOC 2012
PASCAL VOC 2012 provides 21 classes (including background) with 1,464 training and 1,449 validation images. It is a standard benchmark for evaluating segmentation models, though its relatively small size has led to the use of augmented data (e.g., SBD dataset with 10,000+ images).
5.1.2 Cityscapes
Cityscapes focuses on urban street scenes with 30 classes (19 used for evaluation). It contains 5,000 finely annotated images (2,975 train, 500 val, 1,525 test) plus 20,000 coarsely annotated images. Image resolution is 2048×1024. Common metrics are reported on the validation and test sets.
5.1.3 ADE20K
ADE20K (SceneParse150) includes 150 classes covering a wide variety of indoor and outdoor scenes. It has 20,210 training and 2,000 validation images. The dataset is challenging due to fine-grained categories (e.g., bed, lamp, curtain) and large class imbalance.
5.1.4 COCO-Stuff
COCO-Stuff augments the COCO dataset by adding stuff annotations (e.g., sky, road, grass) to the existing thing annotations. It contains 164,000 images with 91 stuff and 80 thing classes, totaling 171 semantic categories. It is used for large-scale panoptic and semantic segmentation tasks.
5.2 Evaluation metrics
5.2.1 Pixel accuracy
Pixel accuracy computes the fraction of correctly classified pixels: \(\frac{\sum_i n_{ii}}{\sum_i t_i}\), where \(n_{ii}\) is the number of pixels of class \(i\) correctly predicted as class \(i\), and \(t_i\) is the total number of pixels of class \(i\). This metric can be misleading in the presence of class imbalance, as a trivial model that predicts the majority class will achieve high accuracy.
5.2.2 Mean intersection over union (mIoU)
mIoU is the primary metric for semantic segmentation. For each class \(i\), the intersection over union is \(\frac{n_{ii}}{t_i + \sum_j n_{ji} - n_{ii}}\), where \(n_{ji}\) is the number of pixels of class \(i\) predicted as class \(j\). The mean is taken over all classes. mIoU penalizes both false positives and false negatives and is robust to imbalance when averaged per class.
5.2.3 Frequency weighted IoU
Frequency weighted IoU (FWIoU) weights each class’s IoU by the frequency of that class in the ground truth. It gives more importance to large classes and is sometimes used when overall pixel-level accuracy is of interest. However, it can obscure performance on rare classes.
6 Applications
6.1 Autonomous driving
6.1.1 Road and lane detection
Semantic segmentation identifies drivable areas, lane markings, and road boundaries. These outputs are crucial for path planning and vehicle control. Models like DeepLab and SegFormer are used to segment road, sidewalk, and lane line pixels in real-time from camera images.
6.1.2 Obstacle segmentation
Segmenting obstacles such as other vehicles, pedestrians, cyclists, and static objects (e.g., traffic cones) provides a dense scene understanding for collision avoidance. Segmentation masks can be fused with LiDAR data to create occupancy grids.
6.2 Medical image analysis
6.2.1 Organ and tumor segmentation
Semantic segmentation of organs (liver, kidney, lung) and tumors from CT, MRI, or ultrasound scans aids diagnosis and treatment planning. U-Net and its variants are standard in this domain due to their ability to work with limited annotated data and produce fine boundary details.
6.2.2 Cell segmentation
In microscopy, semantic segmentation separates cells from background and distinguishes different cell types (e.g., nuclei, cytoplasm). This enables automated counting, morphological analysis, and tracking in biological research.
6.3 Remote sensing and satellite imagery
Semantic segmentation is used to classify land cover (urban, forest, water, cropland) from aerial or satellite images. It supports urban planning, environmental monitoring, and disaster response. Models must handle large spatial extents and multi-spectral input channels (e.g., RGB + near-infrared).
6.4 Robotics and augmented reality
Robots use semantic segmentation to understand their environment (e.g., floor, walls, furniture) for navigation and manipulation. In augmented reality, segmentation enables object-aware rendering, occlusion handling, and interaction with real-world elements (e.g., placing virtual furniture on a segmented floor).
7 Challenges and open issues
7.1 Domain shift and generalization
Models trained on one dataset (e.g., Cityscapes) often perform poorly on images from different cities, weather conditions, or sensor types. Domain adaptation and domain generalization techniques are active research areas. Unsupervised domain adaptation, where labeled source data and unlabeled target data are used, is a common approach.
7.2 Fine-grained segmentation
Distinguishing between visually similar categories (e.g., different animal species, types of vegetation) remains difficult due to subtle texture and shape differences. Fine-grained segmentation requires high-resolution features and large-scale annotated datasets.
7.3 Real-time inference
Many applications (autonomous driving, robotics) require latency of tens of milliseconds. Balancing accuracy and speed is challenging. Lightweight architectures (e.g., MobileNet-based backbones, efficient decoder designs) and model compression techniques (pruning, quantization) are employed.
7.4 Weakly supervised and semi-supervised learning
Pixel-level annotation is expensive and time-consuming. Weakly supervised methods use image-level labels, bounding boxes, or scribbles to train segmentation models. Semi-supervised methods leverage a small labeled set alongside a large unlabeled set. Both approaches aim to reduce annotation cost while maintaining reasonable performance.
8 Future directions
8.1 Foundation models for segmentation (e.g., SAM)
The Segment Anything Model (SAM) is a large-scale foundation model trained on over one billion masks. It can segment any object in an image with zero-shot generalization, using prompts (points, boxes, masks). Foundation models are expected to become the backbone for many segmentation tasks, requiring fine-tuning or distillation for specific domains.
8.2 Integration with large language models
Combining vision-language models (e.g., CLIP) with segmentation enables open-vocabulary segmentation, where arbitrary text queries define semantic categories. Models like LSeg and GroupViT allow users to segment objects described by free-form language, expanding the flexibility of semantic segmentation beyond fixed class sets.
8.3 3D and video semantic segmentation
Extending segmentation to 3D point clouds and video sequences addresses temporal and spatial continuity. Video semantic segmentation leverages motion cues and temporal consistency to improve mask quality. 3D segmentation (e.g., on LiDAR or RGB-D data) supports autonomous navigation and scene reconstruction. Future research will likely unify 2D, 3D, and temporal segmentation under a single framework.