1 Overview of Non-maximum Suppression

1.1 Core intuition: keeping local maxima

Non-maximum suppression (NMS) is a post-processing procedure that retains only the strongest responses within a defined neighborhood while discarding weaker ones. Given a score field—such as edge strength, corner likelihood, objectness, or class probability—NMS identifies peaks that are locally maximal and removes neighboring scores that are not maximal. The result is a sparser representation that emphasizes prominent structures and reduces redundancy.

1.2 Where NMS appears in vision systems

NMS is common across classical image processing and modern computer vision pipelines. In classical edge detectors, it contributes to producing thin edges by selecting gradient-consistent local maxima. In feature extraction, it helps keep only salient keypoints by pruning nearby candidates. In object detection, NMS is widely used to convert dense predictions (e.g., many overlapping bounding boxes) into a smaller set of final detections. In dense prediction tasks, variations of NMS may be applied to heatmaps to extract discrete locations from continuous confidence maps.

1.3 Typical inputs and outputs

A typical NMS input is a grid or set of candidate scores, often accompanied by spatial metadata. For heatmaps and images, the input is usually a 2D array (or 3D array for multiple channels) of values with known pixel coordinates. For object detection, the input includes predicted bounding boxes and their associated confidence scores, sometimes with class labels. The output is either:

  • A thinned map where only local maxima retain their values (with others set to zero or suppressed), or
  • A list of selected detections after removing near-duplicates.

1.4 Relationship to peak detection and filtering

NMS is closely related to peak detection: it identifies local maxima under a neighborhood comparison rule. It can be viewed as a specialized filter that enforces local dominance. Unlike smoothing filters that average information, NMS performs a selective operation that preserves peak locations and suppresses weaker neighbors, often improving interpretability and downstream matching.

2 Mathematical and Algorithmic Foundations

2.1 Local neighborhoods and comparison rules

The defining ingredient of NMS is a neighborhood over which candidates are compared. Conceptually, a point is kept if its score is greater than (or not less than) the scores in the neighborhood according to a rule.

2.1.1 Defining “non-maximum” in 1D vs 2D

In 1D, NMS typically compares each sample to values in a window to the left and right. A position is retained if it is a local peak relative to its immediate neighbors (or relative to a larger span). In 2D, the neighborhood may be a square or circular window, or it may be constrained by direction (e.g., along gradient orientation for edge thinning). A 2D point can be declared a non-maximum if any neighbor in the chosen set has a higher score.

Directional NMS modifies the neighborhood for structured responses: for edges, the comparison can be performed only along the direction where a thin ridge is expected to lie. This makes the “local maximum” notion dependent on context, not just position.

2.2 Thresholding and ranking

NMS often includes a gating step so that very small scores are ignored even if they form a peak.

2.2.1 Hard vs adaptive thresholds

A common approach uses a hard threshold: candidates with scores below a fixed cutoff are discarded early. Adaptive thresholding adjusts the cutoff based on image content, noise level, or score distribution, aiming to maintain a stable number of retained peaks across varying conditions. Adaptive variants can be useful when score magnitudes fluctuate between images.

2.2.2 Score ordering and selection

For detection boxes, NMS is typically greedy: candidates are sorted by confidence, then processed from highest to lowest. Each selected candidate suppresses others that are sufficiently similar by a chosen metric. Because the selection depends on order, using descending scores helps ensure that the kept set prioritizes the most confident detections.

2.3 Suppression mask formulation

In map-based NMS, the output can be expressed using a mask. Let \(S(x)\) be the score at location \(x\), and let \(N(x)\) denote its neighborhood. A binary mask \(M(x)\) might be defined as:

  • \(M(x)=1\) if \(S(x)\) is locally maximal within \(N(x)\) (and passes a threshold),
  • \(M(x)=0\) otherwise.

The suppressed output is then \(S_{\text{out}}(x)=S(x)\cdot M(x)\) or another scheme that records retained maxima.

This formulation clarifies that NMS is fundamentally a local comparison followed by masking.

2.4 Complexity considerations

The computational cost depends on whether NMS is applied to dense maps or to sparse candidate lists. For dense 2D NMS with a fixed neighborhood size \(k \times k\), each pixel may require comparisons against \(k^2\) neighbors, leading to cost proportional to the number of pixels times neighborhood area. Optimizations include:

  • Using efficient max pooling or morphological operations,
  • Restricting neighborhoods to small windows,
  • Applying NMS only to candidates above a low threshold.

For bounding-box NMS, the dominant cost can come from pairwise overlap checks. Greedy processing still may require many comparisons in worst-case scenarios with many candidates, motivating faster approximations, preselection, or alternative post-processing.

3 NMS Variants by Task

3.1 Edge thinning and gradient-based NMS

Edge thinning NMS converts thick or blurred gradient responses into a one-pixel-wide edge map. Since edges have a preferred direction, classic implementations compare along that direction rather than in all directions.

3.1.1 Directional suppression using gradient orientation

Given gradient magnitude and gradient orientation, the algorithm identifies, for each pixel, the two neighbor locations along the gradient-aligned axis. The pixel is retained if its magnitude is greater than those two neighboring values (often after interpolation). This yields a ridge-like selection consistent with the edge’s local orientation.

3.1.2 Interpolation vs discrete pixel comparisons

Because gradient orientation may point between pixel centers, directional sampling can involve interpolation. Instead of comparing strictly at integer grid points, the method may interpolate magnitudes at fractional offsets to better approximate the true ridge intersection. Discrete-only comparisons are faster but can produce slight quantization artifacts in edge positions.

3.2 Keypoint and corner detection NMS

Keypoint detectors often produce many candidate corners clustered around the same physical structure. NMS prunes these by enforcing spatial separation: within a window, only the strongest candidate remains.

3.2.1 Spatial windowing for candidate pruning

A typical approach uses a fixed-radius or fixed-size window around each candidate. After sorting by corner response, the algorithm selects the top candidate and suppresses other candidates within the window radius. The window size controls the minimum distance between final keypoints and influences both coverage and redundancy.

3.3 Bounding-box NMS for object detection

Object detection NMS operates on bounding boxes, removing near-duplicates that overlap heavily.

3.3.1 IoU-based suppression logic

The standard criterion uses Intersection over Union (IoU). For a selected box, any other box with IoU above a threshold is suppressed. IoU quantifies overlap relative to combined area, making it suitable for comparing boxes that refer to the same object.

3.3.2 Class-agnostic vs class-aware NMS

Class-agnostic NMS treats all boxes as belonging to one pool, suppressing overlaps regardless of predicted category. Class-aware NMS performs suppression separately per class, allowing overlapping boxes of different classes to coexist. The choice depends on model behavior and application constraints.

3.4 Multi-scale NMS strategies

When detectors run at multiple image scales or produce multi-level feature pyramid outputs, NMS can be applied per scale or after merging candidates across scales. Per-scale NMS reduces redundancy locally, while global NMS helps enforce consistent suppression across scale levels. Some pipelines incorporate scale-dependent thresholds to reflect changing localization uncertainty at different resolutions.

3.5 Heatmap-based NMS in dense prediction

Dense prediction models often output a heatmap where peaks correspond to discrete entities, such as landmarks or object centers. Heatmap-based NMS keeps local maxima on the grid, sometimes followed by refinement. Because these tasks may involve very dense activations, the suppression neighborhood and thresholds are chosen to balance detecting true points against removing nearby false peaks.

4 Implementation Details and Practical Tips

4.1 Choosing window size and thresholds

Selecting the neighborhood size and thresholding strategy strongly affects results.

4.1.1 Trade-offs: recall vs precision

A larger neighborhood suppresses more aggressively, reducing duplicates but risking missed detections when true peaks are close. A smaller neighborhood preserves more candidates, improving recall but increasing the chance of multiple detections for a single object. Thresholds similarly trade sensitivity for specificity by controlling which peaks are eligible before suppression.

4.2 Handling plateaus and ties

When multiple locations share identical scores, strict “greater than” comparisons can behave inconsistently across platforms. Many implementations handle plateaus by using “greater than or equal to” logic, adding small perturbations, or explicitly defining tie-breaking rules (e.g., keeping the first maximum encountered). This is especially relevant for quantized scores or networks producing low-precision heatmaps.

4.3 Borders, padding, and coordinate conventions

NMS near image borders requires careful neighborhood definition. Options include reducing the neighborhood at boundaries, padding the score map (with zeros or negative infinity), or using masked comparisons. Consistent coordinate conventions—such as whether maxima are reported in pixel indices versus continuous coordinates—are important for compatibility with downstream stages like tracking or geometric fitting.

4.4 Vectorization and acceleration (CPU/GPU)

For dense NMS, performance can be improved by using vectorized operations and GPU-friendly primitives such as max pooling over sliding windows. For bounding-box NMS, acceleration may involve:

  • Pre-filtering by score threshold to reduce candidate count,
  • Using efficient overlap computation kernels,
  • Employing batched processing for throughput.

The goal is to reduce Python-level loops and leverage parallelism.

4.5 Numerical stability and data types

Score maps and confidence values may be stored as float16, float32, or integer types. Comparisons and suppression masks can be sensitive to precision, especially when many values are close. Using adequate numeric precision for the comparison step and ensuring consistent type casting helps avoid unexpected behavior. For IoU computation, stable box parameterization (e.g., using corner coordinates carefully) prevents negative widths or heights due to rounding.

5 Evaluation and Diagnostics

5.1 Measuring effect on detection quality

NMS changes the candidate set, affecting standard metrics such as precision-recall curves and mean average precision (mAP). Evaluation typically compares outputs before and after suppression using the detection task’s ground truth matching criteria. For keypoints, metrics may include localization error and repeatability. For edge maps, evaluation can involve edge alignment scores or F-measure style metrics based on distance thresholds.

5.2 Visualizing suppressed vs retained responses

Diagnostics often use visualization: overlaying retained boxes on images, marking kept keypoints, or displaying heatmaps with suppressed regions zeroed out. For dense maps, showing both the raw response and the NMS-masked output reveals whether the neighborhood rule is too strict or too permissive. For edge NMS, side-by-side comparisons can indicate whether directional suppression is consistent with expected edge orientations.

5.3 Common failure modes

5.3.1 Over-suppression and missed peaks

Overly aggressive NMS settings—such as a too-large window, a high IoU threshold (for box NMS that removes too much due to similarity), or a high score threshold—can eliminate true positives. The result may be fewer detections than expected, lower recall, and spatial gaps in heatmap-derived points.

5.3.2 Under-suppression and duplicates

Under-suppression occurs when the neighborhood or IoU thresholds are too lenient, allowing multiple peaks or overlapping boxes that correspond to the same underlying entity. This increases duplicate detections, raising false positives and potentially confusing downstream association steps in tracking or mapping pipelines.

6 Variants and Alternatives

6.1 Soft-NMS (score decays instead of removal)

Soft-NMS reduces the confidence scores of overlapping candidates rather than discarding them outright. The suppression effect is often modeled as a decay function based on overlap (e.g., IoU). This approach can preserve difficult or partially overlapping detections that hard NMS would eliminate, improving performance in some crowded-object scenarios while still controlling duplicates.

Weighted box fusion combines information from multiple overlapping predictions by averaging box coordinates using confidence-based weights, potentially across multiple model outputs. Instead of selecting one box and suppressing the rest, it merges boxes to produce a single, improved estimate. This is especially common when ensembling multiple detectors or multiple test-time augmentations.

6.3 Comparison with post-processing alternatives

Alternative post-processing strategies include:

  • Greedy matching variants with different similarity metrics,
  • Clustering-based methods that group candidates and replace them with cluster representatives,
  • Learned post-processing layers that predict final selections directly.

Compared with these, classic NMS is simple, fast to implement, and interpretable, but may be less flexible than learned approaches in complex overlap patterns.

6.4 When NMS is replaced by other mechanisms

NMS may be reduced or removed when a model produces inherently non-overlapping outputs, such as anchor-free methods with built-in selection rules, or when training encourages sparse predictions. Some modern architectures use specialized heads that predict a small set of entities directly, using attention or matching mechanisms rather than explicit NMS. Even in those cases, NMS-like ideas sometimes remain as a fallback for final discretization.

7 Use Cases and Examples

7.1 Step-by-step example on a synthetic heatmap

Consider a small 2D heatmap of response scores. Choose a neighborhood rule such as a \(3 \times 3\) window. For each pixel above a low threshold:

  1. Compare its value to all values in the window.
  2. If any neighbor has a higher value, suppress the pixel.
  3. If the pixel is tied for the maximum, apply the chosen tie policy (keep one or keep all equal maxima).

The output is a masked heatmap where peaks remain and nearby lower responses vanish.

7.2 Example: edge thinning from gradient magnitude

Assume an edge detection stage produces a gradient magnitude map and gradient orientation map. For each pixel:

  1. Determine the two sample points aligned with the local gradient direction (possibly using interpolation).
  2. Compare the current magnitude to the magnitudes at those two points.
  3. Keep the pixel if it is greater than both neighbors (or within a small tolerance).

This produces an edge map composed primarily of local ridges at the expected thickness.

7.3 Example: pruning detection candidates with IoU

Suppose an object detector proposes several bounding boxes with confidence scores. NMS proceeds as:

  1. Sort boxes by confidence descending.
  2. Select the highest-scoring box and add it to the final list.
  3. For remaining boxes, compute IoU with the selected box.
  4. Suppress boxes whose IoU exceeds the chosen threshold.
  5. Repeat until boxes are exhausted.

The remaining boxes are typically fewer and less redundant.

7.4 Typical pipeline integration points

NMS is usually inserted after a model produces dense candidates. In classical pipelines, it follows gradient computation or corner response evaluation. In modern detection pipelines, it follows the network’s box and class predictions and precedes label assignment and rendering. In keypoint and heatmap-based systems, it acts as a discretization step before coordinate extraction, refinement, or geometric estimation.