1 Definition and Basic Concepts

1.1 Pixel-to-distance mapping

A depth map is a 2D image whose pixel values encode distance from a chosen viewpoint (camera or sensor) to locations in a 3D scene. Conceptually, each pixel corresponds to a ray that leaves the viewpoint through the image plane; the stored value indicates where that ray intersects the visible surface. As a result, depth maps provide a compact representation of scene geometry that can be processed with image techniques while preserving 3D information.

1.2 Depth vs. disparity vs. inverse depth

Depth maps are closely related to several alternative encodings of distance. Disparity is commonly used in stereo systems and is proportional to inverse distance: closer objects produce larger disparity values. Inverse depth (or depth in reciprocal form) is often used to improve numerical stability in optimization and learning, since variations at near ranges can become more linear in reciprocal space. These encodings differ by monotonic transformations, so many algorithms can convert between them if camera calibration and depth range are known.

1.3 Coordinate systems and camera geometry

Depth is defined relative to a coordinate system. In practice, it depends on the camera model and whether the distance is measured along the camera’s optical axis (often the “z” coordinate in camera space) or as Euclidean range. Using the pinhole camera model, pixel coordinates map to rays through the intrinsic camera matrix, and depth corresponds to the scale at which a 3D point lies along each ray. Consistent use of intrinsics, extrinsics, and the chosen depth convention is essential when transforming depth maps between views or combining them with other sensor data.

2 Acquisition Methods

2.1 Active sensing

2.1.1 Time-of-flight (ToF) depth sensors

Time-of-flight sensors measure distance by estimating the travel time of emitted light, commonly using modulated illumination. The resulting depth image can be produced directly by the sensor hardware, offering dense measurements and moderate real-time performance. Performance depends on ambient light, reflectivity, and multipath effects, which can introduce biased depth where surfaces cause interference.

2.1.2 Structured light projection

Structured light systems project a known pattern (e.g., stripes or grids) onto the scene. A camera observes the deformation of the pattern; triangulation then yields depth. Structured light can produce high accuracy in controlled conditions and is effective at capturing fine geometric detail. However, it may struggle with highly reflective, transparent, or poorly illuminated surfaces.

2.2 Passive sensing

2.2.1 Stereo matching and triangulation

Stereo vision uses two calibrated cameras to acquire synchronized images. Matching corresponding pixels across views and applying triangulation yields disparity and then depth. The depth map quality is determined by texture availability, the accuracy of correspondence, and calibration. Regions with low contrast often become sparse or incorrect due to mismatches and uncertainty.

2.2.2 Multi-view depth reconstruction

When more than two views are available, geometric consistency across multiple images can refine depth estimates. Approaches include multi-view stereo (MVS), where hypotheses of depth are evaluated against photometric or geometric cues across a set of images. Multi-view methods can increase robustness and fill gaps, at the cost of greater computational complexity and sensitivity to camera pose accuracy.

2.3 Learning-based estimation

2.3.1 Monocular depth estimation

Monocular depth estimation predicts depth from a single image using trained models. Since only one view is given, the solution is inherently ambiguous up to scale, and many systems learn priors about typical shapes and lighting. Despite this, modern architectures can output dense depth maps that are useful for downstream applications such as viewpoint synthesis, augmented reality, and scene understanding.

2.3.2 Training data and supervision signals

Learning systems typically rely on labeled depth data from sensors, synthetic scenes, or self-supervised objectives that exploit geometric relations between frames. Supervision signals may include photometric consistency across views, consistency under camera motion, or direct regression to ground-truth depth when available. Choice of dataset, depth range normalization, and augmentation strategies significantly influence generalization to new scenes and domains.

3 Depth Map Representations and Formats

3.1 Metric depth vs. normalized depth

Depth maps may store metric depth (e.g., meters) or a normalized representation scaled to a fixed range, such as [0, 1]. Normalized depth is convenient for storage and for machine learning pipelines but requires the original scaling parameters to recover metric values. Metric depth preserves physical meaning, enabling direct metric measurement when sensors and calibration are accurate.

3.2 Common file formats and channel conventions

Depth may be stored in single-channel images, sometimes using floating-point formats to preserve precision. In common practice, depth maps can appear as grayscale images, EXR files, or depth-specific encodings embedded in image containers. When stored as multi-channel images, the encoding may split depth into bytes or normalize disparity-like values. Channel conventions must be clearly documented to avoid unit and scaling mistakes during integration.

3.3 Handling missing or invalid depth values

Real-world measurements often contain invalid regions due to occlusion boundaries, sensor noise, or failures in correspondence. Depth maps typically mark these pixels as missing using sentinel values (e.g., NaN), specific integer ranges, or mask images. Proper downstream handling includes ignoring invalid pixels during fitting, filling holes only when justified, and preserving uncertainty rather than treating missing data as zeros.

3.4 Quantization, precision, and dynamic range

Depth values can be quantized when stored in integer formats, leading to stair-step artifacts or reduced accuracy at certain ranges. Precision requirements vary by task: rendering may tolerate moderate errors, while robotics and measurement may require high fidelity. Dynamic range also matters because depth uncertainty often grows with distance; logarithmic-like encodings or inverse-depth representations can partially mitigate this effect.

4 Processing and Enhancement

4.1 Denoising and smoothing

Depth maps are frequently noisy due to sensor artifacts, matching errors, or model prediction variance. Denoising methods include edge-preserving filters that reduce random fluctuations while maintaining sharp geometric boundaries. Smoothing can also be formulated as regularization in optimization frameworks, where depth consistency is encouraged spatially but discontinuities at object edges are allowed.

4.2 Depth completion and super-resolution

Depth completion aims to infer missing values in sparse depth maps by leveraging image cues and geometric priors. Techniques may combine RGB guidance with depth propagation, then refine the result with learned models. Depth super-resolution targets increased effective detail, improving local structure beyond the original sensor resolution or disparity discretization.

4.3 Outlier detection and hole filling

Some pixels contain depth values that are inconsistent with neighbors or with multi-view constraints. Outlier detection can be based on statistical residuals, confidence estimates, or geometric constraints such as surface smoothness. Hole filling then interpolates across invalid regions, ideally producing plausible surfaces while avoiding over-smoothing that would erase object boundaries.

4.4 Temporal consistency for video depth

For depth in video, frame-to-frame flicker can degrade user experience and downstream tracking. Temporal consistency methods enforce smooth evolution of depth using motion cues, warping between frames, and regularizers that penalize large temporal changes unless supported by evidence. While this can reduce noise, it must be tuned to avoid lagging behind fast motion.

5 Applications in Graphics and Vision

5.1 Occlusion handling and scene understanding

Depth maps support occlusion reasoning by revealing which surfaces are closer to the viewpoint. In scene understanding pipelines, depth provides structure for segmenting objects, estimating free space, and interpreting spatial relationships. Reliable occlusion cues can improve 3D-aware compositing and help distinguish foreground geometry from background.

5.2 Novel view synthesis and image-based rendering

Novel view synthesis uses depth to reproject pixels from one or more input images into a desired target viewpoint. Correct depth enables approximate rendering of unseen regions and can reduce artifacts in parallax effects. In multi-view settings, depth can be fused or refined to handle visibility changes, while occlusions and disocclusions require careful blending strategies.

5.3 3D effects: parallax and refocusing

In consumer and research applications, depth maps can drive parallax shifting for stereoscopic-like experiences. Refocusing effects simulate a lens-like blur that depends on estimated depth, producing depth-of-field changes without physically capturing new sensor data. These effects rely on smooth depth gradients and robust handling of edges to prevent halo artifacts.

5.4 Relighting and surface-aware rendering

Depth maps can assist rendering tasks that depend on surface orientation. By estimating surface normals from depth (directly or via fitting), algorithms can approximate lighting interactions such as shading and specular highlights. Relighting can reuse geometry information to produce more coherent images under changed illumination, though errors in depth and normals directly affect appearance fidelity.

6 Evaluation and Metrics

6.1 Ground-truth availability and benchmarks

Evaluation depends on the availability of ground-truth depth. Some sensors provide accurate measurements for controlled scenes, while benchmarks use curated datasets with calibrated depth capture. For learning-based monocular systems, ground truth is often available but may differ in scale conventions, necessitating careful alignment before metric computation.

6.2 Error metrics (e.g., absolute/squared relative error)

Common metrics quantify pixel-wise discrepancies between predicted and reference depth. Absolute relative error and squared relative error measure deviations normalized by true depth, emphasizing both overall correctness and proportional accuracy. Root mean squared error variants penalize larger deviations more strongly, which can highlight outliers.

6.3 Scale ambiguity and alignment

Monocular depth estimation frequently suffers from scale ambiguity: a prediction may match shape but not metric size. Evaluation often addresses this by aligning predictions to ground truth using a scale factor, sometimes also allowing an affine correction depending on the benchmark protocol. Without proper alignment, depth quality can be underestimated even when geometry is reasonable.

6.4 Visual quality assessment

Beyond numeric scores, qualitative inspection is often used to judge artifacts such as depth banding, boundary smearing, or flicker in video. Visualization can reveal whether errors are concentrated in challenging regions (e.g., reflective surfaces) or are widespread. For rendering-driven tasks, perceptual quality of synthesized views can serve as a functional assessment of depth usefulness.

7 Computational Considerations

7.1 Resolution trade-offs and memory footprint

Higher resolution depth maps capture finer geometry but increase storage and processing costs. Memory consumption depends on the depth encoding (e.g., floating point versus quantized integers) and any accompanying confidence or mask channels. For real-time systems, designers select an input/output resolution that balances detail with throughput, often using multi-scale strategies.

7.2 Real-time pipelines and latency constraints

In interactive applications like AR, depth must be generated and processed within tight time budgets. Latency arises from sensor acquisition, calibration steps, preprocessing, and any neural inference. Efficient implementations may employ pipelining, asynchronous transfers, and streamlined post-processing to keep end-to-end delay low.

7.3 GPU/CPU acceleration strategies

Depth pipelines often use GPUs for parallel operations such as filtering, reprojecting, and running neural networks. CPU can remain responsible for tasks like camera synchronization, metadata handling, and lightweight bookkeeping. When combining CPU and GPU stages, transfer overhead and synchronization points can dominate performance if not managed carefully.

8.1 Depth images, z-buffers, and point clouds

A depth image is a general term closely aligned with depth maps, often used interchangeably in graphics contexts. A z-buffer stores per-pixel depth values as part of a rasterization pipeline, typically in a projected or normalized form rather than a physically meaningful metric unit. A point cloud converts depth into 3D points, usually using camera intrinsics to back-project pixels into 3D space.

8.2 Surface normals from depth

Surface normals can be estimated from depth by computing local gradients and transforming them into the camera coordinate frame. Methods may use finite differences, plane fitting in neighborhoods, or learned normal prediction. Accurate normals are important for shading and geometry-aware processing, but noise and missing pixels can lead to unstable normal estimates.

8.3 Mesh reconstruction from depth maps

Depth maps can be turned into polygonal meshes by back-projecting pixels and triangulating the grid. Post-processing steps such as smoothing, hole filling, and outlier removal can produce more coherent surfaces. Mesh quality depends on depth resolution, edge preservation, and how the method handles discontinuities at object boundaries.

9 Limitations and Failure Modes

9.1 Reflective or low-texture surfaces

Reflective materials can violate measurement assumptions in active sensors and can also degrade stereo correspondence. Low-texture regions provide insufficient cues for matching, which can yield noisy or missing depth. These limitations often concentrate errors on shiny, transparent, or uniformly colored surfaces.

9.2 Motion blur and rolling shutter effects

Camera motion during exposure can blur edges and reduce the accuracy of feature matching, stereo correspondence, or structured light decoding. Rolling shutter can further distort geometry across rows, complicating depth estimation especially in fast motion. Video-based depth pipelines must account for such temporal artifacts to prevent systematic distortions.

9.3 Domain shift in learning-based depth estimation

Models trained on one set of environments may underperform when tested on different lighting, camera characteristics, or scene types. This “domain shift” can cause systematic bias in predicted depths, such as overly compressed ranges or incorrect uncertainty patterns. Robust training strategies and adaptive calibration can mitigate, but not entirely eliminate, these issues.

9.4 Scale errors in monocular systems

Even when predicted depth captures correct relative structure, monocular systems may output depths at an incorrect scale. Scale errors affect applications that require metric accuracy, including physical measurement and robotics. Approaches include scale calibration using reference objects, multi-frame constraints, or fusion with additional sensors.