GPU‑accelerated computing is the use of a graphics processing unit (GPU) together with a central processing unit (CPU) to accelerate scientific, engineering, and enterprise applications. By offloading computationally intensive portions of code to the GPU, which contains thousands of cores designed for parallel processing, this approach achieves significant performance gains over CPU‑only execution. Originally developed for real‑time graphics rendering, GPU acceleration now supports general‑purpose computing through platforms such as NVIDIA CUDA, AMD ROCm, and the cross‑vendor OpenCL standard. Applications range from deep learning and molecular dynamics to financial simulations and data analytics, making it a cornerstone of modern high‑performance computing.

1 Fundamentals

1.1 Architecture of a modern GPU

1.1.1 Streaming multiprocessors and SIMT execution

A modern GPU is composed of an array of streaming multiprocessors (SMs) in NVIDIA terminology or compute units (CUs) in AMD terminology. Each SM contains multiple scalar processors, or CUDA cores, that execute groups of 32 threads called warps (NVIDIA) or wavefronts of 64 threads (AMD). The single‑instruction, multiple‑thread (SIMT) model allows all threads in a warp to execute the same instruction on different data, though individual threads can diverge at branches, reducing efficiency.

1.1.2 Memory hierarchy (global, shared, local, registers)

The GPU memory hierarchy consists of several levels with different capacities and latencies. Global memory, off‑chip DRAM, is large (tens of gigabytes) but has high latency. Shared memory is a small, on‑chip scratchpad per SM (typically tens to hundreds of kilobytes), accessible by all threads in a thread block with low latency. Local memory is per‑thread storage used for register spilling, while registers themselves are the fastest, per‑thread storage. Additionally, texture and constant memories provide read‑only cached access for specific patterns.

1.2 Parallelism concepts

1.2.1 Data parallelism vs. task parallelism

Data parallelism applies the same operation to many data elements simultaneously, the natural fit for GPU architectures. Task parallelism, where different operations are performed on different data sets concurrently, can be implemented on GPUs but is often less efficient because GPU cores are optimized for uniform work. Heterogeneous computing typically uses the CPU for task‑parallel control flow and the GPU for data‑parallel kernels.

1.2.2 Amdahl's and Gustafson's laws in GPU context

Amdahl’s law states that the speedup of a program is limited by its sequential portion: speedup ≤ 1 / (1 − P), where P is the parallelizable fraction. For GPU acceleration, even a small serial part can dominate if the parallel part is greatly accelerated. Gustafson’s law offers a more optimistic perspective by scaling the problem size: speedup = P × N + (1 − P), where N is the number of processors. In practice, GPU computing often targets large problems where the parallel fraction dominates, mitigating Amdahl’s bottleneck.

1.3 CPU‑GPU interaction model

1.3.1 Host‑device data transfers

The CPU acts as the host and the GPU as the device. Data must be transferred between host memory and device memory across the PCI Express bus (or NVLink for faster connections). These transfers have high latency (microseconds to milliseconds) and limited bandwidth relative to internal GPU memory. Overlapping data transfers with computation is a key optimization.

1.3.2 Asynchronous execution and streams

CUDA streams and OpenCL command queues allow non‑blocking kernel launches and data transfers. By queuing operations in separate streams, developers can overlap kernel execution with data transfers or run multiple kernels concurrently on the same GPU. Streams also enable out‑of‑order execution and event‑based synchronization.

2 Programming Models and Frameworks

2.1 CUDA (NVIDIA)

2.1.1 Kernel launches and thread hierarchy

CUDA extends C/C++ with syntax for defining device functions (kernels). The programmer specifies a grid of thread blocks; each block contains up to 1024 threads (compute capability dependent). Threads within a block cooperate via shared memory and barrier synchronization. The hardware maps blocks to SMs and schedules warps.

2.1.2 CUDA Runtime and Driver APIs

The CUDA Runtime API provides high‑level functions for memory management, kernel launching, and stream creation. The lower‑level Driver API offers more control over context management and module loading, suitable for libraries and frameworks that need fine‑grained resource handling.

2.2 OpenCL

2.2.1 Platform model and kernel compilation

OpenCL provides a standardized, cross‑vendor framework for heterogeneous computing. Its platform model includes a host (CPU) and one or more devices (GPUs, CPUs, accelerators). Kernels are written in a subset of C99 (OpenCL C) and compiled at runtime for the target device using the OpenCL compiler.

2.2.2 Memory objects and synchronization

OpenCL memory objects (buffers and images) reside in device memory and are transferred via command queues. Synchronization is achieved through events, barriers, and clFinish. OpenCL 2.0 introduced shared virtual memory and dynamic parallelism, though adoption varies by vendor.

2.3 ROCm (AMD)

2.3.1 HIP as a portable CUDA alternative

AMD’s ROCm stack includes HIP (Heterogeneous‑Interface for Portability), a C++ runtime API that closely mirrors CUDA. Developers can write HIP code that compiles to both NVIDIA GPUs (via a CUDA backend) and AMD GPUs (via ROCm). HIP provides hipMalloc, hipLaunchKernelGGL, and hipStreamSynchronize, making CUDA‑style programming portable.

2.3.2 Rocclr and the ROCm runtime

The ROCm runtime (rocclr) manages device initialization, memory allocation, and kernel dispatch for AMD GPUs. It sits above the ROCk kernel driver and provides a common interface for higher‑level libraries (e.g., rocBLAS, rocFFT). ROCm also includes ROCgdb for debugging and ROCProfiler for performance analysis.

2.4 Directive‑based approaches

2.4.1 OpenACC

OpenACC uses compiler directives (pragmas) to annotate loops and code regions for offloading to GPUs. It is designed for incremental parallelization of existing Fortran, C, and C++ code. Developers specify data regions and parallel constructs, and the compiler generates GPU kernels. OpenACC is popular in scientific computing (e.g., climate modeling) where quick porting is desired.

2.4.2 OpenMP target offloading

OpenMP 4.0 and later introduced target offloading directives (e.g., #pragma omp target teams distribute) for GPUs. It allows programmers to use the same directive‑based paradigm familiar from shared‑memory parallelism to offload compute to accelerators. Implementation maturity varies by vendor, but recent versions support advanced features like nested parallelism and device memory management.

3 Key Application Domains

3.1 Scientific computing

3.1.1 Molecular dynamics (e.g., GROMACS, NAMD)

Molecular dynamics (MD) simulations benefit from GPU acceleration due to the large number of pairwise force calculations. GROMACS, NAMD, and AMBER have been highly optimized for GPUs, achieving speedups of 10–50× on single‑GPU nodes. Short‑range non‑bonded interactions are especially amenable to parallelization on GPU cores.

3.1.2 Computational fluid dynamics (CFD)

CFD solvers use GPU acceleration for grid‑based (finite difference/element/volume) methods. Libraries like OpenFOAM have GPU backends that solve pressure Poisson equations and compute convective fluxes in parallel. Turbulence modeling and immersed boundary methods also exploit GPU parallelism to handle millions of cells.

3.2 Machine learning and deep learning

3.2.1 TensorFlow and PyTorch

Deep learning frameworks rely on GPU acceleration for training deep neural networks. TensorFlow and PyTorch automatically offload tensor operations (matrix multiplies, convolutions) to GPUs using cuDNN, cuBLAS, and MIOpen (AMD). Automatic differentiation runs on the GPU, and data loaders prefetch inputs to keep the GPU busy.

3.2.2 Training vs. inference acceleration

Training requires both forward and backward passes, often using mixed‑precision and tensor cores to speed up matrix operations. Inference benefits from GPU acceleration for high‑throughput serving, but also from model quantization and pruning. Frameworks like TensorRT and ONNX Runtime optimize inference graphs for GPU execution.

3.3 Data analytics and databases

3.3.1 GPU‑accelerated SQL (e.g., BlazingSQL)

GPU‑accelerated SQL engines leverage GPU parallelism to process large datasets in memory. BlazingSQL, built on RAPIDS cuDF, executes SQL queries on columnar data with GPU‑backed joins, aggregations, and filters. Queries can run orders of magnitude faster than CPU‑only databases for analytics workloads.

3.3.2 Graph analytics with cuGraph

cuGraph, part of the RAPIDS suite, provides GPU‑accelerated graph algorithms (BFS, PageRank, connected components, Louvain clustering). By storing graph adjacency lists in GPU memory and processing vertices/edges in parallel, cuGraph handles graphs with billions of edges on a single GPU.

3.4 Computer graphics and visualization

3.4.1 Real‑time ray tracing (RT cores)

Modern GPUs include dedicated ray‑tracing cores (RT cores in NVIDIA, Ray Accelerators in AMD) that accelerate intersection tests for real‑time ray tracing. These cores are programmable and work with shading cores to compute lighting, reflections, and shadows in games and visualization applications.

3.4.2 Scientific visualization (e.g., ParaView)

ParaView uses GPU acceleration for volume rendering, isosurface extraction, and streamline generation. With large‑scale simulation data (terabytes), distributed GPUs in supercomputers enable interactive visualization. In‑situ visualization tools (e.g., VTK‑m) also leverage GPU parallelism directly within simulation runs.

4 Performance Optimization Techniques

4.1 Memory access patterns

4.1.1 Coalesced vs. uncoalesced accesses

Global memory access is most efficient when threads in a warp access consecutive memory addresses, forming a single coalesced transaction (32 bytes or more). Uncoalesced accesses (strided or random) cause multiple transactions, reducing effective bandwidth. Restructuring data into structures of arrays (SoA) rather than arrays of structures (AoS) improves coalescing.

4.1.2 Shared memory tiling and bank conflicts

Shared memory can be used to cache data from global memory for reuse within a thread block (tiling). However, shared memory is divided into banks (typically 32 of 4 bytes each). When multiple threads access the same bank simultaneously, bank conflicts serialize accesses. Padding or rearranging data (e.g., using __ldg intrinsics) can avoid conflicts.

4.2 Occupancy and warp scheduling

4.2.1 Maximizing active warps per SM

Occupancy is the ratio of active warps to the maximum supported per SM. Higher occupancy can hide memory latency by allowing the scheduler to switch to another warp while a warp waits. However, excessive shared memory or register usage may limit occupancy. The optimal occupancy depends on kernel characteristics; sometimes lower occupancy with less per‑warp resource use yields better performance.

4.2.2 Occupancy calculators and tuning

NVIDIA provides an occupancy calculator (spreadsheet and within Nsight Compute) that estimates occupancy given kernel resource usage. Tuning involves adjusting block size, shared memory allocation, and register usage (via the __launch_bounds__ qualifier). AMD’s ROCProfiler and Omnitrace provide similar occupancy analysis.

4.3 Profiling and debugging tools

4.3.1 NVIDIA Nsight Systems and Compute

Nsight Systems provides a system‑level timeline view of CPU and GPU activity (kernel launches, data transfers, API calls). Nsight Compute is a kernel profiler that reports metrics such as achieved occupancy, memory bandwidth utilization, and instruction mix, enabling per‑kernel optimization.

4.3.2 AMD ROCProfiler and Omnitrace

ROCProfiler is a low‑overhead profiling tool for AMD GPUs, providing counters for kernel execution, memory activity, and hardware events. Omnitrace (formerly known as Omnitrace) is a newer performance analysis framework that traces OpenMP, HIP, and MPI calls, offering timeline views and hardware counter collection for ROCm applications.

5.1 Generational evolution of GPU architectures

5.1.1 NVIDIA Fermi to Hopper and Blackwell

Fermi (2010) introduced CUDA 2.0 with ECC memory and 512 cores. Kepler (2012) improved power efficiency and dynamic parallelism. Maxwell (2014) focused on performance per watt. Pascal (2016) introduced NVLink and unified memory. Volta (2017) added tensor cores and independent thread scheduling. Turing (2018) included RT cores. Ampere (2020) doubled tensor core performance and added third‑generation RT cores. Hopper (2022) introduced Transformer Engine and DPX instructions. Blackwell (2024) further scales with fifth‑generation tensor cores and die‑interconnect technology.

5.1.2 AMD GCN to RDNA and CDNA

AMD’s Graphics Core Next (GCN) architecture (2011) was used for both graphics and compute. RDNA (2019) split the focus: RDNA for gaming, CDNA for compute. CDNA (2020) introduced Matrix Cores (similar to Tensor Cores) and Infinity Fabric for multi‑GPU communication. CDNA 3 (2023) added advanced packaging and chiplets, while RDNA 3 (2022) uses chiplet design for graphics.

5.2 Specialized acceleration units

5.2.1 Tensor Cores (NVIDIA) and Matrix Cores (AMD)

Tensor Cores (NVIDIA) and Matrix Cores (AMD) are dedicated matrix‑multiply‑accumulate units that deliver high throughput for deep learning workloads, supporting mixed‑precision (FP16, BF16, TF32, INT8). Tensor Cores in Hopper and Blackwell also accelerate sparse matrix operations (2:4 structured sparsity) and transformer‑specific computations.

5.2.2 Ray tracing cores

RT cores in NVIDIA GPUs and Ray Accelerators in AMD GPUs accelerate bounding volume hierarchy (BVH) traversal and ray‑triangle intersection. These cores are programmable via ray‑tracing shaders and are essential for real‑time rendering in games and professional visualization.

5.3 Integration with other accelerators

5.3.1 GPU‑FPGA hybrid computing

In hybrid systems, FPGAs handle low‑latency, custom‑pipeline tasks (e.g., network packet processing), while GPUs perform bulk parallel computation. Intel’s oneAPI and Xilinx’s Vitis support unified programming models that target both FPGA and GPU devices, though tight integration remains a research area.

5.3.2 GPU‑DPU (data processing unit) co‑design

Data processing units (DPUs) offload networking, storage, and security functions from CPUs. In GPU‑accelerated servers, DPUs handle data movement between storage and GPU memory, reducing CPU involvement and improving end‑to‑end performance. NVIDIA’s BlueField DPU is an example, integrating with CUDA to enable GPUDirect Storage.

5.4 Cloud and edge GPU computing

5.4.1 GPU instances in AWS, Azure, GCP

Cloud providers offer GPU‑accelerated instances (e.g., AWS P4d, Azure NDv4, GCP A100) for training, inference, and HPC. These instances include multi‑GPU configurations with NVLink/Infinity Fabric. Serverless GPU options (e.g., AWS Inferentia, GCP TPU) provide lower‑cost inference, while NVIDIA GPUs dominate general‑purpose acceleration.

5.4.2 Lightweight GPUs for embedded systems (Jetson, iGPU)

NVIDIA Jetson series (e.g., Xavier, Orin, AGX) integrates GPU, CPU, and memory on a module, targeting robotics, autonomous vehicles, and edge AI. Integrated GPUs (iGPUs) in AMD Ryzen and Intel Core processors provide moderate acceleration for laptop/desktop workloads without a discrete GPU. These systems balance power efficiency with compute capability.