In information technology, batch size refers to the number of data samples or transactions processed together in a single operation, rather than individually. It is a key parameter in machine learning (where it defines the number of training examples used in one forward/backward pass), database operations (e.g., batch inserts or updates), and distributed computing (e.g., batching messages for efficiency). The choice of batch size affects computational performance, memory usage, convergence behavior, and throughput.

1 Definition and background

1.1 Core concept

Batch size is the cardinality of a batch—the set of items processed as a unit. In computing, batching reduces overhead by amortizing setup costs (e.g., I/O calls, kernel launches) across multiple items. The concept appears in numerous domains: neural network training, database transactions, and message queuing.

1.2 Historical context

Early computing systems used batch processing for job scheduling, where batches of punch cards were processed sequentially. In machine learning, the term emerged with gradient-based optimization: the perceptron and early neural networks processed examples one at a time. The mini-batch approach became standard in the 1990s and 2000s as datasets grew and GPUs enabled parallel computation.

2 Batch size in machine learning

2.1 Role in gradient descent optimization

Batch size determines how many training examples contribute to each parameter update in gradient descent. The gradient of the loss function is averaged over the batch, and the average is used to adjust weights. A larger batch provides a more accurate estimate of the true gradient, while a smaller batch introduces noise that can help escape local minima.

2.2 Types of gradient descent by batch size

2.2.1 Stochastic gradient descent (batch size = 1)

SGD updates parameters after every single example. It is highly stochastic, converges quickly per update, but can oscillate. It is memory-efficient because only one example is held at a time, but throughput is low due to limited parallelism.

2.2.2 Mini-batch gradient descent (batch size > 1 and < full dataset)

The most common form, mini‑batch GD uses batches of size 32, 64, 128, or 256. It balances gradient accuracy with computational efficiency, leveraging vectorized hardware (e.g., GPUs) for parallel processing. The noise in the gradient helps generalization.

2.2.3 Full-batch gradient descent (batch size = dataset size)

Full‑batch GD computes the gradient over the entire training set. It produces a deterministic update, but is computationally expensive unless the dataset is small. It can converge to sharp minima and may overfit more easily.

2.3 Effects on training dynamics

2.3.1 Convergence speed and stability

Larger batches reduce variance in gradient estimates, leading to smoother convergence, but each update takes longer because more data must be processed. Smaller batches converge faster per iteration but may require more iterations overall. There is a trade‑off between per‑step computation and total steps.

2.3.2 Generalization gap

Empirically, very large batch sizes often lead to poorer generalization (the “generalization gap”). Researchers attribute this to the lack of noise in the gradient, which pushes the optimizer toward sharp minima. Techniques like learning rate scaling and batch normalization mitigate this effect.

2.3.3 Memory and hardware constraints

Batch size is limited by available memory (GPU, CPU RAM). Larger batches require storing more activations and gradients temporarily. For deep networks or large inputs (e.g., high‑resolution images), the maximum batch size may be small; techniques like gradient accumulation simulate larger batches without increasing memory.

2.4 Batch size in inference

2.4.1 Throughput optimization

During inference, batching multiple inputs together improves throughput because the hardware’s parallel computation units are more fully utilized. The optimal batch size often depends on the model architecture and memory bandwidth.

2.4.2 Real-time vs. batched inference

Real‑time applications (e.g., video processing, interactive voice) require low latency, so they typically use a batch size of 1 to avoid waiting for batched inputs. Offline or batch inference (e.g., large‑scale image classification) accumulates inputs over time to maximize throughput, even at the cost of higher latency.

3 Batch size in database and data processing

3.1 Transaction batching

3.1.1 Batch inserts, updates, and deletes

In relational databases, batching multiple SQL statements into one transaction reduces network round trips and disk I/O overhead. A batch size of a few hundred to a few thousand rows is common; too large a batch can cause locking contention or log file growth.

3.2 Data loading and ETL pipelines

Extract, Transform, Load (ETL) processes often batch rows before writing to a data warehouse. Batch size is tuned to balance memory usage and throughput. Larger batches improve compression and reduce index maintenance costs but may increase the risk of failure (all rows in a batch are rolled back if an error occurs).

3.3 Stream vs. batch processing paradigms

Stream processing (e.g., Apache Kafka, Flink) handles data one record at a time or in micro‑batches (a few milliseconds of data). Traditional batch processing (e.g., Hadoop MapReduce) processes large static batches. The choice depends on latency requirements and data volume.

4 Performance and scaling considerations

4.1 Hardware limitations (GPU/CPU memory, cache)

Memory constrains maximum batch size. For GPUs, the batch size must fit in VRAM; exceeding it causes out‑of‑memory errors. CPU cache size also influences performance: smaller batches that fit in L1/L2 cache can be processed faster than those causing cache misses.

4.2 Throughput and latency trade-offs

Increasing batch size typically increases throughput (samples per second) up to a saturation point, after which memory bandwidth becomes the bottleneck. For latency‑sensitive applications, a smaller batch size is preferred. In distributed systems, batching reduces communication overhead but introduces queuing delay.

4.3 Adaptive batch size techniques

Some frameworks dynamically adjust batch size during training or inference. Examples include “batch size scheduling” (starting small and growing) and “auto‑tuning” that searches for the largest batch that fits in memory while maintaining target throughput. Adaptive batching is also used in stream processing to handle variable input rates.

5 Practical guidelines and tuning

5.1 Rule-of-thumb strategies

In deep learning, common starting batch sizes are 32, 64, or 128. A heuristic is to use the largest batch size that fits in memory while still achieving good generalization. For databases, start with batches of 500–1000 rows and adjust based on response time.

5.2 Empirical tuning in deep learning frameworks

Libraries such as TensorFlow and PyTorch allow easy experimentation. Practitioners train for a few epochs with different batch sizes, monitor loss curves, and evaluate validation accuracy. Tools like learning rate finders help set the learning rate proportionally (see linear scaling rule).

5.3 Common pitfalls (overfitting, out-of-memory errors)

Using a batch size that is too large can lead to overfitting or sharp minima. Out‑of‑memory (OOM) errors occur when batch size exceeds available memory; solutions include gradient accumulation, reducing input size, or using mixed precision. In databases, too large a batch can cause transaction timeouts or deadlocks.

6.1 Epoch vs. iteration vs. batch

An epoch is one complete pass over the entire dataset. An iteration (or step) is one forward and backward pass using one batch. The number of iterations per epoch equals the dataset size divided by the batch size.

6.2 Learning rate scaling (linear scaling rule)

When the batch size is multiplied by k, the learning rate should also be multiplied by k to keep the expected update magnitude constant. This rule helps maintain convergence quality when scaling to large batch sizes in distributed training.

6.3 Batch normalization

Batch normalization reduces internal covariate shift by normalizing activations within each mini‑batch. Its behavior depends on batch size: small batches introduce high variance in the estimated mean and variance, hurting performance. Alternatives include layer normalization and group normalization, which are less batch‑size sensitive.