MNIST dataset

1.1 NIST datasets (Special Database 1 and 3)

The original data used to create MNIST came from two datasets produced by the National Institute of Standards and Technology (NIST). Special Database 1 contained handwritten digits collected from high school students, while Special Database 3 contained digits from Census Bureau employees. These sets differed in writing style and image quality, providing a natural diversity for training and evaluation.

1.2 Creation of MNIST by LeCun et al.

Yann LeCun, Corinna Cortes, and Christopher J. C. Burges extracted a subset of NIST images and pre‑processed them to create a more uniform dataset. They applied size normalization and centering, producing 28×28 grayscale images. The training set was drawn from NIST’s Special Database 3 and additional samples from Special Database 1, while the test set came entirely from Special Database 1. This careful mixing helped balance writer variability between training and testing.

1.3 Standardization and naming convention

The dataset was named “Modified NIST” (MNIST) to distinguish it from the raw NIST collections. The format and splits were fixed in the late 1990s, and the dataset was released online for public research. Over time, MNIST became a standard benchmark in machine learning, often used as a first test for new classification algorithms.

2.1 Image specifications

2.1.1 Resolution and pixel values

Each image is 28 pixels wide and 28 pixels high, totaling 784 pixels per sample. Pixel values range from 0 (black) to 255 (white), with intermediate shades of gray. The images are centered by bounding box and anti‑aliased, though some edge pixels may remain fully white.

2.1.2 Class distribution

The dataset contains ten roughly balanced classes representing digits 0 through 9. The training set has about 6,000 examples per digit, and the test set about 1,000 per digit. Minor variations exist due to the original NIST sampling, but no class is severely underrepresented.

2.2 Official splits

2.2.1 Training set

The training set consists of 60,000 images. These are used to fit model parameters. Common practice is to further split this into a training subset and a validation subset (e.g., 50,000 training, 10,000 validation) when tuning hyperparameters, although the original distribution does not prescribe a validation split.

2.2.2 Test set

The test set contains 10,000 images, never to be used during training. Its purpose is to provide an unbiased estimate of final model performance. The test set labels are withheld during model development and are only revealed for final evaluation.

2.3 File format and storage

2.3.1 IDX file format

The images and labels are stored in the IDX file format, designed for multi‑dimensional arrays. The image file (train‑images‑idx3‑ubyte) uses a 4‑byte magic number, then four dimensions: number of images (60,000), number of rows (28), number of columns (28). The label file (train‑labels‑idx1‑ubyte) has a 2‑byte magic number, then one dimension: number of labels. All values are stored as unsigned bytes.

2.3.2 Metadata and label files

Labels are stored separately from images. Each label is a single byte representing the digit value (0–9). The file for the training set contains 60,000 such bytes; the test set contains 10,000. No additional metadata (e.g., writer identity or original NIST source) is included in the standard distribution.

3.1 Pre‑processing techniques

3.1.1 Normalization and centering

Pixel values are often normalized to a [0,1] range or centered around zero (e.g., subtracting the mean). Some practitioners use global normalization based on the whole dataset, while others apply per‑image standardization. These steps help gradient‑based optimizers converge faster and improve stability.

3.1.2 Data augmentation

To combat overfitting, especially on small models, common augmentations include slight rotation (up to ±15°), translation, scaling, and elastic distortion. Augmentation generates additional training samples without collecting new data. For MNIST, affine transformations are particularly effective because the original images are already size‑normalized and centered.

3.2 Common models applied

3.2.1 Linear classifiers

3.2.1.1 Logistic regression

Multinomial logistic regression treats MNIST as a 10‑class softmax problem. It learns a linear decision boundary per pixel, achieving around 92–93% test accuracy. Despite its simplicity, it serves as a baseline for measuring the advantage of non‑linear models.

3.2.1.2 Support vector machines

Support vector machines (SVMs) with a radial basis function (RBF) kernel can reach about 98–99% accuracy on MNIST. The RBF kernel allows non‑linear separation. However, training an SVM on 60,000 samples is computationally expensive, and performance is sensitive to hyperparameter tuning.

3.2.2 Neural networks

3.2.2.1 Multilayer perceptrons

A multilayer perceptron (MLP) with one or two hidden layers typically achieves 98–99% accuracy. Common architectures use 300–1000 hidden units and ReLU activations. Dropout and batch normalization help prevent overfitting. MLPs represent a step up from linear models, capturing simple non‑linearities in digit shapes.

3.2.2.2 Convolutional neural networks

Convolutional neural networks (CNNs) are the standard high‑performance method for MNIST. A typical design includes two or three convolutional layers with max‑pooling, followed by fully connected layers. State‑of‑the‑art CNNs achieve test errors below 0.2%. The spatial structure of digits makes convolutions especially effective.

3.2.3 Other approaches

3.2.3.1 K‑nearest neighbors

K‑nearest neighbors (KNN) with a distance metric (e.g., Euclidean or tangent distance) can achieve about 97–98% accuracy. The method is straightforward: classify a test image by the majority label of its k closest training images. However, inference is slow because it requires computing distances to all training examples.

3.2.3.2 Random forests

Random forests, an ensemble of decision trees, typically reach 93–97% accuracy on MNIST. While competitive with linear models, they are often outperformed by neural networks. Random forests are robust to overfitting and handle pixel‑wise features without extensive pre‑processing.

3.3 Benchmark results and leaderboards

3.3.1 Historical milestones

Early MNIST benchmarks were dominated by linear classifiers (∼92% in 1998) and boosted stumps. In the 2000s, SVMs raised accuracy to about 99%. The 2010s saw neural networks surpass human‑level performance; LeNet‑5, a CNN developed by Yann LeCun, achieved a 0.95% error rate. Later, deep ensembles and specialized architectures pushed errors below 0.25%.

3.3.2 Current state‑of‑the‑art

As of the late 2010s, the best published results for MNIST achieve less than 0.2% test error, often using very deep CNNs, residual connections, or data augmentation. Some methods report 0.17% or lower. However, many practitioners consider the dataset “solved” and focus on other benchmarks for comparing new techniques.

4.1 Simplicity and overfitting risks

MNIST is considered an easy classification problem. Modern models can achieve near‑perfect accuracy with relatively simple architectures. This simplicity means that methods which overfit on more challenging tasks might still perform well on MNIST, giving a false sense of generalization. Researchers caution against using MNIST as the sole measure of model quality.

4.2 Lack of intra‑class variation

The dataset contains only ten classes with relatively uniform styles (centered, grayscale, isolated digits). Real‑world handwriting includes variations in rotation, slant, noise, and background clutter. MNIST’s clean images do not reflect the full difficulty of handwriting recognition, limiting its representativeness.

4.3 Comparison with modern datasets

Contemporary datasets such as CIFAR‑10, ImageNet, or SVHN offer greater complexity (color images, diverse objects, natural backgrounds). MNIST’s small size (70k images) and low resolution (28×28) make it less suitable for benchmarking modern deep‑learning architectures. Many researchers now use MNIST primarily for debugging or educational purposes.

5.1 EMNIST (Extended MNIST)

EMNIST (Extended MNIST) expands MNIST to include handwritten letters and digits. It offers six splits, covering digits, uppercase, lowercase, and a merged set. The total number of images is over 800,000. EMNIST retains the same 28×28 format, making it a direct drop‑in replacement for more complex character recognition tasks.

5.2 Fashion‑MNIST

Fashion‑MNIST is a drop‑in replacement for MNIST but contains grayscale images of clothing items (e.g., T‑shirts, trousers, shoes) instead of digits. It has the same size (60k/10k) and resolution (28×28). Fashion‑MNIST is designed to be more challenging than the original digit task, since clothing categories exhibit greater intra‑class variation.

5.3 KMNIST (Kuzushiji‑MNIST)

KMNIST (Kuzushiji‑MNIST) provides handwritten Japanese cursive characters (kuzushiji) as 28×28 grayscale images. It retains the same dataset size and structure as MNIST. The dataset aims to preserve historical documents and introduces a non‑Latin script, offering a fresh challenge for machine learning models accustomed to Roman characters and digits.

The original MNIST dataset is available from Yann LeCun’s website and from the LISA lab at the University of Montreal. It is also included in many machine‑learning libraries (e.g., TensorFlow, PyTorch, scikit‑learn) for convenient loading. MNIST is released under a permissive license; it can be used for research, education, and commercial purposes without restriction. Variants such as Fashion‑MNIST and EMNIST are similarly available under open‑source licenses.