Ablation study on MNIST and CIFAR-10: 99.47% and 93.87% accuracy

Problem

MNIST and CIFAR-10 are the two classic image classification benchmarks. The challenge here was not just reaching a number, but knowing what produces it: start from a simple model, propose improvements and measure each one’s contribution separately.

MNIST (28 × 28 handwritten digits, 60,000 for training and 10,000 for test): beat 99.4% accuracy with no convolutional layers, using only multilayer perceptrons (MLPs).
CIFAR-10 (32 × 32 colour images in 10 classes, 50,000 and 10,000): beat 92%.

Four handwritten MNIST digits: 5, 0, 4 and 1, white on a black background. — MNIST

Twenty-four small CIFAR-10 images with their classes: ship, dog, deer, bird, cat, horse, frog, automobile and truck. — MNIST

My contribution

Solo project: I designed the architectures and data augmentation pipelines, implemented them in PyTorch, ran the 11 experiments (100 epochs each) and documented each study in a technical report.

Technical approach

MNIST: an MLP that behaves like a modern network

The baseline is a four-layer MLP (784 → 1024 → 1024 → 1024 → 10) with ReLU. On top of it I proposed two independent changes:

Architecture (model B): batch normalisation, 20% dropout and two residual blocks of 1024 units, like a ResNet’s but with linear layers.
Data augmentation: mild rotations (±10°, so that 6 and 9 are not confused), 10% translations and scaling, elastic deformations that mimic natural variation in handwriting, and Gaussian noise.

Diagram of the MNIST model: a linear layer from 784 to 1024 with normalisation, ReLU and dropout; two residual blocks with ReLU; and a final linear layer from 1024 to 10. — Model B for MNIST: an MLP with batch normalisation, dropout and two residual connections (5 M parameters).

CIFAR-10: three architectures, data augmentation and a scheduler

The baseline is a simple CNN with five convolutional blocks (32 → 512 channels) and no data augmentation. I compared three architectures implemented from scratch and two training improvements:

ResNet-18, WideResNet-32 (wider and shallower) and DenseNet-100: three dense blocks of 16 layers in which every layer receives the outputs of all the previous ones.
Data augmentation: random crops with 4 px padding, horizontal flips and random erasing, which masks parts of the image so the model cannot rely on a single detail.
Training: a cosine learning-rate scheduler and stronger weight decay (5 × 10⁻⁴).

Diagram of the DenseNet-100: initial convolution from 3 to 24 channels, three dense blocks separated by transition layers, normalisation, ReLU, pooling and a linear layer from 342 to 10. — DenseNet-100: three dense blocks joined by transition layers that reduce channels and resolution. Only 1.05 M parameters.

Diagram of ResNet-18: initial convolution and four residual layers of two blocks, from 64 to 512 channels. — ResNet-18

Diagram of WideResNet-32: initial convolution and three residual layers of five blocks, from 32 to 128 channels. — ResNet-18

The key piece of DenseNet is the dense layer: instead of adding the input like a ResNet, it concatenates it, so every layer reuses all previous features with very few new parameters.

# CIFAR/CIFAR.ipynb — dense layer (growth_rate = 12)
class DenseLayer(nn.Module):
    def __init__(self, in_channels, growth_rate):
        super().__init__()
        self.bn = nn.BatchNorm2d(in_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv = nn.Conv2d(in_channels, growth_rate,
                              kernel_size=3, padding=1, bias=False)

    def forward(self, x):
        out = self.conv(self.relu(self.bn(x)))
        # Concatenate instead of add: every layer sees all previous ones
        return torch.cat([x, out], dim=1)

Results

Baseline98.31%
Model B98.83%
Data augmentation99.40%
Model B + data augmentation99.47%

Scale from 98.0% to 99.5%

MNIST: test accuracy of the four combinations of model (baseline / B) and preprocessing (without / with data augmentation).

On MNIST, data augmentation matters more than architecture: on its own it already reaches the target (99.40%), while model B adds half a point. Together they reach 99.47%. That looks small, but test errors drop from 169 to 53 images: three times fewer.

Baseline (simple CNN)82.20%
ResNet-1885.45%
WideResNet-3286.45%
DenseNet-10089.40%
CNN + data augmentation87.82%
CNN + cosine scheduler82.53%
DenseNet-100 + data augmentation + scheduler93.87%

Scale from 80% to 95%

CIFAR-10: test accuracy of each configuration. The first three improvements change only the architecture; the next two, only the training of the baseline CNN.

On CIFAR-10, by contrast, architecture contributes the most: DenseNet-100 gains 7.2 points with 1.05 M parameters, fewer than the baseline CNN (1.84 M) and ten times fewer than ResNet-18 (11.2 M), which stops at 85.45%. Data augmentation adds 5.6 points to the baseline CNN; the scheduler alone, only 0.3. Everything together reaches 93.87%, almost two points above the target.

Evidence

Repository with the notebooks for the final model and the ablation on each dataset, including all training outputs.
Technical reports: MNIST and CIFAR-10.

Professional application

Improving a model through measurable steps instead of trial and error: knowing which component pays off before spending more compute or data.
Choosing efficient architectures when model size or inference cost matters: here, the best model is also one of the smallest.
Designing domain-specific data augmentation, which on its own gave the biggest jump on MNIST and more than five points on CIFAR-10.