Batch Normalisation — Explained Simply

What Batch Normalisation Actually Is

You Are Running a Restaurant Kitchen

Five chefs — each responsible for one course. Each works at a completely different pace and scale. The starter chef produces tiny delicate portions. The soup chef produces huge vats. The side dish chef is erratic — sometimes tiny, sometimes enormous.

The person assembling the final dish receives wildly different quantities in wildly different formats. The result is inconsistent and unreliable every time.

This is what happens between layers in a neural network without batch normalisation.

The Solution — A Standardisation Station

Install a station between each chef that does two things:

Step 1 — Normalise: Scale everything to a standard reference point regardless of how it arrived. Step 2 — Adjust: Apply a small learned correction so the food still tastes right for this kitchen specifically.

Now every station receives food in a consistent format. The kitchen runs dramatically faster.

That station is batch normalisation.

What Is Actually Being Standardised

As training progresses the distribution of activations between layers shifts constantly. Each layer has to constantly readjust to the changing outputs of the previous layer.

This is called internal covariate shift. Training becomes slow and unstable.

Batch normalisation fixes this by standardising activations between layers — so each layer always receives inputs in a consistent distribution.

The Two Steps

Step 1 — Normalise to Mean 0 and Std 1

Batch activations: [2.3, 8.7, 1.1, 15.2, 0.4, 9.8]

Mean = 6.25
Std  = 4.81

Normalised = [-0.82, 0.51, -1.07, 1.86, -1.22, 0.74]

Every activation now has: Mean = 0, Std = 1

Step 2 — Scale and Shift With Learned Parameters

Final output = γ × normalised + β

γ (gamma) and β (beta) are learned during training
If γ=1 and β=0 → stays normalised
If γ=2 and β=1 → network learned it needs a different scale

In Code

model = nn.Sequential(
    nn.Linear(input_size, 256),
    nn.BatchNorm1d(256),        # Normalise after linear layer
    nn.ReLU(),                  # Then activation

    nn.Linear(256, 128),
    nn.BatchNorm1d(128),        # Same pattern
    nn.ReLU(),

    nn.Linear(128, 3)           # No batch norm on output
)

# BatchNorm1d → tabular data and text (1D features)
# BatchNorm2d → images (2D spatial features)
# LayerNorm   → transformers (normalise per sample not per batch)

The Three Benefits

1. Faster Training

Without batch norm:  Need lr = 0.0001 to stay stable → very slow
With batch norm:     Can use lr = 0.01 → much faster convergence

2. Less Sensitive to Weight Initialisation

Without batch norm:  Bad weight init → training diverges
With batch norm:     Bad weight init → batch norm corrects → training continues

3. Mild Regularisation

Each batch has slightly different statistics → model sees slightly different normalised values → acts as mild noise → reduces overfitting slightly.

Train vs Eval Mode

model.train()   # Batch norm uses CURRENT BATCH statistics
model.eval()    # Batch norm uses RUNNING AVERAGE statistics

# Critical: always call model.eval() before validation
# Forgetting this = unreliable validation scores

Batch Norm vs Layer Norm

Batch Normalisation:
  Normalises ACROSS the batch
  Used in: CNNs, feedforward networks
  Problem: fails with small batches or single samples

Layer Normalisation:
  Normalises ACROSS the features of ONE sample
  Used in: Transformers — BERT, GPT, LLaMA, Claude
  Works with any batch size including 1

nn.BatchNorm1d(256)    # For feedforward nets and CNNs
nn.LayerNorm(256)      # For transformers ← you will use this most

The Real Words Mapped to the Story

The One Thing to Remember

Batch normalisation standardises the outputs between layers so each layer always receives inputs in a consistent format. This makes training faster, more stable, and less sensitive to weight initialisation. In transformers the same idea is called layer normalisation — it is in every modern LLM including GPT, BERT, LLaMA, and Claude.