Regularisation — Explained Simply

What Regularisation Actually Is

You Are a Manager Reviewing Employee Expense Reports

Employees who submit large complicated expense reports with charges in every possible category are often gaming the system. Employees with simple focused reports tend to be honest.

So you create a rule:

Any report that is unnecessarily complex gets an automatic penalty. Keep it simple or face consequences.

That penalty for unnecessary complexity is regularisation.

The Expense Report Is Your Model's Weights

Without constraints, weights grow as large as needed to minimise training loss — even if that means memorising examples rather than learning patterns.

Regularisation adds a penalty for large weights. It forces weights to stay small and focused — which forces the model to learn general patterns rather than memorise specific examples.

The Two Types

L2 Regularisation — The Gentle Penalty

Regularised loss = how wrong + λ × sum of (every weight²)

Large weights → heavy penalty
Small weights → barely penalised
Result: all weights stay small but none go to zero

# Built into AdamW — one line
optimizer = torch.optim.AdamW(
    model.parameters(),
    lr           = 0.001,
    weight_decay = 0.01    # ← this IS L2 regularisation
)

L1 Regularisation — The Ruthless Penalty

Regularised loss = how wrong + λ × sum of |every weight|

Result: most weights pushed to exactly zero
Only the truly important connections survive (sparsity)

# Manual in PyTorch
l1_lambda  = 0.001
l1_penalty = sum(param.abs().sum() for param in model.parameters())
loss       = base_loss + l1_lambda * l1_penalty
loss.backward()

L1 vs L2 — Which to Use

L2 (weight_decay in AdamW):
  → Keeps all weights but makes them small
  → Use this by default ✅
  → Fine-tuning LLMs, standard neural nets

L1:
  → Pushes many weights to exactly zero
  → Good for feature selection
  → Less common in deep learning

Lambda — How Hard You Penalise

λ too low   → penalty barely matters → weights still grow → overfitting
λ too high  → weights crushed to zero → model stops learning → underfitting
λ just right → weights small and meaningful → generalises well ✅

weight_decay = 0.01     # Good starting point
weight_decay = 0.1      # If still overfitting → increase
weight_decay = 0.001    # If model not learning → decrease

The Visual Difference

Without regularisation:
  Weights: 0.002, 15.3, -8.7, 0.001, 23.1, -12.4
           ↑ tiny       ↑ huge       ↑ huge — memorising

With L2 regularisation:
  Weights: 0.002, 0.87, -0.43, 0.001, 1.12, -0.67
           All small and proportional — generalising ✅

With L1 regularisation:
  Weights: 0.000, 0.87, 0.000, 0.000, 1.12, 0.000
           Most eliminated — only important ones survive

Regularisation vs Dropout

Regularisation (L1/L2):
  → Penalises weights for being large
  → Applied to the weights themselves
  → Active during BOTH training and evaluation

Dropout:
  → Randomly switches off neurons
  → Applied to activations
  → Only active during TRAINING

Same problem — overfitting — completely different approaches. Most models use both.

Complete Training Loop With Regularisation

model = nn.Sequential(
    nn.Linear(input_size, 256), nn.ReLU(),
    nn.Linear(256, 128), nn.ReLU(),
    nn.Linear(128, 3)
)

# L2 regularisation via weight_decay — no extra code needed
optimizer = torch.optim.AdamW(
    model.parameters(),
    lr           = 0.001,
    weight_decay = 0.01
)

for epoch in range(50):
    model.train()
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(X_batch), y_batch)
        loss.backward()
        optimizer.step()

    val_loss, val_f1 = evaluate(model, val_loader)
    print(f"Epoch {epoch+1} | Val Loss: {val_loss:.4f} | F1: {val_f1:.4f}")

How to Know If It Is Working

Before regularisation:
  train_loss = 0.05   val_loss = 0.89   ← overfitting badly

After weight_decay=0.01:
  train_loss = 0.18   val_loss = 0.22   ← well fitted ✅

Val loss still high → increase weight_decay to 0.1
Train loss also high → decrease weight_decay to 0.001

The Real Words Mapped to the Story

The One Thing to Remember

Regularisation adds a penalty for large weights. This forces the model to keep its weights small and general rather than large and specific. In PyTorch you get this for free by using AdamW with weight_decay. One line. Always use it.