Dropout — Explained Simply

What Dropout Actually Is

You Are a Football Coach

Your team of 11 players trains together every session. They get brilliant at playing with each other specifically — Player 7 always passes to Player 9, Player 3 always covers for Player 5.

Match day comes. Player 7 is injured. Player 9 falls apart — their entire game was built around Player 7. The team collapses.

Brilliant in training. Fragile in the real game.

This is what happens to a neural network without dropout.

The Solution — Randomly Bench Players Every Session

Before each training session, randomly send some players to the bench:

Session 1:  Players 1,2,3,4,5,6,8,9,10,11 train  (7 benched)
Session 2:  Players 1,2,4,5,7,8,9,10,11 train    (3,6 benched)
Session 3:  Players 2,3,5,6,7,8,9,10,11 train    (1,4 benched)

Now Player 9 cannot rely on Player 7 — sometimes Player 7 is not there. Every player becomes independently capable. When match day comes and Player 7 is injured — no problem. They have trained without Player 7 many times.

That is dropout.

How It Works in Code

Without dropout — all neurons always active:
Input → [N1, N2, N3, N4, N5, N6, N7, N8] → Output

With Dropout(p=0.3) — 30% randomly zeroed each step:
Step 1: Input → [N1, 0,  N3, N4, 0,  N6, N7, N8] → Output
Step 2: Input → [N1, N2, N3, 0,  N5, N6, 0,  N8] → Output
Step 3: Input → [0,  N2, N3, N4, N5, 0,  N7, N8] → Output

No neuron can rely on any specific other neuron. Each must become independently useful.

The Critical Rule — Dropout Off During Evaluation

model.train()    # Dropout ON — neurons randomly zeroed during training
model.eval()     # Dropout OFF — all neurons active during evaluation

Forgetting model.eval() before validation is one of the most common bugs in PyTorch. Your validation scores will be randomly degraded by active dropout — giving you unreliable results.

The p Parameter

p = 0.0  → No dropout
p = 0.2  → Light — for mild overfitting
p = 0.3  → Standard — good starting point ✅
p = 0.5  → Strong — for severe overfitting
p = 0.9  → Too aggressive — model learns nothing

Where to Put Dropout

model = nn.Sequential(
    nn.Linear(input_size, 256),
    nn.ReLU(),
    nn.Dropout(0.3),        # ← After activation, before next layer ✅

    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(0.3),        # ← Same pattern ✅

    nn.Linear(128, 3),
    # No dropout on output layer ✅
)

Rules:

• After activation functions in hidden layers ✅
• Never on the input layer ✅
• Never on the output layer ✅

Complete Example — Dropout + Regularisation Together

model = nn.Sequential(
    nn.Linear(input_size, 256),
    nn.ReLU(),
    nn.Dropout(0.3),

    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(0.3),

    nn.Linear(128, 3)
)

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr           = 0.001,
    weight_decay = 0.01     # L2 regularisation
)

for epoch in range(50):
    model.train()                           # ← Dropout ON
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(X_batch), y_batch)
        loss.backward()
        optimizer.step()

    model.eval()                            # ← Dropout OFF
    with torch.no_grad():
        val_loss = evaluate(model, val_loader)

Dropout vs Regularisation — Clear Difference

Regularisation (L2 / weight_decay):
  → Penalises weights for being too large
  → Active during training AND evaluation
  → Works by constraining weight magnitude

Dropout:
  → Randomly removes neurons during training
  → Active during TRAINING ONLY
  → Works by forcing redundancy and independence

Using both together is very common and usually better than either alone ✅

How to Know If It Is Helping

Without dropout:
  train_loss = 0.03   val_loss = 0.87   ← severe overfitting

After Dropout(0.3):
  train_loss = 0.21   val_loss = 0.24   ← well fitted ✅

Val loss still high → increase to Dropout(0.5)
Train loss also very high → too strong → reduce to Dropout(0.1)

The Real Words Mapped to the Story

The One Thing to Remember

Dropout randomly switches off neurons during training so each neuron is forced to become independently useful. During evaluation all neurons are active. Always call model.train() before training and model.eval() before evaluation — forgetting this is one of the most common bugs in PyTorch.