Overfitting and Underfitting — Explained Simply

What Overfitting and Underfitting Actually Are

You Are Preparing For a Job Interview

You have a study guide with 200 possible questions. Two friends studied the same guide and both failed — for completely opposite reasons.

Friend 1 — Did Not Study Enough Glanced at the guide for 30 minutes. Got every question roughly wrong — both in practice and in the real interview. This is underfitting.

Friend 2 — Memorised Instead of Understanding Memorised every practice question word for word. Got 100% on practice. Then the interviewer asked two slightly different questions — froze completely. Failed the real interview. This is overfitting.

The goal: Genuinely understand the concepts. Good on practice AND on the real interview. This is a well-fitted model.

In ML Terms

# Underfitting
train_loss = 0.85   # High — bad on training data
val_loss   = 0.87   # Also high — bad on new data too

# Overfitting
train_loss = 0.02   # Very low — perfect on training data
val_loss   = 0.91   # Very high — terrible on new data

# Well-fitted
train_loss = 0.12   # Low
val_loss   = 0.15   # Also low — small gap ✅

Why Overfitting Happens

Too much model capacity relative to the amount of data.

1000 training samples + model with 10 million parameters
→ Model just memorises every training example
→ 100% training accuracy
→ Useless on new data

Why Underfitting Happens

Model too simple, trained too little, or wrong learning rate.

1000 training samples + model with 10 parameters
→ Not enough capacity to learn the pattern
→ 55% training accuracy — barely above random
→ Also 55% on new data — consistently bad

The Five Ways to Fix Overfitting

Fix 1 — More Data

# Collect more Nigerian Pidgin reviews
# 500 samples → collect 2000
# More data almost always reduces overfitting

Fix 2 — Simpler Model

# Overfitting
model = nn.Sequential(
    nn.Linear(input_size, 1024), nn.ReLU(),
    nn.Linear(1024, 512), nn.ReLU(),
    nn.Linear(512, 3)
)

# Simpler
model = nn.Sequential(
    nn.Linear(input_size, 64), nn.ReLU(),
    nn.Linear(64, 3)
)

Fix 3 — Dropout

model = nn.Sequential(
    nn.Linear(input_size, 256),
    nn.ReLU(),
    nn.Dropout(0.3),        # Randomly zeros 30% of neurons
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(0.3),
    nn.Linear(128, 3)
)

Fix 4 — Regularisation (Weight Decay)

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=0.001,
    weight_decay=0.01    # L2 regularisation — penalises large weights
)

Fix 5 — Early Stopping

best_val_loss = float('inf')
patience, no_improve = 5, 0

for epoch in range(100):
    train_loss = train_epoch(model, train_loader)
    val_loss   = evaluate(model, val_loader)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        torch.save(model.state_dict(), 'best_model.pth')
        no_improve = 0
    else:
        no_improve += 1
        if no_improve >= patience:
            print(f"Early stopping at epoch {epoch+1}")
            break

The Two Ways to Fix Underfitting

Fix 1 — Bigger Model

model = nn.Sequential(
    nn.Linear(input_size, 256), nn.ReLU(),
    nn.Linear(256, 128), nn.ReLU(),
    nn.Linear(128, 3)
)

Fix 2 — Train Longer or Better Learning Rate

for epoch in range(200):    # More epochs
    ...

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

Diagnosing From the Loss Curve

Both losses high + similar    → Underfitting → bigger model, train longer
Train low, val high           → Overfitting  → dropout, regularisation, more data
Both losses low + similar     → Well-fitted  → done ✅
Val loss improving then rises → Overfitting starting → early stopping

The Bias-Variance Tradeoff

Underfitting = High Bias
Model too rigid — wrong assumptions about the data

Overfitting = High Variance
Model too sensitive to specific training examples

Goal: Low bias AND low variance
Simple enough to generalise + complex enough to learn real patterns

The Real Words Mapped to the Story

The One Thing to Remember

Overfitting is memorising. Underfitting is not learning enough. The goal is genuine understanding — good performance on training data AND on data the model has never seen. Always keep a separate validation set and watch both losses throughout training.