Learning Rate — Explained Simply

What a Learning Rate Actually Is

You Are Trying to Find the Perfect Temperature for Your Shower

You step into the shower. The water is freezing cold. You need to adjust the tap to find the perfect temperature — not too hot, not too cold, just right.

Person A — Giant Turns Feels cold. Turns the tap all the way to hot. Now it is burning. Turns it all the way back to cold. Now freezing again. They keep swinging back and forth, never landing on the right temperature.

Person B — Tiny Adjustments Feels cold. Turns the tap just a little toward hot. Still a bit cold. Turns it a tiny bit more. Getting closer. One more tiny turn. Perfect.

The learning rate is how much you turn the tap each time.

What Happens With Each Learning Rate

Too High — The Bouncy Ball

Loss
 │
 │  *               *
 │      *       *
 │          *
 │                      ← never settles
 └──────────────────→ training steps

The model keeps bouncing over the minimum
Never converges — or gets worse over time

Too Low — The Glacier

Loss
 │ *
 │  *
 │   *
 │    *
 │     *
 │      *
 │       *  ← still going, very slowly
 └──────────────────→ training steps

The model is learning but painfully slowly

Just Right — The Sweet Spot

Loss
 │ *
 │  **
 │    ***
 │       ****
 │           *******
 │                  **********─── levels out
 └──────────────────→ training steps

The Numbers in Practice

# Too high — loss will explode or bounce
optimizer = torch.optim.Adam(model.parameters(), lr=1.0)

# Too low — will train but very slowly
optimizer = torch.optim.Adam(model.parameters(), lr=0.000001)

# Good starting point for Adam — works for most problems
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Standard for fine-tuning LLMs
optimizer = torch.optim.AdamW(model.parameters(), lr=0.00002)

Learning Rate Schedules — Changing Over Time

At the beginning you need bigger steps to make fast progress. Near the end you need tiny steps to dial in precise weights.

Learning Rate
    │ ████
    │     ████
    │         ████
    │             ████
    │                 ████
    └──────────────────────→ training steps
           Decaying over time

The Warmup — Used in Every LLM Training Run

At the very start of training, weights are random and gradients are unstable. Starting with the full learning rate can push weights in completely wrong directions.

The fix: start tiny and gradually increase to the full learning rate over the first few hundred steps.

Learning Rate
    │                    ████████████████── then decays
    │                 ███
    │              ███
    │           ███
    │        ███
    │     ███
    │  ███
    └──────────────────────→ training steps
      warmup  → full LR → decay

from torch.optim.lr_scheduler import OneCycleLR

scheduler = OneCycleLR(
    optimizer,
    max_lr          = 0.001,
    steps_per_epoch = len(train_loader),
    epochs          = 50,
    pct_start       = 0.3    # 30% of training = warmup
)

# In training loop
optimizer.step()
scheduler.step()

Diagnosing Learning Rate Problems From the Loss Curve

Symptom                        Diagnosis              Fix
───────────────────────────────────────────────────────────────
Loss goes up immediately       LR too high            Divide by 10
Loss bounces up and down       LR too high            Divide by 10
Loss barely moves              LR too low             Multiply by 10
Loss goes down then explodes   LR slightly too high   Divide by 3
Loss goes down smoothly        LR is good             Leave it
Loss goes down then plateaus   LR decay needed        Add scheduler

The Real Words Mapped to the Story

The One Thing to Remember

The learning rate decides how fast your model walks down the mountain. Too fast and it keeps tripping over the valley. Too slow and it takes forever to get there. Start with 0.001, watch the loss curve, and adjust from there.