Gradient Descent — Explained Simply

What Gradient Descent Actually Is

You Are Lost on a Mountain at Night

It is completely dark. You cannot see anything. You are standing somewhere on a huge mountain and you need to get to the bottom — the valley.

You cannot see the full mountain. You cannot see where the bottom is. The only thing you can feel is the slope of the ground directly under your feet right now.

So you use the only strategy available to you:

Feel which direction is downhill under your feet. Take one small step in that direction. Repeat.

That is gradient descent. Nothing more.

The Mountain Is Your Loss

The weights in your neural network are the position on the mountain. The loss — how wrong the predictions are — is the height.

High on mountain  =  High loss  =  Bad predictions
Low in valley     =  Low loss   =  Good predictions

The network starts at a random position — random weights — and needs to find its way to the bottom — the weights that produce the lowest possible loss.

The Slope Is the Gradient

The gradient tells you two things:

• Which direction the loss is increasing
• How steeply it is increasing in that direction

Since you want to go downhill — reduce the loss — you move in the opposite direction of the gradient.

Gradient points uphill   →   You step downhill
Gradient says go right   →   You step left
Gradient says go forward →   You step backward

One Step at a Time

for epoch in range(10000):
    predictions = model(X)                  # Stand on mountain
    loss        = loss_fn(predictions, y)   # Measure your height

    optimizer.zero_grad()   # Clear last step's slope reading
    loss.backward()         # Feel the slope (compute gradient)
    optimizer.step()        # Take one step downhill

After thousands of steps you arrive at the bottom — the weights that produce the best predictions.

Why Small Steps and Not One Giant Leap

Reason 1 — You cannot see the full mountain You only know the slope at your current position. A huge leap might fly past the valley and land somewhere worse.

Reason 2 — The mountain has many valleys Taking small steps lets you carefully navigate toward a good valley rather than overshooting everything.

Too large a step:               Just right:

Loss                            Loss
  │ *                             │ *
  │   *   * ← jumped past        │   *
  │       *                      │     *
  └──────────→ weights           └──────────→ weights
    Diverges                       Converges well

The Learning Rate — How Big Each Step Is

Learning rate too HIGH  →  Steps too large  →  Keep overshooting the valley
Learning rate too LOW   →  Steps too small  →  Take forever to reach the bottom
Learning rate just right →  Steps are good  →  Steady progress to the valley

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
#                                                 ↑
#                                    0.001 = small steps
#                                    0.1   = larger steps
#                                    1.0   = often too large

Three Versions of Gradient Descent

Batch Gradient Descent

See ALL training examples → Compute gradient → Take one step
Pros: Very accurate        Cons: Extremely slow

Stochastic Gradient Descent (SGD)

See ONE training example → Compute gradient → Take one step
Pros: Very fast           Cons: Very noisy

Mini-Batch Gradient Descent — The Standard

See 32 examples (one batch) → Compute gradient → Take one step
Pros: Fast AND reasonably accurate

# Mini-batch handled automatically by DataLoader
loader = DataLoader(dataset, batch_size=32, shuffle=True)

for X_batch, y_batch in loader:
    optimizer.zero_grad()
    loss = loss_fn(model(X_batch), y_batch)
    loss.backward()
    optimizer.step()

What Can Go Wrong

Getting Stuck in a Local Minimum

Loss │
     │        *
     │      *   *
     │    *       *   *
     │  *           *   *         *
     │*               *   * * * *   * ← deeper valley here
     └────────────────────────────→ weights
           ↑
     Stuck here (local minimum)
     Never found the real bottom (global minimum)

Modern optimisers like Adam help escape these shallow valleys.

The Real Words Mapped to the Story

The One Thing to Remember

Gradient descent is a blindfolded person on a mountain who can only feel the slope under their feet. They take small steps downhill, over and over, until they reach the bottom. The gradient is the slope. The loss is the height. The weights are the position. Training is the walk.