Gradient Descent: How Large Language Models Learn to Generate Text from Random Guesses

An explanation of gradient descent—one of the central ideas in LLM training—for programmers without a machine-learning background.

This article discusses public, general principles of LLM training. It does not cover unpublished model architectures, training data, or training recipes.

Summary

When programmers first encounter LLM training, they often meet a string of abstract terms: parameters, loss functions, gradients, backpropagation, gradient descent, and optimizers.

Once those concepts are connected, the central training process of a language model can be compressed into one sentence:

Have the model predict the next token, measure how seriously it was wrong, then adjust its internal parameters slightly so that it is more likely to predict correctly next time.

Repeated many times, this process moves a model from nearly random predictions toward learning patterns in language, code, and the texts used for training. Public descriptions of language-model pretraining commonly describe it as predicting the next token from preceding tokens, while training continuously adjusts a model’s large collection of numerical parameters.¹²

The tool that answers “which direction should these parameters move next?” is the subject of this article: gradient descent.

Contents

• Do Not Make LLMs Seem More Mysterious Than They Are
• What Question Does an LLM Actually Solve During Training?
• The Model Produces a Probability Distribution, Not a Single Word
• The Loss Function: Scoring the Model’s Error
• What Exactly Is a Gradient?
• Gradient Descent: Move in a Direction That Reduces Error
• Calculate One Step with a Single Parameter
• Backpropagation and Gradient Descent Are Not the Same Thing
• A Minimal LLM Training Loop
• Why Training Uses Mini-batches
• The Learning Rate Determines How Far Each Step Goes
• Why Real Training Also Needs Optimizers
• Why Can Next-token Prediction Produce Complex Capabilities?
• Why Is Post-training Needed after Pretraining?
• Gradient Descent Does Not Mean the Model Knows It Was Wrong
• Gradient Descent Cannot See the Whole Mountain
• Put the Whole Training Process Together
• The Four Concepts to Remember

Do Not Make LLMs Seem More Mysterious Than They Are

From a programmer’s perspective, begin by viewing an LLM as a very complicated function:

a sequence of input tokens
        ↓
   large language model
        ↓
a probability distribution over the next token

In simplified mathematical form:

Here, $\theta$ represents all of the model’s trainable parameters.

For now, you can think of them as an enormous floating-point array:

parameters = [
    0.0182,
    -0.7314,
    1.2057,
    # ...
]

Those numbers are not hand-written rules such as:

if user_asks_about_france:
    return "Paris"

Language ability arises from the joint behavior of many parameters. Training mainly consists of repeatedly adjusting them. OpenAI’s public explanation describes weights or parameters as many internal numerical values that are adjusted in small ways according to relationships found in training data.¹

One number usually cannot be said to “store” a particular fact. But taken together, the numbers can produce complex behavior, much as a large program’s configuration values and coefficients work together.

What Question Does an LLM Actually Solve During Training?

Suppose a training document contains the sentence:

The capital of France is Paris.

The tokenizer first turns it into tokens. For illustration, assume:

[France, 's, capital, is, Paris, .]

Real tokenization can differ: a token can be a word, part of a word, one or more Chinese characters, punctuation, or another common text fragment.

Training turns the sentence into many next-token questions automatically:

See: France                  Predict: 's
See: France 's               Predict: capital
See: France 's capital       Predict: is
See: France 's capital is    Predict: Paris

Therefore pretraining data normally needs no manually written answer for every sentence. The later tokens in the text are themselves the answers for earlier tokens.

At implementation level, one token sequence is shifted by one position:

Input:  [France, 's, capital, is]
Target: ['s, capital, is, Paris]

One forward pass can train multiple positions at once. The GPT-4 Technical Report describes its pretraining objective as predicting the next token in a document.²

The Model Produces a Probability Distribution, Not a Single Word

When a model sees:

The capital of France is

it does not internally obtain one unique answer, “Paris.” It first assigns scores to many vocabulary candidates and converts them into a probability distribution. Here is an invented example:

Paris       10%
London       8%
Tokyo        5%
city         4%
Beijing      3%
other       70%

The training data says that the correct target is Paris, but the model assigned it only 10%. The training system now needs a single number that says how poor that prediction was. That number is the loss.

The Loss Function: Scoring the Model’s Error

Language-model training commonly uses cross-entropy loss. For a correct token at one position, it can first be understood as:

where $p_{\text{correct}}$ is the probability assigned to the correct answer.

If the correct answer is Paris, giving it 1% probability produces a large loss; giving it 80% produces a smaller loss. In short:

Lower probability for the correct token  → larger loss
Higher probability for the correct token → smaller loss

In actual training, a system normally sums or averages losses for many tokens in a batch. PyTorch’s optimization tutorial describes training as repeatedly adjusting parameters so that the error at each training step decreases.³

The goal can therefore be written as:

But a model has many parameters. Which ones should change, and should each become larger or smaller? The answer is the gradient.

What Exactly Is a Gradient?

Shrink a real model to a toy model with three parameters:

w₁ = 0.5
w₂ = -0.8
w₃ = 1.2

After computing a loss, ask three local questions:

If w₁ increases slightly, how does loss change?
If w₂ increases slightly, how does loss change?
If w₃ increases slightly, how does loss change?

Mathematically these rates of change are partial derivatives:

Suppose the result is:

∂L/∂w₁ = +0.7
∂L/∂w₂ = -0.2
∂L/∂w₃ = +0.05

Together those values are the gradient vector. In programmer-friendly language, a gradient is local change information for every parameter: near the current point, it tells how the loss tends to change when each parameter changes by a tiny amount.

For one parameter:

• A positive gradient means increasing the parameter tends to increase loss, so decreasing it tends to reduce loss.
• A negative gradient means increasing the parameter tends to decrease loss, so increasing it tends to reduce loss.

The “descent” in gradient descent means reducing loss, not reducing every parameter value. In one update, some parameters can become smaller and others larger.

Gradient Descent: Move in a Direction That Reduces Error

The standard metaphor is descending a mountain in dense fog:

your position              → current model parameters
your altitude              → current loss
the slope under your feet  → gradient
how far you walk           → learning rate
a lower valley             → lower loss

The gradient points in the locally fastest uphill direction. Moving in the opposite direction usually reduces loss:

gradient direction      → local uphill direction
negative gradient       → local downhill direction

The most basic update rule is:

where $\eta$ is the learning rate. In code-like form:

parameters = parameters - learning_rate * gradients

That is the core meaning of gradient descent: use the gradient’s local direction to make a small parameter adjustment that tends to lower the loss. Deep-learning training commonly turns learning into an optimization problem and uses gradient information to search for parameters with lower loss.⁴

Calculate One Step with a Single Parameter

Let a toy model have one parameter $w$ and this loss function:

Its minimum is clearly at $w=3$, where the loss is zero. But suppose the model starts at $w=0$; it does not “know” the answer is 3.

At that point:

The derivative is:

At $w=0$, the gradient is $-6$. With learning rate $0.1$:

The new loss is $5.76$, down from 9. Repeating gives:

0 → 0.6 → 1.08 → 1.464 → 1.7712 → … → gradually approaches 3

The model never suddenly “understands that the answer is 3.” It repeats: measure the slope, move a small step downhill, then measure the slope again. A real LLM adjusts an enormous number of parameters in a very high-dimensional space, but the core idea is the same.

Backpropagation and Gradient Descent Are Not the Same Thing

These terms often appear together, but they play different roles in one training step.

Forward Pass: First Make a Prediction

logits = model(input_tokens)

Tokens pass through the model’s layers and produce candidate-token scores for every position.

Loss Function: Judge How Bad the Prediction Was

loss = cross_entropy(logits, target_tokens)

The loss function compares predicted scores with correct tokens and produces a loss value.

Backpropagation: Calculate Each Parameter’s Contribution to Loss

loss.backward()

Starting from the final loss, backpropagation traverses the computation graph backward and applies the chain rule to calculate the loss gradient for every trainable parameter. PyTorch describes autograd as the automatic-differentiation engine that computes gradients over a computation graph.⁵

Optimizer Step: Actually Change the Parameters

optimizer.step()

The optimizer reads the gradients and applies an update rule. A useful debugging analogy is:

loss              → how far the result was from the target
backpropagation   → trace how intermediate values affected that error
gradients         → local change information for each parameter
optimizer step    → actually modify parameters using that information

Backpropagation computes gradients; gradient descent or another optimizer uses them to update parameters.

A Minimal LLM Training Loop

Ignoring distributed training, mixed precision, memory management, and gradient accumulation, the loop is:

for batch in training_data:
    # Clear gradients left from the previous step.
    optimizer.zero_grad()

    # Forward pass: predict the next token at every position.
    logits = model(batch.input_tokens)

    # Compare predictions with correct target tokens.
    loss = cross_entropy(logits, batch.target_tokens)

    # Backpropagation: compute parameter gradients.
    loss.backward()

    # Update parameters using those gradients.
    optimizer.step()

Its essential cycle is:

predict → compute loss → compute gradients → update parameters

PyTorch’s standard optimization flow likewise clears gradients, performs backpropagation, and updates through the optimizer.³ Large systems add parallel devices, communication, fault tolerance, and numerical-stability machinery, but still repeat this mathematical loop efficiently and reliably.

Why Training Uses Mini-batches

In theory, we could calculate an exact average gradient over the whole training set before every update. For large training runs that would be extremely expensive:

read all training data → compute one full average gradient → update once

Instead, training divides data into mini-batches:

training data
├── batch 1
├── batch 2
├── batch 3
├── batch 4
└── …

Each update uses one batch to estimate a gradient. Like estimating an online service’s average latency from a sample rather than rescanning every historical request, it is not the exact whole-dataset average, but it is far cheaper and supports many rapid updates. That estimate is noisy, which is why the method is broadly called stochastic gradient descent (SGD).⁴

Strictly, a one-example update and a mini-batch update differ, but deep-learning practice often uses “stochastic gradient” broadly for any update estimated from a subset of training data.

The Learning Rate Determines How Far Each Step Goes

The $\eta$ in the update formula is the learning rate: the stride length for descending the foggy mountain.

Learning Rate Too Large

One step can cross a valley floor:

left slope → jump past the minimum → right slope → jump back

Loss can oscillate violently or grow without bound; training can diverge.

Learning Rate Too Small

Every step is cautious, but progress can be extremely slow:

many updates → only tiny parameter changes → loss falls very slowly

Real training therefore often uses schedules: gradually raise the learning rate at the start, then lower it later or change decay speed at specific stages. Intuitively: probe with small steps, move faster in the middle, then slow near a good region. The learning rate directly affects speed, stability, and final quality.

Why Real Training Also Needs Optimizers

Plain SGD is:

parameter -= learning_rate * gradient

In real models, different parameters can have gradients at different scales, different noise, and different histories. A single mechanical rule can be inefficient or unstable, so practice commonly uses SGD with Momentum, Adam, or AdamW.

Plain gradient descent looks only at the slope under its feet at this moment. Momentum also considers recent direction, reducing pointless back-and-forth due to local noise. Adam maintains first- and second-moment estimates of gradients to adapt the update scale for different parameters. The original Adam paper describes it as a first-order method for stochastic objectives using adaptive moment estimation.⁶

Optimizers are not the opposite of gradient descent. They are concrete, improved choices for how to use gradients to update parameters:

compute gradients → decide an update from them → try to reduce the objective

Exact recipes vary by model and stage, and commercial-model developers often do not disclose them completely. The GPT-4 Technical Report explicitly does not disclose full model size, hardware, training compute, data construction, or training methods.²

Why Can Next-token Prediction Produce Complex Capabilities?

The pretraining task looks simple: predict later tokens from earlier tokens. Yet improving this prediction across diverse text requires the model to exploit many patterns.

For example, continuing:

for item in items:

requires learning Python syntax, indentation and block structure, common loop bodies, and relations between names and context. Continuing “HTTP status code 404 usually means” requires statistical regularities in technical documentation. Continuing “Because the first two conditions are true, therefore…” requires tracking conditions, conclusions, and familiar inference forms.

These abilities are not inserted as hand-written if-else rules. They emerge gradually through many examples and many gradient updates. The GPT-3 paper showed that scaling autoregressive language models could substantially improve few-shot performance across translation, question answering, cloze tasks, and some reasoning or domain-adaptation tasks.⁷

Accuracy matters here: complex behavior on language tasks does not prove that a model understands the world exactly as people do. What types of internal representations models form remains an active research question.

Why Is Post-training Needed after Pretraining?

Next-token prediction mainly teaches a model what often follows what in similar text. A user, however, needs an assistant that follows instructions, is helpful, satisfies requirements, and is as safe as possible. Those objectives are not identical.

Internet and other text can contain unanswered questions, factual errors, hostile or unsafe material, low-quality verbosity, and writing unlike an assistant. So pretrained models normally undergo post-training. A widely cited public workflow from InstructGPT is:

pretrained model
    ↓
supervised fine-tuning on human demonstrations
    ↓
reward model trained from human rankings
    ↓
further optimization with reinforcement learning

The InstructGPT paper explicitly distinguishes predicting the next token on internet pages from following user instructions helpfully and safely. It uses human demonstrations, ranked model outputs, and further training with human feedback.⁸

The data and objective change during post-training, but many steps still abstract to:

model produces output → compute a training objective or loss → backpropagate → optimizer updates parameters

Gradient methods thus underpin pretraining as well as many forms of supervised fine-tuning, reward-model training, and preference optimization.

Gradient Descent Does Not Mean the Model Knows It Was Wrong

This distinction is important. A model does not think:

I answered London for the capital of France.
That was a geography mistake.
I should remember Paris next time.

The system first obtains a number such as:

Loss = 4.61

Backpropagation turns the aggregate numerical error into gradients:

parameter_1.grad = ...
parameter_2.grad = ...
parameter_3.grad = ...

The optimizer then makes a numerical update. The model is not directly learning human concepts of “right” and “wrong”; it is optimizing a mathematical objective chosen by its trainers.

Therefore:

training loss decreases ≠ the model is necessarily better in every real setting

Poor data can teach bad patterns; a train–use mismatch can harm generalization; requirements absent from the loss may not emerge automatically; and optimizing what people prefer is not identical to optimizing factual truth. Good training also needs quality data, suitable objectives, independent evaluation sets, safety and reliability testing, and ongoing error analysis.

Gradient Descent Cannot See the Whole Mountain

A gradient says only which direction near the current parameters may reduce loss. It does not show a complete map of the loss landscape or reveal the global minimum.

cannot see the whole mountain
→ feel the local slope
→ walk one step downhill
→ measure the slope again

Neural-network loss landscapes are high-dimensional, complex, and usually non-convex. They can contain flat regions, steep regions, narrow valleys, saddle points, and mini-batch noise. Real training adds momentum, adaptive learning rates, schedules, gradient clipping, weight decay, normalization, and better initialization to improve efficiency and stability.⁴

Gradient descent does not guarantee that every mini-batch update lowers total loss, that every evaluation metric improves simultaneously, or that a mathematically global optimum is found. It does provide a useful local direction instead of blindly trying every direction in an enormous parameter space.

Put the Whole Training Process Together

The basic LLM training pipeline is:

raw text
   ↓
convert to tokens
   ↓
construct next-token targets
   ↓
forward pass through the model
   ↓
scores and probabilities for candidate tokens
   ↓
loss function measures prediction error
   ↓
backpropagation computes parameter gradients
   ↓
optimizer updates parameters
   ↓
read the next batch and repeat

After many iterations, correct tokens generally receive higher probability, language-model loss falls, and parameters develop reusable language and knowledge patterns. Supervised fine-tuning, preference training, safety training, and other post-training can then make the model more useful as an assistant.

At the lowest level, the loop remains:

guess → measure the error → calculate how parameters should change → change a little → guess again

The Four Concepts to Remember

Parameters

The many internal numerical values that training can modify.

Loss

A number measuring how far the model’s prediction is from its training target.

Gradient

Information about how loss changes when each parameter changes a little near the current point.

Learning Rate

The size of one parameter-update step.

Gradient descent can now be defined in one sentence:

Gradient descent is a family of optimization methods that repeatedly makes small adjustments to parameters using the gradient of a loss function so that training error gradually decreases.

LLMs become powerful not because any one update is exceptionally clever, but because this simple loop is executed stably an enormous number of times on large models, rich data, and large-scale computation.

Quick Reference Diagram

input tokens
   │
   ▼
forward pass
   │
   ▼
probabilities for the next token
   │
   ▼
compute loss
   │
   ▼
backpropagation
   │
   ▼
gradients for all parameters
   │
   ▼
optimizer updates parameters
   │
   └──────────────► next batch of training data

References

Source check date: 2026-07-26.