Where Everything Begins: Linear Transformations
In deep learning, one operation is more fundamental and more frequent than any other: the linear transformation. At its core, it does one thing—uses matrix multiplication to turn one vector into another:
y = Wx
x is the input vector, W is a matrix whose values are parameters learned during training, and y is the output vector. That is all. Yet this simple operation has extremely rich geometric meaning and forms the foundation of deep learning.
Why Is It Called “Linear”?
“Linear” has a strict mathematical definition involving two conditions.
Additivity: Transforming two inputs separately and then adding the results gives the same answer as adding the inputs first and transforming the sum: f(a + b) = f(a) + f(b).
Homogeneity: Scaling the input by k before transforming it gives the same answer as transforming it first and scaling the result: f(k × a) = k × f(a).
Together, these conditions mean that a linear transformation preserves the basic structure of vector space. It does not “warp” space; it performs uniform operations such as rotation, scaling, projection, and shearing. A line remains a line after transformation, parallel lines remain parallel, and the origin never moves.
By contrast, an operation such as f(x) = x² is nonlinear. You can verify that f(2+3) = 25, while f(2) + f(3) = 4 + 9 = 13; the results are not equal.
Build Intuition with Concrete 2D Numbers
Suppose we have a two-dimensional vector x = [1, 0] and a 2×2 matrix W:
W = | 2 0 |
| 0 3 |
y = Wx = | 2×1 + 0×0 | = | 2 |
| 0×1 + 3×0 | | 0 |
This W doubles the x direction and triples the y direction. It is a scaling transformation.
Now consider another matrix:
W = | 0 -1 |
| 1 0 |
For x = [1, 0]: y = | 0×1 + (-1)×0 | = | 0 |
| 1×1 + 0×0 | | 1 |
For x = [0, 1]: y = | 0×0 + (-1)×1 | = | -1 |
| 1×0 + 0×1 | | 0 |
[1, 0] becomes [0, 1], and [0, 1] becomes [-1, 0]. This is a 90-degree counterclockwise rotation. The key point is that the values in W determine how the space is transformed, while the computation is always the same mechanical matrix multiplication.
Dimensions Can Change—and This Is Extremely Important
The previous examples mapped 2D to 2D, but a linear transformation can absolutely change a vector’s dimensionality. An m×n matrix can turn an n-dimensional vector into an m-dimensional vector:
Shape of W: (3, 2) Shape of x: (2,) Shape of y = Wx: (3,)
W = | 1 0 |
| 0 1 | x = | 1 | y = | 1 |
| 1 1 | | 2 | | 2 |
| 3 |
The two-dimensional vector has been “lifted” into three-dimensional space. Conversely, a (2, 3) matrix can “compress” a three-dimensional vector into two dimensions. This ability to change dimensions appears everywhere in deep learning. Every nn.Linear(in_features, out_features) you have encountered is essentially a matrix multiplication with a matrix of shape (out_features, in_features) plus a bias vector.
Understanding “Projection”: Start with a Shadow
In Transformer papers and tutorials, “projection” is a frequent term. What does it actually mean?
The Most Intuitive View: A Shadow
Imagine sunlight shining straight down while you hold a chopstick. It casts a shadow on the ground. The chopstick is an object in three-dimensional space—a 3D vector—while the shadow on the ground is its projection onto a two-dimensional plane.
The key observation is that the shadow preserves the chopstick’s horizontal information but loses its vertical information. You cannot infer how high the chopstick is from the shadow’s length; you can only tell how long it is in the horizontal direction.
That is the essence of projection: compress high-dimensional information into a lower-dimensional space, selectively preserving some aspects while inevitably discarding others.
Walk Through the Numbers
Suppose we have a 3D vector v = [3, 4, 5] and want to project it onto the xy plane—the “ground.” This is equivalent to discarding the z component:
Projection matrix W = | 1 0 0 |
| 0 1 0 |
Projection = W × v = | 1×3 + 0×4 + 0×5 | = | 3 |
| 0×3 + 1×4 + 0×5 | | 4 |
The 3D vector [3, 4, 5] becomes the 2D vector [3, 4]. Information in the z direction, the value 5, is discarded completely.
That is only the crudest form of projection. The more interesting case is a projection direction not aligned with the coordinate axes.
Which Wall You Project Onto Determines What You See
Imagine standing in a room holding the chopstick. Sunlight from above casts its shadow onto the floor—that is a projection onto the floor. If light instead shines from the left, the shadow appears on the right wall—a projection onto a different plane. The two shadows preserve different information and lose different information.
Mathematically, the projection matrix W fully determines which subspace receives the projection. Different matrices are like light sources at different angles, revealing different “sides” of the same vector.
An Analogy for Deeper Understanding
Imagine a person standing in front of you. That person is a “high-dimensional object,” with height, weight, hairstyle, facial expression, clothing, voice, and countless other dimensions of information.
- A front-facing photograph is one projection. It shows the expression and clothing but not the hairstyle at the back of the head.
- A side photograph is another projection. It shows the facial profile and height proportions but not the frontal expression.
- An audio recording without a photograph is another projection. It preserves only the voice dimension and discards all visual information.
Every projection is a simplified view of the same high-dimensional object. You lose information but gain a clearer, more focused representation of one particular aspect. That is why projection matters so much in deep learning. Information in a high-dimensional space is too rich; depending on the task, different projection matrices extract the particular side you need now.
How Does This Relate to Q, K, and V in a Transformer?
When we write:
Q = X × Wq
K = X × Wk
V = X × Wv
the same token embedding X is projected by three different matrices into three different subspaces. It is like placing three walls at different angles: the same chopstick casts a different shadow on each because each wall captures a different aspect of it.
- The shadow projected by Wq emphasizes “what I am looking for”
- The shadow projected by Wk emphasizes “what kinds of queries can match me”
- The shadow projected by Wv emphasizes “the actual content I carry”
Note: In strict mathematics, a “projection” has an additional condition: projecting twice must give the same result as projecting once,
P² = P. In deep-learning usage, “projection” is usually less strict and closer to its everyday meaning: mapping data from one space to another in order to extract a particular aspect of the information.
Four Key Roles of Linear Transformations in a Transformer
Before continuing, let us establish the big picture. Linear transformations play four distinct roles in a Transformer.
Role 1: The embedding layer. It maps discrete token IDs to continuous vectors. It is essentially a large lookup table, but can also be understood as a one-hot vector multiplied by an embedding matrix—a linear transformation.
Role 2: Q/K/V projections. They project a general token representation into the different semantic subspaces required by attention.
Role 3: The feed-forward network. It places a nonlinear activation between two linear transformations and performs complex feature transformations independently at every position.
Role 4: The final output layer. It projects the final layer’s hidden state into a vector with one dimension per vocabulary token. Each dimension is a token score, and softmax converts those scores into a probability distribution.
Why Linear Transformations Are Not Enough: The Need for Nonlinear Activations
If a neural network contained only linear transformations, it would have a fatal limitation: a composition of linear transformations is still a linear transformation.
y = W2 × (W1 × x) = (W2 × W1) × x = W_combined × x
A two-layer network is mathematically equivalent to a one-layer network. Stacking 100 layers would not help; they could all be merged into one matrix. A nonlinear activation function, such as ReLU or GELU, is therefore added after linear transformations so the network can express complex nonlinear patterns. A Transformer’s feed-forward network has this form:
FFN(x) = GELU(x × W1 + b1) × W2 + b2
It first applies a linear transformation, usually projecting to four times the dimension; then GELU introduces nonlinearity; then another linear transformation projects back to the original dimension. This expand–activate–compress process performs complex feature combination and selection in a higher-dimensional space before compressing the result.
A Hand-Calculated Demonstration: A Sentence’s Complete Journey Through a Transformer
Now let us use an extremely simplified but complete example to trace every step from input to output. The dimensions are tiny so the arithmetic can be done by hand, but the logic of every step is identical to a real model.
Mini-Transformer Setup
Vocabulary size: 6 words → {I, love, cats, hate, dogs, .}
Embedding dimension: 4 (real models use 768–4096)
Only 1 attention head (real models use 8–128)
Q/K/V dimension: 4 (simplified, without dimensionality reduction)
Input sentence: “I love cats”
Step 1: Tokenization—Turn Text into Numbers
A model does not recognize text; it recognizes numbers. The tokenizer maps each word to its vocabulary ID:
"I" → token ID: 0
"love" → token ID: 1
"cats" → token ID: 2
Input sequence: [0, 1, 2]
This step is more complex in real models, using BPE and other subword segmentation methods, but the essence is the same: text becomes a sequence of integers.
Step 2: Token Embedding—Look Up a Vector from an ID
The model has an embedding matrix of shape (vocab_size, d_model) = (6, 4). Every row is a vector representation of one word. These values are learned, not manually assigned:
Embedding matrix E (6×4):
dim0 dim1 dim2 dim3
ID 0 (I): [ 1.0, 0.2, -0.5, 0.3]
ID 1 (love): [ 0.1, 0.9, 0.8, -0.2]
ID 2 (cats): [ 0.6, -0.1, 0.4, 0.7]
ID 3 (hate): [-0.1, 0.8, -0.7, 0.2]
ID 4 (dogs): [ 0.5, -0.3, 0.3, 0.8]
ID 5 (.): [ 0.0, 0.0, 0.1, 0.0]
Use the token IDs to look up rows:
"I" → [1.0, 0.2, -0.5, 0.3]
"love" → [0.1, 0.9, 0.8, -0.2]
"cats" → [0.6, -0.1, 0.4, 0.7]
Each word is no longer a bare ID but a point in four-dimensional space carrying semantic information about the word.
Step 3: Add Positional Encoding—Inject Position Information
Why is this necessary? Because attention is permutation invariant. A Q·K dot product considers only the directional relationship between two vectors and knows nothing about their sequence positions. Without position information, “I love cats” and “cats love I” would look identical to the model.
Encoding for position 0: [0.00, 0.01, 0.00, 0.01]
Encoding for position 1: [0.84, 0.05, 0.04, 0.05]
Encoding for position 2: [0.91, -0.04, 0.08, 0.03]
Add it directly to each token embedding:
X0 ("I", pos=0) = [1.00, 0.21, -0.50, 0.31]
X1 ("love", pos=1) = [0.94, 0.95, 0.84, -0.15]
X2 ("cats", pos=2) = [1.51, -0.14, 0.48, 0.73]
The vector for the same word now differs when it appears at different positions. This is positional awareness.
Step 4: Use Linear Projections to Produce Q, K, and V
These are the “three walls at different angles” described earlier. Each weight matrix has shape (4, 4) and is learned during training:
Wq = | 0.5 0.0 0.1 0.0 | Wk = | 0.3 0.1 0.0 0.2 | Wv = | 0.1 0.0 0.5 0.0 |
| 0.0 0.6 0.0 0.1 | | 0.0 0.4 0.1 0.0 | | 0.0 0.3 0.0 0.2 |
| 0.1 0.0 0.4 0.0 | | 0.1 0.0 0.5 0.1 | | 0.2 0.0 0.4 0.1 |
| 0.0 0.1 0.0 0.5 | | 0.0 0.1 0.0 0.3 | | 0.0 0.1 0.0 0.6 |
For example, calculate the query of “I,” X0, by hand:
Q0 = X0 × Wq = [1.00, 0.21, -0.50, 0.31] × Wq
Q0[0] = 1.00×0.5 + 0.21×0.0 + (-0.50)×0.1 + 0.31×0.0 = 0.45
Q0[1] = 1.00×0.0 + 0.21×0.6 + (-0.50)×0.0 + 0.31×0.1 = 0.157
Q0[2] = 1.00×0.1 + 0.21×0.0 + (-0.50)×0.4 + 0.31×0.0 = -0.10
Q0[3] = 1.00×0.0 + 0.21×0.1 + (-0.50)×0.0 + 0.31×0.5 = 0.176
Q0 ≈ [0.45, 0.16, -0.10, 0.18]
Repeat the same process three times for every token, using Wq, Wk, and Wv:
Q ("what I seek") K ("what can match me") V ("my actual content")
"I" : [0.45, 0.16,-0.10, 0.18] [0.36, 0.12,-0.20, 0.13] [0.05, 0.12,-0.09, 0.21]
"love" : [0.55, 0.56, 0.43,-0.01] [0.37, 0.46, 0.51, 0.05] [0.51, 0.25, 0.53,-0.02]
"cats" : [0.81,-0.02, 0.34, 0.35] [0.60, 0.02, 0.39, 0.25] [0.39, 0.11, 0.49, 0.42]
A token’s Q, K, and V are three different vectors—the different shadows cast by the same chopstick on three walls.
Step 5: Calculate Attention Scores—the Dot Product of Q and K
Each token’s Q takes a dot product with every token’s K to measure “whom should I attend to?” From the perspective of “love,” using Q1:
score(love→I) = Q1 · K0 = 0.55×0.36 + 0.56×0.12 + 0.43×(-0.20) + (-0.01)×0.13
= 0.198 + 0.067 - 0.086 - 0.001 = 0.178
score(love→love) = Q1 · K1 = 0.55×0.37 + 0.56×0.46 + 0.43×0.51 + (-0.01)×0.05
= 0.204 + 0.258 + 0.219 - 0.001 = 0.680
score(love→cats) = Q1 · K2 = 0.55×0.60 + 0.56×0.02 + 0.43×0.39 + (-0.01)×0.25
= 0.330 + 0.011 + 0.168 - 0.003 = 0.506
Scale by dividing by √d_k = √4 = 2 so large dot products do not push softmax into a gradient-saturation region:
scaled scores = [0.178/2, 0.680/2, 0.506/2] = [0.089, 0.340, 0.253]
Step 6: Softmax—Turn Scores into a Probability Distribution
exp(0.089) = 1.093
exp(0.340) = 1.405
exp(0.253) = 1.288
sum = 1.093 + 1.405 + 1.288 = 3.786
attention_weights = [1.093/3.786, 1.405/3.786, 1.288/3.786]
= [0.289, 0.371, 0.340]
This tells us that when “love” creates its new representation, it assigns 28.9% of its attention to “I,” 37.1% to itself, and 34.0% to “cats.” “Love” attends most strongly to itself and “cats”; verbs and their objects have a naturally strong relationship.
Step 7: Take a Weighted Sum of V—Extract Information
Use the attention weights to calculate a weighted sum of every token’s value vector:
output_love = 0.289 × V_I + 0.371 × V_love + 0.340 × V_cats
= 0.289 × [0.05, 0.12, -0.09, 0.21]
+ 0.371 × [0.51, 0.25, 0.53,-0.02]
+ 0.340 × [0.39, 0.11, 0.49, 0.42]
dim0: 0.289×0.05 + 0.371×0.51 + 0.340×0.39 = 0.014 + 0.189 + 0.133 = 0.336
dim1: 0.289×0.12 + 0.371×0.25 + 0.340×0.11 = 0.035 + 0.093 + 0.037 = 0.165
dim2: 0.289×(-0.09)+0.371×0.53+ 0.340×0.49 =-0.026 + 0.197 + 0.167 = 0.338
dim3: 0.289×0.21 + 0.371×(-0.02)+0.340×0.42= 0.061 - 0.007 + 0.143 = 0.197
output_love ≈ [0.336, 0.165, 0.338, 0.197]
This new vector no longer represents “love” alone. It has combined contextual information from the entire sentence: it contains components from “I” and “cats,” mixed according to the attention weights. This is attention’s output—a context-aware representation.
Step 8: Feed-Forward Network—Deep Feature Transformation
The attention output passes through a two-layer feed-forward network:
First expand: 4 dimensions → 16 dimensions (multiply by W1)
Nonlinearity: GELU activation
Then compress: 16 dimensions → 4 dimensions (multiply by W2)
Expansion enables complex feature combinations in a higher-dimensional space, GELU introduces nonlinearity so the network can learn complex patterns, and compression returns the information to the original dimension. Residual connections, which add the input back to the output, and LayerNorm appear between these steps.
In a real model, the steps above—attention + FFN—are stacked many times. GPT-3 has 96 layers, while LLaMA models have 32–80. Every layer applies attention and an FFN to the preceding layer’s output, gradually constructing increasingly abstract semantic representations.
Hidden States: Intermediate Products Inside the Model
After those stacked layers, we need to understand a key concept: the hidden state.
After embedding and positional encoding, each of the three tokens in “I love cats” has a four-dimensional vector. Those three vectors are the layer-0 hidden states.
They then enter the first attention + FFN layer. Each token’s vector changes because it combines context and undergoes feature transformation. The three new vectors are the layer-1 hidden states. They enter layer 2 and change again.
Input embedding: [1.00, 0.21, -0.50, 0.31] ← initial representation of "I"
│
After layer 1 Attention+FFN: [0.72, 0.35, -0.12, 0.48] ← hidden state (layer 1)
│
After layer 2 Attention+FFN: [0.55, 0.61, 0.20, 0.33] ← hidden state (layer 2)
│
After layer 3 Attention+FFN: [0.82,-0.15, 0.63, 0.91] ← hidden state (layer 3, final layer)
Every layer’s hidden state remains four-dimensional. Its dimensionality never changes, but the values inside it are updated at every layer.
Why Is It Called “Hidden”?
Because these intermediate vectors are not visible externally. As a user, you see only the final output token, such as “.” You cannot see the vectors for each token in layers 1 and 2; they are “hidden” inside the model.
What you can see: input "I love cats" → output "."
What you cannot see: every token's intermediate vector at every layer ← these are hidden states
Transformers did not invent this term. It already existed in the RNN era. An RNN also produces an internal vector at each time step and passes it to the next; that vector is called a hidden state. Transformers retained the terminology.
Why Is the Final Layer’s Hidden State Special?
In an autoregressive GPT-style model, each position predicts the next token. The hidden state at the final token, “cats,” has passed through every layer and fully absorbed information from the entire sentence “I love cats,” so it is used to predict the next word:
Final hidden state at position 0 ("I") → predict what follows "I" → "love"
Final hidden state at position 1 ("love") → predict what follows "love" → "cats"
Final hidden state at position 2 ("cats") → predict what follows "cats" → "."
Step 9: The Final Output Layer—Turn a Vector Back into a Word
Suppose the final hidden state at the “cats” position after every layer is:
hidden_state = [0.82, -0.15, 0.63, 0.91]
A (4, 6) matrix now projects it to the vocabulary size—six dimensions—where every dimension is the score for one word:
logits = hidden_state × W_output
Suppose the result is:
logits = [0.1, 0.3, 0.1, -0.8, 0.5, 1.2]
"I" "love" "cats" "hate" "dogs" "."
Softmax converts the scores into probabilities:
probabilities ≈ [0.08, 0.10, 0.08, 0.03, 0.12, 0.59]
"I" "love" "cats" "hate" "dogs" "."
The model predicts that “.” is the most likely next token, with 59% probability. A period after “I love cats” is natural.
Every Matrix Is Learned
Every matrix encountered in the full process—the embedding matrix; Wq, Wk, and Wv; the FFN’s W1 and W2; and the final output layer’s W_output—is learned during training. The values inside these matrices are the model parameters. When we say GPT-3 has 175 billion parameters, we mean the total number of all values in these matrices.
At the beginning of training, every matrix is initialized randomly, so the model’s output is essentially nonsense. Training is fundamentally a loop:
1. Feed the model a piece of text, such as "I love cats"
2. Using its current parameters, the model predicts the next token, perhaps assigning "dogs" the highest probability
3. The correct answer is ".", so the model is wrong
4. Calculate how wrong it was: the loss
5. Use backpropagation to calculate the direction and amount by which every parameter should change
6. Adjust the values in every matrix slightly
7. Return to step 1 with a new piece of text and repeat billions of times
After repeated adjustment on enormous datasets, the matrix values gradually change from random numbers into meaningful ones. The final layer’s W_output learns which hidden states should map to which words, Wq learns how to extract query information, and the embedding matrix learns that semantically similar words should have nearby vectors.
There is an interesting detail: in many models, the final output matrix W_output and the initial embedding matrix E are actually transposes of the same matrix. This technique is called weight tying. It makes intuitive sense: embedding maps a token ID to a vector, while the output layer maps a vector back to a token ID. They are inverse processes. Sharing parameters both saves memory and keeps the semantic spaces at the two ends consistent.
Full-Process Overview
"I love cats"
│
▼
① Tokenization: [0, 1, 2] text → numbers
│
▼
② Token Embedding: three 4D vectors numbers → vectors (lookup)
│
▼
③ + Positional Enc: inject position information vectors + position
│
▼
④ Linear → Q, K, V: three projections three different views
│
▼
⑤ Q · Kᵀ: attention scores who should attend to whom?
│
▼
⑥ Softmax: probability distribution how much attention?
│
▼
⑦ Weighted sum of V: combine context create a new representation
│
▼
⑧ FFN: deep feature transformation stacked ×N layers
│
▼
⑨ Output Projection: 4D → 6D (vocabulary) vector → probabilities
│
▼
Next token: "." (highest probability)
Closing Thoughts
Understanding a Transformer does not require understanding every mathematical derivation from the start. What matters more is grasping several central intuitions.
Linear transformations are the basic operation. Matrix multiplication maps vectors from one space to another and forms the model’s most fundamental building block.
Projection selectively extracts information. Different projection matrices extract different aspects of the same input, like photographs taken from different angles.
Attention lets every token directly see every other token. Q·K matching finds relevant tokens, and V extracts their information to create a new representation containing global context.
Nonlinear activation lets a network exceed the limits of linearity. Without it, a network of any depth is equivalent to one layer.
Every parameter is learned. Through repeated trial and error on enormous datasets, the model gradually learns what every matrix should look like.
Once you understand these ideas, more complex variants in papers—Multi-Head Attention, RoPE, FlashAttention, and Group Query Attention—become local optimizations on top of the same basic framework. The central logic does not change.