Step 1 Text becomes tokens
A model never sees letters or words directly. Your text is first chopped into tokens — common chunks of characters (a whole short word, a word-piece, a space, a punctuation mark). Each distinct token maps to an integer ID. A model has a fixed vocabulary of typically 30k–200k tokens.
Try it — type and watch it tokenize
This is a simplified stand-in for real byte-pair encoding (BPE). Notice that rare or long words get split into several tokens, while common words stay whole — and a leading space is part of the token. That is why token counts rarely equal word counts.
Step 2 Tokens become vectors (embeddings)
Each token ID is looked up in an embedding table and turned into a list of numbers — a vector of maybe 768 to 12,288 dimensions. These numbers are learned, and they place tokens in a space where meaning becomes geometry: words used in similar ways end up close together, and directions in the space capture relationships.
A token's vector (8 of its dimensions shown)
Real vectors have hundreds or thousands of these numbers — far too many to read. What matters is not any single number but the overall direction.
Meaning as geometry — click a word to see its nearest neighbours
A real embedding space is high-dimensional; this is a flattened 2D sketch. Similar words cluster; the famous example king − man + woman ≈ queen works because relationships are consistent directions.
Step 3 Attention — words look at other words
A word in isolation is ambiguous. "It" refers to something; "bank" could be a river or money. Self-attention lets every token gather information from the other tokens that matter to it. For each token the model builds a query and compares it against every token's key; the better the match, the more of that token's value it pulls in.
Click a word to see what it attends to
Click any word above. Brighter = more attention paid to that word.
Models run many attention "heads" in parallel, each learning a different kind of relationship (grammar, coreference, topic). Stacked across layers, this is what lets a model resolve that "it" means "the cat".
Step 4 The transformer stack
Attention is one piece. A transformer block wraps it together with a small neural network (a feed-forward layer), plus residual "shortcut" connections and normalization that keep training stable. The model stacks dozens to over a hundred of these identical blocks; each one refines the representation a little more.
Watch data flow through one block (repeated ×N)
After the final block, the last position's vector is multiplied by a big output matrix to produce one score (a logit) for every token in the vocabulary — that is what step 5 turns into a choice.
Step 5 Predicting the next token
The logits are turned into probabilities with softmax, then the model samples one token from that distribution, appends it, and runs the whole process again — this is why generation is left-to-right, one token at a time (autoregressive). Two knobs reshape the distribution before the draw:
- Temperature — flattens (high) or sharpens (low) the probabilities. Low = safe and repetitive; high = creative and risky.
- Top-p (nucleus) — keeps only the most likely tokens whose probabilities add up to p, and ignores the long tail.
Prompt: The weather today is …
Bars show the probability of each candidate after your settings. Greyed-out bars were cut by top-p.
Generate one token at a time
Each click runs softmax + your temperature/top-p on a tiny hand-built table and draws a word. Same settings, different runs, different sentences — that randomness is the "sampling".
Step 6 Where it all comes from
None of the numbers above are hand-set. They are learned. During pre-training the model is shown enormous amounts of text and asked, billions of times, to predict the next token. Every wrong guess is nudged via backpropagation and gradient descent, tweaking billions of weights a tiny bit each time until the predictions get good.
| Term | In one sentence |
|---|---|
parameters | The learned numbers (weights) — a model's "7B" or "70B" counts these. |
context window | How many tokens the model can attend to at once — its working memory. |
pre-training | Learning language by predicting the next token across the internet's text. |
fine-tuning / RLHF | A later, smaller stage that makes the raw model helpful, harmless and follow instructions. |
inference | Actually running the trained model to generate text — steps 1–5 above. |
That is the whole loop: tokenize → embed → attend → stack → predict → sample, repeated for every token you read. Everything else — bigger vocabularies, more layers, longer context, clever training — is refinement of these same ideas.