How LLMs actually work
Tokens, context windows, probabilities and temperature — the moving parts, explained with no math and no hand-waving.
The whole machine, in one paragraph
Text in → tokens (the model’s real alphabet) → numbers the model can operate on → the model proposes probabilities for the next token → it picks one (that’s where temperature comes in) → the new token is appended → repeat. Everything else — the billion parameters, the GPUs, the data centers — exists to make that loop fast and good.
Step 1: your text becomes tokens
The model never sees “hello”. It sees a token like hello, and “unbelievable” becomes un + believ + able. Roughly 3/4 of a word per token — that’s why pricing is per million tokens, not per word, and why “The” costs you something.
Step 2: tokens become geometry
Each token is turned into a vector — a list of hundreds or thousands of numbers. Similar meanings land close together in this space. This is where “meaning” becomes math: the model’s entire job is to manipulate those numbers so that the next token comes out reasonable.
Step 3: the model bets on the next token
For any given context, the model doesn’t pick the next token — it produces a probability distribution: “Paris” 40%, “Lyon” 8%, “a café” 5%… Then it draws from that distribution. Two consequences you’ll actually notice:
- Two runs can give different answers — it’s sampling, not looking up.
- Temperature is the dice roll’s shape. Low temperature → the draw barely deviates from the top pick (deterministic, safe, boring). High temperature → long shots become possible (creative, wild, occasionally nonsense). A practical map:
| Temperature | Personality | Good for |
|---|---|---|
| 0.0 – 0.3 | Precise, repetitive | JSON, code, extraction |
| 0.5 – 0.7 | Conversational | Chat, drafting |
| 0.8 – 1.2+ | Inventive, risky | Brainstorming, names, stories |
Step 4: context window = the model’s short-term memory
The model can only “see” what fits in its context window — prompt, conversation history, and its own answer so far. When the window overflows, something has to go: usually the oldest messages get dropped or compressed. That’s why a long chat that “forgot” your constraints isn’t a bug of the model — it’s the geometry of the setup.
Memory ≠ storage. Everything the model “remembers” is inside the current context. Nothing persists between conversations unless you put it back in.
What the model learned (and didn’t)
During pre-training, the model played the next-token game on an ocean of text. What it picked up: grammar, style, structure, lots of facts, some reasoning patterns. What it didn’t pick up: truth. Fluency is a measurable skill; correctness is not what the game optimizes. That gap is the source of every hallucination you’ll ever see.
Why answers stream in
Generation is sequential — each token needs the previous one. There’s no parallelism in the output, which is exactly why chat interfaces stream text one word at a time: it’s not a loading trick, it’s the actual computation happening in front of you.