Home / Chapter 2 · Where a model’s knowledge comes from
Last edited 28 September 2026 · 7 min read

Training

A model is built in three stages. Pretraining provides language and knowledge through next-token prediction, SFT teaches the assistant role, and RL polishes behaviour and reasoning. After training the weights are frozen, so anything the model doesn’t know has to be given to it in context.

In plain wordsFirst, years of reading everything in sight. Then a vocational course on example conversations, after which it knows how an assistant behaves. Finally, graded practice: it tries on its own and an examiner rewards good results, so it learns whatever earns points.

Click a stage and compare what the model does with the same prompt

Answers written by hand, for illustration.

Pretraining

The task: predict the next token across a huge corpus of text and code (Llama 3: about 15 trillion tokens). The loss is cross-entropy, i.e. −log of the probability the model gave the correct token. Backpropagation computes the gradient, and an optimiser, usually AdamW, nudges every weight slightly towards a smaller error. Every position in the text is a separate example, so one document teaches thousands of predictions at once.
The cost is about 6 × parameters × tokens operations. Chinchilla (2022) found that for a fixed budget the optimum is about 20 tokens per parameter. Today models are trained much longer (Qwen3, 2025: 36 trillion tokens for every size, about 60,000 per parameter for the 0.6B model; Llama 3 8B: almost 2000 per parameter), because a smaller model trained for longer is cheaper at inference over its whole lifetime.
Before training the data is filtered: Llama 3 removed duplicates at the URL, document (MinHash) and line level, dropped junk with heuristics and picked high-quality text with classifiers. Public benchmarks sit on the same web and leak into the corpus. On GSM1k, fresh problems in the style of GSM8K, some models scored up to 8% lower, so a public score can overstate what you will see on your own task (see “Evals”).
Towards the end, the model is trained on progressively longer sequences to extend its context, and on a small portion of the best data, e.g. code and maths, with a decaying learning rate (annealing). This stage is also called mid-training.
The result, the base model, is trained to continue documents, not to answer. Given “How do I bake bread?” it may add more questions from a forum. Pretraining data now includes plenty of question–answer and synthetic instruction text, so modern base models often answer anyway, but unreliably and without the assistant role.

Post-training: SFT and RL

SFT uses the same loss, but on conversations written in the chat template (see “How a model sees a chat”) and computed only on the assistant’s answer tokens. Hundreds of thousands to millions of examples, written by people and generated by stronger models, teach format, role and tool calling.
RLHF: people compare pairs of answers, a reward model is trained on those comparisons, and the language model is optimised against its score (classically with PPO), with a KL penalty for drifting away from the SFT model. In InstructGPT (2022) a 1.3-billion-parameter model trained this way beat the 175-billion-parameter GPT-3 in human ratings. DPO learns directly from better/worse pairs, without a separate reward model. Today the comparisons are often made by a model judge following written principles (RLAIF, Constitutional AI); in Tülu 3, GPT-4o rated the answers.
RLVR: the reward is checked by a program, such as tests or a comparison with the correct result. The model generates many attempts, and the better ones are reinforced. GRPO compares attempts on the same prompt with each other, without a separate critic. This is how reasoning models came about: a long chain of thought pays off because it raises the chance of a reward (see “Reasoning models”).
RL in environments: the same idea for agents. Multi-turn tasks with tools (a terminal, a repository with tests, simulated APIs) are rewarded for the end state; Kimi K2 (2025) had a joint RL stage in real and synthetic environments. SFT teaches the tool-call format, and this is where the model practises long, multi-step runs.
Distillation: a smaller model learns from the answers or probability distributions of a larger one, which is why small models beat what their size suggests (see “Distillation”).

What this means in practice

After training the weights are frozen. The model doesn’t learn from the conversation, and “memory” in apps is notes appended to the context (see “Context engineering and memory”). About the world after its cutoff date it knows only what it gets in context.
RL optimises the reward, not the intent (reward hacking). A model rewarded for passing tests can learn to game them instead of fixing the code, and one rewarded by human ratings learns to agree with people (sycophancy). In RLHF a KL penalty and diverse rewards limit this but don’t eliminate it. RLVR for reasoning often drops the KL term (DAPO, 2025), because a long chain of thought has to move far from the starting model, so there the quality of the verifier is what holds reward hacking back.
Your own training starts from an existing model. Fine-tuning is good at teaching style, format and narrow tasks, and poor at teaching new knowledge. Knowledge that changes or needs a source goes into the context (see “Fine-tuning and LoRA” and “RAG”).

Check yourself

How is an LLM made: what is the difference between pretraining, SFT, RLHF and RLVR?

Pretraining teaches next-token prediction on trillions of tokens of text with a cross-entropy loss. That gives language and knowledge, but the base model only continues documents. SFT on conversations in the chat template teaches the assistant role and the tool format. RLHF optimises against a reward model learned from comparisons made by people or a model judge, with a KL penalty for drifting away from the SFT model. RLVR optimises against a reward checked by a program, such as tests: that is how reasoning models are made, and, in multi-turn tool environments, models for agents. The risk of RL is reward hacking. After training the weights are frozen, so current knowledge has to be supplied in context.

Po polsku

Pretraining uczy przewidywać następny token na bilionach tokenów tekstu, stratą cross-entropy. Stąd język i wiedza, ale model bazowy tylko dokańcza dokumenty. SFT na rozmowach w szablonie czatu uczy roli asystenta i formatu narzędzi. RLHF optymalizuje pod model nagrody wyuczony z porównań ludzi albo modelu-sędziego, z karą KL za odejście od modelu po SFT. RLVR optymalizuje pod nagrodę sprawdzaną programem, np. testami: tak powstają modele rozumujące, a w wieloturowych środowiskach z narzędziami także modele do agentów. Ryzykiem RL jest reward hacking. Po treningu wagi są zamrożone, więc bieżącą wiedzę trzeba podać w kontekście.

Follow-up questions (4)

RLHF versus RLVR?: RLHF takes its reward from a model trained on the preferences of people or a model judge: good for style and helpfulness, but the reward model can be fooled. RLVR takes its reward from an automatic check, such as tests or the task’s result: harder to fool and easy to scale, but only where a program can verify the result.
PPO, DPO, GRPO?: PPO is classic RL with a reward model and a separate critic that estimates the value of a state. DPO learns directly from “better/worse answer” pairs, with no reward model and no generation in the loop. GRPO (DeepSeek) generates several answers to the same prompt and compares them with each other, so it needs no critic.
Why a KL penalty in RL?: It keeps the model close to the SFT model. Without it, optimisation quickly finds answers that the reward model rates highly and people rate poorly, and the model loses general skills. RLVR for reasoning often drops it (DAPO), because the model has to move far from its starting point; there a good verifier is what stops the model gaming the reward.
When fine-tuning and when RAG?: Fine-tuning for style, format and narrow skills. RAG for knowledge that changes and needs a source (see “Fine-tuning and LoRA” and “RAG”).

Sources

Report an error · Suggest a fix