10
Reference Tables
Big-picture maps of the training pipeline — stages, methods, and their tradeoffs.
Stage Overview
| Stage | Typical Data Unit | Main Training Signal | Main Goal | What It Mostly Teaches |
|---|---|---|---|---|
| Pre-training | Document chunk / text sequence | Predict next token (or denoising target) | Learn language and broad knowledge | Fluency, facts, syntax, style patterns, code patterns |
| Continued pre-training | Domain-specific document chunk | Same as pre-training | Adapt the base model to a domain | Legal/medical/finance/company knowledge, jargon, domain style |
| Supervised fine-tuning (SFT) | Prompt → ideal response, or multi-turn chat | Supervised target answer | Teach assistant behavior | Instruction following, tone, formatting, task completion |
| Preference learning / reward modeling | Prompt + multiple candidate answers + preference/score | Relative or scalar quality label | Teach which answer is better | Helpfulness, harmlessness, honesty, brevity, style preferences |
| Reinforcement learning (RL) | Full interaction trajectory / episode | Reward from humans, reward model, verifiers, or environment success | Optimize behavior across steps | Tool use, long-horizon planning, persistence, success-oriented behavior |
| Safety tuning | Unsafe prompt → refusal/safer completion | Supervised or preference label | Teach boundaries | Refusal, de-escalation, safer redirection |
| Distillation (cross-cutting) | Prompt → teacher model output | Teacher-generated target | Transfer capability efficiently | Faster learning, style transfer, specialist behavior |
Training Methods Comparison
| Training Type | What One Row Looks Like | Labeling Burden | Usually Used In | Best For | Weakness |
|---|---|---|---|---|---|
| Self-supervised (next-token) | Raw text chunk | None (self-labeled) | Pre-training | Broad capability and knowledge | Does not automatically make a good assistant |
| Denoising / span corruption | Corrupted text → original span/text | None (self-labeled) | Some pre-training setups | Strong text modeling | Less directly tied to chat behavior |
| Supervised fine-tuning | Prompt/messages → target response | Medium to high | Post-training | Instruction following, style, output format | Copies the quality ceiling of the target data |
| Distillation | Prompt → teacher answer | Lower human effort | Post-training | Scale, cheap specialization | Can copy teacher errors and quirks |
| Preference optimization (DPO/RLHF) | Prompt + chosen answer + rejected answer | Medium to high | Post-training | Learn what "better" looks like | Depends heavily on annotation quality |
| Reward modeling | Prompt + answer + score/rank | Medium to high | Before or alongside RL | Turn preferences into a reward signal | Reward hacking is possible |
| Reinforcement learning | State/prompt → action/response → reward → next state | Often high setup cost | RL stage | Multi-step behavior, tool use, agent tasks | Expensive, unstable, easy to mis-specify reward |
| Contrastive / retrieval training | Query + positive doc + negative doc | Medium | Retriever/reranker training | Better search, grounding, RAG | Helps retrieval more than generation |
| Process supervision / verifier | Problem + steps/answer + correctness labels | High | Reasoning-heavy systems | Reliability in math/code/logic | Annotation and verification are costly |