Training Pipeline
LLM training happens in stages. Each stage uses different data formats and teaches different capabilities. The sections that follow cover each stage in detail.
Broad world knowledge, language patterns, code patterns, general fluency
Domain-specific knowledge: legal, medical, finance, company jargon and style
How to follow directions and format answers usefully
Which answers are better or worse
Policies that optimize for outcomes over sequences of actions
Boundary enforcement, tool interaction patterns, and agent-level task completion
Transferring capability from a larger teacher model to a smaller student efficiently
Distillation transfers knowledge from a large "teacher" model to a smaller "student" model. This can happen at any training stage: pre-training (synthetic textbook generation), SFT (teacher-generated instruction responses), or preference learning (teacher-ranked outputs).
The student trains on the teacher's outputs (or soft probability distributions), learning to approximate the teacher's behavior at a fraction of the inference cost. Distillation is attractive because it scales cheaply, but it needs strong filtering to avoid copying teacher-model errors and quirks.