Pre-Training
The broad foundation stage. The model reads huge amounts of data and learns to predict the next token. That sounds simple, but it teaches grammar and style, world knowledge, reasoning patterns, code syntax, and rough answer structure.
| Dataset Type | Example Unit | Source | Turn Type | What It Teaches | Behaviors | Weaknesses | Curation Concerns | |
|---|---|---|---|---|---|---|---|---|
| General text corpus | Document chunk | Web, books, articles, forums | Non-conversational | General language and facts | Fluency, paraphrasing, summarization | Noisy, inconsistent quality | Dedup, spam removal, licensing | |
| Code corpus | File chunks, docstrings | Public repos, docs, Q&A | Non-conversational | Syntax, patterns, library usage | Code generation, refactoring, APIs | Learns bugs and bad practices too | License, secret leakage, test filtering | |
| Dialogue/transcript | Chat or transcript turns | Customer support, forums, chat logs | Multi-turn | Turn-taking patterns | Dialogue flow, tone matching | Noisy and sometimes low quality | Toxicity, privacy, spam | |
| Domain corpus | Domain documents | Medical, legal, finance, enterprise | Non-conversational | Specialist vocabulary and style | Domain language, jargon control | Can overfit, lose generality | Authority, recency, jargon balance | |
| Synthetic text | Model-generated text | Distillation, augmentation | Non-conversational | Coverage of rare forms | Better distribution shaping | Polished-looking mediocrity | Filtering, diversity, confidence | |
| Multilingual corpus | Text in many languages | Web, books, translated corpora | Non-conversational | Cross-lingual knowledge | Translation, multilingual QA | Language imbalance | Language ID, script balance | |
| Multimodal paired data | Image/audio/video + text | Captioned media, transcripts | Cross-modal | Visual/audio grounding | Vision-language, audio understanding | Alignment noise, OCR errors | Modality tags, alignment labels | |
| Structured-to-text | Serialized tables/JSON | Databases, APIs, spreadsheets | Non-conversational | Schema familiarity | Structured data comprehension | Fragile formatting habits | Canonicalization, schema consistency |
Good Curation Focuses On
- • Coverage: many topics, languages, writing styles
- • Quality filtering: remove spam, corruption, duplicates
- • Deduplication: avoid training too heavily on repeats
- • Freshness and provenance: keep data current
- • Safety filtering: remove disallowed content
- • Balance: don't let one source dominate
What Pre-Training Does Not Do
Pre-training alone does not make a model reliably helpful, obedient, or safe. It makes the model knowledgeable and fluent, but not necessarily aligned with what a user wants.
Continued pre-training (sometimes called domain-adaptive pre-training) is a distinct stage where a base model is further trained on domain-specific text using the same next-token prediction objective. The goal is not to teach new behaviors but to shift the model's knowledge distribution toward a target domain (medicine, law, finance, or internal company knowledge).
The key difference from initial pre-training is that you start from an already-capable model and use a smaller, targeted corpus. This is how teams build specialized models without retraining from scratch.
Not all pre-training uses next-token prediction. Encoder and encoder-decoder models (like BERT and T5) use denoising objectives: the model receives text with randomly masked or corrupted spans and must reconstruct the original. This teaches bidirectional context understanding rather than left-to-right generation.
T5's span corruption, for example, randomly replaces contiguous spans with sentinel tokens, then the model generates only the missing spans. This approach is less common in modern decoder-only LLMs but remains important for understanding the full range of pre-training methods.