Curation Best Practices
The biggest returns come not from more data but from better curation. A smaller, cleaner dataset often beats a larger messy one.
What does the model need to know about language and the world?
Pre-training data (books, web text, code, domain documents) teaches language patterns, facts, and reasoning. If the model lacks domain knowledge, continued pre-training on specialized text fills those gaps.
How should the model respond to a user?
Supervised fine-tuning data (prompt-answer pairs and multi-turn chats) teaches the model to follow instructions, format answers, and behave like a helpful assistant instead of just completing text.
Which response is better?
Preference data (chosen vs. rejected answers, rankings, or quality scores) teaches the model to distinguish good answers from acceptable ones. This is how you improve tone, helpfulness, and safety after basic fine-tuning.
How should the model behave over a sequence of actions to achieve a goal?
RL trajectory data (sequences of actions, observations, and rewards) teaches the model to plan over multiple steps, use tools, recover from errors, and optimize for end-task success rather than single-turn quality.
- 01Start with a high-quality evaluation set before you start training.
- 02Deduplicate aggressively, both exact and near-duplicate.
- 03Quality-filter before scaling. 10K clean examples often beat 100K noisy ones.
- 04Track provenance so you can debug surprising model behaviors back to their training data.
- 05Keep labels consistent. One ambiguous label can poison an entire category.
- 06Verify objective tasks: use unit tests for code, math checkers for math.
- 07Version your tool schemas. An outdated API signature will fail at inference.
- 08Separate train/validation/test carefully to prevent leakage.
Before collecting more data, ask: What does the model need to know? (pre-training),How should it respond? (SFT), Which response is better? (preference), and How should it act over multiple steps? (RL). Then curate ruthlessly. Clean data and consistent labels matter more than volume.