Curating Training Data for LLMs

Behavior View

Identify the behavior you want to improve, then look up which dataset types and stages are most effective for training it.

Behavior to Train	Best Dataset Types	Typical Stage
General fluency	Raw text corpus	Pre-training
Coding ability	Code corpus, code repair traces, execution loops	Pre-training + post-training
Instruction following	Prompt-response SFT	Fine-tuning
Multi-turn conversation	Dialogue SFT	Fine-tuning
Concise helpful answers	Preference data + style SFT	Post-training
Safe refusals	Safety SFT + preference data	Post-training
Grounded factual answers	Retrieval-grounded QA	Post-training
Tool calling	Tool trajectories	Fine-tuning + RL
Planning over steps	RL trajectories	RL
Domain expertise	Domain corpus + domain SFT	Continued PT + fine-tuning
Self-correction	Critique-revision data	Post-training
Honesty under uncertainty	Preference data + rubric scoring	Post-training
Structured outputs	Schema-labeled tasks	Task fine-tuning
Reasoning reliability	Verified tasks, process labels, outcome labels, verifier models	RL + preference
Code reliability	Code tasks with test suites + outcome/reward signals, execution-based verification	Fine-tuning + RL