11
Metadata Fields
Every training example needs more than just content. These metadata fields help you filter, balance, deduplicate, and debug your datasets.
| Field | Description | Why It Matters |
|---|---|---|
| Source / provenance | Where the data came from | Enables filtering by quality, license compliance, and bias auditing |
| License / permission | Usage rights for the data | Prevents legal risk; some sources restrict commercial use |
| Split (train/valid/test) | Train / validation / test assignment | Prevents data leakage between training and evaluation |
| Language / locale | Language and regional variant | Controls multilingual balance and prevents unexpected language mixing |
| Domain / task tag | Subject area or task category | Enables domain-specific training runs and balanced sampling |
| Quality score | Automated or human quality rating | Supports quality-weighted sampling; a smaller clean set often beats a larger messy one |
| Deduplication cluster ID | Near-duplicate group identifier | Prevents the model from memorizing repeated content and inflating metrics |
| Safety / PII flags | Content safety and personal data markers | Required for compliance (GDPR, etc.) and safety filtering |
| Difficulty level | Estimated complexity of the example | Supports curriculum learning: easy examples first, hard examples later |
| Ground truth / reference | Verified correct answer or label | Enables held-out evaluation and reward verification |
| Tool / schema version | API version or schema the tool call targets | Prevents training on outdated API signatures that will fail at inference |
| Freshness / timestamp | When the content was created or last verified | Facts go stale; freshness metadata enables time-aware filtering |
| Annotator agreement | Inter-annotator agreement score | Low agreement = ambiguous example; filtering by agreement improves signal quality |
Curriculum Learning
Curriculum learning presents training examples in order of difficulty: easy examples first, then progressively harder ones. This is particularly relevant for reasoning and math training, where the "difficulty level" metadata field enables sorting examples by complexity.
Research suggests curriculum learning can improve both training efficiency and final accuracy, especially when combined with process supervision. The key requirement is having reliable difficulty annotations, which is why the metadata field matters.