07
Safety & Tools
Boundaries & Agents
These two concerns, safety and tool use, are grouped because both are typically layered on after core alignment. Safety teaches the model where boundaries are and how to stay useful without causing harm. Tool-use teaches the model to act as an agent: searching, calling APIs, reading files, and executing code. They use different data formats but share the same training stage.
Safety Dataset Types
| Dataset Type | Example Unit | Turn Type | Label | Goal | Behaviors | Weaknesses | Curation Concerns | |
|---|---|---|---|---|---|---|---|---|
| Refusal examples | Unsafe prompt + safe refusal | Single-turn | Preferred refusal | Stop disallowed behavior | Refusing harmful requests | Over-refusal on benign queries | Policy coverage, edge cases | |
| Safe completion | Risky prompt + harmless answer | Single-turn | Preferred safe answer | Continue safely | Redirection, safer assistance | Can feel evasive | Nuance in partial-help cases | |
| Borderline cases | Ambiguous prompt + policy answer | Single/multi-turn | Best interpretation | Improve consistency | Better handling of gray areas | Annotator disagreement | Clear policy, rater calibration | |
| Adversarial prompts | Jailbreaks, injections | Single/multi-turn | Safe outcome | Robustness | Resistance to manipulation | Ongoing contest with attackers | Attack type diversity, coverage | |
| De-escalation | Emotional/dangerous input | Multi-turn | Helpful safe response | Human-centered safety | Supportive, calm, bounded | Can flatten emotional nuance | Severity levels, cultural sensitivity |
Tool-Use & Agent Datasets
| Dataset Type | Example Unit | Turn Type | Format | Goal | Behaviors | Weaknesses | Curation Concerns | |
|---|---|---|---|---|---|---|---|---|
| Function-calling | Prompt + schema + tool call | Multi-turn/action | Structured JSON/action | Call tools correctly | Tool selection, argument formatting | Tied to specific tool versions | Schema versioning, execution tests | |
| Retrieval QA | Query + passages + answer | Single-turn | Context-grounded answer | Use evidence | Grounded answering, citations | May overfit to citation format | Evidence mapping, freshness | |
| Browser/search | Search terms, clicks, reads | Multi-step | Sequential observations | Find and synthesize info | Search planning, source selection | Fragile to UI changes | Realistic page snapshots | |
| Code execution | Prompt, code, result, repair | Multi-step | Action-observation loop | Learn from runtime feedback | Debugging, iterative repair | Overfitting to specific tests | Strong hidden test suites | |
| Workflow orchestration | Multi-tool business flow | Multi-step | State transitions | Complete real tasks | Planning and action chaining | Complex credit assignment | End-to-end success metrics |
Detailed Descriptions
Key Point
A good safety dataset does not just teach the model to say "I can't help with that." It teaches:
- when refusal is necessary
- when safer partial help is possible
- how to redirect productively
- how to stay calm under adversarial prompting