A golden dataset for agent evals is a small, curated set of 20-50 real interaction cases — each labeled with the expected trajectory, not just the final output — that a lean PM team can build in a week without a research team, a labeling vendor, or a data science function. The discipline is curation, not volume.
Quick answer: Start with 20-30 cases pulled from real usage and near-miss edge cases, label the expected reasoning path and tool calls (not just the answer), and grow the set by triaging production failures weekly. A tight 30-case set beats a noisy 500-case set almost every time.
Most teams get this backwards. They try to build a comprehensive benchmark before they've shipped anything, then abandon it because maintaining hundreds of loosely-labeled examples takes more time than the eval saves. The better path is smaller, sharper, and grows organically from what the agent actually does wrong.
Why a small curated set beats a large noisy one
A golden dataset of 30 well-labeled cases will catch more regressions than 300 scraped, unlabeled ones, because signal density matters more than sample size when a human still has to trust the result. Noise in the set means noise in your confidence.
Large, loosely-assembled datasets fail for a predictable reason: nobody agrees on what "correct" means for each case, so scores drift and the team stops trusting them. Andrej Karpathy has described eval sets in production ML systems as needing to be treated like unit tests — small, deterministic, and owned — precisely because bloated suites become noise nobody debugs. The same logic applies to agent trajectories, where the failure surface (tool selection, intermediate reasoning, stopping conditions) is much larger than a single text output.
A curated set also forces you to be honest about what "good" looks like before you start grading. That's uncomfortable, but it's the actual work. If you can't write down what the agent should have done in 25 cases, you don't yet understand your product well enough to grade 250.
The volume trap, quantified
| Approach | Setup time | Maintenance burden | Trust level | Typical failure mode |
|---|---|---|---|---|
| 20-50 curated cases, trajectory-labeled | 3-5 days | Low — reviewed weekly | High | Set goes stale if not fed new failures |
| 200+ scraped/logged cases, output-only labels | 1-2 days | High — nobody re-labels | Low | Scores drift, team ignores dashboard |
| Vendor-labeled large benchmark | Weeks + budget | Medium, but opaque | Medium | Labels don't match your product's actual definition of correct |
| Synthetic-only cases (LLM-generated) | Hours | Low | Low-Medium | Misses real edge cases, over-indexes on obvious paths |
This isn't an argument against ever growing the set — it's an argument for sequencing. Curate first, scale later, and only scale with cases that earned their place through real failure.
How to source your first 20-50 cases
Your first cases should come from three places: real usage logs, deliberately constructed edge cases, and known failure modes from adjacent systems — weighted roughly 50/30/20 in favor of real usage. Anything else risks building a benchmark that's easy to pass and irrelevant to production.
Start with real usage logs. Pull the last 100-200 agent sessions and read them, not just the outputs. You're looking for:
- Sessions where a human intervened or corrected the agent mid-task
- Sessions with long tool-call chains (more steps = more chances to drift)
- Sessions users abandoned or rated poorly, if you have that signal
- The most common 5-10 task types, sampled proportionally to actual frequency
Then add deliberate edge cases. This is where product judgment matters most — you're encoding the failure modes you already suspect exist:
- Ambiguous instructions where the agent must ask a clarifying question instead of guessing
- Requests that require the agent to decline or escalate rather than act
- Multi-step tasks where an early tool call returns an error or empty result
- Tasks near the edge of the agent's declared autonomy — see our agent autonomy levels framework for how to define those boundaries before you grade against them
- Cases that probe whether guardrails hold under adversarial or malformed input, per our guide to agent action guardrails
Finally, borrow known failure modes. If you're building a support or research agent, look at published failure taxonomies from similar systems — tool-selection errors, premature termination, hallucinated tool arguments — and write one case per mode even if you haven't seen it in your own logs yet. Anthropic's own agent-building guidance (in its published engineering posts on building effective agents) repeatedly stresses testing the decision points an agent hits, not just end-to-end success — which is exactly the trajectory-labeling habit the next section covers.
Aim for 20 cases as a floor and 50 as a ceiling for version one. Below 20, you can't tell signal from noise across a model or prompt change. Above 50, review fatigue sets in and cases stop getting maintained.
Why you must label trajectories, not just outputs
Output-only labels ("was the final answer correct?") miss the majority of agent failure modes, because an agent can reach a right answer through a dangerous or unreliable path, or reach a wrong answer despite reasoning correctly given bad tool data. Trajectory labels catch both.
This distinction matters more for agents than for single-turn LLM calls, precisely because of the non-determinism agents introduce at every step — a topic we cover in depth in agent vs. workflow non-determinism. Two runs of the same agent on the same input can take different paths and still arrive at the same output, which means an output-only eval tells you almost nothing about reliability.
What a trajectory label actually captures
A trajectory label should specify, for each case:
- Entry state: what the agent knew and had access to at the start
- Expected tool calls: which tools should be invoked, in roughly what order, and with what argument shape
- Decision points: where the agent should branch, ask for clarification, or stop
- Acceptable variance: which parts of the path are flexible (phrasing, tool order when order doesn't matter) versus fixed (must call the lookup tool before the write tool)
- Terminal condition: what "done" looks like, including cases where "done" means declining to act
This is more labeling work per case than a simple output check — which is exactly why the set has to stay small. You cannot trajectory-label 500 cases with a two-person PM team and keep them current.
The case-card template
Use one card per case, kept in a shared doc, spreadsheet, or your eval tool of choice. Consistency across cards is what makes the set gradable by more than one person.
Case ID: EVAL-014
Task: User asks agent to cancel a subscription but account has an active dispute
Source: Production log, session #8821 (escalated to human)
Entry state: User message + account context with dispute_status=open
Expected trajectory:
1. Call get_account_status before attempting cancellation
2. Detect dispute_status=open
3. Decline to cancel automatically; explain why
4. Offer to route to a human reviewer
Acceptable variance: wording of the explanation; order of steps 3-4 may swap
Fixed requirements: must NOT call cancel_subscription tool at all
Terminal condition: human handoff message sent, no cancellation executed
Failure severity if missed: high (billing/legal exposure)
Last reviewed: 2026-06-02
Every card should carry a severity tag. Not every miss is equal — a wrong tone is a minor deduction, an unauthorized action is a blocking failure. Weighting your scoring by severity keeps the aggregate score honest instead of averaging trivial and catastrophic misses into a meaningless number.
How to grow the set from production failures
The golden set should grow only from cases that demonstrate a gap the current set didn't catch, added on a fixed weekly or biweekly cadence rather than continuously, so the set stays reviewable instead of becoming another unmaintained backlog.
The feedback cadence
| Cadence step | Who | What happens |
|---|---|---|
| Daily | On-call or agent owner | Flag any session with a correction, escalation, or user complaint |
| Weekly triage (30-45 min) | PM + one engineer | Review flagged sessions, decide which represent a new failure mode vs. a known one |
| Case authoring | PM | Write a case card for each new failure mode using the template above |
| Set review (monthly) | PM + team | Retire cases that no longer discriminate (every model version passes them trivially); confirm severity tags still hold |
The weekly triage step is the one teams skip, and it's the one that matters most. Without it, the golden set freezes at whatever it looked like at launch, while the agent's real failure surface keeps moving as you ship new tools, prompts, and model versions.
A practical rule of thumb: only add a new case if it fails against your current agent version. If the agent already passes it, it's not adding discriminating signal — it's adding maintenance cost. This keeps the set lean even as it grows, and it's the single highest-leverage discipline in this whole practice.
Where to source the failures worth promoting
Not every user complaint deserves a case card. Prioritize failures that are:
- Reproducible — you can reconstruct the entry state well enough to replay it
- Representative — likely to recur, not a one-off freak input
- Consequential — tied to a real cost: trust, safety, revenue, or support load
- Novel — not already covered by an existing case's trajectory definition
What this looks like at different team sizes
A one-person PM team should cap the golden set at 20-30 cases and review it monthly; a team with a dedicated eval owner can sustain 50-80 cases with weekly triage, but going much beyond that without dedicated tooling usually means the set quietly stops being maintained.
Scaling the practice isn't about scaling the case count first — it's about scaling the cadence and ownership. A single owner who triages weekly will outperform a team that "owns evals" collectively but reviews quarterly. Diffuse ownership is how eval sets rot.
If your agent's task space is genuinely broad — spanning multiple job types or customer segments — it's worth grounding your case selection in the actual jobs users are hiring the agent to do, rather than sampling failures uniformly. Our complete guide to jobs-to-be-done and customer journey mapping are useful lenses here: cases should cover the moments in the journey where trust is won or lost, not just the moments that are easiest to log.
For a broader grounding in what agent evals need to cover beyond this dataset practice — determinism, autonomy boundaries, guardrail testing — our complete guide to AI agents is a useful companion to this piece.
Key Takeaways
- Start with 20-50 cases sourced roughly 50% from real usage logs, 30% from deliberate edge cases, and 20% from known failure modes in comparable systems.
- Label the expected trajectory — entry state, tool calls, decision points, terminal condition — not just whether the final output was correct.
- Use a consistent case-card template so any reviewer can grade a case the same way another reviewer would.
- Weight cases by failure severity; a wrong tone and an unauthorized action should never score the same.
- Grow the set on a fixed weekly or biweekly triage cadence, and only promote a failure to a case if the agent currently fails it.
- Retire cases that no longer discriminate between agent versions so the set doesn't calcify into dead weight.
- A small, honestly-labeled set beats a large, loosely-labeled one for every team size below "dedicated eval platform team."
Frequently Asked Questions
How many cases do I need for a golden dataset?
Twenty to fifty cases is enough for most single-agent products in their first eval cycle. Fewer than 20 makes it hard to distinguish real regressions from noise; more than 50 without a dedicated owner tends to go unmaintained and stops earning trust.
What's the difference between output evals and trajectory evals for agents?
Output evals check whether the final answer was correct; trajectory evals check whether the agent took an acceptable path to get there — which tools it called, in what order, and where it should have branched or stopped. Agents need trajectory evals because non-deterministic paths can produce a right answer through a wrong or unsafe process.
How often should I add new cases to my golden dataset?
Weekly or biweekly triage is the sustainable cadence for most lean teams — frequent enough to catch drift, infrequent enough to stay reviewable. Only add a case if the current agent version actually fails it; otherwise it adds maintenance cost without adding signal.
Can I use an LLM to generate my golden dataset cases automatically?
Synthetic cases can supplement your set but shouldn't be its foundation, because LLM-generated edge cases tend to cluster around obvious, already-covered failure modes rather than the genuinely surprising ones real usage surfaces. Treat synthetic generation as a way to stress-test coverage gaps you've already identified from real logs, not as a replacement for reading real sessions.
Do I need a data science team to build agent evals?
No — the case-card template and weekly triage cadence in this article are designed to run with a PM and, ideally, one engineer for reproducing sessions. The bottleneck is product judgment about what "correct" means, not statistical infrastructure, which is exactly the kind of judgment a PM is positioned to own.