Stocking Your Eval Set With the Cases That Break Things

Load a fixed slice of your golden set—commonly 20-30%—with inputs engineered to fail: ambiguous requests, out-of-scope asks, injection attempts, and long-context traps. Score that slice separately from your happy-path average, because a strong-looking 92% overall pass rate can mask a 40% failure rate on exactly the inputs most likely to end up screenshotted online.

Quick Answer: Don't just sample representative inputs for your eval set. Deliberately stock a stress slice—ambiguous, out-of-scope, injection, long-context cases—and report its pass rate separately from the main average, so a strong blended number can't hide a weak adversarial one.

Why a Representative Sample Alone Won't Catch What Breaks in Production

Representative sampling optimizes for the average case, but average performance tells you nothing about tail behavior—and tail behavior is exactly what triggers outages, screenshots, and lost trust. A golden set built only from typical usage logs will always make the system look better than it behaves once real, curious, or malicious users show up.

Most teams build their first golden set the right way for the wrong reason: they pull from support logs, ask subject-matter experts for "typical" requests, and stop there. That's a fine starting discipline—see our guide to building your first golden dataset from 50 examples—but a set built entirely this way inherits three blind spots.

1. Logs only show what users have already tried. They can't show you what a user will try next week, especially once the feature is public and someone deliberately probes it.
2. Rare-but-catastrophic events are statistically invisible. A case that happens 1% of the time in production can still be the case that ends up in a bug report with your company's name on it.
3. Adversarial behavior is qualitatively different from normal usage, not just rarer. It's optimized against your system, not incidental to it.

Microsoft's AI Red Team, which has run structured adversarial testing across more than 100 of the company's own generative AI products, has published that the most damaging failures were rarely exotic jailbreaks—they were simple things like a model confidently answering outside its intended scope, or failing on a request that was ambiguous rather than malicious. That's the pattern a purely representative golden set is structurally blind to: it never asked the question in the first place.

This isn't a fringe opinion among practitioners, either. Hamel Husain's widely circulated writing on LLM evaluation argues that teams under-invest in error analysis precisely because their test sets never surface the errors worth analyzing—if every example in the set is one the model already handles well, there's nothing left to learn from.

This is why a mature eval practice, as laid out in our complete guide to AI evals, treats "representative" and "adversarial" as two different sampling jobs, not one. You don't fix this by adding a few more typical examples. You fix it by deliberately going looking for the inputs designed to break things.

A Taxonomy of Break-It Categories to Seed Your Golden Set

Eight categories cover most of the ways an LLM feature fails once it meets real usage: ambiguity, out-of-scope requests, prompt injection, long-context degradation, adversarial phrasing, format-breaking input, policy edge cases, and near-duplicate confusables. Seed your golden set with real examples from each—not just the ones you can imagine off the top of your head.

Prompt injection deserves its own callout: the OWASP Top 10 for LLM Applications lists it as LLM01, the single most cited risk category in the framework, precisely because it's the one category attackers actively engineer against rather than stumble into.

The long-context row isn't hypothetical, either. Research on "lost in the middle" behavior, documented by Liu et al. in a widely cited Stanford and UC Berkeley study, found that models reliably retrieve facts placed near the start or end of a long input but get measurably worse at facts buried in the middle—well within the model's stated context window, not past it.

Where to Find Real Examples for Each Category

Don't invent every case from imagination alone—that tends to produce cases that are adversarial in theory but not in the way real users or attackers actually behave.

• Mine support tickets and escalations for genuinely confusing past requests—these are real ambiguity, not hypothetical ambiguity.
• Run a red-team session with a skeptical teammate whose only job for thirty minutes is trying to break the feature.
• Borrow published attack patterns from the OWASP LLM Top 10 and adapt them to your specific product surface.
• Use long real documents—actual contracts, transcripts, or tickets—for long-context tests, not synthetic filler text.
• Ask another model to propose break-it cases, then have a human review and curate before anything goes into the golden set.

How Much of Your Eval Set Should Be Adversarial

Aim for roughly 20-30% of a mature golden set to be adversarial or edge-case examples, scaling that share up for high-stakes or public-facing features and down for low-risk internal tools. If you're just building your first set, start smaller—even 5-10 stress cases—and grow the ratio as the feature approaches launch.

The ratio matters less than the discipline behind it. A set that's 25% adversarial by accident, because someone got bored and added a few weird cases, isn't the same as a set that's 25% adversarial by design, refreshed every release against the taxonomy above.

The ratio should also shift across a launch cycle. Pre-launch, weight the set heavily toward adversarial and edge cases, since that's the only window to catch failures before real users do. Post-launch, keep replenishing the adversarial slice from production incidents even as the happy-path slice grows to reflect how the feature is actually being used.

Scoring Adversarial Cases Separately From Your Main Average

Report two numbers, not one: a primary pass rate on your representative, happy-path slice, and a separate pass rate on your adversarial slice. Never blend them into a single average—blending lets a handful of easy wins mathematically cancel out a cluster of dangerous failures, which is exactly the failure mode averages are known for hiding.

A model that's 98% correct on typical requests and 55% correct on adversarial ones can still report a 90%+ blended average. The number goes up on a dashboard while the actual risk—what happens when someone tries to break it—stays exactly where it was.

A two-track scorecard makes this visible instead of buried:

Setting the actual pass/fail line for each slice is its own decision. See our piece on setting numeric thresholds for failable evals for how to pick a number you can defend rather than one that just feels right.

Adversarial cases also tend to need a different grading approach. "Did the model refuse helpfully, without being preachy or unhelpfully rigid" is a nuanced call that a human rubric often handles better than a blunt automated check. And even where you do use a model as the grader, you should know its limits—our breakdown of when to trust an LLM-as-judge covers exactly where that method holds up and where it doesn't.

Two real frameworks back this separation. NIST's AI Risk Management Framework explicitly recommends measuring and tracking risks discretely rather than folding them into a single aggregate score, on the grounds that aggregation is where risk visibility goes to die. And Anthropic's own published red-teaming research (Ganguli et al., 2022) found that scaling a model up doesn't reliably reduce adversarial failure rates on its own—which is exactly the argument for testing and reporting adversarial performance as its own dedicated track, not assuming it improves quietly alongside everything else.

Turning Break-It Cases Into a Living Backlog, Not a One-Time List

Treat the adversarial slice as a living backlog, not a list you write once before launch. Every production incident, support escalation, and near-miss becomes a candidate new eval case within a day of discovery, and the taxonomy above gets revisited every release to check which categories have gone stale.

A few habits keep the backlog alive instead of frozen:

1. Log the input, not just the incident. When something breaks in production, the exact input that broke it should become an eval case before the ticket is closed, not sometime later.
2. Review the taxonomy every release cycle. Ask which of the eight categories hasn't had a new example added in the last quarter—that's usually the one nobody's been actively attacking.
3. Assign ownership. A backlog with no owner quietly stops growing the moment the person who cared about it moves to another project.

Ambiguity, in particular, is often a symptom of a job you haven't fully mapped. If users keep phrasing requests in ways your model misreads, the real gap may not be the model—it may be that you haven't nailed down the underlying job the user is hiring the feature to do. Our complete guide to jobs-to-be-done is a useful lens for turning "the model keeps guessing wrong" into "we haven't defined what done looks like."

A single mishandled edge case also rarely lands in isolation from the rest of a user's experience. It helps to know where in the customer journey's emotion curve these moments tend to fall—an edge case hit at an already-frustrated low point does more trust damage than the identical failure hit at a neutral one.

Where Prodinja Fits In

This is the shift Prodinja's Evals tool is built to encourage. Instead of defaulting to a happy-path-only golden set, the Evals workspace—currently shipping as an interactive prototype—walks you through structuring a golden set with adversarial and edge-case categories built into the plan from the start.

Hard cases end up part of the design instead of an afterthought bolted on after a near-miss. It doesn't run your model and hand you a scored verdict; it's designed to help you plan and organize the set so hard cases have a place before you ever start grading.

Key Takeaways

• Representative sampling alone can't surface tail-case failures—it only reflects what users have already tried, not what a determined or confused one will try next.
• Seed your golden set from a fixed taxonomy: ambiguous intent, out-of-scope requests, prompt injection, long-context degradation, adversarial phrasing, format-breaking input, policy edge cases, and near-duplicate confusables.
• Aim for roughly 20-30% adversarial share in a mature golden set, scaling higher for public-facing or agentic features and lower for low-risk internal tools.
• Never blend adversarial and happy-path scores into one average—report them as two separate numbers so a strong blended score can't mask a weak adversarial one.
• Give the adversarial slice its own explicit launch gate, not just a contribution to an overall pass rate, and expect it to need more human judgment in grading.
• Treat break-it cases as a living backlog: every production incident becomes a candidate eval case within a day, and the taxonomy gets revisited every release.