Your Best Eval Set Is Already in Production: Mining Golden Datasets From Live Traffic

Your best eval set already exists—it's sitting in your logs. Production traffic contains the messy phrasing, edge-case intents, and low-confidence responses that synthetic test cases rarely anticipate, because synthetic prompts are usually written by people who already know the "right" way to ask.

Quick Answer: Stop writing eval cases from imagination. Sample real production traffic in strata (by intent, length, channel), mine the low-feedback and low-confidence tail for edge cases, and dedup by semantic intent instead of exact text—then use that curated set to design pass/fail criteria.

Why Synthetic Eval Sets Test the Easy Path

Synthetic eval sets systematically test what the writer already believes the model can handle, because a human generating test cases anchors on the canonical, well-formed version of a request. This creates a blind spot exactly where failures cluster: ambiguous phrasing, mid-conversation context switches, and domain jargon real users produce without thinking twice.

The failure mode isn't laziness—it's a structural bias. Anyone writing test prompts from a spec will reconstruct the spec's own mental model of the task, not the chaotic reality of how people actually type into a text box. Nielsen Norman Group's usability research has documented for decades that real user input diverges from designer assumptions in query phrasing, abbreviation, and intent-blending—the same gap shows up when PMs hand-author LLM eval prompts instead of users.

Three ways synthetic sets go soft on your system:

• Vocabulary mismatch — writers use domain-correct terms; real users use slang, typos, and partial sentences.
• Single-intent framing — hand-written prompts ask one thing; production requests often blend two or three intents in one message.
• No adversarial pressure — a synthetic set rarely includes the "trying to break it" or "confused and frustrated" traffic that shows up naturally once you have real users.

Production traffic doesn't have this bias. It's generated by people solving their actual problems, under actual time pressure, with actual prior context—which is precisely why it's a better substrate for an eval set than anything you can write from a whiteboard. This is the same insight behind closing the loop from failures to eval cases: the richest source of test cases is the system's own failure history, not a hypothetical one.

A Curation Framework: Stratified Sampling First

A good production-mined eval set starts with stratified sampling—pulling examples proportionally across the dimensions that actually predict behavior differences, not a random dump of the last thousand requests. Random sampling over-represents the common case and under-represents the tail, which is exactly backward for eval purposes.

Pick strata that map to known behavior forks in your system. Common ones:

Sample a fixed quota from each cell rather than letting the largest bucket dominate. If "search" queries make up 70% of traffic but only 10% of your worst-performing outputs, an eval set built from raw frequency will bury the failures you most need to catch.

How Many Examples Per Stratum

There's no universal number, but a workable starting heuristic: enough per cell that a single bad example doesn't swing the pass rate more than a couple of percentage points—commonly 15-30 examples per stratum for an initial pass, expanding cells that show high variance. Treat the eval set as a living document, not a one-time export; revisit strata quarterly as usage shifts.

Edge-Case Mining via Low-Feedback Signals

The single richest vein of hard eval cases is the traffic your system already told you it struggled with—thumbs-down ratings, low retrieval scores, abandoned sessions, or unusually long agentic loops. Mining that tail directly surfaces the examples synthetic generation would never think to write.

Concretely, rank production interactions by any of these proxies for difficulty:

1. Explicit negative feedback (thumbs down, low star rating, "this didn't help" free text)
2. Implicit friction signals (user rephrased the same question 2+ times, session ended abruptly, user copy-pasted the output elsewhere without acting on it)
3. Model-internal confidence (low log-probability on generated tokens, retrieval similarity scores below threshold, agent tool-call retries)
4. Latency or cost outliers (unusually long chains-of-thought often correlate with the model "struggling")

None of these signals is perfectly clean—thumbs-down can mean the answer was correct but unwelcome, not wrong—so treat them as a ranking, not a filter. Pull the top decile by each signal, then have a human spot-check before promoting anything into the golden set. This is the same discipline behind good production observability generally: as covered in the complete guide to LLMOps observability, the goal is turning raw signal into a triage queue a human can actually work through, not an automated verdict.

Low feedback scores are a compass, not a court. They point you toward the interesting cases; a human still has to decide what "correct" means for each one.

Why Low-Confidence Cases Compound in Value

A single low-confidence production example is worth more to your eval set than ten synthetic hard cases, because it's already been observed happening at the input distribution your users actually generate. It's also evidence the failure mode isn't hypothetical—which matters when you're trying to convince a skeptical engineering team a fix is worth prioritizing. Left unaddressed, this is exactly how quality drift happens with no stack trace—the model degrades on real inputs while every synthetic regression test keeps passing.

Dedup by Intent, Not by Text

Deduplicating an eval set by exact string match misses the real redundancy problem: hundreds of differently-worded prompts that all express the same underlying intent, inflating your set's size without adding coverage. Dedup by semantic intent cluster, keeping a small representative sample per cluster instead of every near-duplicate.

A practical approach:

1. Embed each production prompt using a sentence-embedding model (any reasonably modern general-purpose embedding model works for this—precision matters less here than consistency).
2. Cluster the embeddings with a density-based method like HDBSCAN (preferred over k-means here because you don't know the true number of intents in advance, and HDBSCAN naturally flags noise points that don't belong to any cluster).
3. Label each cluster with a short human-written intent description by sampling 5-10 examples from it.
4. Keep a capped sample per cluster (e.g., 5-10 examples) prioritizing the highest-difficulty-signal ones from the previous section, and discard near-duplicates within the same cluster.
5. Flag noise points for manual review—unclustered outliers are disproportionately likely to represent a genuinely novel intent nobody has named yet.

This is where the framework earns its keep: clustering doesn't just compress your set, it surfaces intents you didn't know existed.

Worked Example: Finding an Untested Intent via Clustering

Say a support-copilot product clusters three months of production prompts and finds twelve clean clusters matching known intents—password reset, billing questions, feature how-tos, and so on. But one cluster of roughly 40 prompts doesn't map cleanly to any existing category: users are asking the assistant to compare their current plan against a competitor's, something the product's spec never anticipated as a use case.

That cluster is a signal, not noise. It means real users are pushing the assistant into a task nobody designed pass criteria for—likely because the assistant is being used as a decision-support tool in a moment product mapping never accounted for. The clustering step is what makes this visible; a stratified sample built only from expected intents would never have surfaced it, because nobody would have thought to write a synthetic prompt for it.

This mirrors what good customer journey mapping and jobs-to-be-done analysis already teach: the job a user is actually hiring your product to do is often broader than the job it was scoped to do, and production behavior is the most honest evidence of that gap.

Turning Curated Examples Into Pass Criteria

A curated production example is only half the work—the other half is deciding what "pass" looks like for that specific case, which is harder for edge cases than for the easy path precisely because there's no obvious canonical right answer. This is where most eval-building efforts stall, because writing pass criteria for a clean synthetic prompt is easy and writing it for an ambiguous real one requires judgment.

A workable process for turning a mined example into a graded eval case:

• State the user's actual goal, inferred from surrounding context (prior turns, what they did next), not just the literal prompt text.
• Define what "good enough" looks like, distinguishing a strict correctness bar (facts, numbers, actions taken) from a softer tone/format bar.
• Write the rubric as a checklist, not a single yes/no—most production edge cases have 2-4 independent things a response needs to get right.
• Note what a bad response predictably does wrong, since that's often more diagnostic for debugging than the pass criteria alone.

The table makes the tradeoff explicit: production mining costs more per-example effort in writing criteria, but that cost buys realism synthetic sets structurally cannot. It's worth paying once you've decided the eval set needs to actually predict production quality, not just pass a smoke test. And this connects to the broader discipline of watching quality continuously—see three dashboards every AI product needs on day one for how eval results should feed back into ongoing monitoring rather than sitting as a one-time gate.

Where Prodinja Fits Into This Workflow

Curated production examples—stratified, edge-mined, and deduped by intent—give Prodinja's Evals concept realistic cases to design pass criteria around, instead of synthetic prompts that never fail. As part of the interactive prototype at pmsynapse.in, Prodinja's Evals experience walks through what it would look like to take a set of representative production-shaped cases and structure them into graded criteria—an intended workflow for turning messy real inputs into something a team can actually grade against, rather than a working AI system that has evaluated your traffic for you. The value of that workflow is entirely dependent on feeding it examples like the ones this framework produces: representative, edge-heavy, and honestly labeled by a human who understands the underlying intent.

Key Takeaways

• Synthetic eval sets test the easy path because writers anchor on the canonical version of a request, missing the vocabulary, blended intent, and adversarial pressure real users generate.
• Stratified sampling across intent, input length, channel, and feedback score prevents your eval set from over-representing the common case and burying the tail.
• Low-feedback and low-confidence signals—thumbs down, rephrased queries, low retrieval scores—are the densest source of hard, realistic edge cases; treat them as a ranking to triage, not an automatic filter.
• Dedup by semantic intent cluster, not exact text match, using embeddings plus a density-based method like HDBSCAN so near-duplicate phrasings don't inflate your set.
• Clustering surfaces untested intents—a cluster of production prompts that doesn't map to any known category is often evidence of a real, unscoped use case, not noise to discard.
• Writing pass criteria for edge cases is harder than for the easy path, and that added effort is exactly what buys the realism a synthetic set can't.

Frequently Asked Questions

How many production examples do I need for a golden eval set?

There's no fixed number, but a workable starting point is 15-30 examples per stratum (intent, length, channel, feedback tier), expanding any cell showing high result variance. Total set size matters less than whether every meaningful behavior fork in your system has enough examples to detect regressions reliably.

Can I build a golden dataset without user feedback scores?

Yes—use implicit signals instead, like rephrased queries, abandoned sessions, retrieval similarity scores, or unusually long agent tool-call chains as proxies for difficulty. Explicit feedback is the cleanest signal but not the only one; any proxy for "the system struggled here" works for triage purposes.

What's the difference between deduping by text and deduping by intent?

Text-based dedup only catches near-identical wording, missing hundreds of differently-phrased prompts expressing the same underlying request. Intent-based dedup embeds prompts and clusters them semantically, so you keep a representative sample per true intent rather than redundant phrasings of the same one.

How often should a production-mined eval set be refreshed?

Revisit strata and re-mine edge cases on a regular cadence—quarterly is a reasonable default for most products—since user behavior and intent distribution shift as the product and its audience evolve. Treat the eval set as a living artifact tied to ongoing observability, not a one-time export frozen at launch.

Should synthetic test cases be discarded entirely in favor of production mining?

No—synthetic cases still have a role for testing known requirements before a feature ships to real users, since production data won't exist yet. The point isn't to eliminate synthetic cases but to stop treating them as sufficient once real traffic exists, and to prioritize production-mined examples as the primary source of truth once it does.