Synthetic data for evaluation is both: a smart shortcut when you have zero real examples yet, and a comfortable lie the moment you stop replacing it. Generated by an LLM and verified by a human, it can bootstrap a golden dataset in days. Left unreplaced, it grades your model against its own blind spots.
Quick Answer: Synthetic data is a legitimate way to bootstrap an eval set before real user data exists — but only if a human verifies every label and a migration plan swaps in production examples over time. An LLM generating the inputs, producing the outputs, and grading the outputs in one closed loop isn't evaluation. It's a mirror.
The Cold-Start Problem Synthetic Data Actually Solves
Before a product ships, you have no production logs, no support tickets, no failure transcripts — just a spec and a hypothesis about how people will use it. That makes it structurally impossible to build the real-data golden dataset that rigorous evaluation calls for. Synthetic data solves exactly this window: a testable eval set on day one, before any real traffic exists.
This isn't a hack unique to LLM products. Software teams have long written synthetic test fixtures before a feature has real users, and QA teams generate boundary-value inputs precisely because production traffic that would exercise those edges hasn't arrived yet. What's new with LLMs is that the generator itself can be the same kind of model you're testing — which is where the honesty problem starts, not where it ends.
The complete guide to AI evals frames this trade-off directly: an eval set with zero examples is worse than one with fifty imperfect ones, because a team with nothing to grade against ships blind. Synthetic data is the fastest legitimate way to get from zero to fifty. The rest of this article is about what has to be true for those fifty to stay honest.
Why "No Eval Set" Is the Worse Failure Mode
Teams often delay evals until they have "enough real data," which in practice means never, because without an eval set nobody notices regressions worth prioritizing. That's the argument behind building your first golden dataset with 50 examples: a small, honest set beats a large, imagined one. Synthetic generation is how you fill that first fifty when real examples don't exist yet.
Synthetic vs. Real: What Each One Actually Buys You
Neither source is strictly better — they trade off on availability, accuracy, and honesty in opposite directions, which is why the right answer is almost always "both, in sequence" rather than "pick one."
| Property | Synthetic (LLM-generated) | Real (production-sourced) |
|---|---|---|
| Availability before launch | High — unlimited, on demand | None — doesn't exist yet |
| Label accuracy | Only as good as human verification behind it | Reflects actual outcomes, still needs review for ambiguous cases |
| Match to real user distribution | Approximate, biased toward common patterns | Exact, by definition |
| Cost to produce | Low — compute plus review time | Requires a shipped product and real traffic |
| Risk of self-referential blind spots | High if the same model generates and grades | Low — grounded in an external source of truth |
| Best use in the eval lifecycle | Cold-start bootstrap, adversarial stress tests | Ongoing eval set, regression detection |
Read the table as a lifecycle, not a menu: synthetic data earns its keep in the top-left cell — high availability, low cost — while real data earns its keep in the bottom-right — accuracy and low blind-spot risk. The migration plan later in this article is the mechanism for moving from one side of the table to the other on purpose.
Generating Diverse Synthetic Inputs With an LLM
Generating eval examples with an LLM works best when you constrain it with a taxonomy of real user intents first, then prompt for edge cases inside each bucket. Asking broadly for "50 examples" reliably produces clustered, samey outputs that look diverse but aren't. Structure has to come before volume.
Start by defining the taxonomy, not the examples. Two frameworks do this well:
- Jobs to be Done identifies the discrete tasks users are "hiring" your product to do — the complete guide to Jobs to Be Done walks through surfacing these jobs, including Ulwick-style opportunity scoring for which jobs are underserved. Each job becomes a generation bucket, so synthetic inputs map to actual use cases instead of the LLM's default assumptions about your product.
- Customer journey mapping adds the emotional dimension: a frustrated user mid-task writes differently than a first-time user exploring, and both write differently than a power user hitting an edge case. The customer journey framework gives you the stages to seed prompts against, so your synthetic set covers tone and urgency, not just topic.
This isn't a new idea in software testing generally — OpenAI's open-source evals framework, one of the earliest widely adopted templates for this work, structures test cases into typed registries rather than a single unsorted list, precisely so a generation pass can't quietly collapse everything into one bucket. Borrow the instinct even if you don't borrow the tool: a taxonomy with named categories is what makes "diverse" a checkable property instead of a vibe.
Once you have buckets, generate inside constraints, not open-ended:
- Persona-seed each prompt. "Generate a query from a first-time user who hasn't read the docs" produces a different distribution than an unconstrained prompt.
- Ask for adversarial and boundary cases explicitly. Malformed input, contradictory instructions, and off-topic requests need their own prompts — an LLM won't volunteer them unprompted.
- Generate negatives, not just positives. Include inputs the system should refuse or decline to answer confidently, since most real eval sets are thin on these.
- Batch small, then dedupe. Generating ten at a time and checking for near-duplicates catches mode collapse before it pollutes two hundred examples.
- Rotate the generating model or prompt phrasing. A single model with a single prompt template produces a narrower distribution than mixing generation strategies.
Each of these levers pushes against the same failure: an LLM asked to imagine user inputs defaults to the median case, repeatedly, because the median is what it has seen most. Constraint is what forces it toward the tails — exactly where evals earn their keep.
A concrete example: asked to generate customer-support queries for a billing feature, an unconstrained prompt tends to produce a cluster of near-identical "how do I update my payment method" variants. Seeded instead with a Jobs to Be Done bucket for "dispute a charge I don't recognize" plus a customer-journey stage of "already frustrated, this is my second contact," the same model produces a sharper, more adversarial, and far more useful input. The taxonomy did the work; the LLM just filled it in.
The Self-Referential Trap: Grading an LLM's Homework With the Same LLM
The core risk with synthetic eval data isn't that it's fake — it's that a closed loop can form where the same model, or a close relative, generates the inputs, produces the outputs, and grades them. The model's blind spots become invisible, because the same blind spots wrote the exam.
This is well documented outside of evals, too. Shumailov et al.'s widely cited research on model collapse, published in Nature in 2024, found that models trained recursively on their own generated data progressively lost the ability to represent low-probability, tail-end patterns within just a few generations — the exact minority cases an eval set exists to catch. Evaluation has a parallel failure mode: grade a model only on inputs shaped like its own outputs, and it will look great, because you've measured it against a distribution it already fits.
LLM-as-judge compounds the risk when the grader is the same family of model as the one being graded. Zheng et al.'s influential MT-Bench and Chatbot Arena study documented self-enhancement bias — judge models systematically rating outputs from their own model family more favorably than equivalent human-preferred alternatives.
Stack that on top of self-generated inputs and you get a two-layer echo: the model wrote the question in a shape it's comfortable with, then graded its own answer favorably. Neither layer is doing evaluation. For a fuller breakdown of when a model-graded score is trustworthy, see when to trust LLM-as-judge.
The tell that you're in the trap: every synthetic failure case your eval set catches looks like something the model would plausibly generate itself. Real production failures are messier, dumber, and weirder than that — typos, contradictory requirements, users asking three things in one sentence.
Consider a support-summarization feature evaluated entirely on synthetic transcripts, generated and graded by the same model family. The eval set will reliably reward summaries that match the generating model's own stylistic preferences — sentence length, level of formality, which details it considers worth keeping. A real customer transcript with sarcasm, code-switching, or a garbled voice-to-text capture never enters the picture, so the score looks strong right up until real users arrive.
Human Verification: The Step That Makes a Synthetic Golden Dataset Trustworthy
A synthetic golden dataset earns that name only when a human has verified every label — a synthetic input paired with an unverified LLM-generated "correct" answer isn't a golden example, it's a guess wearing a costume. An unverified label just moves the model's error into the answer key instead of removing it.
Hamel Husain's widely read guidance on AI product evaluation makes a related point about real data: the highest-leverage early activity isn't writing more evals, it's a human directly reading dozens of real transcripts and doing structured error analysis before trusting any automated grade. The same discipline applies to synthetic examples — a human has to read the generated input and independently decide what the right output should be.
In practice, verification looks like this:
| Verification Step | What It Catches |
|---|---|
| Read the synthetic input cold, without the LLM's proposed output | Whether the input is even realistic or coherent |
| Independently write or confirm the expected output | Whether the "answer key" inherited the model's own errors |
| Tag ambiguous cases separately | Inputs with no single correct answer, which make bad eval rows |
| Score with a hard pass/fail, not a vibe | Whether the criterion is actually testable |
That last row matters more than it looks. A synthetic example graded on vague quality feels rigorous but isn't — see why numeric thresholds make evals failable for why a testable pass/fail bar, even an imperfect one, beats a fuzzy quality score every time. Verification is wasted effort if the criterion it produces can't actually fail.
Budget for this properly rather than treating it as an afterthought. Verifying fifty synthetic examples well — reading each one, writing an independent expected output, flagging the ambiguous ones — typically takes longer than generating five hundred did. That asymmetry is normal, not a sign something went wrong; generation is cheap precisely because verification is where the real judgment gets applied.
The Migration Plan: From Synthetic to Real Production Data
Treat synthetic data as scaffolding with an expiration date, not a permanent foundation. Decide upfront what share of your eval set is synthetic at launch, define a concrete trigger for when real examples start replacing it, and keep shrinking the synthetic share until it covers only genuine long-tail edge cases.
A workable version of this plan has four phases:
| Phase | Synthetic Share | Real Share | Trigger to Advance |
|---|---|---|---|
| Pre-launch | ~100% | 0% | Product ships to any users |
| Early access | 60-70% | 30-40% | First 100-500 real queries logged |
| Public beta | 20-30% | 70-80% | Real traffic covers most input categories in your taxonomy |
| Steady state | <10% (edge cases only) | 90%+ | Real data covers the long tail; synthetic kept only for rare adversarial cases |
Three practices keep this migration honest instead of aspirational:
- Log real production inputs from day one, even before you have time to review them — you can't migrate to data you didn't capture.
- Retire synthetic examples explicitly, not just dilute them. If a synthetic case is now covered by a similar real one, remove the synthetic version rather than keeping both.
- Re-run the full eval set at each phase transition and expect the pass rate to move. A model that scored well against synthetic data and drops against real data just told you something synthetic data alone never could.
That last practice is the actual test of whether your synthetic set did its job. A stable score across the migration is a mild good sign; a drop isn't a failure of the plan — it's the plan working exactly as intended.
Set this expectation with stakeholders before launch, not after a score drops. A leadership team that hears "we'll re-baseline against real data at 500 queries and the number may move" reacts very differently than one blindsided by a metric falling three weeks post-launch. The migration plan is as much a communication artifact as a technical one — it's the difference between a planned recalibration and an unexplained regression.
Where a Golden Dataset Tool Fits Into This Workflow
Most teams don't lack the concept of an eval set — they lack the trigger to build one before it's overdue. Prodinja's Evals tool is designed to flag a missing golden dataset as a gap in a project's readiness, rather than a silent omission that's easy to skip without a checklist forcing the question.
From there, the workflow this article describes is the intended use: synthetic examples generated to bootstrap the missing set, human-verified before they count as golden, and flagged for replacement as real cases accumulate. The tool is designed to surface the gap and structure the bootstrap. The migration discipline above is still the team's job, not something any tool does for you.
Key Takeaways
- Synthetic data solves a timing problem, not a data-quality problem — it exists to fill the gap before real examples do, not to replace them permanently.
- Unconstrained generation clusters around the median case. Seed prompts with a
Jobs to Be Donetaxonomy and customer-journey stages to force coverage of edge cases and emotional tone, not just topics. - The self-referential trap is the real danger, not fakeness itself: a model that generates its own inputs and grades its own outputs is measuring itself against a distribution it already fits.
- Every synthetic label needs independent human verification before it counts as golden — an unverified label just relocates the model's error into the answer key.
- A migration plan with concrete triggers, not vague intentions, is what keeps synthetic data from quietly becoming permanent.
- A dropping pass rate when you switch to real data is useful information, not a setback — it's the exact blind spot synthetic data couldn't show you.
Frequently Asked Questions
Is synthetic data reliable for LLM evaluation?
Synthetic data is reliable for bootstrapping an eval set, not for running one indefinitely. It's trustworthy when every label is human-verified and a plan exists to replace it with real examples — unverified or permanent synthetic data reliably hides the model's own blind spots instead of exposing them.
How many synthetic examples do I need before I have real data?
Enough to cover your input taxonomy at least once — commonly a few dozen examples per major user intent, in line with the "first 50 examples" approach to building a golden dataset. Volume matters less than whether each example maps to a real, distinct use case rather than a variation on the same one.
Can an LLM grade its own synthetic eval set?
Not safely as the sole check. Research on LLM-as-judge, including the MT-Bench and Chatbot Arena studies, has documented self-enhancement bias, where a model rates outputs from its own family more favorably. Use human review for at least the labels, even if a model assists with initial grading at scale.
What's the real difference between a synthetic and a real golden dataset?
A real golden dataset reflects the actual distribution of things users ask, including the messy, ungrammatical, contradictory inputs a model wouldn't generate for itself. A synthetic one approximates that distribution but skews toward patterns the generating model considers plausible — rarely the full picture of production traffic.
How do I know when to stop relying on synthetic eval data?
When real production data covers most of your input taxonomy and your pass rate has been re-measured against it at least once. If synthetic examples still make up more than a small minority of your eval set after real traffic exists, that's a sign the migration plan stalled, not a sign synthetic data is still needed.