Build the Eval Set Before the Feature: How PMs Define 'Working' for AI

An AI feature is "working" when it clears a score you wrote down before anyone touched a prompt or a training run — not when a demo looks convincing in a review. Getting there means building an eval set first: a fixed collection of representative inputs, expected outputs or grading criteria, deliberately hard edge cases, and a numeric target the feature must clear before it ships.

Quick answer: An eval set is a dataset — representative inputs, expected outputs or graders, hard cases, and a target score — that defines what "working" means for an AI feature before it's built. The PM owns it, because it encodes what "good enough" means to the business and the user, not just what the model can technically produce.

Why an Eval Set Beats a Vibe Check

Vibe-checking an AI feature — trying a dozen prompts, nodding at the outputs, shipping — fails because it tests the demo, not the distribution. An eval set fixes that by forcing you to decide, in advance, what counts as correct across the full range of inputs real users will actually send, then scoring against that definition every time the prompt or model changes.

The failure mode is familiar to anyone who has watched a promising AI demo turn into a support queue three weeks after launch. It happens in a predictable order:

1. A handful of hand-picked prompts look great in a demo, because they were unconsciously chosen to look great.
2. The feature ships on the strength of that impression, with no written definition of failure.
3. Real users send inputs nobody tried — messier, shorter, angrier, in a different language, missing context.
4. Someone swaps the model or tweaks the prompt for cost or latency reasons, with no way to tell if quality moved.
5. A support ticket or a churn signal is the first real feedback the team gets on whether the feature actually works.

Each of those steps is a symptom of the same root cause: nobody wrote down a scoreable definition of "working" before building. Cursor-style prompt tinkering and side-by-side eyeballing are useful for exploration, but they don't produce anything you can rerun, diff, or defend in a launch review.

The regression problem nobody notices until it's a ticket

Every prompt edit, every model upgrade, every added tool call is a change that can silently move quality in either direction. Without an eval set, teams have no way to detect a regression until a user reports one — by which point it's already shipped to everyone.

Software engineer Andrej Karpathy has argued publicly that evals function as the unit tests of the LLM era: without them, teams are effectively shipping untested code and calling it "prompting." Anthropic's own published guidance on building evaluations makes a similar case, framing a small, well-curated eval set as the first artifact to build — before extensive prompt iteration — precisely so that iteration has something to measure against.

That reframing matters for PMs specifically. A PRD without acceptance criteria isn't a spec, it's a wish list; an AI feature without an eval set is the same problem wearing a different outfit. The AI feasibility work you do before committing to build the feature should already be surfacing the inputs and failure modes that belong in this eval set — the eval set is where that feasibility analysis gets turned into something scoreable.

The Four Building Blocks of a Working Eval Set

A working eval set has exactly four parts: representative inputs pulled from real usage, expected outputs or a grading rubric that defines correctness, a curated set of hard cases designed to break the feature, and a target score the feature must clear before and after every change. Skip any one of the four and the eval set can't do its job.

A few things about each column are easy to get wrong in practice.

Representative inputs need a real distribution, not a curated one

Pulling examples from production logs or support transcripts beats inventing them, because invented examples unconsciously mirror what the team expects rather than what users actually do. If your team has already run a data readiness audit for this feature, that's usually the fastest source of a representative sample — it already surfaces what real inputs look like and where the gaps are.

Interview-based sources work too. Grounding representative inputs in the tasks users are actually trying to get done — the lens jobs-to-be-done research provides — keeps the sample anchored to real intent rather than to whatever's easiest to scrape.

Expected outputs need a rubric, not just an answer key

For classification, expected outputs are close to a normal answer key: one correct label per input. For anything generative, "expected output" has to become a rubric — a short list of scoreable criteria (accurate, on-tone, complete, no fabricated details) rather than one golden sentence, because no two acceptable answers to an open-ended prompt will ever be identical.

Hard cases come from where things already went wrong

The best source of hard cases is rarely a data scientist inventing edge cases in the abstract — it's the moments in the customer journey where things already go sideways: the angry follow-up message, the request that spans two categories, the ambiguous one-liner. Mapping the journey's friction points onto your hard-case list turns a generic stress test into one calibrated to your actual users.

Target score needs a business number attached, not just a percentage

A target score in isolation ("95% accuracy") is meaningless without knowing what a miss costs. Weighting the score by the business impact of each error type — a missed P1 ticket is not the same failure as a mislabeled P4 — turns an arbitrary-feeling number into a defensible one. And a quality target is incomplete without a paired constraint: a feature that's 96% accurate but takes eleven seconds to respond needs its latency budget defined alongside the accuracy bar, or you'll ship something technically "correct" that nobody wants to use.

A Worked Eval Table: Classifying Support Tickets by Urgency

A small eval table makes the abstraction concrete: for a support-ticket urgency classifier, each row is one input, its expected label, the model's actual output, whether they match, and a note on why a miss matters. Below is a trimmed sample — a real eval set for this feature would run 100-300 rows deep, not eight.

Reading the table like a PM, not like a data scientist

Raw accuracy on this sample is 5/8, or 62.5% — well under a plausible 92% target, which is the point of building the table before shipping rather than after. But the more useful read isn't the aggregate number; it's which rows missed and what that costs.

• Two of three misses under-classified severity (P1 read as P2 or P3), which is the expensive direction to get wrong — a missed P1 becomes an escalation, while an over-classified P4 just gets triaged down a level later.
• The pattern across misses is legible: tone-masked urgency, multi-issue tickets, and non-English input are all systematic gaps, not random noise — which means they're fixable with targeted prompt or data changes, not just "more data."
• The vague ticket miss is arguably the hardest case in the set, and a reasonable eval design might grade it on a partial-credit rubric rather than a strict match, since even a human triager might disagree.

This is the judgment an eval table exists to force: not just "did it pass," but where it failed and whether that failure pattern is one the business can tolerate at launch or must fix first.

Golden Sets vs. LLM-as-Judge for Generative Tasks

Classification evals have a ground truth to check against; generative tasks — summaries, drafted replies, chat responses — usually don't have one right answer, so PMs need either a golden set of human-graded reference outputs or an LLM-as-judge grader, and each comes with real caveats.

A golden set is a curated collection of inputs paired with a human-written or human-approved ideal response, scored against a rubric (accurate, on-tone, complete, free of fabricated detail) rather than an exact-match string. It's slow and expensive to build — every entry requires a subject-matter expert's time — but once built, it's a stable, trustworthy benchmark that doesn't drift or develop its own biases.

LLM-as-judge uses a second model to score the first model's outputs against a written rubric, instead of a human doing it. Researchers at UC Berkeley and LMSYS, in the widely cited MT-Bench and Chatbot Arena work (Zheng et al., 2023), found that a strong judge model agreed with human preference judgments at a rate approaching how often two human raters agreed with each other — good enough to be genuinely useful, not good enough to treat as ground truth.

The caveats that come with LLM-as-judge

The same research, and follow-on work citing it, documented specific, recurring biases that any team using this method needs to design around:

• Position bias — judges systematically favor whichever answer is shown first in a side-by-side comparison, regardless of content.
• Verbosity bias — longer answers get rated as better even when they aren't more correct or more useful.
• Self-preference bias — a model asked to judge tends to rate outputs written in its own style more favorably.
• Weak grounding for facts — a judge model without retrieval access can't reliably verify claims against current, external truth; it can only check internal consistency and rubric adherence.

Mitigations are straightforward but easy to skip under deadline pressure: randomize answer order, anchor the rubric to specific, checkable criteria rather than a vague "which is better," and run a periodic human-audit sample against the judge's verdicts to catch drift. Google's PAIR Guidebook and Chip Huyen's writing on AI system evaluation both make the same point from different angles: automated judges are a scaling mechanism for a rubric a human already validated, not a replacement for having validated one.

In practice, most teams run both: a small, human-graded golden set as the trusted anchor, and a larger LLM-as-judge pass for day-to-day regression testing, with the golden set periodically used to check whether the judge itself is still calibrated. Stanford's HELM project (Holistic Evaluation of Language Models) is a useful reference point here — it standardizes evaluation across many models and tasks precisely because ad hoc, one-off comparisons are so easy to get wrong.

Make the Eval Set a Spec Input, Not a Post-Build Afterthought

The eval set belongs in the feature spec next to acceptance criteria, written before backlog grooming starts — not bolted on after the model ships and something goes wrong. That's the real shift: defining "working" becomes a product deliverable the PM owns and writes down, not a QA checklist someone else fills in after the fact.

Treating it as a spec input changes what a PM actually hands to engineering. Instead of "make it good at classifying tickets," the brief becomes: here are 200 representative inputs, here's the rubric, here are the twelve hard cases we know will break a naive approach, and here's the score we need before this ships.

That's a concrete, testable spec — much closer to the kind of clarity the questions engineering needs from an AI spec actually calls for. It removes an entire category of back-and-forth about what "done" means.

Where this fits in Prodinja

It's a simulated version of the exact discipline this article is arguing for: evaluation as a spec input, defined up front, not an afterthought bolted on once something breaks.

Ownership matters here more than tooling. A data scientist can build the grading pipeline, but only the PM has the context to say what a miss actually costs the business — which is the judgment call every target score ultimately encodes.

Key Takeaways

• An eval set is four things, not one: representative inputs, expected outputs or a grading rubric, hard cases, and a target score — missing any one leaves the definition of "working" incomplete.
• Vibe-checking a demo tests the wrong thing. It evaluates a handful of cherry-picked prompts, not the real distribution of inputs users will send once the feature ships.
• Hard cases should come from where things already went wrong — past incidents, journey-map friction points, and known ambiguous inputs — not from abstract invention.
• Classification tasks get a golden answer key; generative tasks need a rubric, because open-ended outputs rarely have exactly one correct answer to match against.
• LLM-as-judge scales evaluation but carries real, documented biases — position, verbosity, and self-preference — that require randomized ordering and periodic human audits to catch.
• A target score without a business-cost weighting is just a percentage. Weight it by what each type of error actually costs, and pair it with a latency or cost budget.
• The eval set belongs in the spec, before the build starts — that's the shift from evaluation as QA afterthought to evaluation as a PM-owned deliverable.