Defining 'Done' for a Task an Agent Might Solve Five Ways

"Done" for an agent task isn't one expected output — it's a rubric of must-haves and nice-to-haves that any acceptable answer has to clear. Write required elements, quality thresholds, and disqualifiers explicitly, then check outputs against the rubric instead of comparing them to a reference answer.

Quick Answer: When a task has multiple valid solutions, define "done" as a checklist — required sections, factual grounding, disqualifying errors — not a single expected output. Grade against the rubric, not a reference.

Why a single expected output breaks for agentic tasks

A single expected output works when a task has one correct shape: a database migration either runs or it doesn't. It fails the moment a task admits multiple valid solutions, because comparing an agent's real output to one canned answer flags every legitimate alternative as wrong.

Think about a task like "write a competitive analysis of three rivals." Five capable analysts would produce five genuinely different documents — different section orders, different emphasis, different framing — and all five could be correct. A string-match or exact-output test against any one of them fails the other four.

This is the same trap teams fall into with evals for open-ended generation tasks. If your test suite defines success as "output equals this fixture," you're not testing quality — you're testing whether the agent happened to converge on the same phrasing as whoever wrote the fixture. That tells you almost nothing about whether a different correct answer would also pass.

• Exact-match tests work for deterministic tasks (parse this JSON, return this value).
• Rubric-based tests work for generative tasks (write this report, draft this response).
• Most teams building agents default to the first pattern because it's easier to write, then wonder why their evals reject good outputs.

The fix isn't a smarter model. It's a different definition of the target. As covered in our complete guide to agentic workflows, the reliability of an agent system depends less on model capability and more on how precisely you've specified what a good outcome looks like.

The shift: from expected output to acceptance rubric

An acceptance rubric is a structured list of conditions an output must satisfy to count as done, split into tiers by how non-negotiable each one is. It replaces "does this match the answer key" with "does this clear the bar," which is a fundamentally different — and more honest — question for creative or analytical work.

The three-tier structure

A workable rubric has three tiers, borrowed loosely from the MoSCoW prioritization method (Must have, Should have, Could have, Won't have) that's been used in requirements engineering since the 1990s:

Must-haves are your non-negotiables: required sections, mandatory citations, a specific data point that has to appear. If the output lacks one, it's not "less good" — it's not done, full stop.

Should-haves are where legitimate variation lives. Tone, ordering, phrasing, level of detail — an agent can make different reasonable choices here and still pass. This tier is what lets you accept five different-looking outputs as equally "done."

Disqualifiers deserve their own tier because they're not the inverse of must-haves. A report can include every required section (satisfying every must-have) and still be worthless if it fabricates a statistic. Disqualifiers catch the failure modes that a checklist of presence/absence won't.

A worked example: the report-writing agent

Take an agent tasked with producing a competitive landscape report. Instead of a reference document, the rubric looks like this:

Must-haves:

1. Includes sections on pricing, feature comparison, and market positioning
2. Every named competitor claim links to a cited source
3. Report length falls within a stated range (so "done" isn't a two-paragraph shrug)

Should-haves:

1. Opens with an executive summary that previews the conclusion
2. Uses comparison tables rather than prose paragraphs for feature parity
3. Flags confidence level on claims where source data is thin

Disqualifiers:

1. Any claim not traceable to a real source (fabricated statistics or invented quotes)
2. Recommending a competitor's product without disclosing the conflict of framing
3. Omitting a competitor the brief explicitly named

Notice what this rubric does not specify: exact wording, paragraph order within a section, or which comparison framework to use. That's deliberate — those are the axes on which five valid reports would legitimately differ, and locking them down would just recreate the single-expected-output problem in disguise.

Making "good enough" explicit before the agent runs

A rubric only works if it's written before the agent runs, not reverse-engineered from whatever output you happen to get. Writing it up front forces you to decide, in advance, what actually matters — which is a genuinely harder exercise than it sounds.

Ask these questions while drafting the rubric

1. What would make this output unusable, even if everything else were perfect? That's your disqualifier list.
2. What structural elements does every acceptable version need, regardless of style? That's your must-haves.
3. Where am I tempted to specify exact wording, and is that actually necessary? If not, that's a should-have or it's out of scope entirely.
4. How would I know if the agent lied or made something up? Grounding checks belong in disqualifiers, not should-haves — factual integrity isn't a nice-to-have.
5. What does "too short" or "too shallow" look like concretely? Vague quality bars ("comprehensive," "thorough") need a measurable proxy — a minimum section count, a required number of cited sources, a word-count floor.

Grounding as its own rubric line

Factual grounding deserves explicit treatment because it's the failure mode most likely to hide inside an otherwise well-structured output. A report can satisfy every must-have — right sections, right length, right format — while quietly inventing a market-share figure that sounds plausible.

Treat grounding as checkable: does every quantitative claim trace to a source the agent was given or a source it cites? This is closer to a fact-check pass than a style review, and it belongs in the disqualifier tier precisely because no amount of polish should compensate for a fabricated number.

Bertrand Russell's observation that "the fool is very sure and the wise man very doubtful" is a useful gut-check for agent grading — an output that reads confidently is not evidence it's grounded. Rubrics need to check the claim, not the tone of the claim.

Writing rubric criteria that are actually checkable

A rubric item is only useful if someone — or something — can look at an output and determine pass or fail without a debate. Vague criteria ("is comprehensive," "sounds professional") don't survive contact with a real evaluation pipeline.

Turn vague criteria into checkable ones

Each checkable version can be evaluated by a human reviewer, a rule-based script, or an LLM-as-judge pass without requiring a shared, unstated sense of what "professional" means. That specificity is also what makes a rubric portable across reviewers — two different people grading the same output should reach the same verdict.

This connects directly to how you scope the task in the first place. A rubric with unclear must-haves is usually downstream of a goal statement that was never precise to begin with — see our guide on how to write an agent goal without drift for the upstream half of this problem. It also pairs with the broader five-part agent spec structure, where acceptance criteria are one of five components a complete spec needs — not a standalone artifact bolted on afterward.

Where rubric-based acceptance fits in the larger agent spec

A rubric is not a replacement for scoping the agent's tool access or its input boundaries — it's the acceptance layer of a spec that also needs a clear goal, defined inputs, and constrained permissions. Treating "done" as a rubric doesn't reduce the need for the rest of the spec to be equally precise.

How this connects to the rest of your agent design

• Goal statement defines what the agent is trying to accomplish — the rubric defines how you'll know it succeeded. Confusing the two leads to goals that quietly smuggle in acceptance criteria, or rubrics that restate the goal without adding checkable detail.
• Tool access shapes what evidence the agent can actually gather to meet grounding must-haves; an agent scoped to least-privilege tool access that can't reach the sources your rubric requires will fail disqualifiers it had no way to avoid.
• Input boundaries determine what "complete" even means — a rubric requiring coverage of "every named competitor" only works if the input brief actually names them.

Rubrics also travel well outside pure agent work. The same must-have/should-have/disqualifier structure maps cleanly onto acceptance criteria for a Jobs-to-be-Done analysis or a customer journey deliverable — anywhere a team is tempted to grade a deliverable against one reference example instead of the underlying job it needs to do. Our guides on Jobs-to-be-Done and customer journey mapping both hit this same tension between one canonical answer and a family of acceptable ones.

Turning a rubric into a repeatable eval

A rubric written once and never checked is just documentation. The value shows up when every output — not just the first one you happen to review — gets graded against the same must-have, should-have, and disqualifier list, consistently, run after run.

Practical steps to operationalize a rubric

1. Write the rubric before the first agent run, not after seeing an output you liked or disliked.
2. Separate the three tiers explicitly in whatever tool or document holds the rubric — don't let disqualifiers hide inside a general "quality" bucket.
3. Assign each item a checker: rule-based (word count, section presence), human review, or LLM-as-judge — and note which, since each has different failure modes.
4. Re-run the rubric against every new output, not just spot-checks, so drift in agent behavior surfaces early rather than at the point a stakeholder notices.
5. Revisit the rubric when disagreements happen — a reviewer disagreement over a pass/fail call usually means a criterion wasn't checkable enough, not that the reviewer was wrong.

This is precisely the gap Prodinja's Agentic Workflows tool is built to close in the prototype: it walks you through stating an agent's goal precisely enough that "done" becomes something you can actually check, and that same precision is designed to feed directly into the eval criteria you'd use to grade every run against. The tool doesn't grade outputs for you — it's aimed at getting the goal and acceptance definition sharp enough that grading becomes possible in the first place.

Key Takeaways

• Replace single expected outputs with acceptance rubrics whenever a task admits more than one valid solution — comparing to one reference answer fails every legitimate alternative.
• Structure rubrics in three tiers: must-haves (non-negotiable), should-haves (quality signals, not fatal if missed), and disqualifiers (specific failures that override everything else).
• Treat factual grounding as its own disqualifier, not a should-have — a confidently written output is not evidence it's accurate.
• Write every criterion so it's checkable by a human, a script, or an LLM-as-judge — vague quality language ("comprehensive," "professional") doesn't survive real evaluation.
• Write the rubric before the agent runs, not reverse-engineered from an output you happen to like.
• The rubric is one component of a full agent spec — it doesn't substitute for a precise goal statement, scoped tool access, or clear input boundaries.
• Rubric thinking extends beyond agents — it applies anywhere a deliverable has a family of acceptable forms, from JTBD analysis to customer journey mapping.

Frequently Asked Questions

How do you define acceptance criteria for an AI agent's output?

Define acceptance criteria as a tiered rubric — must-haves an output can't lack, should-haves that improve quality without being fatal if missing, and disqualifiers that fail an output regardless of anything else it gets right. Write it before the agent runs.

What's the difference between a rubric and a test case for agent tasks?

A test case checks a specific input-output pair, usually exact-match, and suits deterministic tasks. A rubric checks whether any output satisfies a set of conditions, which suits generative tasks with multiple valid solutions, like report or content writing.

How many must-haves should an agent task rubric have?

Keep must-haves to the smallest list that would genuinely make an output unusable if missing — usually 3-6 items. Overloading must-haves turns legitimate stylistic variation into false failures; those belong in should-haves instead.

Can an LLM grade its own rubric as an evaluator?

Yes, using an LLM-as-judge pattern, but disqualifiers involving factual grounding need an independent check against real sources — an LLM judging its own factual claims without external verification can rubber-stamp a fabrication as grounded.

Why does a rubric-based agent still need a precise goal statement?

Because the rubric defines success conditions, not what the agent is trying to do — an unclear goal produces outputs so divergent that no single rubric can fairly evaluate all of them, and vague goals often smuggle acceptance criteria in informally instead of making them checkable.