Acceptance criteria for AI agents should stop asserting a single expected output and start asserting three things instead: a pass rate across a representative test set, a rubric score for quality on a scale, and a short list of binary never-do constraints that admit zero exceptions. This converts "did it match?" into "how often, how well, and did it ever cross a hard line?"

Quick Answer: Replace exact-match given-when-then with three layers — a numeric pass rate over a test set (e.g., 95%+), a weighted rubric for subjective quality, and absolute never-do rules evaluated as 0% tolerance, not percentages.

Why Given-When-Then Stops Working for Agents

Given-when-then assumes one input maps to one correct output, which is true for deterministic software and false for most agent behavior. An LLM-based agent can take different reasoning paths, phrase responses differently, or choose between several valid tool calls on every single run. Testing it against one fixed expected string will fail runs that are functionally correct and pass runs that got lucky.

The root issue is that traditional acceptance criteria conflate two questions: "is the behavior correct?" and "is the output identical?" For deterministic code these are the same question. For a generative agent they are not, and treating them as one causes QA teams to either write criteria so loose they catch nothing, or so strict they fail on cosmetic variation.

This isn't a new problem in disguise — it's closer to what statistical process control and quality assurance in manufacturing solved decades ago: you don't inspect every unit for a perfect match to a platonic ideal, you sample, measure variance, and set control limits. W. Edwards Deming's work on statistical quality control is the direct ancestor of "pass rate over a sample" thinking, and it maps onto agent testing more cleanly than most QA teams initially expect.

The Three Failure Modes of Exact-Match Criteria

Exact-match acceptance criteria for agents tend to fail in three predictable ways: false negatives on correct-but-differently-phrased outputs, false positives when a rigid regex happens to match malformed reasoning, and silent drift where nobody notices quality degrading because the test was never re-run against new model versions.

  • False negative: the agent resolves the ticket correctly but summarizes it in different words than the fixture expected.
  • False positive: a keyword-matching assertion passes even though the agent's underlying reasoning was wrong.
  • Silent drift: criteria written once, never revisited, stop reflecting what "correct" means after a model or prompt update.

Rebuilding Acceptance Criteria as Rates, Not Matches

Rate-based acceptance criteria specify a target percentage of a defined test set that must meet a bar, rather than a single expected transcript. This absorbs natural output variance while still being falsifiable: "95% of 200 test tickets resolved without escalation" is a claim you can measure and refute, unlike "the agent should handle tickets well."

The core shift is moving the unit of testing from one interaction to a set of interactions. You define a fixed, versioned test set — real or synthetic tickets, queries, or tasks — and run the agent against all of them every time behavior, prompt, or model changes. The acceptance criterion attaches to the aggregate, not to any individual case.

Building the Test Set

A test set only produces a meaningful pass rate if it's representative of production distribution, not just easy cases. Include common paths, known edge cases, and adversarial inputs in proportions that roughly mirror what the agent will actually see.

  1. Sample real historical cases where available — actual support tickets, actual queries — anonymized as needed.
  2. Add synthetic edge cases for scenarios too rare to appear naturally but too risky to skip (refund disputes, contradictory instructions, ambiguous intent).
  3. Tag each case with its expected category so failures can be sliced by scenario type, not just aggregated.
  4. Freeze a version of the set before each evaluation run so results are comparable across time — this is the same discipline behind the five-part agent spec structure, where the test set is treated as a first-class spec artifact, not a throwaway script.

Setting the Threshold

The percentage you pick should reflect the cost asymmetry of the task, not a round number that feels reassuring. A support agent that occasionally over-explains costs little; one that occasionally issues wrong refunds costs real money — the thresholds should not be equal.

Task risk profileExampleReasonable pass-rate targetRationale
Low stakes, easily reversibleDrafting a first-pass email reply80-85%Human review catches the rest before send
Moderate stakes, some cost to errorTicket resolution without escalation90-95%Escalation is a safety valve, not free
High stakes, hard to reverseAutonomous refund issuance98%+ pass rate, plus hard never-do rulesFinancial and trust cost of a miss is high
Irreversible or safety-criticalDeleting data, sending irreversible actionsNever-do constraint, not a rate at allNo percentage is acceptable for the wrong action

A rate target answers "how often is this good enough." It does not answer "is there anything this must never do" — that requires a different, non-probabilistic mechanism, covered next.

Rubric Scoring for Everything a Pass/Fail Can't Capture

A rubric decomposes subjective quality into a small number of scored dimensions — tone, completeness, policy adherence, reasoning transparency — each rated on a fixed scale by a human reviewer or an LLM-as-judge, then aggregated into a quality score the acceptance criterion can threshold against.

Pass/fail alone can't distinguish "technically resolved but curt and unhelpful" from "resolved and genuinely good." A rubric captures gradient quality that a binary test collapses to a single bit, which matters most for agents whose job is inherently qualitative — tone, judgment calls, written communication.

Designing a Usable Rubric

Keep rubrics to 3-5 dimensions with concrete, example-anchored descriptions per score level — a rubric that takes a reviewer more than a couple of minutes per case won't get run consistently, and inconsistent scoring is worse than no rubric.

  • Dimension: name the quality being scored (e.g., "policy adherence," "tone appropriateness," "reasoning clarity").
  • Scale: use a small integer scale (1-4 or 1-5) — finer granularity mostly adds reviewer disagreement, not signal.
  • Anchors: write one concrete example per score level so two reviewers converge on similar numbers for the same output.
  • Weight: decide whether dimensions are equally weighted or whether one (often policy adherence) dominates the composite score.

A useful acceptance criterion built on this looks like: "average rubric score ≥ 3.4/4 across the test set, with no individual case scoring below 2 on policy adherence." That combines an aggregate quality bar with a floor, so one catastrophically bad case can't be diluted into invisibility by ninety good ones.

Human Judge vs. LLM Judge

Both scoring methods have a place, and most mature evaluation setups use both rather than picking one permanently. Human review is the ground truth you calibrate against; LLM-as-judge is the scale mechanism you run frequently once calibrated.

ApproachStrengthWeaknessBest use
Human reviewerGround-truth judgment, catches nuanceSlow, expensive, doesn't scale to every runCalibrating the rubric; spot-checks on releases
LLM-as-judgeFast, cheap, runs on every changeCan share blind spots with the agent it's judgingHigh-frequency regression checks between human audits
HybridCombines bothRequires periodic recalibrationMost production agent eval pipelines

If you use an LLM judge, periodically audit its scores against a human-scored sample — judge drift is a real failure mode, and an uncalibrated judge quietly rubber-stamps a degrading agent.

Never-Do Rules Stay Binary — No Percentage, No Exceptions

Never-do constraints are the one category of acceptance criteria that should remain deterministic and absolute, evaluated as pass/fail with a 0% tolerance for violation, because some failures are unacceptable regardless of how rarely they occur. A 99% pass rate on "don't issue unauthorized refunds" still means real money leaves the business on the 1%.

The mistake teams make is treating everything probabilistically once they've adopted rate-based thinking for the rest of the eval. Never-do rules are the exception on purpose — they exist precisely because averaging away one bad outcome among many good ones is the wrong model for that specific class of risk.

What Belongs in a Never-Do Rule

Reserve never-do constraints for actions with asymmetric, hard-to-reverse downside — not for quality preferences, which belong in the rubric instead. A never-do rule should be checkable mechanically wherever possible, not left to subjective judgment.

  1. Financial limits: "0% of resolutions issue a refund over the policy limit without human approval."
  2. Scope boundaries: "0% of interactions access or reference data outside the customer's own account."
  3. Irreversible actions: "0% of runs delete a record without an explicit, logged confirmation step."
  4. Compliance lines: "0% of responses make a medical, legal, or financial guarantee the product isn't licensed to make."
  5. Tone floors: "0% of responses include language flagged by the toxicity/harassment filter."

Writing never-do rules as checkable assertions rather than vague principles is what makes them testable at all — "don't overpromise" is not a never-do rule, "never state a delivery date without checking the fulfillment system first" is. This same tight-scoping discipline is what writing an agent goal without drift is fundamentally about: a goal that's too broad can't be checked, and neither can a constraint that's too vague.

Never-do rules are also where least-privilege tool access does real work: an agent that structurally cannot call the refund API above a threshold can't violate that rule no matter what it reasons its way into. See least-privilege agent tool access for how scoping permissions turns some never-do rules into non-issues by construction rather than by hoping the model behaves.

Putting the Three Layers Together in One Acceptance Criteria Document

A complete agent acceptance criteria document states a pass rate target, a rubric threshold, and a list of never-do rules together, because each layer catches a failure mode the other two miss — rate catches frequency, rubric catches quality gradient, never-do catches catastrophic edge cases that a rate would average away.

Skipping any one layer creates a predictable blind spot: rate-only criteria miss quality nuance, rubric-only criteria have no hard floor against real harm, and never-do-only criteria say nothing about whether the agent is actually good at its job most of the time.

A Worked Example

Consider a customer support agent handling billing tickets. A complete acceptance criteria set might read:

  • Pass rate: 95% of a 200-ticket frozen test set resolved without human escalation.
  • Rubric: average tone/clarity/policy-adherence composite ≥ 3.5/4, no individual case below 2 on policy adherence.
  • Never-do: 0% of resolutions issue a refund exceeding the policy limit; 0% disclose another customer's account details.

Each line is independently measurable, independently falsifiable, and targets a different failure mode. That's the actual test of whether acceptance criteria are agent-ready — can you point to the specific number, score, or rule that would be violated, and by which mechanism you'd catch it.

From Written Rubric to Checkable Spec

This structure only holds up if the rubric dimensions and never-do rules are written specifically enough to run against a test set repeatedly, not restated informally every time someone builds a new eval. Prodinja's Agentic Workflows tool asks you to define an agent's checkable goal and constraints up front, and that same goal-and-constraints structure translates directly into rubric dimensions and never-do assertions for your eval set — the acceptance criteria and the agent spec end up describing the same thing from two angles.

Framing acceptance criteria this way also connects back to why the agent exists in the first place. A rubric dimension like "resolves the customer's underlying need, not just the literal request" only makes sense if you've already done the work of the jobs-to-be-done complete guide to define what that underlying need actually is — otherwise you're scoring against a guess.

Key Takeaways

  • Exact-match acceptance criteria fail for generative agents because one input can produce multiple valid outputs, causing false negatives on correct variation and false positives on lucky keyword matches.
  • Pass rate over a frozen, representative test set replaces single-transcript matching, with the threshold scaled to the cost of a miss — 80-85% for low-stakes drafts, 98%+ for high-stakes financial actions.
  • Rubric scoring on 3-5 anchored dimensions captures quality gradient a binary test collapses away, and works best as a human-calibrated, LLM-scaled hybrid.
  • Never-do rules stay binary and absolute — 0% tolerance, no averaging — reserved for financial limits, scope boundaries, irreversible actions, and compliance lines.
  • A complete acceptance criteria document states all three layers together, because rate, rubric, and never-do rules each catch a different failure mode the others miss.
  • A well-scoped agent goal and constraint set doubles as the eval spec — the same discipline that prevents goal drift also produces checkable rubric dimensions and never-do assertions.

Frequently Asked Questions

How do you write acceptance criteria for AI that isn't deterministic?

Write acceptance criteria as a pass rate over a fixed test set, a rubric score for subjective quality, and a set of binary never-do rules, instead of asserting one exact expected output. Each layer targets a different kind of failure the others can't catch alone.

What's a good pass rate threshold for agent testing unpredictable behavior?

It depends on the cost of an error: 80-85% is reasonable for low-stakes, easily reversible tasks like a draft reply, while high-stakes actions like autonomous refunds warrant 98%+ pass rates plus separate never-do constraints, since no pass rate below 100% is acceptable for the wrong irreversible action.

Should acceptance criteria ever use percentages for critical failures?

No. Critical, hard-to-reverse failures — unauthorized refunds, data leakage, irreversible deletions — should be written as never-do rules with 0% tolerance, not as a rate target, because averaging one bad outcome into a large pass rate hides real harm.

How is rubric scoring different from a simple pass/fail test?

A rubric scores multiple quality dimensions (tone, completeness, policy adherence) on a scale, capturing gradient differences between "barely acceptable" and "genuinely good" that a single pass/fail bit collapses into one outcome.

Can an LLM judge replace human review for agent acceptance testing?

An LLM-as-judge can run frequent, cheap regression checks once its scoring is calibrated against human review, but it shouldn't fully replace periodic human audits, since a judge model can share the same blind spots as the agent it's evaluating.