A non-deterministic feature spec swaps "expected result: X" for two layers: properties every output must satisfy (cites a source, stays under a word limit, stays on policy) and a target rate across an eval set of representative cases. Acceptance shifts from "matches this exact answer" to "satisfies these invariants often enough."
Quick Answer: Stop writing one expected output. Define properties the output must always satisfy (grounding, length, policy) plus a target pass rate across an eval set — acceptance becomes "meets these rules on 95%+ of cases," not "equals this string."
Why "Expected Result: X" Doesn't Survive Contact With a Generative Feature
A generative feature answers a fixed prompt differently on two consecutive runs, so a spec built on exact-match equality fails almost immediately — not because the feature is broken, but because the spec asked the wrong question. The fix isn't a looser string match; it's a different kind of acceptance criterion entirely.
Traditional PRDs inherit their acceptance format from deterministic software: given input A, the system returns output B, and QA writes a test asserting actual === expected. That format assumes the same input always produces the same output — true for a checkout flow, false for anything routed through a language model with temperature above zero.
Teams that don't update the spec format tend to land in one of two failure modes:
- Over-fitting to one example. The spec locks in a single "golden" response, engineering builds to pass that exact case, and the feature ships brittle — it handles the demo prompt perfectly and falls apart on the next one.
- Abandoning acceptance criteria altogether. Frustrated by constant false failures, the team drops formal testing and ships on vibes, which means nobody notices when quality quietly degrades after a model or prompt change.
Both failure modes trace back to the same root cause: treating a probabilistic system as if it were deterministic. This is exactly the kind of mismatch six clarifying questions engineering needs before it will commit to an AI feature spec is meant to surface early, before a sprint gets planned around an acceptance format that can't hold.
The tell that you're specifying the wrong thing
If your draft spec reads "the assistant should respond with something like [example]," you're describing a sample, not a contract. A contract has to hold across every input in the feature's domain, not just the one you happened to test in the demo.
Specify by Properties and Constraints, Not Exact Strings
Replace the single expected output with a checklist of properties — testable, binary-or-scored conditions the output must satisfy regardless of its exact wording. A property doesn't care what the model says; it cares whether what the model said obeys the rule.
Properties generally fall into a handful of recurring categories. Naming them explicitly in the spec is what lets engineering build validators instead of guessing at intent.
| Property type | What it constrains | Example rule |
|---|---|---|
| Grounding | Every factual claim traces to a retrieved source | "Must cite a source document for each claim" |
| Length/format | Output shape stays within bounds | "Must not exceed 150 words"; "must be valid JSON matching schema" |
| Policy/safety | Output stays inside allowed content boundaries | "Must not recommend an unapproved product substitute" |
| Tone/voice | Register matches brand or context | "Must not use exclamation marks in a refund-denial message" |
| Structural | Machine-parseable shape for downstream systems | "Must include exactly one next_step field" |
A well-formed property is checkable by a script, a classifier, or a short rubric — not by a human re-reading every response for a vibe check. If you can't write a test for it, it isn't a property yet; it's a hope.
Where the properties come from matters as much as the properties themselves. A constraint invented in a spec-writing session tends to be arbitrary; a constraint derived from what the user is actually trying to get done is defensible when engineering pushes back on it. Grounding a feature's must-cite-a-source rule in the underlying job — say, "help me trust a recommendation before I act on it" — is the same discipline covered in a complete guide to jobs-to-be-done: the property exists to serve the job, not the other way around.
Constraints vs. properties — a useful distinction
- Constraints are hard boundaries: violate one and the response is rejected outright, no exceptions (a policy breach, a schema failure).
- Properties can be scored on a spectrum: a response can partially satisfy "helpful tone" and still ship, while a rate of satisfaction is what gets tracked over time.
Keep constraints short and enforceable in code or a classifier. Keep properties few enough that QA can actually maintain test coverage for each one — a spec with forty properties is a spec nobody will finish testing.
Move Acceptance From Equality to a Distribution Over an Eval Set
Once properties replace exact-match strings, the second shift follows naturally: acceptance is no longer "does this one response pass," it's "what percentage of responses across a representative sample pass." A single response passing or failing tells you almost nothing about a probabilistic system; a rate across dozens or hundreds of cases tells you what to expect from the population of real user interactions.
This isn't a new idea invented for AI — it's the same logic behind statistical process control, the manufacturing quality discipline Walter Shewhart formalized in the 1920s and W. Edwards Deming later popularized industry-wide. A factory doesn't reject a production line because one bolt measured 0.001mm off spec; it tracks the distribution of measurements and flags the line only when the process drifts outside control limits. Generative features need the same posture: individual variance is expected, and the acceptance question is whether the process stays in control.
Practically, that means building an eval set: a curated, representative sample of inputs the feature is expected to handle, run through the feature repeatedly, with each output scored against the properties defined earlier. OpenAI Evals, the open-source framework OpenAI published for exactly this purpose, formalizes the pattern — a test suite of prompts, a grading function per property, and an aggregate pass rate instead of a pass/fail verdict on any single case.
What replaces "expected: X" in the actual spec document
| Deterministic feature spec | Non-deterministic feature spec |
|---|---|
| "Given input A, return exact output B" | "Given input A, return output satisfying properties P1–Pn" |
| One test case per requirement | An eval set of N representative cases per requirement |
| Pass = output equals expected string | Pass = property holds for a given case |
| Acceptance = 100% of tests pass | Acceptance = property holds at ≥ target rate across the eval set |
| Regression = any test flips red | Regression = pass rate drops below threshold or shifts distribution |
Building a genuinely representative eval set is where most of the real work sits — a set skewed toward easy inputs will report a pass rate the feature can't actually hold in production. This is the same representativeness problem covered in a data-readiness audit for AI features: if the data feeding your eval set doesn't reflect production traffic, your target rate is measuring the wrong population.
A target rate is only as honest as the eval set it's measured against. An eval set built from ten hand-picked happy-path prompts will always look better than production traffic ever will.
How QA Tests Properties and Monitors Aggregate Quality
QA's job on a generative feature splits into two distinct activities that used to be one: writing property-level test assertions (does this specific response satisfy this specific rule) and monitoring aggregate quality over time (is the rate at which responses satisfy that rule holding steady in production). Golden-output diffing does neither well.
Property-level testing
For each property named in the spec, write an automated checker:
- A word-count property is a trivial script — count tokens, flag anything over the limit.
- A citation property checks whether every factual sentence links back to a retrieved source ID, which is enforceable in code if the retrieval pipeline tags its outputs.
- A policy property is often a classifier call: run the output through a smaller model or rules engine trained to flag violations, then treat that classifier's verdict as the test oracle.
- A tone property is the hardest to automate cleanly and often needs either a rubric-based LLM-as-judge scorer or periodic human spot-checks, scored on a scale rather than pass/fail.
Hamel Husain, an ML engineer widely cited for his writing on evaluation practice for LLM products, has argued that teams skip this step at their peril — without a structured evaluation harness, a team is effectively debugging in production by vibes, unable to tell whether a prompt tweak made things better or worse. That argument is exactly why property tests need to run in CI, not just at ship time.
Aggregate monitoring in production
A test suite passing in CI doesn't guarantee the same pass rate holds once the feature meets real traffic, so monitoring has to continue after launch:
- Sample production outputs continuously — not every response, but a statistically meaningful slice, scored against the same properties as the eval set.
- Track pass rate as a time series, the same way you'd track latency or error rate, with alerting when it drops below the agreed target.
- Re-run the full eval set on every model or prompt version change, before rollout, so a regression is caught pre-deploy rather than discovered by a support ticket.
- Watch for distribution shift, not just average pass rate — a feature can hold a 95% average while failing consistently on one input segment nobody sampled.
Google's 2017 "The ML Test Score" research (Breck et al.) proposed exactly this dual structure for production ML systems: a scored rubric spanning roughly two dozen checks across data, model, infrastructure, and — critically — ongoing monitoring, not a one-time pre-launch gate. The paper's broader observation, that most production ML systems it surveyed satisfied only a fraction of the full checklist, still tracks with how generative features get shipped today: heavy investment pre-launch, comparatively little in continuous monitoring.
The same target-rate logic already lives in a metric most PMs accept without argument: a p95 latency budget is itself a non-deterministic acceptance criterion — "95% of requests finish under 400ms," not "every request takes exactly 400ms." The guide to latency budgets for AI features is worth reading alongside this one, because it's the same distribution-over-equality mindset applied to a different property.
Setting Target Rates: How Good Is Good Enough
A target rate — 90%, 95%, 99% — isn't a number you pick by instinct; it's a risk decision tied to what happens when the property fails and where in the user's experience the failure lands. A tone slip in a casual suggestion is a shrug; a missing citation in a compliance-adjacent answer is a real problem.
Three inputs typically drive the number:
- Severity of a single failure. A wrong policy answer that leads a user to break a rule warrants a stricter target than a slightly off-brand tone in a low-stakes suggestion.
- Where the moment sits in the user's journey. A first-run onboarding tip tolerates more variance than a moment right before a purchase or a support escalation — mapping the feature against the customer journey's emotion curve helps identify which moments actually need the strictest targets.
- Cost and feasibility of getting closer to 100%. Pushing a property from 95% to 99.5% often costs disproportionately more engineering effort than the first 95 points did — a feasibility conversation, not just a QA one.
| Target rate | Typical use case | Failure mode if missed |
|---|---|---|
| ~85–90% | Low-stakes suggestions, tone/style properties | Mildly off-brand response, easily dismissed |
| ~95% | Most content-generation and summarization features | Occasional visible quality dip, recoverable |
| ~99%+ | Compliance-adjacent claims, safety/policy constraints | Regulatory, trust, or safety exposure |
Because these targets are a feasibility tradeoff as much as a quality one, they belong in the same conversation as the rest of the feature's technical risk assessment — see the complete guide to AI feasibility for how target rates fit alongside latency, cost, and data-readiness constraints in a single go/no-go call.
Framing Acceptance Before You Ever Touch an Eval Set
The hardest part of this shift usually isn't building the eval harness — it's getting a spec-writing conversation to stop reaching for a single expected output in the first place. Old habits pull toward "here's what it should say," and that instinct is hard to break mid-meeting without something concrete to reframe against.
This is the gap Prodinja's Feature-to-Feasibility walkthrough in the prototype is designed to address: rather than asking you to fill in an expected result, it's built to prompt you toward framing acceptance as ranges and properties from the start — a target rate instead of a single output, a constraint instead of a sample answer. It won't write your eval set for you, but it's meant to nudge the spec-writing conversation toward the mindset a non-deterministic feature actually requires, before engineering shows up asking what "done" means.
Key Takeaways
- Replace "expected result: X" with properties — testable, checkable rules like "must cite a source," "must not exceed N words," and "must stay on policy" that any valid output has to satisfy.
- Separate hard constraints from scored properties — constraints reject a response outright; properties can be measured on a spectrum and tracked as a rate.
- Acceptance becomes a target rate over an eval set, not a pass/fail verdict on one response — commonly 85–99%+ depending on the failure's severity.
- Build the eval set from representative production-like inputs, not hand-picked happy paths, or the target rate measures the wrong population entirely.
- QA's job splits in two: automated property checks per response, plus ongoing aggregate-quality monitoring in production, not a one-time pre-launch gate.
- Set the target rate as a risk decision, weighing failure severity, the journey moment it lands in, and the marginal cost of pushing the rate higher.
- Treat regression testing as a distribution question — a regression is a pass-rate drop or a shift in failure pattern, not a single flipped test.
Frequently Asked Questions
How do you write acceptance criteria for an AI feature that gives a different answer every time?
Define properties the output must always satisfy — grounding, length, policy, tone — rather than one expected string, then require those properties to hold at a target rate (commonly 90–99%) across a representative eval set instead of on any single response.
What is a non-deterministic feature spec?
A non-deterministic feature spec is a PRD or acceptance-criteria document written for a feature whose output varies by design (generative AI, ranking, recommendation). It replaces exact expected outputs with testable properties plus a target pass rate over an eval set.
How do you test probabilistic AI output without a golden answer to compare against?
You test each output against a checklist of properties (does it cite a source, stay under the word limit, stay on policy) using scripts, classifiers, or rubric scoring — then aggregate pass/fail results across an eval set into a single rate, rather than diffing against one golden response.
What pass rate should a generative feature be required to hit?
There's no universal number — it depends on failure severity and where the feature sits in the user journey. Low-stakes tone properties often tolerate 85–90%, most content-generation features target roughly 95%, and compliance-adjacent or safety properties typically need 99% or higher.
Can QA still write meaningful test cases for a generative AI feature?
Yes, but the test case changes shape: instead of asserting an exact output, QA writes an automated checker per property (a word-count script, a citation validator, a policy classifier) and reports pass rate across an eval set, plus ongoing production monitoring for drift.