Same Prompt, Different Answer: Making Peace With Non-Deterministic AI

Identical prompts produce different outputs because LLMs sample from a probability distribution over possible next tokens rather than computing one fixed answer, and because GPU batching, floating-point math, and silent model updates introduce variance even at temperature=0. Treat every output as a draw from a distribution, not a deterministic return value — and build QA around acceptable ranges, not exact strings.

Quick Answer: LLMs generate text by sampling from a probability distribution, and that sampling process — plus non-deterministic GPU batching and floating-point rounding — means the same prompt can legitimately produce different valid outputs. QA for AI features should score output distributions against a rubric, not assert one golden string.

Traditional software is a calculator: same inputs, same output, every time, forever. An LLM is closer to a dice roll with a memory — weighted heavily toward likely answers, but never mechanically guaranteed to land on the same one twice. Treating an AI feature like a deterministic function is the single most common QA mistake teams make when they ship their first LLM-powered feature, and it produces test suites that are perpetually red for the wrong reasons.

Why do LLMs give different answers to the same prompt?

LLMs give different answers to identical prompts because the model doesn't select one "correct" next token — it computes a probability distribution over the entire vocabulary and then samples from it, and that sampling step is where determinism dies. Even when the underlying weights never change, the decoding process is designed to introduce variation.

The sampling mechanism, briefly

At each generation step, the model outputs a probability for every possible next token — tens of thousands of candidates. Decoding parameters decide how that distribution gets turned into an actual choice:

• Temperature scales the distribution before sampling. temperature=0 collapses it toward always picking the single highest-probability token (greedy decoding), while higher temperatures flatten the distribution and increase the chance of picking a lower-probability token.
• Top-p (nucleus sampling) restricts sampling to the smallest set of tokens whose cumulative probability crosses a threshold, discarding the long tail.
• Top-k simply caps the candidate pool to the k most likely tokens before sampling.

Each of these is a deliberate design choice, not a bug — a fixed function would make the tokenization and vocabulary mechanics described in our tokens explainer irrelevant to output diversity, and diverse, natural-sounding text is often the entire point of the product.

Why temperature 0 still isn't fully reproducible

Setting temperature=0 should, in theory, make decoding fully greedy and deterministic — always pick the highest-probability token. In practice it usually gets you closer to reproducible, not guaranteed reproducible, for reasons that have nothing to do with the sampling algorithm itself:

1. GPU batching non-determinism. Inference providers batch multiple users' requests together for throughput. The order and grouping of operations within a batch can shift slightly run to run, and floating-point arithmetic is not strictly associative — (a + b) + c can differ infinitesimally from a + (b + c) — so tiny numerical differences compound across billions of operations.
2. Mixture-of-experts routing. Some model architectures route each token through a subset of specialized sub-networks chosen partly based on what else is in the current batch, meaning the same input token can occasionally get routed differently depending on unrelated concurrent traffic.
3. Silent backend updates. Providers routinely update serving infrastructure, quantization strategies, or the underlying model checkpoint behind a stable-looking API name, without a version bump you can pin against.
4. Hardware and kernel differences. The same model run on different GPU generations, or with different low-level kernel implementations, can produce different floating-point rounding — invisible until you compare outputs side by side.

The upshot for anyone testing an AI feature: "deterministic" is a spectrum, not a switch. temperature=0 narrows the variance dramatically, but it is not a substitute for a QA process that tolerates a range of acceptable outputs.

Why does this matter for QA — what breaks when you test AI features like normal software?

Testing an LLM feature the way you'd test a login form — assert the output equals an expected string — breaks in two directions at once. False failures flag legitimate, high-quality outputs that simply used different phrasing than your golden string. False passes let a genuinely wrong answer through if it happens to match brittle keyword or format checks. Both erode trust in the test suite until people stop looking at it.

The golden-string trap

A "golden-string" test hardcodes one expected output and fails anything that doesn't match character-for-character (or close to it). This pattern is inherited from deterministic software testing, and it's tempting because it's cheap to write and easy to reason about. It fails an LLM feature almost immediately:

Golden-string tests also create a second-order problem: engineers learn to distrust red builds, because so many are false alarms from paraphrasing rather than real regressions. Once a team starts routinely overriding or skipping failing AI tests "because it's probably fine," you've lost the entire point of having a test suite.

What "acceptable range" QA actually looks like

Reframing QA from exact-match to acceptable-range means defining, up front, what a family of correct answers looks like rather than one instance of it. In practice this means:

• Scoring rubrics that check for required facts, tone, and constraints (e.g., "mentions the refund policy," "stays under 200 words," "does not promise a specific dollar figure") rather than matching text.
• LLM-as-judge evaluation, where a second model scores each output against a rubric — useful for scale, but it introduces its own non-determinism and needs periodic human spot-checking to stay calibrated.
• Multiple-sample testing, running the same prompt N times and measuring the pass rate across the batch, not a single pass/fail.
• Regression sets with reference answers, used for similarity scoring (e.g., semantic similarity via embeddings) rather than identity matching.
• Guardrail checks for hard failure modes — PII leakage, unsafe content, broken JSON schema — which should remain closer to exact-match, because these are binary correctness properties, not stylistic ones.

Not everything about an AI feature is non-deterministic in the ways that matter. Schema validity, safety constraints, and factual claims about your own product are legitimate places to hold a hard line — reserve exact-match testing for those, and reserve range-based evaluation for everything shaped by natural language variation.

How should PMs think about reproducibility when writing acceptance criteria?

Acceptance criteria for an AI feature should describe the properties a correct answer must have, not the literal text it must contain — because "the model must return this exact sentence" is a criterion that will fail the moment the underlying model updates. Write criteria the way you'd brief a human reviewer, not the way you'd write a unit test assertion.

Practical rewrite pattern

A useful discipline is translating every "must output X" criterion into a rubric line before it goes into a spec:

1. Instead of: "Response must say 'Your refund will be processed in 5-7 business days.'" Write: "Response must state a specific refund timeframe consistent with policy (5-7 business days), without inventing an alternate figure."
2. Instead of: "Response must match this exact JSON: {"status": "approved", "reason": null}" Write: "Response must be valid JSON matching this schema, with status in {approved, denied, pending} and reason populated when status != approved."
3. Instead of: "Summary must be this sentence, verbatim." Write: "Summary must cover the three required facts, stay under 60 words, and avoid speculative claims not present in the source."

This isn't just a testing nuance — it changes how a PM writes the spec in the first place. A living PRD that captures acceptance criteria as rubric statements ages far better than one full of hardcoded example outputs, because the rubric survives a model swap and the literal string doesn't.

Model updates as a first-class QA input, not a surprise

Because providers update models behind stable-looking endpoint names, a QA plan for an AI feature needs an explicit answer to "what happens when the underlying model changes under us." That includes re-running your eval suite against any new model version before adoption, tracking knowledge cutoff shifts that might change factual answers, and version-pinning where the provider allows it, with a deliberate review gate before moving off a pin.

What does "good" LLM QA actually look like day to day?

Good LLM QA runs a small suite of deterministic guardrail checks alongside a larger suite of distribution-based evals, tracks pass rate and variance over time rather than a single pass/fail signal, and treats occasional low-severity drift as expected — reserving escalation for drops in aggregate score, not any single failed sample. It's an ongoing measurement discipline, closer to monitoring than to a binary release gate.

A minimal day-to-day loop

• Run the eval set on every meaningful change — prompt edits, model version bumps, RAG/embedding index updates — not just at release time.
• Sample N ≥ 5 generations per test case where output variability is expected, and score the aggregate rather than a single draw.
• Separate guardrail failures from quality failures in your reporting, since a PII leak and a slightly-verbose-but-correct answer are not the same severity.
• Track score trend lines over time, so a slow quality regression (e.g., from an upstream model update) is visible before a user complains.
• Keep a small human-reviewed calibration set to sanity-check any LLM-as-judge scoring, since the judge model has its own non-determinism.

This is a genuinely different operating rhythm from a CI suite that's either green or red. It's closer to how a data or ML team monitors a recommendation model in production — continuous measurement against a moving target — than to how a frontend team runs Jest.

Where Prodinja fits into this

Key Takeaways

• LLMs sample from a probability distribution rather than computing one fixed answer, so identical prompts can legitimately produce different valid outputs — this is a design property, not a bug.
• temperature=0 reduces variance but doesn't guarantee reproducibility, because GPU batching, floating-point non-associativity, mixture-of-experts routing, and silent backend updates all introduce non-determinism outside the sampling algorithm.
• Golden-string tests are the wrong tool for AI features — they produce false failures on valid paraphrases and false passes on wrong answers that happen to match a keyword or format.
• Reframe acceptance criteria as rubrics describing required properties, not literal expected text, so specs survive model updates instead of breaking on the next version bump.
• Reserve hard exact-match testing for binary correctness properties — schema validity, safety, and factual claims about your own product — and use distribution/rubric-based scoring for everything shaped by natural language variation.
• Track pass rate and score trends over time, not single pass/fail runs, since occasional low-severity drift is expected and escalation should be reserved for aggregate regressions.
• Model updates behind stable API names should be a planned QA input, with a re-run of your eval suite before adopting any new model version.