When the Model Returns Garbage: Retry, Repair, and Validate Malformed Outputs

When a model returns garbage, the fix isn't a better prompt — it's a resilience layer: validate every response against a schema, attempt one automated repair via a follow-up prompt, retry with backoff inside a hard budget, and degrade gracefully if all else fails. Treat malformed output as an expected failure mode, not an anomaly, and design for it the way you'd design for a flaky network call.

Quick Answer: Don't chase a prompt that never fails — build a four-rung ladder (validate → repair → retry → degrade) with hard retry limits and a cost ceiling, so a bad response becomes a handled path instead of a user-facing error.

Malformed Output Is an Engineering Problem, Not a Prompt Problem

Even a well-designed prompt with a tight schema will occasionally return output that doesn't parse, and no amount of prompt tuning eliminates that tail risk. Language models are probabilistic text generators; a truncated response, a stray comment, an extra field, or a hallucinated enum value is a normal output of that process, not a bug you can prompt your way out of. The moment you accept that framing, the design question changes from "how do I get a perfect prompt" to "how do I build a system that survives an imperfect one."

This distinction matters because teams that treat malformed output as a prompt-engineering failure keep iterating on wording indefinitely, chasing a zero that doesn't exist. Teams that treat it as a reliability engineering problem instead build the same kind of defense-in-depth they'd build around any unreliable external dependency — a database that times out, a third-party API that occasionally 500s. Anthropic's own guidance on tool use and structured outputs is explicit that consuming code should validate and handle malformed calls rather than assume conformance, and OpenAI's structured-outputs documentation frames constrained decoding as a way to reduce, not eliminate, schema violations.

A useful mental model borrows from Michael Nygard's Release It!, which popularized designing distributed systems around the assumption that any given call will eventually fail, and asking what happens next rather than whether it happens. Apply the same posture to an LLM call: assume a fraction of responses will be malformed, and design the surrounding system so that fraction never reaches a user as a raw error.

Where This Fits in the Prompt Lifecycle

Output resilience sits downstream of prompt design, not instead of it. A tight system prompt with explicit output constraints — the discipline covered in why the system prompt is the new PRD — genuinely lowers the malformed-output rate. But "lower" isn't "zero," and the resilience layer is what closes that remaining gap. If you haven't yet nailed down the foundational prompt structure, the complete guide to prompt design is the right starting point before layering on retry logic.

Validate First: Catching Garbage Before a User Ever Sees It

Validation is the first and cheapest rung of the ladder: check every LLM response against a strict schema before any downstream code touches it, and treat "parses but fails validation" as equally serious as "doesn't parse at all." A response that's syntactically valid JSON but has the wrong types, a missing required field, or an out-of-range enum is just as dangerous as unparseable text — it's the failure mode that slips past a naive JSON.parse try/catch.

Build validation as its own explicit step, not an afterthought inside a try/catch block:

1. Parse — attempt to deserialize the raw string; catch syntax errors immediately.
2. Schema-check — validate the parsed object against a strict schema (types, required fields, enums, ranges).
3. Semantic-check — apply business-logic rules a schema can't express (a discount_pct field that's technically a valid number but is 150%).
4. Classify the failure — log which rung failed and why; this classification is what feeds your repair prompt and your metrics later.

This is exactly the discipline covered in turning structured outputs into shippable JSON: a schema isn't documentation, it's an enforceable contract, and the contract only has teeth if something actually checks it on every call.

What to Validate Beyond "Is It JSON"

Skipping the semantic layer is the most common gap. A response can sail through schema validation — every field present, every type correct — and still be operationally wrong, like a priority field the model set to "urgent" for every single row.

The Repair Ladder: Validate, Repair, Retry, Degrade

When validation fails, don't reach straight for a full retry — attempt a cheap, targeted repair first, because a follow-up prompt describing the exact validation error usually fixes the response for a fraction of the cost of starting over. The four rungs, in strict order of increasing cost, are validate, repair-via-follow-up, retry-from-scratch, and graceful degradation.

Rung 1 — Validate (covered above) is where you decide whether you have a problem and what kind.

Rung 2 — Repair sends the model its own malformed output plus the specific validation error, and asks it to fix only that. This is cheaper than a fresh retry because the model already did the hard reasoning work; it's usually just being asked to correct a formatting slip.

The previous response failed validation with this error:
"Missing required field 'severity' at path $.incidents[2]"

Here is the original response:
<paste raw output>

Return a corrected version that fixes only this issue,
preserving all other content exactly.

Rung 3 — Retry discards the malformed response entirely and re-issues the original request, optionally with a slightly reworded prompt or a lower temperature, on the theory that the failure was a one-off sampling artifact rather than a systematic misunderstanding.

Rung 4 — Degrade is what happens when repair and retry both exhaust their budget: return a partial result, fall back to a simpler deterministic path, or surface a clear "we couldn't complete this, here's what we have" state — never a raw stack trace or a silent blank screen.

A repair prompt should never repeat the entire original instructions from scratch — that's a retry wearing a repair costume, and it costs the same as one.

Repair vs. Retry: When Each One Wins

Setting Retry Limits and Cost Budgets

Cap every repair-and-retry loop at a small, fixed number of attempts — commonly 1 repair attempt plus 1-2 retries — and track the cost of the whole loop against a per-request budget, not just the cost of a single call. Without an explicit ceiling, a resilience loop can quietly turn one failed call into five or six billed calls before anyone notices, and Google's Site Reliability Engineering literature on error budgets is a useful frame here: reliability work itself has a cost, and past a certain point more retries buy diminishing returns while burning budget that should go to genuinely new requests.

Concrete rules worth adopting as defaults:

• Cap total attempts per request at 3, structured as: 1 original call, 1 repair, 1 retry (adjust the mix, not the total, to your failure profile).
• Use exponential backoff between retries — even a few hundred milliseconds — since a transient rate-limit or overload condition is a common co-occurring cause of malformed truncated responses.
• Track cost-per-successful-response, not cost-per-call. A loop that averages 1.3 calls per success is healthy; one averaging 2.5 is a sign your validation or prompt needs work, not more retry budget.
• Set a wall-clock timeout on the whole loop, separate from the per-call timeout, so a user-facing request never hangs waiting for a third repair attempt.
• Log every rung a request passes through (validate-fail, repair-attempted, repair-succeeded, retry-attempted, degraded) as structured telemetry — this is the data that lets you calibrate limits with evidence instead of guesswork.

Circuit-breaker thinking applies at the aggregate level too: if malformed-output rates spike across many requests in a short window, that's a signal to pause and investigate — a model version change, a prompt regression, an upstream data-format shift — not to individually retry harder on every request.

This is also where the discipline of versioning prompts like code and testing them like features pays off directly: a spike in the repair-loop's log volume is often the first symptom that a prompt change shipped a regression, well before anyone files a bug report.

Worked Example: Turning a 5% Parse-Failure Rate Into a Handled Path

A structured-extraction feature that turns free-text customer feedback into a fixed JSON shape (sentiment, theme, priority) is a realistic case where an unconstrained prompt sees roughly a 5% malformed-output rate, and a validate-then-repair loop can bring the user-visible failure rate close to zero without changing the underlying model. This kind of extraction task — free text in, strict schema out — is exactly where malformed output shows up most, because the input is unpredictable even when the output contract is rigid.

Here's the shape of a loop that handles it, in order:

1. Call the model with a system prompt specifying the exact JSON shape and a request to return only that JSON.
2. Validate the response against a schema (required fields: sentiment enum, theme string, priority enum). Roughly 1 in 20 responses fails here — most commonly a missing priority field or a rogue explanatory sentence prepended before the JSON.
3. Repair by sending back the specific validation error and the raw output, asking for a corrected JSON-only response. In this kind of scenario, the repair step alone typically resolves the majority of the initial failures, because the model already extracted the right information — it just needs to reformat it.
4. Retry the small remainder that repair doesn't fix — usually a case where the input text itself was ambiguous enough that the model needs to try the whole extraction again, sometimes with a slightly clarified prompt.
5. Degrade the rare final failures (well under 1% of total requests) into a needs_manual_review status rather than a dropped record or a user-facing error — the record still gets processed, just by a human instead of silently vanishing.

The result: what started as a 1-in-20 failure rate at the raw-model layer becomes, from the user's perspective, a system that essentially never visibly fails — it just occasionally routes a record to manual review, which is a wildly different experience than an error toast.

This pattern generalizes well beyond feedback extraction. Any workflow that maps unstructured input to structured output — support-ticket triage, meeting-notes-to-action-items, requirements-to-schema — hits a similar failure shape and benefits from the same four-rung ladder.

Designing the Loop So It Doesn't Spiral

A retry-and-repair loop is itself a feedback loop, and like any feedback loop it can reinforce itself into a runaway spiral if the reinforcing path (retry more) isn't checked by a balancing path (a hard limit); thinking about it explicitly in loop terms — not just as "add a retry" — is what keeps it stable. A loop with unlimited retries and no cost ceiling is a reinforcing loop with no brake: more failures trigger more retries, more retries generate more load and more cost, and under load, latency and error rates often get worse, not better.

The practical translation of "balancing loop" for this specific problem is everything in the retry-limits section above: a hard attempt cap, a wall-clock timeout, and a graceful-degrade path that stops the loop from ever reaching an unbounded state. None of that requires the visualization to design correctly — but naming the failure shape as a reinforcing loop is what makes the fix (a balancing constraint) obvious rather than an afterthought bolted on after an incident.

Key Takeaways

• Malformed output is a normal, expected failure mode of any LLM call, not a sign your prompt is broken — design the system around that assumption rather than chasing a perfect prompt.
• Validation has at least three layers — syntax, schema, and semantic — and skipping the semantic layer is the most common gap that lets operationally wrong-but-well-formed output through.
• Repair before you retry. A targeted follow-up prompt describing the exact validation error is cheaper and usually more effective than discarding the response and starting over.
• Cap total attempts (commonly around 3) and track cost per successful response, not per call, so the resilience loop doesn't quietly inflate your bill or your latency.
• Graceful degradation is a first-class rung, not an admission of failure — routing an unresolvable case to manual review beats a silent drop or a raw error message every time.
• Think of the loop itself as a feedback loop and make sure it has a balancing constraint (a hard limit), or a reinforcing failure spiral is only a bad day away.