Your Prompt Isn't Portable: Surviving a Model Migration Without a Behavior Rewrite

A prompt tuned to one model is a set of instructions calibrated against that model's specific quirks, not a universal specification. Upgrading or switching models changes formatting adherence, verbosity, and refusal thresholds even when the prompt text stays identical. Treat every model migration as a behavior change requiring regression testing, not a drop-in upgrade you get for free.

Quick Answer: A prompt is not portable across models by default. Freeze an eval set, run it against both the old and new model, diff the outputs on format/verbosity/refusals, and only cut over once the new model's failure modes are understood and patched.

Why Your Prompt Isn't Actually Portable

A prompt's words are only half of what makes it work; the other half is the model's implicit priors about how to interpret those words, learned during training and reinforcement. Two different models reading the exact same system prompt will resolve ambiguity differently, because their fine-tuning shaped different defaults for tone, structure, and caution.

This is the same lesson articulated in system-prompt-is-the-new-prd: a prompt functions like a specification, and specifications are read differently by different interpreters. A junior engineer and a senior engineer reading the same underspecified PRD ship different code — not because the PRD changed, but because the reader's judgment filled the gaps differently. Models do the same thing with prompts.

Three categories of implicit behavior tend to break on migration:

1. Format adherence — how strictly the model honors structural instructions like "respond only in JSON" or "use exactly three bullet points."
2. Verbosity — how much hedging, preamble, and explanation the model adds around the core answer.
3. Refusal thresholds — where the model draws the line on ambiguous, borderline, or adversarial requests.

None of these are documented per-prompt anywhere. You discover them empirically, and a migration resets that discovery process to zero unless you've built the harness to catch it.

The Illusion of "Better Model, Better Results"

Model providers market upgrades as strictly-better: higher benchmark scores, larger context windows, improved reasoning. That framing is true in aggregate and can still be false for your specific prompt. A model that's better at multi-step reasoning can simultaneously be worse at terse, rigid output formatting, because those two capabilities are often in tension during training.

Anthropic's own model deprecation and migration documentation explicitly recommends re-evaluating prompts and outputs when moving between model versions, rather than assuming compatibility — a tacit admission that even the same vendor's successive models are not behavior-equivalent. If the maker of the model tells you to re-test, that is not a courtesy suggestion.

What Tends to Break When You Migrate Models

Format adherence, verbosity, and refusal calibration are the three failure modes that show up most often in real migrations, and each has a distinct signature you can watch for. Understanding the mechanism behind each helps you write eval cases that actually catch the regression instead of ones that happen to pass by luck.

Format Adherence

Newer, more capable models are often trained with heavier emphasis on being "helpful" in a conversational sense, which can mean they editorialize around a requested format even when told not to. A model that reliably returned bare JSON on the old version might now wrap it in a markdown code fence, add a one-line summary before it, or occasionally include a trailing note like "let me know if you'd like adjustments."

This is precisely the failure mode covered in structured-outputs-shippable-json: the moment output stops being deterministically parseable, every downstream system consuming it breaks silently. A parser that expected clean JSON now throws on a stray markdown fence, and nobody notices until a support ticket arrives.

Common format regressions after migration:

• Extra prose wrapped around a requested JSON or XML payload
• Inconsistent list formatting (numbered vs. bulleted) where the old model was consistent
• Added disclaimers or caveats inside a field meant to hold clean data
• Subtly different key casing or field ordering in structured responses
• Markdown formatting (bold, headers) appearing inside outputs meant to be plain text

Verbosity Drift

Verbosity shifts in both directions and both are dangerous. A model can become chattier, padding answers with restated context and hedging language, which breaks prompts tuned for terse output in a chat UI with token or latency budgets. Or it can become terser, dropping detail your downstream logic depended on, such as reasoning traces used for auditability.

Verbosity drift symptoms to watch for:

Refusal Threshold Shifts

Every model has a different calibration for what counts as a borderline request — medical, legal, security, or otherwise sensitive topics near a policy boundary. A newer model trained with different safety tuning can refuse a request the old model handled fine, or conversely, comply with something the old model declined. Either direction is a behavior change your users will notice immediately, often at the worst moment: mid-conversation, mid-workflow.

A refusal that appears only on the new model is functionally identical to a crash from the user's point of view — the task silently stops working.

The Migration Playbook: Freeze, Run, Diff, Adjust

Migrating a prompt safely follows a four-step loop borrowed from standard regression-testing discipline: freeze your evaluation set, run it against both models in parallel, diff the outputs systematically, and adjust the prompt or routing logic before cutover. Skipping any one of these steps is how regressions reach production undetected.

Step 1: Freeze an Eval Set Before You Touch Anything

Before you even open a prompt editor, assemble a representative case set: real inputs (anonymized), including edge cases, adversarial inputs, and the inputs that have historically broken things. This set should be frozen — not modified — for the duration of the migration, so the diff you produce later is attributable to the model change alone, not to a shifting target.

This is the same discipline argued for in version-prompts-like-code-test-like-features: a prompt without a test suite is a function without unit tests, and you'd never ship a refactor of critical business logic without running the existing tests first. A model migration is exactly that kind of refactor, except the "refactor" is happening inside a vendor's training pipeline instead of your own codebase.

A usable eval set for a migration typically includes:

1. The 10-20 most common real request patterns your product actually receives
2. Every case that has previously caused a production incident or bug report
3. Deliberately ambiguous or borderline inputs near your refusal boundary
4. Inputs that stress format constraints (long lists, nested structures, unusual characters)
5. At least a few adversarial or prompt-injection-style inputs if your product is user-facing

Step 2: Run Both Models Against the Same Set

Run the frozen case set through the old model and the new model with the identical prompt, identical parameters (temperature, max tokens, system message), and identical inputs. Resist the temptation to "fix" the prompt for the new model before this baseline run — you need an unmodified comparison first, or you won't know what actually changed.

Capture full outputs, not just a pass/fail judgment, for every case. You need the raw text to diff against later, and you need it stored somewhere durable enough to compare a week from now if the migration stalls.

Step 3: Diff Systematically, Not Anecdotally

Comparing a handful of outputs by eye catches obvious breaks but misses the subtle ones — a slightly longer verbosity creep, a format shift that only appears in 1 of 20 cases. Structure the diff along the three failure axes above:

Nassim Taleb's writing on fragility is useful framing here: a system that looks robust under typical conditions but breaks under a distributional shift was never actually robust — it was untested outside its comfort zone. A prompt that "looks fine" on five happy-path examples but was never run against your edge cases has the same false robustness. The diff has to include the edge cases on purpose, not just the easy ones.

Step 4: Adjust Before Cutover, Not After

Once the diff surfaces regressions, you have three real options, roughly in order of preference: tighten the prompt's instructions to re-anchor the new model's behavior (adding explicit formatting constraints, few-shot examples, or stricter output schemas), add a lightweight validation/retry layer downstream that catches and corrects format drift, or delay the migration for that specific use case until the new model's behavior is better understood.

What you should not do is ship the new model with the old prompt unchanged just because most cases passed. "Most cases" is not the bar — the cases that broke are usually the ones with the highest cost when they fail in production, precisely because they were edge cases nobody thought to double-check by hand.

A Worked Example: When "Better" Regressed a Format

Consider a common real scenario: a PM migrates a customer support triage prompt from an older model to a newer, more capable one, expecting free improvement on reasoning quality. The prompt instructs the model to output a strict JSON object with three fields: category, priority, and summary, nothing else.

On the old model, this worked reliably across thousands of runs — bare JSON, every time, because that model's instruction-following for rigid format constraints was tight and its "helpfulness" instinct rarely overrode an explicit format directive. The downstream ticketing system parsed the output directly with JSON.parse() and had never needed a fallback path.

On the new model, the eval diff revealed something the PM hadn't anticipated: in roughly 1 in 8 cases, the model wrapped the JSON in a markdown code fence (```json ... ```) and, in a smaller fraction, appended a one-sentence note like "Let me know if you'd like me to adjust the priority." Both additions are, in isolation, reasonable helpful behavior — and both silently broke JSON.parse() on every affected ticket, routing them to a dead-letter queue instead of a support agent's queue.

What made this regression dangerous rather than just annoying:

• It didn't happen on every call, so early smoke testing (a handful of manual runs) missed it entirely.
• The failure was silent — no error thrown by the model, just malformed output the parser choked on downstream.
• It disproportionately hit longer, more nuanced tickets, which correlates with exactly the tickets a support team most needs triaged correctly.

The fix, once the eval diff caught it, was two-pronged: the prompt was tightened with an explicit constraint ("Output raw JSON only. Do not use markdown code fences. Do not include any text before or after the JSON object.") plus a few-shot example showing the exact expected output, and the downstream parser added a defensive strip-and-retry step that removes code fences before parsing as a safety net. Neither fix alone was sufficient; the prompt fix reduced the frequency, and the parser fix caught what the prompt fix missed.

Catching Regressions Before Users Do

Running this migration playbook manually against a spreadsheet of outputs is workable for a one-time switch, but most teams migrate models more than once — a provider deprecates a version, a new one launches with better pricing, or a competitor's benchmark makes a switch worth evaluating. Re-deriving the eval set and comparison process from scratch each time is where the discipline quietly erodes.

That only works, of course, if the eval set itself was built well in the first place — which loops back to understanding what your prompt is actually specifying. The foundational patterns behind writing prompts precise enough to have a meaningful diff in the first place are covered in prompt-design-complete-guide.

Key Takeaways

• A prompt encodes model-specific quirks, not universal instructions — treat any model change as a behavior change requiring regression testing.
• Three failure modes recur across migrations: format adherence, verbosity drift, and refusal threshold shifts — each has a distinct signature worth checking separately.
• A "better" model on benchmarks can still regress your specific prompt, because capabilities like reasoning depth and rigid format adherence are often in tension during training.
• Freeze your eval set before touching the prompt, so any diff you observe is attributable to the model change alone, not a shifting comparison target.
• Diff systematically along format, verbosity, and refusals rather than spot-checking a handful of outputs by eye — subtle regressions hide in the cases you don't manually review.
• Fix before cutover, not after: tighten the prompt, add downstream validation, or delay the migration for that use case until the new model's behavior is understood.
• Silent format breaks are the most dangerous regression because there's no error thrown — just malformed output a parser chokes on downstream, discovered only when tickets pile up.