Fine-Tune, Prompt, or Retrieve? A Cost-Benefit Decision Tree

Fine-tuning changes a model's weights; prompting and retrieval-augmented generation (RAG) change what the model sees at inference time, leaving weights untouched. Reach for fine-tuning only when you have a stable task, thousands of labeled examples, a tight latency budget, and volume high enough to justify hosting. Everyone else — most teams, most of the time — should max out prompting and RAG first.

Quick answer: Start with prompting. Add RAG the moment answers depend on facts that change or live outside the model. Reach for fine-tuning only when the task is stable, the data volume is large, latency is tight, and cost-per-token at your volume favors it — and even then, pair it with RAG for anything fact-based.

The Three Levers: What Prompting, RAG, and Fine-Tuning Actually Change

Prompting, RAG, and fine-tuning aren't three strengths of the same dial — they modify different layers of the system. Prompting and RAG both operate at inference time and leave the model itself untouched. Fine-tuning is the only one of the three that changes weights. Confusing these three is why so many fine-tuning projects fail to fix the problem they were funded to solve.

• Prompting (in-context learning): Everything the model needs — instructions, examples, output format — lives in the request itself. Nothing about the model changes between calls; you're steering a fixed system with better inputs.
• RAG (retrieval-augmented generation): A retrieval step (vector search, BM25, or a hybrid of both) pulls relevant passages from an external corpus and inserts them into the prompt before generation. Still no weight changes — it widens what the model can "see" per call.
• Fine-tuning (full or adapter-based, e.g. LoRA): Additional training on labeled examples adjusts the model's weights, baking a behavior, tone, or output format into the model itself rather than into the prompt.

Not all fine-tuning is equally expensive. Full fine-tuning retrains every weight in the model; adapter methods like LoRA — introduced by Microsoft researcher Edward Hu and co-authors in their 2021 paper "LoRA: Low-Rank Adaptation of Large Language Models" — freeze the base model and train a small set of additional parameters instead. That difference alone can cut training cost and hosting complexity by a wide margin, which is worth knowing before assuming "fine-tuning" means the expensive, full-retrain version.

Fine-tuning is the most expensive, slowest-to-iterate lever of the three — reach for it last, not first.

Here's the myth worth killing: teams treat fine-tuning as the "serious" option because it sounds like heavier engineering. Andrej Karpathy's widely cited "State of GPT" talk lays out the opposite order — prompting first, then retrieval and tools, with fine-tuning reserved for last, after cheaper rungs are exhausted. It's the most expensive and slowest-to-iterate lever, not the default one.

The knowledge question settles this further. A 2023 research paper from Microsoft and Technion researchers, Ovadia et al.'s "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs," found retrieval consistently outperformed fine-tuning at injecting both existing and entirely new factual knowledge into a model. If your problem is "the model doesn't know something," fine-tuning is often the wrong tool even before you weigh cost.

For a deeper look at how these customization choices sit inside a broader AI program, our complete guide to AI data strategy covers the data-readiness work that has to happen before any of this decision matters.

The Decision Tree: Four Questions That Determine Your Path

The decision comes down to four variables, asked in order: whether your answers depend on facts that change (freshness), whether you have enough labeled examples (volume), how tight your latency budget is, and what a query costs at your real volume. Walk through them in sequence and the right answer is usually obvious long before you reach fine-tuning.

Q1: Do correct answers depend on facts that change over time
    or live outside the model's training data?
      ├─ YES → You need RAG. Fine-tuning cannot substitute for retrieval here.
      │         (Also fine-tune only if Q2-Q4 below are ALSO yes.)
      └─ NO  → Continue to Q2.

Q2: Do you have 1,000+ stable, labeled examples of the exact
    behavior or output format you want?
      ├─ NO  → Stay with prompting. Revisit once you've collected more data.
      └─ YES → Continue to Q3.

Q3: Is your latency budget tight (sub-200ms) at meaningful,
    sustained volume?
      ├─ NO  → Prompting is enough; add RAG only if knowledge is the gap.
      └─ YES → Continue to Q4.

Q4: Does per-token API cost at your real volume exceed the cost
    of hosting a fine-tuned or distilled model?
      ├─ NO  → Prompting/API still wins on total cost of ownership.
      └─ YES → Fine-tune. Pair with RAG if any part of the task is fact-based.

Freshness: the question that eliminates "fine-tune only" fastest

If the correct answer changes next week — prices, inventory, policy, a support article that just got updated — fine-tuning alone is the wrong tool. A fine-tuned model is a snapshot; it doesn't know what happened after its training cutoff. RAG is what keeps answers current without retraining anything.

Data volume: the threshold before fine-tuning is even viable

Fine-tuning needs a real dataset, not a handful of examples. As a rough floor, fewer than a few hundred labeled examples rarely produces a fine-tune that beats a well-built prompt — you'll overfit to quirks in a small sample. Thousands of clean, representative examples is where it starts to pay off.

Latency budget: when shorter beats smarter

Long few-shot prompts and retrieved context both add tokens, and tokens add latency and cost. A fine-tuned model can bake the "few-shot" behavior into its weights, shrinking the prompt to just the new input. That only matters if your use case genuinely needs sub-200ms responses at real concurrency — most chat and drafting experiences don't.

Cost per token: where volume flips the math

API pricing is usage-based; hosting is a fixed cost that has to be justified by throughput. Below roughly tens of thousands of queries a month, the fixed cost of hosting rarely beats pay-per-token API pricing. Above that, and with prompts that are long or repetitive, the balance can shift — which is exactly what the cost table below quantifies.

The Real Cost Table: What Each Approach Costs at Scale

Cost isn't just the sticker price of a training run — it's setup time, per-query cost at your real volume, and the ongoing labor of keeping the system correct. RAG and fine-tuning both add a new maintenance surface that prompting alone doesn't have, just different ones.

A few honest caveats sit underneath this table:

1. Fine-tuning's training cost is usually the smallest line item. According to OpenAI's own fine-tuning documentation and pricing pages, the training run itself is typically far cheaper than the ongoing hosting and inference cost of serving a custom model at volume.
2. Inference costs keep falling for everyone. Stanford HAI's AI Index Report has tracked the cost of GPT-3.5-class inference falling by roughly two orders of magnitude between 2022 and 2024 — which quietly moves the breakeven point in favor of "just call the API" every year.
3. Enterprises are voting with their deployments. a16z's 2024 survey of enterprise generative AI buyers, "16 Changes to the Way Enterprises Are Building and Buying Generative AI," found retrieval-based customization far more commonly deployed than fine-tuning — the market has already made the "prompting + RAG first" call.

A quick illustrative example (hypothetical numbers, to show the shape of the math): say a team runs 50,000 classification calls a day through a mid-tier API. At a few cents per call, that's roughly $1,000–$3,000 a month in inference cost alone, before any retries or retrieval overhead.

A small fine-tuned or distilled model, self-hosted on a modest GPU instance, could plausibly bring per-call cost down by an order of magnitude. But only once you've absorbed the training run and a few hundred to a few thousand dollars a month in hosting. The crossover point is a real calculation you should run with your own numbers, not an assumption.

For the deeper unit-economics behind these numbers — how to model cost-per-query against the value of an interaction — see our guide to AI economics. And if the real bottleneck turns out to be your prompt rather than your architecture, our complete guide to prompt design covers how far a well-built prompt can go before you need anything heavier.

Two Scenarios Walked Through the Tree

Frameworks are only useful once you run a real case through them. Here are two — a customer support bot and a domain classifier — that land on opposite ends of the tree, on purpose.

Scenario 1: The customer support bot

A support bot has to answer questions about policies, pricing, and order status that change weekly, using a conversational tone, at volume that scales with support ticket load.

• Q1 — Freshness: Yes. Prices, order status, and policy documents change constantly. RAG is mandatory regardless of anything else in the tree.
• Q2 — Data volume: Usually thin. Most teams have a large document corpus (help docs, policy pages) but comparatively little labeled conversation data showing exactly how to phrase answers.
• Q3 — Latency: Moderate. A one-to-two-second response is fine for a chat interface; this doesn't push toward fine-tuning.
• Q4 — Cost per token: Tolerable. Each deflected ticket is worth real support cost, so per-token API pricing is easy to justify.

Verdict: prompting + RAG, no fine-tuning. The bottleneck is knowledge freshness and retrieval quality, not model behavior — fine-tuning would bake in facts that are wrong within a month. This is exactly the pattern a16z's enterprise survey found dominating real deployments.

Scenario 2: The domain classifier

A domain classifier sorts inbound documents, tickets, or transactions into a fixed, stable taxonomy — a narrow, repetitive task run continuously at high throughput.

• Q1 — Freshness: No. The taxonomy is stable; classification doesn't depend on facts that change week to week.
• Q2 — Data volume: Large. Years of historical, labeled examples usually already exist as a byproduct of the business process — often tens of thousands of rows.
• Q3 — Latency: Tight. Synchronous processing at high queries-per-second is common for this kind of pipeline.
• Q4 — Cost per token: Volume-sensitive. Millions of classifications a month makes per-token API pricing add up fast; a smaller, fine-tuned or distilled model amortizes better.

Verdict: fine-tune (or distill) a compact model. RAG buys almost nothing here — there's no external fact to look up, only a pattern to recognize. This is the case the decision tree is built to catch: fine-tuning as the correct, not the default, answer.

Notice what didn't change between the two: the tree, the questions, the honesty about trade-offs. What changed was the answers, because the underlying data and latency reality were genuinely different. That's the whole point of running the questions in order instead of picking an architecture because it's fashionable.

If either scenario also needs to handle spoken input — a voice-driven support line, for instance — the retrieval and latency math shifts again; our guide to multimodal and voice AI walks through what changes when text stops being the only modality. And whichever path you choose, bound the failure modes explicitly — our AI safety guardrails guide covers how to catch a bad retrieval or a bad fine-tune before it reaches a user.

Document the Trade-off Before You Commit Engineering

The riskiest moment in this decision isn't choosing wrong — it's choosing before anyone has written down why. A one-line Slack message ("let's fine-tune it") skips past the freshness question, the data-volume threshold, and the cost table entirely.

This is a spec problem as much as a technical one. Before engineering starts building a retrieval pipeline or curating a fine-tuning dataset, someone should have written the trade-off down.

At minimum, that write-up should cover:

• What freshness the task actually needs, and whether that rules out "fine-tune only."
• How much labeled data really exists — not what's hoped for.
• What the true latency budget is, measured, not assumed.
• What the approach costs at real projected volume, not a demo's worth of traffic.

As your production system runs, the data it generates — which retrievals actually got used, which classifications needed correction — becomes the raw material for whichever path you take next. Our piece on turning usage into a data flywheel covers how to capture that loop deliberately instead of losing it.

Key Takeaways

• Prompting and RAG change what the model sees; fine-tuning changes the model. Match the lever to the actual problem — a knowledge gap needs retrieval, not retraining.
• Freshness is the fastest disqualifier for "fine-tune only." If facts change, you need RAG regardless of what else the tree says.
• Fine-tuning needs real data volume to pay off — thousands of clean, representative labeled examples, not a few hundred.
• Fine-tuning is not the "serious" default. Karpathy's prompting-first ladder and Ovadia et al.'s knowledge-injection research both point the same direction: exhaust the cheaper rungs first.
• Cost is a volume game. API pricing scales with usage; hosting is a fixed cost that only pays off above a real throughput threshold — do the arithmetic before committing.
• The two scenarios in this piece land on opposite answers on purpose. The tree is only useful if you run your actual numbers through it instead of copying someone else's verdict.
• Write the trade-off down before engineering starts building. A visible, revisable spec beats a one-line architecture decision every time.