RAG or Fine-Tune? A Feasibility Guide for Getting Your Data Into an LLM Feature

Retrieval-augmented generation and fine-tuning solve different problems. RAG injects fresh, external knowledge into a prompt at the moment of the query, while fine-tuning bakes a fixed style, format, or skill into the model's weights. If your feature needs facts that change, reach for RAG; if it needs consistent behavior, fine-tune; most production systems eventually need both.

Quick answer: Use RAG when the underlying facts change — product docs, prices, policies, inventory. Use fine-tuning when you need consistent tone, structure, or a narrow skill the model doesn't have out of the box. The two are not competitors; most mature LLM features combine a retrieval layer with a lightly fine-tuned (or well-prompted) model, and the real feasibility risk sits in retrieval quality, not in picking the "right" technique.

This question sits inside the larger feasibility question every technical PM eventually has to answer before committing engineering time — covered in more depth in our complete guide to AI feature feasibility. This piece narrows in on one recurring decision: once you know your feature needs the model to "know" something about your company, how do you decide whether that knowledge belongs in a retrieval index or in the model's weights?

RAG vs. Fine-Tuning: The Core Distinction

Retrieval-augmented generation and fine-tuning change different parts of the system. RAG retrieves relevant text from an external store and stuffs it into the prompt at inference time — the model's weights never change. Fine-tuning updates the weights themselves, teaching the model new patterns of behavior that persist with no retrieval step at all.

The distinction traces back to the term's origin. Patrick Lewis and coauthors at Facebook AI Research (now Meta AI) coined "retrieval-augmented generation" in their 2020 NeurIPS paper, pairing a pretrained sequence-to-sequence model with a dense vector index so the model could pull in passages it was never trained on. Their finding still holds: retrieval-augmented models produce more specific, more verifiable, and more current answers than models relying purely on what's compressed into their parameters — because the knowledge lives outside the model, not inside it.

Fine-tuning solves a different problem entirely. It doesn't add facts; it adjusts behavior. OpenAI's own fine-tuning guidance is explicit about this: fine-tuning is best suited to teaching a model a specific tone, output format, or complex multi-step instruction pattern, while injecting new or frequently changing knowledge is better handled through retrieval. Fine-tune a model to sound like your brand voice, and it will still get last week's pricing wrong if pricing changed yesterday.

The table makes the split concrete: RAG is an infrastructure problem (index, retrieve, rank), and fine-tuning is a data-curation problem (examples, evaluation, re-training cadence). Neither is inherently cheaper — they're expensive in different currencies. RAG spends effort on retrieval engineering; fine-tuning spends it on labeled examples and re-training discipline.

Reach for RAG when:

• The underlying facts change weekly, daily, or in real time
• You need per-customer or per-tenant knowledge the base model was never trained on
• You need to cite or show sources for what the model says
• The knowledge base is too large or too dynamic to fit in a training set

Reach for fine-tuning when:

• The model needs to reliably follow a rigid output format (JSON schema, a specific report structure)
• You need a consistent voice or persona across thousands of interactions
• The task requires a narrow skill the base model performs inconsistently (a domain-specific classification, a house style of summarization)
• The "knowledge" in question is really a pattern of reasoning, not a fact that could go stale

When Retrieval-Augmented Generation Is the Right Call

RAG is the right call whenever the feature's value depends on knowledge that's larger than a context window or that changes faster than you can retrain a model. It works by converting your documents into embeddings, storing them in a vector index, and pulling the most relevant chunks into the prompt at query time — nothing about the base model itself changes.

Three signals mean you're looking at a RAG problem:

1. The source material updates on its own schedule. Product documentation, pricing tables, policy pages, and support macros change independent of your release cycle — a fine-tuned model would go stale within weeks.
2. You need traceability. If the feature has to show "here's the doc this answer came from," that citation only exists if you retrieved a real passage rather than generated one from memory.
3. The corpus is too big to compress into training examples. A wiki with ten thousand articles doesn't fit into a few hundred fine-tuning examples, but it fits fine into a vector index.

The mechanics are deceptively simple to describe: chunk your documents, embed them, store the vectors (in something like pgvector, a managed vector database, or a well-tuned BM25 keyword index), retrieve the top-k matches for a query, and pass them into the prompt alongside the user's question. That's the whole pipeline in one sentence.

Executing it well is another matter. Every one of those steps has failure modes — bad chunk boundaries, mismatched embeddings, a top-k that's too small or too noisy — and each one degrades the final answer in a way that's easy to miss in a demo and painful in production.

Before greenlighting a RAG build, it's worth running the kind of structured check laid out in our AI feature data-readiness audit — confirming your source documents are clean, current, and structured enough to chunk sensibly. A RAG system built on top of stale, duplicated, or poorly formatted documentation will retrieve confidently and answer wrong; the model isn't the weak link, the corpus is.

When Fine-Tuning Is the Right Call

Fine-tuning is the right call when the problem isn't "the model doesn't know something," it's "the model doesn't behave the way this feature needs it to." That's a subtler diagnosis than it sounds, because it's easy to mistake a formatting problem for a knowledge problem and reach for retrieval when what you actually need is better examples.

Good fine-tuning candidates share a pattern: the desired behavior is stable, repeatable, and expressible as consistent input-output pairs. A support team that always closes tickets with a specific structure, a legal team that needs summaries in a fixed clause format, a classifier that needs to sort tickets into your company's specific taxonomy — none of these require new facts, they require the model to reliably do the same kind of thing every time.

Signals you're looking at a fine-tuning problem:

• Prompting alone produces inconsistent formatting or tone across otherwise-identical requests
• The task is a narrow, repeatable skill (classification, structured extraction, a house style) rather than open-ended knowledge lookup
• You have (or can generate) a few hundred to a few thousand high-quality labeled examples
• The behavior needs to be fast and cheap at inference — no retrieval round-trip, no extra prompt tokens

Cost and effort here run in the opposite direction from RAG. Lightweight approaches like LoRA (low-rank adaptation) can fine-tune an open-weight model for a narrow task at a modest compute cost, while a full fine-tune of a larger model is a meaningfully bigger and slower undertaking — directionally, an order of magnitude more in both compute spend and iteration time. Either way, the ongoing cost isn't hosting an index; it's the discipline of re-training when behavior needs to shift, plus an evaluation harness that catches regressions before they ship.

The Support-Assistant Case Study: Weekly Docs, Fixed Voice

A customer-support assistant is the cleanest real-world illustration of why this is rarely an either/or decision. The product documentation it answers from changes weekly as features ship and pricing updates — that's a RAG problem. The tone, structure, and escalation behavior it should exhibit every single time is fixed — that's a fine-tuning problem, or at minimum a heavily-templated prompting problem that behaves like one.

Think about what the assistant is actually being hired to do — the job the customer brings to it, in the language of our complete guide to jobs-to-be-done. The customer's job is "resolve my issue without waiting," not "receive a stylistically perfect answer." That reframing matters: it tells you which failure mode to fix first. Wrong or stale facts fail the job outright; an inconsistent tone merely annoys.

Notice the last two rows: policy language needs to be retrieved (so it's never stale) but delivered in a consistent, trained voice — that's the hybrid pattern most production support assistants actually converge on. The retrieval layer keeps facts current; the fine-tuned (or tightly prompted) layer keeps the experience consistent.

That consistency matters more than it looks like on paper, because tone inconsistency compounds across a customer's full journey, not just a single ticket. A customer who gets a warm, on-brand reply on their first contact and a stiff, generic one on their third notices the seam — the kind of friction our customer journey guide frames as an emotion-curve dip, not just a formatting nitpick. Facts can be wrong and get corrected in the next message; a broken voice erodes trust across the whole relationship.

Why Retrieval Quality Is the Hard Part Nobody Budgets For

The hardest part of a RAG feature isn't choosing to do RAG — it's making retrieval actually return the right passage, consistently, across the messy long tail of real questions. Teams routinely budget for the vector database and skip budgeting for the weeks of tuning that make retrieval trustworthy, which is where most RAG projects quietly underperform their demo.

The model usually isn't the weak link in a struggling RAG feature — the retrieval pipeline feeding it is.

Three specific failure modes account for most of the gap between a RAG demo and a RAG feature that holds up in production:

• Chunking that splits meaning across boundaries. Cut a policy document in the wrong place and the retrieved chunk contains half a rule, which the model then confidently misapplies.
• Embeddings that miss the query's real intent. A customer asking "can I get my money back" and a doc titled "Refund Policy" are semantically close, but plenty of real queries are phrasing edge cases that a naive embedding search misses.
• Long context that dilutes what was retrieved. Stanford researcher Nelson Liu and coauthors, in their widely cited 2023 "Lost in the Middle" study, found that model performance on information placed in the middle of a long context degrades noticeably compared to information at the start or end — even in models built for long context windows. Stuffing in more retrieved chunks isn't a free win; it can bury the one that mattered.

Anthropic's own applied research on retrieval, published as "Contextual Retrieval" in 2024, is a useful data point on how much engineering effort retrieval quality actually absorbs. Adding context to chunks before embedding them, combining that with keyword-based retrieval, and layering a reranking step on top each meaningfully cut the rate of failed retrievals in their internal testing.

Directionally, each additional technique took a real bite out of the failure rate, with the full stack cutting failures by roughly half to two-thirds depending on configuration. None of that is exotic technology — it's disciplined engineering on top of a simple idea, and it's the difference between a RAG feature that works in a demo and one that works for a real support queue.

Retrieval also isn't free at inference time — every retrieved chunk adds tokens and every retrieval call adds a round-trip before generation even starts. If your feature has a hard response-time requirement, work through the tradeoffs in our guide to latency budgets for AI features before committing to a multi-step retrieval-then-rerank pipeline; each added retrieval stage buys you accuracy at the direct cost of speed.

A Feasibility Framework: Freshness, Effort, and Cost

The decision comes down to three questions asked in order: how often does the underlying knowledge change, how much curated example data do you already have, and what's your tolerance for added inference latency and ongoing engineering cost. Answer those honestly and the RAG-versus-fine-tune choice mostly makes itself.

1. How often does the knowledge change? Weekly or faster points hard toward RAG. Rarely-changing, stable behavior points toward fine-tuning.
2. Do you have (or can you generate) enough labeled examples? Fine-tuning needs a real training set — a few hundred good examples at minimum, ideally more. No examples, no fine-tune, at least not yet.
3. What's your latency and cost budget? RAG adds a retrieval round-trip on every request; fine-tuning adds none at inference but costs more upfront in curation and retraining discipline.
4. Does the feature need to cite its sources? If yes, you need retrieval regardless of anything else — a fine-tuned model can't point to the document it "remembered" a fact from, because it didn't retrieve one.

Once you've made the call, the decision needs to survive contact with an engineering team that has to build it. That's a separate feasibility conversation — worth working through using the six questions engineering needs answered before scoping an AI feature, because "we're doing RAG" and "we're fine-tuning" are starting points for a spec, not the whole spec.

This is exactly the kind of explicit tradeoff Prodinja's Feature-to-Feasibility tool is designed to walk through as a prototype experience. Rather than defaulting to whichever approach a PM heard about most recently, it frames the model-approach decision as an explicit choice, prompting you to reason through freshness, effort, and cost side by side before a single line of retrieval or training code gets written.

It doesn't run the retrieval pipeline or the fine-tune for you. It's designed to make sure you've asked the right questions before someone else has to build the answer.

Key Takeaways

• RAG and fine-tuning solve different problems — RAG injects knowledge that changes, fine-tuning bakes in behavior that's fixed.
• The freshness test is the fastest filter: if the source material updates weekly or faster, retrieval beats retraining almost every time.
• Citations require retrieval — a fine-tuned model can't point back to the document a fact came from, because none was retrieved.
• Retrieval quality, not model choice, is where most RAG projects underperform — chunking, embedding relevance, and context dilution do more damage than a "wrong" model pick.
• Fine-tuning needs real labeled data — without a genuine training set of good examples, there's no fine-tune to run yet.
• Most mature systems combine both — retrieved facts delivered in a consistently trained or tightly templated voice, as the support-assistant example shows.
• Latency and cost trade off in opposite directions — RAG spends at inference (retrieval round-trips), fine-tuning spends upfront (curation and retraining).