The Unit Economics of RAG: What Each Answer Really Costs

Every RAG answer carries five stacked costs: embedding the query, storing and indexing vectors, retrieving candidates, reranking them, and generating the final response from a token-stuffed context window. Generation and reranking usually dominate, often 70-90% of per-query spend, because they scale with context size, not document count.

Quick answer: A typical RAG query costs roughly $0.001-$0.02, driven mostly by generation tokens (the retrieved context you stuff into the prompt) and, if used, a reranking pass. Embedding and storage are usually rounding errors by comparison.

Most teams price and package their RAG product before they've priced a single query. That's backwards. If you're accountable for margins, you need a line-item model — not a vibe — before your infrastructure bill or your API costs surprise you in month three.

Why RAG Cost Modeling Is Different From Plain LLM Cost Modeling

A plain chatbot's cost is one line: input tokens plus output tokens, multiplied by a per-token price. RAG adds four more stages before generation even starts, each with its own pricing unit and its own scaling curve.

The trap is that these stages don't scale together. Storage scales with your document set. Retrieval scales with query volume. Generation scales with context budget — how many retrieved tokens you cram into the prompt — which is a design choice, not a fixed cost of doing business.

That's the crux: two teams building "the same" RAG feature can have 10x different unit economics purely because one team retrieves 3 chunks at 500 tokens each and the other retrieves 12 chunks at 1,000 tokens each. Same architecture diagram, wildly different bill.

For the broader architectural picture — chunking, indexing, retrieval strategy — see this complete guide to RAG and knowledge systems. This article isolates the money question inside that architecture: what does one answer actually cost, and where does that cost hide?

The Five-Stage Cost Stack

Every RAG query, regardless of vendor stack, passes through the same five billable stages:

1. Query embedding — converting the user's question into a vector.
2. Storage / index — amortized cost of holding vectors in a vector database.
3. Retrieval — the similarity search that returns candidate chunks.
4. Reranking (optional but common) — a cross-encoder pass that re-scores candidates for relevance.
5. Generation — the LLM call that reads the retrieved context and writes the answer.

Model each separately. Summing them without decomposition is how teams miss that one stage is quietly eating the margin.

Line 1-2: Embedding and Storage Costs Are Almost Never the Problem

Embedding a query typically costs a fraction of a cent, and per-document storage in a vector database runs pennies per thousand vectors per month — these two lines rarely explain a margin problem, but teams still audit them first out of habit.

Embedding cost is a function of query length and the embedding model's per-token price. A typical embedding model prices in the range of $0.01-$0.13 per million tokens, per OpenAI's and Cohere's published embedding pricing tiers as of their current public rate cards. A 20-word query is roughly 30 tokens — the embedding cost per query is effectively negligible, well under $0.00001.

Storage cost depends on your vector database's pricing model — some charge per vector stored, some per GB, some per "pod" or compute unit regardless of volume (Pinecone's serverless and pod-based tiers illustrate both models). For a corpus of 100,000 documents chunked into 500,000 vectors, storage typically lands in the tens to low hundreds of dollars per month — again, usually immaterial per query once amortized across query volume.

Where embedding does bite: if you re-embed your entire corpus frequently (daily reindexing of a fast-changing knowledge base), embedding becomes a fixed corpus cost, not a per-query cost — model it separately, and revisit your chunking strategy if reindexing frequency is driven by chunk sizes that are too small or too granular.

Line 3-4: Retrieval and Reranking Are Where Precision Fights Cost

Retrieval itself is cheap per query, but reranking — the step that improves answer quality by re-scoring candidates — routinely becomes the second-largest cost line because it runs a heavier model over every candidate chunk, not just the winners.

Retrieval is an approximate nearest-neighbor (ANN) search — the compute cost is small and mostly absorbed into your vector database's service fee. The real retrieval decision that affects downstream cost is how many candidates (top_k) you pull back. Pull back 50 candidates instead of 10, and you've just handed your reranker 5x more work.

Reranking trades cost for precision. A cross-encoder reranker (like Cohere Rerank or an open-source bge-reranker model) scores each query-candidate pair individually — it's more accurate than vector similarity alone but scales linearly with candidates. Reranking 50 candidates instead of 10 can easily triple that stage's cost.

This is the classic precision-cost tradeoff in RAG: more candidates and reranking generally improve answer relevance, but every additional candidate is billable compute. It's the same tension Andrew Ng and other applied-AI practitioners have flagged in retrieval pipeline write-ups — bigger top_k values as a bluntly effective knob for quality that teams turn up without pricing what it costs.

A Worked Retrieval-to-Rerank Example

Say your retriever pulls top_k = 20 candidate chunks, and you rerank all 20 before selecting the top 5 to send to generation.

• Retrieval compute: negligible, bundled into vector DB pricing.
• Reranking 20 pairs at a typical cross-encoder rate (roughly $1-$2 per 1,000 pairs on hosted rerank APIs): ~$0.02-$0.04 per query if you're on a metered API, less if self-hosted at scale.
• If you instead retrieve top_k = 10 and rerank all 10, that line roughly halves — with a measurable but often acceptable quality tradeoff, especially for narrow, well-chunked corpora.

The lesson isn't "never rerank." It's that top_k is a cost dial, and most teams set it once during a demo and never revisit it once they're at production volume.

Line 5: Generation Tokens Are the Line That Actually Determines Your Margin

Generation cost is driven almost entirely by how many retrieved tokens you stuff into the context window plus how long the output runs — and because most teams retrieve generously "to be safe," this is usually the single largest and most controllable cost line in the entire stack.

Generation is priced per input and output token, and input tokens in a RAG system are dominated by the retrieved context, not the user's question. A 20-token question wrapped in 3,000 tokens of retrieved chunks means you're paying almost entirely for the retrieval, not the query.

Why Context Budget Is the Real Lever

Consider the same question answered with two different context budgets — the total retrieved-token allowance you allow into the prompt:

That's a 4.5x cost swing for the same question, purely from context budget, using representative frontier-model pricing (roughly $3 per million input tokens, $15 per million output tokens, in the range major model providers publish). At scale — a million queries a month — that's the difference between an $8,000 generation bill and a $36,000 one.

This is also where quality and cost genuinely fight each other. Under-retrieving causes hallucination and missed context; over-retrieving buries the model in irrelevant tokens, which research on long-context degradation (including the "lost in the middle" findings from Liu et al. at Stanford) shows can actually hurt answer quality even as it inflates cost. More tokens isn't a free quality upgrade — it's a cost with diminishing and sometimes negative returns.

Getting this right requires treating context budget as a first-class product decision, alongside the other structural calls every RAG team has to make — covered in the four product decisions RAG forces you to make — and it interacts directly with who's allowed to see which chunks in the first place, which is its own cost-and-risk surface covered in access control and permissions for RAG.

Building the Spreadsheet: A Per-Query Cost Model You Can Actually Use

A usable cost model has one row per stage, one column per pricing variable, and outputs a per-query total you can multiply by projected volume — most teams can build this in under an hour once they have real pricing from their vendors.

Here's the structure, filled with illustrative numbers for a mid-size RAG assistant (adjust the rates to your actual vendor contracts):

At 1 million queries a month, that's roughly $56,000 — and the model just told you reranking and generation input tokens are 92% of it. That's the number your pricing tier needs to cover, with margin, before you decide what to charge.

Turning the Model Into Pricing and Packaging Decisions

Once you know the real per-query cost, three packaging questions become answerable instead of guessed at:

• Should you price per seat or per query? If cost is volume-driven and usage is uneven across customers, flat per-seat pricing quietly subsidizes your heaviest users.
• Should "high-precision mode" (larger top_k, reranking on) cost more? If it's 3-4x the base cost, it probably deserves a premium tier, not a free toggle.
• What's your gross margin floor? If infrastructure is 40% of query revenue, you have less room than a typical SaaS margin structure assumes — model it before your board does the math for you.

This is also a design decision, not just a finance one — the same way understanding what a customer is actually trying to accomplish reshapes a roadmap, understanding what a query actually costs reshapes a pricing page. If you haven't mapped the underlying jobs your RAG feature serves, the complete guide to jobs-to-be-done is worth pairing with this cost model, and mapping cost against the customer journey shows you exactly which moments are expensive to serve versus which are expensive to under-serve.

Where Prodinja Fits: Treating Context Budget as an Explicit Lever

Given that generation cost tracks context budget more than any other variable, the tool that lets you see and adjust that budget directly is the highest-leverage place to intervene — which is what Prodinja's Context Engineering tool is built for.

Rather than treating "how much retrieved context to send" as an invisible default buried in a retrieval config, Prodinja's Context Engineering tool is designed to let you treat the retrieved-token budget as an explicit, adjustable lever — so you can see the tradeoff between context size, answer quality, and cost as a deliberate decision rather than an accident of your top_k setting. It's one input into the kind of per-query modeling this article walks through, not a replacement for building the spreadsheet yourself.

Key Takeaways

• Generation tokens, driven by context budget, are usually the largest cost line in a RAG query — often 60-80% of total per-query cost.
• Reranking is the most commonly under-priced stage — it scales with candidates reranked, not with corpus size, and teams often set top_k once and never revisit it.
• Embedding and storage are rarely the margin problem — audit them once, then stop spending analysis time there.
• Context budget is a product decision, not just an engineering default — bigger isn't automatically better; long-context research shows quality can degrade even as cost rises.
• Build a five-line spreadsheet before you set pricing — embedding, storage, retrieval, reranking, generation — so packaging reflects real unit economics.
• Revisit the model whenever you change chunk size, top_k, or reranking policy — each is a cost lever disguised as a quality lever.

Frequently Asked Questions

How much does a single RAG query typically cost?

Most production RAG queries cost between $0.001 and $0.05, depending on context budget and whether reranking is used. Generation tokens (driven by how much retrieved context you send) and reranking together typically account for 70-90% of that total, with embedding and storage rarely mattering much per query.

What's the biggest lever for reducing RAG cost per query?

Context budget — how many retrieved chunks and tokens you send to the generation model — is typically the biggest lever. Reducing top_k and tightening chunk relevance before generation can cut generation cost by 2-4x with a manageable, testable quality tradeoff.

Does reranking always make RAG more expensive?

Reranking adds a cost line, but it can lower total cost if it lets you retrieve fewer candidates upfront or send fewer, higher-quality chunks to generation. The net effect depends on your top_k before and after reranking — model both scenarios rather than assuming either direction.

Should RAG pricing be per query, per seat, or usage-based?

It depends on how uneven usage is across customers and how volume-sensitive your cost stack is. If generation and reranking dominate cost and usage varies widely, usage-based or tiered pricing typically protects margin better than flat per-seat pricing.

How do I know if my context budget is too large?

If retrieved tokens per query regularly exceed what a human would actually read to answer the question, or if answer quality plateaus (or drops) as you add more chunks, your context budget is likely oversized relative to the value it's adding — test smaller top_k values against your eval set before assuming more context is safer.