How to Actually Calculate Cost Per Inference for an AI Feature

Cost per inference is the fully loaded price of producing one successful, usable AI output — input tokens (system prompt, retrieved context, few-shot examples) plus output tokens, priced at your model's per-token rate, then multiplied by however many attempts it actually takes. The provider's list price is only the first term in that equation.

Quick answer: cost = (input_tokens × in_price) + (output_tokens × out_price), summed across every attempt, then multiplied by your expected retry factor. Most first cost models undercount by 30–60% because they price only the "happy path" call and skip retries, guardrail passes, and the system-prompt/RAG overhead riding on every request.

What "Cost Per Inference" Actually Means

Cost per inference is not the number on a model provider's pricing page. It is the total spend required to produce one successful, usable output for the user — including every retry, safety check, and re-generation along the way. Most AI PMs price the happy path and then get blindsided by their real COGS.

The Sticker-Price Trap

A first-draft cost model usually looks like this: take the provider's per-million-token rate, multiply by an assumed prompt length, and call it done. That number is real, but it is not your cost — it is your cost for a request that never fails, never gets flagged, and never needs a second pass.

What that estimate typically leaves out:

• The full system prompt, which runs on every single call, not just the example you tested with.
• Any retrieved (RAG) context injected before the model ever sees the user's question.
• Few-shot examples used to steer formatting or tone.
• Retries triggered by schema failures, low-confidence flags, or guardrail rejections.
• Guardrail and eval passes — separate model calls that check the output before a user sees it.

AI engineer and author Chip Huyen, whose book AI Engineering is a common practitioner reference, has made a related point: in production retrieval-augmented systems, input tokens frequently outweigh output tokens, precisely because teams size their cost estimate on the visible answer and forget everything feeding into it.

From Unit Price to Fully Loaded Cost Per Outcome

The mental-model shift that matters here is small to state and easy to skip in practice: stop pricing the API call and start pricing the successful outcome. A request that fails validation and retries doesn't cost the same amount twice by accident — it costs you the full input again, plus new output, plus whatever checked it the first time.

This distinction — unit price versus fully loaded cost — is the foundation of any credible unit-economics model for an AI feature, which is why it anchors the broader framework in our complete guide to AI economics. Everything below is that framework applied to a single request.

The Token Line Items: System Prompt, RAG Context, Few-Shot, and Output

Every inference call carries four token-based cost lines: the system prompt, retrieved context, few-shot examples, and generated output. Each is priced separately by your provider, and each is a line item a different team controls — which is exactly why it gets missed in a first pass.

Input Tokens: The System Prompt Tax

A system prompt that runs 500, 1,500, or 3,000 tokens costs you that many input tokens on every request, forever, regardless of what the user actually asked. A verbose, conversational system prompt is not a stylistic choice — it's a recurring line item.

Treating the system prompt as a product surface, not a scratchpad, is usually the fastest way to shrink this line. That discipline — versioning prompts, testing changes, and trimming instructions that no longer earn their tokens — is exactly what our guide to prompt design walks through.

Input Tokens: RAG Context

Retrieval-augmented generation injects document chunks so the model can ground its answer in real data. More chunks generally means better grounding and a higher hallucination-resistance floor, but each chunk is priced identically to any other input token.

How many chunks you retrieve, how they're sized, and how cleanly your source data is structured is fundamentally a data strategy decision, not a prompting one — the tradeoffs are covered in our AI data strategy guide.

Input Tokens: Few-Shot Examples

Few-shot examples — sample input/output pairs included to steer format or tone — are a common fix for inconsistent output. They also run on every call, which means a 3-example block you added to fix one edge case is now a permanent tax on 100% of requests, not just the edge case.

Output Tokens: The Part Everyone Remembers

Output tokens are usually priced several times higher per token than input tokens across most provider tiers, which is the one line item most first-draft models actually do account for. What they miss is volume: a system prompt that asks the model to "explain your reasoning" or return verbose chain-of-thought before the answer can quietly double or triple output length.

The Hidden Multipliers: Retries and Guardrail Passes

Two cost lines rarely survive a first-draft model: retries, when a request fails validation and has to run again, and eval or guardrail passes, additional model calls that check output before a user sees it. Both multiply your base per-request cost — sometimes by 2–3x.

Retries: Why Failure Costs 2–3x

A retry happens when an output fails a JSON schema check, violates a business rule, trips a guardrail, or gets flagged low-confidence by an eval step. Whatever the trigger, the retry re-runs the entire input cost — system prompt, RAG context, few-shot examples, all of it — plus a fresh output.

The math is unforgiving because it compounds with your failure rate, not against it:

A feature that looks fine at 85% first-attempt success is already paying an 18% cost tax nobody budgeted. Drop to a 1-in-3 success rate on a genuinely hard extraction or reasoning task, and you're paying triple the sticker price for every accepted answer.

If your feature spans multiple stages — speech-to-text, generation, then text-to-speech, for instance — each stage can fail independently, and the retry math compounds across the whole pipeline. That's a materially different cost model than a single text call, and it's why we treat it separately in our multimodal and voice AI guide.

Eval and Guardrail Passes: The Invisible Extra Calls

Production AI features commonly run at least one of the following before a response reaches a user: a moderation or safety classifier, an LLM-as-judge eval scoring the response against a rubric, or a second model cross-checking a factual claim. Each of these is its own inference call, with its own input and output tokens, even when it runs on a cheaper model.

Layering in the guardrail and moderation passes covered in our AI safety guide is often the right call for a regulated or high-stakes feature. Just budget it as its own line item — teams that skip this step consistently underprice their COGS by treating a whole extra model call as a rounding error.

The Formula and a Worked Line-Item Example

The core formula is cost = (input_tokens × in_price) + (output_tokens × out_price), summed across every attempt a request takes, then divided by the number of successful outcomes it produced. Below is a full line-item breakdown for one realistic support-triage request that fails validation once and succeeds on retry.

The Base Formula

To apply the formula honestly, work through it in order:

1. Count tokens per attempt, broken into the four input lines plus output — don't estimate, count.
2. Price each line at your provider's actual per-token rate (input and output are priced separately, and differently).
3. Add any guardrail or eval calls as their own input/output token cost, even on a cheaper model.
4. Multiply by your measured retry factor, from the table above or your own production logs.
5. Divide the total by successful outcomes, not total attempts, to get your true cost per successful inference.

Line-Item Table: One Request, Fully Loaded

The rates below are round, illustrative numbers, not a specific vendor quote — swap in your own provider's published per-token pricing before you trust the total.

The naive, sticker-price estimate lands at roughly $0.0082. The fully loaded cost of the outcome the user actually received is about $0.0169 — just over 2x the naive number, entirely from one retry and two small eval calls. That gap is real and directional: it shows up whether your naive estimate is a fraction of a cent or tens of cents.

Building Your First Cost-Per-Inference Model

Building a usable model takes five inputs: your provider's per-token rates, real token counts for each prompt component, your measured retry rate, whether you run guardrail passes, and your target request volume. Most teams can assemble a working first version from existing logs in under a day.

1. Count your tokens for real. Run an actual tokenizer — tiktoken or your provider's equivalent — against logged production prompts, not the shorter example you tested with in a notebook.
2. Separate the four input lines so you know which one to attack first: system prompt, RAG context, few-shot examples, user input.
3. Measure your actual retry rate from production logs — validation failures, guardrail rejections, low-confidence flags — rather than guessing.
4. Price eval and guardrail calls separately. They usually run on a cheaper model, but they're not free, and they're not optional to count.
5. Multiply by volume to turn a per-request number into a monthly COGS forecast — that's the number finance actually needs.

Sequoia Capital's widely circulated analysis of AI infrastructure spend versus realized application revenue made this point at the industry level: the economics only work out if someone is tracking the cost of every inference, not just the aggregate compute bill. The FinOps Foundation has reached a similar conclusion from the practitioner side, extending its cost-allocation discipline into a dedicated FinOps for AI practice built around exactly this kind of per-request, per-feature unit tracking.

Where This Fits Next to Your Spec

Once you have a credible cost-per-outcome number, the next decision is how you recover it — usage-based pricing, value-based pricing, or some blend — a tradeoff covered in our guide to usage-based vs. value-based AI pricing.

This is also where Prodinja's simulated credits economy is meant to help before you've built any of the above. Each action inside the prototype experience is designed to carry a per-inference credit cost — system prompt, context, retry, eval pass — giving you a concrete unit to reason with before you touch a spreadsheet or a provider's live pricing page.

It's a way to build intuition for where a feature's cost actually accumulates, not a replacement for a production model built on your own logs and retry rates. Documenting the line-item table above next to the requirement it prices out, rather than in a spreadsheet nobody reopens, is the kind of detail a living spec like Prodinja's Spec Studio is built to keep visible across revisions.

Andreessen Horowitz's research on generative AI infrastructure economics has argued that inference behaves nothing like traditional software COGS — it's a variable cost that scales with usage, not a fixed cost that scales with headcount — which is precisely why AI-native products often post thinner gross margins than SaaS peers charging comparable prices.

Key Takeaways

• Cost per inference is the fully loaded cost of a successful outcome, not the provider's per-token sticker price.
• Four line items make up every request — system prompt, RAG context, few-shot examples, and output tokens — and a different team controls each one.
• Retries are the biggest hidden multiplier: a 70% first-attempt success rate costs roughly 1.4x sticker price; a 33% rate costs roughly 3x.
• Guardrail and eval passes are separate inference calls with their own token cost — budget them as a line item, not a footnote.
• The formula is (input_tokens × in_price) + (output_tokens × out_price), summed across attempts, divided by successful outcomes.
• Build your model from actual logged token counts and retry rates, not estimates from a prompt draft in a notebook.
• A credible cost-per-outcome number is also the direct input to your pricing model — you cannot price value you haven't first costed.