AI Feature Feasibility and Estimation: The Complete Guide to Turning 'Make It Smarter' Into an Estimable Spec

An estimable AI feature spec answers six questions before anyone writes code: what data proves the idea works, which model approach fits the job, what latency and cost it can tolerate, how it fails and how you'll catch that, and how much build-plus-maintenance effort it costs. Skip one, and "make it smarter" stays a wish, not a plan.

Quick answer: To estimate an AI feature, translate the vague ask into six concrete answers — data needs, model approach, latency budget, failure modes, effort, and a maintenance plan — before engineering is asked for a number. Each answer should be falsifiable, not aspirational.

From "Make It Smarter" to a System With a Bill

A vague AI ask is unestimable because it hides five decisions inside one sentence. "Make it smarter" sounds like a direction; to an engineer asked to size it, it's an unbounded set of unknowns about data, models, and failure. The fix is a structural mental shift, not a better meeting.

Stop treating AI as a feature toggle and start treating it as a system with a data contract, an error budget, and a maintenance bill. A toggle is either on or off. A system has inputs it depends on, a defined tolerance for being wrong, and a running cost that doesn't stop at launch. That reframe alone changes what a PM is required to bring to the estimation conversation.

This shift matters because AI features fail estimation for structural reasons traditional software doesn't share:

• The behavior is probabilistic, so "does it work" is a distribution, not a boolean.
• The dependency is data, which means the spec is unstable until the data is inspected.
• The cost recurs, since every inference costs money and every drift event costs attention.

Martin Zinkevich's widely circulated engineering rules for machine learning at Google open with a deliberately unglamorous instruction: don't be afraid to launch without machine learning first, and keep the first model as simple as the infrastructure allows. That's not anti-AI caution — it's an estimation discipline. Complexity should be earned by evidence, not assumed by ambition.

We'll carry one running example through every dimension below: "auto-summarize support tickets so agents can triage faster." It's a common, innocent-sounding ask that hides real decisions the moment you try to size it.

Why so many AI features stall before they're even scoped

This isn't a hypothetical risk. Gartner analysts have repeatedly warned that a large share of generative AI pilots will be abandoned before reaching production, and the cited reasons are almost never model quality — they're unclear ROI, unresolved data readiness, and escalating costs discovered late. Those are exactly the gaps an estimable spec is built to close before a team commits budget, not after.

The six dimensions below don't weigh equally for every feature type, which is itself a useful diagnostic. A retrieval-heavy assistant lives or dies on data needs and context assembly; a synchronous in-line suggestion lives or dies on latency; an agentic, multi-step workflow lives or dies on failure modes, because each step compounds the last one's error rate. Knowing which dimension is your feature's actual constraint — before you estimate — tells you where to spend the interrogation's time.

If you want the condensed, engineer-facing version of this same interrogation, our companion piece on the six questions engineering needs before it will size an AI feature distills it into a checklist you can drop straight into a ticket.

Dimension 1: Data Needs — What's the Data Contract?

An AI feature is only as estimable as its data is inspectable, so this dimension asks what data exists, how much, how clean, and who's allowed to see it. Answering this before model selection prevents the single most common estimation failure: assuming data that doesn't exist in usable form.

For the ticket-summarization example, "data" isn't one thing — it's at least four separate questions:

1. Volume: How many tickets per day/week, and across how many channels (email, chat, phone transcripts)?
2. Ground truth: Do agent-written summaries already exist anywhere, even informally in resolution notes, that could seed an evaluation set?
3. Structure: Are tickets consistent in format, or do three support tools each export differently?
4. Sensitivity: Do tickets routinely contain PII — names, account numbers, payment details — that constrains storage, logging, and which model providers are even eligible?

"Data first" isn't a slogan — it's triage. If the labeled examples you need don't exist, your first deliverable isn't a model, it's a labeling plan.

The data audit checklist

Before anyone estimates model work, run a fast audit:

• Source inventory — where does the raw data live, and who owns write access to it?
• Volume estimate — is there enough to build an eval set (dozens), a few-shot prompt (single digits), or a fine-tune (thousands)?
• Quality pass — sample 20-30 real records by hand; count how many are actually usable without cleanup.
• Label availability — does "correct" already exist, or does someone need to write it?
• Access and residency constraints — can this data leave your infrastructure to reach a model provider at all?

This is also where context engineering earns its keep: for many AI features, the real data-needs question isn't "do we have training data" but "can we assemble the right context — the specific ticket, the customer's history, the product area — at inference time." Our deep dive on context engineering covers how to design that assembly pipeline, which is frequently the more expensive half of "data needs" for retrieval-based features.

For the ticket summarizer, the honest data answer is usually: plenty of volume, some structural inconsistency across channels, thin explicit ground truth, and real PII exposure that needs redaction before anything touches an external model. That single paragraph is worth more to an estimating engineer than a page of feature description.

One more thing worth naming explicitly: data isn't static. Ticket categories shift when a new product ships, phrasing shifts when a new support tool rolls out, and a data contract that was accurate at spec time can be stale within a quarter. Treat the data-needs answer as a snapshot with an expiration date, not a permanent fact — which is precisely why revisiting it belongs on the maintenance side of the estimate, not just the build side.

Dimension 2: Model Approach — Prompt, Retrieval, Fine-Tune, or Build?

The model-approach question isn't "which model" — it's "how much machinery does this task actually require," and the honest default for most PM-initiated features is the cheapest approach that clears the bar, escalating only when evidence demands it. Overbuilding here is the single biggest source of blown estimates.

Four approaches sit on a ladder of increasing cost, control, and maintenance burden:

For auto-summarization, the task is one general-purpose LLMs are already competent at, so the estimable default is prompt-only, with retrieval of similar past tickets if agents need precedent, not fine-tuning. Fine-tuning becomes justified only if evaluation shows a specific, recurring quality gap — say, consistently missing an escalation flag — that prompting can't close after real iteration.

That escalation trigger is the estimable part. A defensible model-approach answer names:

1. The default approach and why it's sufficient for the task's complexity.
2. The specific quality signal that would justify moving up the ladder.
3. The fallback if the model approach underperforms in production (e.g., route to a human, or degrade to a simpler extractive summary).

Chip Huyen's writing on production ML systems makes a related point worth internalizing here: the model is rarely the bottleneck in shipping AI features — the surrounding data pipeline, monitoring, and serving infrastructure usually is. Sizing "model approach" honestly means sizing that infrastructure, not just picking an API.

It's also worth deciding, at this stage, what "good enough" looks like relative to a non-AI baseline. A rules-based extractive summary — pull the first customer message and the last agent reply, truncate to two lines — costs almost nothing to build and sets the floor your model approach has to beat. If a general-purpose model's output isn't meaningfully better than that floor for a meaningful share of tickets, the honest estimate is "not yet," and the spec should say so rather than quietly shipping a marginal upgrade at real ongoing cost.

Dimension 3: Latency and the Cost Budget

Every AI feature has an implicit latency and cost budget dictated by where it sits in the user's workflow, and naming that budget explicitly — not "make it fast" but a P50/P95 number in milliseconds plus a per-call cost ceiling — turns a vague performance hope into something engineering can architect against.

The budget question has two halves that get confused constantly:

• Latency: how long can the user wait before the feature stops feeling useful or starts feeling broken?
• Cost: what does this cost per call, multiplied by expected volume, and does that number survive scale?

For the ticket summarizer, the real answer depends on when the summary appears. If it's synchronous — the agent opens a ticket and waits for a summary to render — the budget is unforgiving, something like a P95 under 2-3 seconds before agents simply stop trusting or waiting for it. If it's precomputed asynchronously the moment a ticket is created or updated, the latency budget nearly disappears and the constraint becomes freshness (how stale can a summary be after a new reply) rather than raw speed.

That distinction — synchronous vs. precomputed — is frequently the single highest-leverage decision in the whole spec, because it can turn a hard latency problem into a non-problem. It's also where cost compounds: precomputing on every ticket update multiplies calls by edit frequency, which can quietly dominate the cost line even when per-call cost looks cheap.

A usable latency-and-cost line in a spec states:

• Target P50 and P95 latency, tied to where in the flow the AI runs (sync vs. async).
• Expected call volume per day/month, and cost per call at current model pricing.
• The resulting monthly cost estimate, with a note on how it scales with usage growth.

Our detailed breakdown of latency budgets for AI features walks through how to set these numbers by interaction type and how they interact with model-approach choice — a synchronous chat reply and a batch-processed nightly digest have almost nothing in common as latency problems, even if the underlying model call is identical.

Cost deserves the same specificity latency gets, and rarely receives it. "It's just an API call" hides a multiplier: volume growth, retries on failure, and any retrieval step that adds its own calls before the model even runs. A spec that states cost per summary but skips the retry and retrieval multipliers will understate the real monthly bill, often by a wide margin once the feature is at scale.

Dimension 4: Failure Modes and the Error Budget

AI features don't fail like normal software — they don't crash, they quietly produce a plausible-sounding wrong answer — so this dimension requires naming specific failure modes in advance, ranking them by damage, and assigning each a detection method before launch, not after a customer complaint surfaces it.

The most expensive AI failures are never the ones that throw an error. They're the confident, well-formatted, wrong ones nobody notices until a customer does.

For the ticket summarizer, a realistic failure-mode list looks like this:

Naming five failure modes isn't pessimism — it's the only way to define an error budget: an explicit, agreed tolerance for how often the feature is allowed to be wrong before it's paused, reviewed, or rolled back. Without a stated tolerance, any single bad example becomes a crisis instead of an expected, planned-for event.

This is squarely the territory the NIST AI Risk Management Framework's "Measure" function is built for — it pushes teams to define and track failure categories as a standing practice, not a pre-launch checklist item that gets forgotten. Two of our other deep dives exist specifically to make this dimension rigorous:

• Adversarial thinking, for systematically red-teaming a feature before real users find the gaps for you.
• AI evals, for building the measurement harness that turns "it feels better" into a number you can regression-test against.

Neither is optional for an estimable spec. A failure-mode list without a detection method attached to each item is a list of worries, not a plan.

It's worth being explicit, too, about who is accountable when a failure mode fires in production. "The model" isn't an answer — a person needs to own the escalation path: who gets paged when the PII scan flags a leak, who reviews the weekly eval score, who has authority to pause the feature if the error rate crosses the agreed line. An error budget without an owner tends to quietly become nobody's problem until it's everybody's problem.

Dimension 5: Effort and the Maintenance Bill

The effort dimension is where most AI estimates quietly lie, because they price the build and forget the bill that starts the day after launch — model updates, prompt drift, data shift, and the ongoing review cycle a probabilistic feature demands forever, not just at ship time.

Sculley et al.'s well-known paper on hidden technical debt in machine learning systems put a name on this problem years ago: ML systems accumulate a distinct, compounding technical debt beyond normal software, because changing data, changing entanglement between components, and configuration debt keep piling up quietly after launch. Estimating "effort" without pricing that ongoing debt understates the true cost, often badly.

A defensible effort estimate separates two numbers instead of one:

1. Build effort — engineer-weeks to ship a first estimable version: prompt/retrieval engineering, integration into the existing UI, an eval harness, and the failure-mode mitigations from the previous dimension.
2. Maintenance effort — a recurring, ongoing allocation: who reviews eval scores on a schedule, who owns re-prompting when the underlying model updates, and who's paged when cost or error rate drifts outside the agreed budget.

For the ticket summarizer, build effort typically centers on prompt iteration, redaction pipeline work, and eval-set construction rather than model training — which is exactly why the model-approach decision earlier in the spec drives most of the effort number. Maintenance effort centers on periodic eval re-runs, monitoring cost as ticket volume grows, and revisiting the prompt whenever the underlying model is upgraded by its provider.

This is also the argument for treating the spec itself as a living artifact rather than a document you write once and file away. As data shifts, models update, and failure modes surface in production, the spec needs to change with them — which is the whole premise behind our guide to living specs: a spec that tracks changes over time, the way code does, rather than one that goes stale the week after launch.

A practical rule of thumb: if the maintenance line in your estimate is zero, the estimate is wrong, not lean. Even a feature that never gets a fine-tune or a model swap still needs someone checking eval scores on a schedule and someone who notices when the underlying model provider silently updates the model behind the API — which happens more often than most specs assume.

The One-Page Estimable Spec Template

A spec becomes estimable the moment every field below has a specific, falsifiable answer instead of an aspirational one — copy this table, fill it in against your own feature, and hand it to engineering as the starting point for a real sizing conversation rather than a debate about scope.

Keep this to one page on purpose. A spec engineering can't read in five minutes doesn't get read carefully, and a feature that can't fit these eight fields with real answers usually isn't ready to be estimated yet — it's ready for another round of data audit or scoping first.

Running the Interrogation as a 90-Minute Session

Turning six dimensions into a spec doesn't require a multi-week discovery phase — a single, structured 90-minute session with a PM, a tech lead, and whoever owns the relevant data can produce a first-draft estimable spec, provided the conversation is sequenced deliberately instead of free-form.

A workable sequence:

1. 0-15 min — State the feature in one sentence and the success metric. Resist the urge to discuss solutions yet.
2. 15-35 min — Data audit: walk the checklist from Dimension 1 out loud; assign follow-ups for anything unknown.
3. 35-55 min — Model approach: pick the default, and write down the specific evidence that would justify escalating.
4. 55-70 min — Latency, cost, and failure modes together, since they constrain each other (a tighter error tolerance often demands a slower, more careful pipeline).
5. 70-85 min — Effort split into build vs. maintenance, plus a named kill criterion.
6. 85-90 min — Fill the one-page template live, on screen, so everyone leaves with the same document.

If you're building this alongside other AI features on a shared roadmap, the same prioritization work that decides which feature gets a feasibility session in the first place — RICE or Kano scoring, weighed against effort estimates like these — is worth doing before, not after, you invest a 90-minute session in every AI idea that gets pitched.

Key Takeaways

• Treat every AI feature as a system with a data contract, an error budget, and a maintenance bill — not a toggle you turn on.
• An estimable spec answers six questions: data needs, model approach, latency, failure modes, effort, and a named kill criterion.
• Default to the cheapest model approach that clears the bar (prompt-only or retrieval) and reserve fine-tuning or custom models for a documented, evidence-based escalation.
• Split effort into build and maintenance — the ongoing bill is where most AI estimates quietly understate true cost.
• Name failure modes and their detection methods before launch; the costliest AI failures are confident and wrong, not crashed and obvious.
• Keep the spec on one page and treat it as living, versioned documentation that updates as data, models, and failure modes change.