Most AI features don't need to run in real time — they need to feel real time, and those are different engineering problems. Online inference costs more, takes longer to build, and demands operational maturity batch never touches; batch precomputes answers on a schedule for a fraction of the cost. Decide which the user need actually requires before you scope anything.

Quick Answer: If a user is actively waiting on screen for one personalized answer, you likely need online inference. If the value can be computed ahead of the interaction — rankings, scores, recommendations — batch is cheaper, simpler, and usually just as good.

What "Real-Time" Actually Means in an AI Feature Request

"Real-time" in a feature request usually bundles three separate demands: low latency (the user is waiting on screen right now), freshness (the answer reflects recent data, not last week's), and context-awareness (the output depends on what just happened in this session). Untangling which one is actually required changes the entire build.

Stakeholders say "real-time" as shorthand for "good," not as a precise technical spec. Your job is to decompose the request before you estimate it:

  • Latency — does a human have to wait, right now, for this specific output? Measured in milliseconds to a few seconds.
  • Freshness — does the underlying data need to be minutes old rather than a day old? This is about how often the source data updates, not how the model runs.
  • Personalization-in-session — does the answer depend on something the user just did (typed a query, added an item to a cart) that didn't exist when any batch job last ran?

Only the first and third actually force online inference. Freshness alone — "make sure it reflects this morning's numbers" — is usually solvable by running batch more often, not by making it synchronous. This is the single most common architecture-scoping mistake: treating a freshness complaint as a latency requirement and building a much more expensive system than the ask demands.

This is exactly the kind of ambiguity worth resolving with the six questions engineering needs answered before it will scope an AI feature — one of which is, almost verbatim, "does this need to happen while the user is looking at the screen?" Get a concrete answer before anyone estimates a sprint count. Pair it with an explicit latency budget for the feature: a number in milliseconds, not an adjective.

Online Inference: What You're Actually Signing Up For

Online (real-time) inference runs the model synchronously inside the request path — a user acts, the model scores that specific input, and the response returns before the page moves on. It buys immediacy and full personalization, and it costs you in infrastructure, latency engineering, and ongoing operational load that a batch job never asks for.

Concretely, going online means committing to:

  1. Always-on serving infrastructure — a model endpoint (GPU or CPU pool) that's up and autoscaling 24/7, not a job that runs and exits.
  2. A p95/p99 latency SLO, not just an average — tail latency is what users actually feel, and it's the hardest number to hit reliably under load.
  3. Serving-training skew management — the exact features available at request time have to match what the model trained on, computed under a much tighter time budget.
  4. On-call and incident response — a model endpoint that degrades under load is a production incident, with the same urgency as any other API outage.
  5. Cold-start and fallback logic — what does the endpoint return when the model is slow, unavailable, or has no signal for a brand-new user?

None of this is exotic engineering — it's the same discipline as any low-latency API — but it's discipline your team pays for continuously, not once. Most of this list surfaces within the first fifteen minutes of a structured feasibility interview with an ML engineer, which is exactly why running one before scoping pays for itself. Ask what the p95 looks like under 3x expected load, not just under a demo.

The economics compound too: every model call in the online path is billed per request, scales with traffic, and needs headroom for spikes you can't smooth out with a queue. Contrast that with a batch job, which you can run whenever compute is cheap and size for average, not peak, load.

Batch and Precomputed: The Boring Option That Usually Wins

Batch inference runs on a schedule — nightly, hourly, every few minutes — against your full dataset, writes the results to a lookup table or cache, and your application just reads that table instantly at request time. It's slower to update and can't react mid-session, but it's dramatically cheaper, simpler to test, and easier to operate than an always-on endpoint.

The appeal isn't just cost. A batch pipeline is a data pipeline — the kind of system most engineering teams already know how to build, monitor, and debug, running on infrastructure they already have (a cron job, an orchestrator like Airflow, a scheduled query). There's no new latency SLO, no new on-call rotation, and no serving-training skew to chase, because scoring happens in a controlled offline environment with as much compute time as it needs.

The tradeoff is staleness. If a user's situation changes between batch runs — they browse a new category, their account status changes, a competitor drops a price — the precomputed answer won't know until the next run. For a large share of AI features, that lag is invisible to the user; for some, it's the whole feature failing. The work is figuring out which case you're actually in, which is a job for understanding the underlying need, not the interface.

That's a Jobs to Be Done question as much as an engineering one: the user isn't hiring your feature for "instant," they're hiring it to get a specific outcome, and the Jobs to Be Done framework is the sharpest tool for separating the outcome from the delivery mechanism they assumed it needed.

The Hybrid Pattern: Precompute Plus Fallback

The hybrid pattern precomputes the common case in batch and reserves a lightweight online path only for what batch can't cover — new entities with no history, or a signal that only exists at request time. It captures most of batch's cost savings while closing the gap on the cases that actually need freshness.

This is closer to how most production recommendation and personalization systems actually work than either pure extreme. The pattern looks like this:

  • Offline layer (batch): nightly or hourly job scores every known user against every known item, writes results to a fast key-value store.
  • Online layer (thin): at request time, read the precomputed result in single-digit milliseconds; only fall back to a live model call for cases the batch layer has no answer for (a brand-new user, a just-published item, an out-of-catalog query).
  • Refresh trigger (optional): an event — a purchase, a support ticket, a plan change — can invalidate a single user's cached result and queue a targeted recompute, without touching the nightly job for everyone else.

This "batch layer plus speed layer" split isn't a new idea — it's close to what Nathan Marz named the Lambda Architecture over a decade ago, precisely because most real-time-feeling systems are actually a fast, thin online layer sitting on top of a much larger offline computation. Uber's public engineering writeups on its Michelangelo ML platform describe the same offline/online feature-serving split for exactly this reason: most of the value comes from the batch layer, and the online layer only needs to handle what batch genuinely can't.

The hybrid pattern costs more than pure batch — you're still running two systems — but it's a fraction of the operational load of making everything synchronous, because the online path only has to be fast and reliable for a narrow slice of traffic.

At a glance, the three patterns trade off along the same handful of dimensions every time:

DimensionBatchHybridOnline
Latency to userN/A — precomputedFast for most, model call for edge casesBound by model + network on every request
FreshnessHours to a day staleFresh for edge cases, else batch-freshAlways current
Build effortLowModerateHigh
Ongoing ops burdenLow (job monitoring)Moderate (two systems)High (SLOs, on-call, autoscaling)
Typical fitRankings, digests, most recommendationsRecommendations with cold-start casesFraud checks, live pricing, chat

Read down the "ongoing ops burden" row before you read anything else — it's the cost stakeholders forget to ask about, and it's the one that compounds every week the feature stays live, not just at launch.

A Worked Example: Nightly Batch vs. Live-Scored Recommendations

Take a straightforward feature request: "show each user a personalized list of recommended items." Scored live, it's a real-time inference system with a latency SLO and an always-on endpoint. Computed nightly, it's a batch job and a lookup table. The two estimates for "the same feature" typically differ by an order of magnitude in both cost and build time.

DimensionNightly batchLive-scored (online)
Read latencySingle-digit ms (cache read)100-500ms+ per request (model call)
FreshnessUp to ~24 hours staleReflects this session's activity
Infra to stand upScheduled job + a cache/tableServing endpoint, autoscaling, load balancing
New operational surfaceJob-failure monitoringLatency SLOs, on-call rotation, canary deploys
Typical build effortDays to a few weeksSeveral weeks to a few months
Ongoing cost driverCompute for one run/day, sized for total volumeCompute per request, sized for peak traffic
Failure modeStale list until next runSlow or failed response the user sees immediately

The lookup-table row is doing the real work here: once a nightly job has scored every user, serving a recommendation is just a cache read, which is why its latency and infra cost look nothing like the online column. Read the row-by-row difference as your negotiating tool with stakeholders who default to "just make it real-time" without knowing what they're actually asking to pay for.

Before you accept either estimate, translate it into an explicit ask:

"Instant" is usually a want, not a need. The question isn't whether real-time inference is possible — it almost always is — it's whether the user's actual job is better served by that cost, or whether a nightly batch run delivers the same perceived value for a fraction of the build.

Two real, frequently-cited data points are worth keeping in the room for this conversation, without treating either as a universal law. Amazon engineer Greg Linden's widely-referenced internal finding was that every 100 milliseconds of added latency measurably cost sales; Google's own reported experiment (discussed publicly by Marissa Mayer) found that an extra half-second of search latency cut traffic by a double-digit percentage.

Both are directional evidence that latency has a real cost — not proof that every feature needs a sub-second response. A stale recommendation list is a completely different failure mode than a slow-loading page, and it's worth not conflating the two when the estimate comes back.

How to Decide: A Framework for Making the Call

Three questions determine which architecture actually fits a given feature, and they should be answered before anyone estimates a build. If the answer to all three is no, batch is enough — and probably better, because it's cheaper to build, easier to operate, and easier to reason about when it breaks.

  1. Does the value change meaningfully within a single session? If a user's next click should change what they see next, you need an online component somewhere in the loop.
  2. Can the user tolerate staleness measured in hours, not seconds? If "as of last night" is genuinely fine — most ranking, digest, and recommendation features qualify — batch wins outright.
  3. Does the output depend on something only knowable at request time — the current cart, the current search query, a live sensor reading — that no batch job could have precomputed? If yes, that specific slice needs an online path; the rest of the feature usually still doesn't.

Answering these honestly requires knowing what the user is actually trying to accomplish, not just what the request as written says. Mapping the customer journey and its emotion curve around the moment this feature fires is often more revealing than the feature request itself — a user who's frustrated and re-checking a page ten times in one sitting has a real latency need, while a user opening an app once a day for a digest usually doesn't.

If you haven't already framed the broader tradeoff, a complete framework for assessing AI feasibility walks through where architecture choice fits alongside data readiness and model risk.

A shorthand worth keeping on a sticky note

  • No + no + no → batch. Build the cheap version first.
  • Any single yes, narrowly scoped → hybrid. Keep the online path thin and reserved for the case that actually needs it.
  • Multiple yeses, core to the feature → online. Budget for the SLO, the on-call rotation, and the higher unit cost from day one — and say so in the estimate.

Where Prodinja Fits

Feasibility conversations like this one are exactly what Prodinja's Feature-to-Feasibility tool is designed to walk you through in the prototype: you describe the feature, and it's built to help you trace the latency requirement back to an architecture pattern — online, batch, or hybrid — so the cost and effort conversation happens before a sprint gets scoped around the wrong assumption, not after.

Key Takeaways

  • "Real-time" is usually three different requests bundled together — latency, freshness, and in-session context — and only some of them force online inference.
  • Online inference means committing to ongoing operational cost, not just a bigger one-time build: an SLO, on-call rotation, autoscaling, and serving-training skew management.
  • Batch is a data pipeline your team likely already knows how to run, at a fraction of the cost, with the tradeoff being staleness between runs.
  • The hybrid pattern — precompute plus a thin online fallback — covers most real production systems, including the pattern behind the Lambda Architecture and Uber's Michelangelo feature-serving split.
  • The same feature request can produce estimates that differ by an order of magnitude depending on which architecture it's scoped against — get that decision made before anyone commits to a timeline.
  • Latency has a real, well-documented cost — but a stale list and a slow page are different failure modes, and treating every freshness complaint as a latency complaint over-builds the system.

Frequently Asked Questions

How do I know if my AI feature actually needs real-time inference?

You need real-time inference only if the output must react to something that happened in the current session — a typed query, an item just added to a cart — and the user is actively waiting on screen for that specific answer. If the underlying value only needs to be a few hours fresh, batch almost always covers it.

Is batch inference the same as a cron job?

Functionally, yes for most feature-level purposes: a batch inference pipeline is a scheduled job that scores your dataset and writes results somewhere fast to read, whether that scheduler is a simple cron entry or a managed workflow orchestrator. The "inference" part is just what the job computes, not a different execution model.

How much more expensive is real-time AI inference than batch?

Directionally, an always-on serving endpoint typically costs several times more to build and run than an equivalent nightly batch job, because it has to be sized for peak traffic and staffed for ongoing operational load rather than sized for average volume once a day. Exact multiples vary by traffic pattern and model size, so treat any number here as a planning range, not a quote.

Can I start with batch and move to real-time later?

Yes, and it's often the better sequencing: shipping the batch version first validates that users actually want the feature before you invest in serving infrastructure, and the hybrid pattern (batch plus a thin online fallback) is a natural next step rather than a rewrite. Most teams that go straight to online inference are paying for freshness nobody asked for.

What's the difference between latency and freshness in an AI feature?

Latency is how long the user waits for a response once they've triggered it; freshness is how recently the underlying data was updated before that response was computed. A feature can have excellent latency (a fast cache read) on very stale data (last night's batch run), and the two need to be evaluated — and budgeted — separately.