Rate Limits Explained: Why Your AI Feature Breaks at Scale, Not in Demo

Rate limits are the ceiling most PMs never check until launch day: every LLM API caps how many requests per minute (RPM) and tokens per minute (TPM) an account can send, and demos rarely get close to either. At real concurrency, bursty traffic hits those caps in seconds, requests queue or fail, and a feature that felt instant in the demo starts timing out for real users.

Quick Answer: LLM providers throttle usage two ways — a request-count ceiling (RPM) and a token-volume ceiling (TPM) — set per API key and per pricing tier. Demos use one user sending occasional requests, so neither ceiling matters. Production has many users firing at once, so you must model peak concurrent throughput, add queuing and exponential backoff, and request tier increases before launch, not after.

What Are RPM and TPM Limits, Exactly?

RPM (requests per minute) and TPM (tokens per minute) are the two independent throttles that cap how much traffic your API key can send to an LLM provider in a rolling 60-second window. You can be under one limit and still blocked by the other. A feature sending few but massive requests can exhaust TPM while barely touching RPM, and vice versa.

Providers enforce both because a single request's cost varies enormously. A one-line classification prompt might use 50 tokens; a document-summarization call with a long context window attached might use 50,000. Counting requests alone would let one huge call starve every other tenant on shared infrastructure, so tokens are the real currency being metered — a concept covered in depth in why tokens, not words, are the AI billing unit.

Why Two Separate Ceilings Exist

• RPM protects against request-storm patterns — thousands of tiny calls hammering an endpoint, common in polling loops or retry storms gone wrong.
• TPM protects against compute-storm patterns — fewer calls, but each one demanding enormous inference compute (long prompts, large outputs, or both).
• Providers publish both because optimizing only one metric is easy to game; together they approximate actual GPU load.
• Limits typically apply per API key, sometimes per organization, and almost always per model — your GPT-4-class limit and your smaller-model limit are separate pools.

Anthropic, OpenAI, and Google all structure limits this way, though exact numbers and tiering differ by provider and account age — always check your specific dashboard rather than assume a number from a blog post, including this one.

Why Bursty Usage Trips Limits That "Should" Be Enough

Rate limits are almost always expressed as an average over a rolling window, but real usage is never a smooth average — it arrives in bursts tied to human behavior, batch jobs, and retries, which is exactly what trips a limit that looks generously sized on paper. A limit of 10,000 TPM sounds like plenty until 40 users load a page simultaneously at 9:00 a.m.

Three burst patterns show up in almost every AI feature:

1. Synchronized human behavior — a scheduled report, a Monday-morning dashboard, a webinar CTA — that sends hundreds of users into the feature within the same 60-second window.
2. Fan-out from a single user action — one click that triggers five parallel LLM calls (summarize, classify, extract, translate, embed) multiplies your effective request rate by five instantly.
3. Retry amplification — a naive retry-on-failure loop that resends immediately, doubling or tripling load on a system that is already over its limit, which is the single most common way a minor overage becomes a full outage.

A queueing-theory concept worth knowing here is Little's Law (L = λW), formalized by mathematician John Little: the number of requests in flight equals arrival rate times average wait time. It explains, mathematically, why a small increase in arrival rate during a burst can spike queue depth far faster than intuition suggests.

Queuing, Retries, and Backoff: The Three Defenses

Queuing, retries with exponential backoff, and jitter are the three standard mechanisms that keep a rate-limited AI feature usable instead of throwing errors at users the moment traffic spikes. None of them raises your actual limit — they change how gracefully you fail and recover when you hit it.

Why Naive Retries Make Outages Worse

A retry loop without backoff is the most common root cause of a rate-limit incident escalating into a full outage. If 500 requests fail at once and all retry after a flat 1-second delay, you've just recreated the exact same burst a second later — this is the thundering herd problem, well documented in distributed-systems literature going back to early Unix networking research. Exponential backoff with jitter — a technique popularized by Google's site reliability engineering practices — spreads retries across a widening, randomized window so they don't re-collide.

Designing the Queue Layer

• Set a maximum queue depth and a user-facing message ("processing, this may take a moment") rather than letting requests pile up invisibly until the whole service times out.
• Prioritize by request type — a real-time chat response should queue ahead of a background batch-summarization job.
• Emit a distinct error state for rate-limit rejections (HTTP 429) so client code can back off intelligently instead of treating it as a generic failure.
• Log every 429 with its token count, not just its count — this is the raw data you need for capacity planning, covered next.

Per-Tier Limits and How They Actually Scale

Every major LLM provider assigns RPM/TPM ceilings in tiers tied to cumulative spend, account age, or a manual enterprise agreement, and the jump between tiers is often 5-10x rather than incremental. A new account on a free or entry paid tier can be throttled at limits that comfortably serve a demo and nothing resembling production traffic.

This has a direct product-planning implication: tier upgrades are not instant, and some require a support conversation or a minimum usage history before the provider will approve a jump. If your launch date assumes a higher tier than your account currently has, that's a dependency with its own lead time — the same way a data migration or a security review would be.

1. Audit your current tier in the provider console before committing to a launch date.
2. Request a tier increase or dedicated capacity early, especially before a marketing push or public launch.
3. Model degraded-mode behavior for the gap between "current tier" and "target tier," since the increase may not land on your timeline.
4. Revisit tiering whenever you add a new model or feature — TPM pools are usually per-model, so adding a bigger context-heavy feature can silently starve an existing one sharing the same account.

Capacity Planning: A Worked Example

Capacity planning for an AI feature means translating expected peak concurrent users into a token-per-minute number you can compare directly against your provider's tier limits — and if you skip this step, you find out the ceiling exists in production, not in planning. The math is simple; the discipline of doing it before launch is what's usually missing.

Say a support-ticket summarization feature expects 200 peak concurrent users, each triggering one summarization call roughly every 30 seconds during active use, and each call averages 1,200 input tokens plus 300 output tokens (1,500 total).

If your current tier caps you at 200,000 TPM, this feature is structurally unable to serve its own stated peak load, no matter how well the code is written. That gap has to be closed by a tier upgrade, a smaller/cheaper model for part of the flow, response caching, or a hard concurrency limit on the feature itself — a product decision, not just an engineering fix.

Where This Math Usually Goes Wrong

• Underestimating output tokens — long-form generation (reports, drafts, plans) often produces far more tokens than the input prompt, and teams frequently plan against input size alone.
• Ignoring fan-out — if one user action triggers multiple calls (retrieval + generation + embedding via embeddings explained for PMs), multiply accordingly.
• Planning for average, not peak — average load rarely trips a limit; peak concurrent load is the number that matters, and it's usually 3-5x the daily average for consumer-facing features.
• Forgetting shared pools — if multiple features share one API key, they compete for the same TPM budget, and a spike in one silently degrades another.

Making Throughput Assumptions Explicit Before Hand-Off

Most rate-limit incidents trace back to one root cause: the throughput assumption was never written down anywhere engineering could see it before build started. It lived in a PM's head as "should be fine" rather than as a number checked against the provider's actual tier limits.

This is precisely the gap Prodinja's Spec Studio is designed to close. Its readiness gates prompt you to make assumptions like peak concurrent users, expected tokens per call, and target RPM/TPM explicit inside the living PRD — as a checklist item engineering can confirm or challenge — before the spec moves to hand-off, rather than discovering the mismatch during a load test or, worse, on launch day.

Tying Rate Limits Back to the Rest of the AI Stack

Rate limits don't sit in isolation — they interact with everything else about how your feature consumes an LLM. A feature with an unnecessarily long context window burns TPM faster per call; understanding tokens as the real billing unit makes the capacity math above intuitive rather than arbitrary. If you're new to how these pieces fit together, the complete guide to LLM fundamentals is the right starting point before diving deeper into any one constraint.

It's also worth mapping rate limits against the actual usage pattern your feature creates. A customer journey with a single high-intent moment (e.g., "generate my report") behaves very differently under load than a jobs-to-be-done pattern with constant background AI calls throughout a session — the first is bursty and spiky, the second is steadier but higher-volume overall, and each demands a different capacity-planning approach.

Key Takeaways

• RPM and TPM are separate, independent ceilings — you can be well under one and still blocked by the other, so model both.
• Bursty human behavior, not steady-state average traffic, is what trips rate limits — plan against peak concurrency, not daily averages.
• Naive retries without backoff and jitter turn a minor rate-limit event into a full outage by re-synchronizing traffic into a new burst.
• Tier upgrades have their own lead time and sometimes require a support conversation or usage history — treat them as a launch dependency, not a same-day fix.
• Capacity planning is simple arithmetic — concurrent users × calls per minute × tokens per call — but it only works if someone actually does it before build starts.
• Writing throughput assumptions into the spec itself, the way Spec Studio's readiness gates encourage, catches the mismatch during planning instead of during a launch-day incident.