Rate-Limit an LLM API Without Dropping User Requests

For: A backend engineer at a 15-person B2B SaaS startup who just shipped an LLM-powered feature to their first 200 users and is now watching 429 errors spike in Sentry every morning during the US East Coast business-hours surge

The right fix for OpenAI 429 errors is not a retry loop. It's a token-budget ledger sitting in front of your API client that decides — before a request leaves your server — whether it fits inside your current TPM/RPM window. If it doesn't, the request waits in a local priority queue until the budget refills. Users see 300ms of extra latency instead of a red toast. Your retry code stops existing. This post walks through building that in Python with asyncio and Redis, in a form you can drop into a FastAPI service today.

The mental model shift matters: you're not handling an HTTP error, you're managing a queue against a known resource budget. Once you internalize that, most of the "LLM rate limit handling" folklore (exponential backoff with jitter, tier bumps, sleeping in the request handler) becomes obviously wrong for user-facing traffic.

Why retries are the wrong primitive

When you catch a 429 and retry, three bad things happen. First, you've already burned latency on a round-trip that was doomed. Second, if every instance of your app retries independently, you create a thundering herd against a limit that's already exhausted. Third, exponential backoff turns a 2-second problem into a 30-second problem for the unlucky user who retries into a still-full window.

Bumping your OpenAI tier helps until it doesn't. Usage tiers raise the ceiling but do not smooth bursts. A morning surge from US East Coast users will still cluster into the same 60-second window regardless of tier. You need shaping, not more headroom.

The correct primitive is a token bucket — the same algorithm CDNs and API gateways use — but adapted to two dimensions (requests and tokens) and to the fact that LLM requests have variable token cost you must estimate before sending.

What we're building

1. A shared Redis-backed sliding-window ledger tracking tokens-per-minute and requests-per-minute consumed across all your app instances.
2. A local async priority queue in each worker that holds pending LLM calls when the ledger is full.
3. A pre-flight token estimator so we know a request's cost before we admit it.
4. A dispatcher that pulls from the queue as budget refills.

What this is bad at: it does not help you if your steady-state demand exceeds your tier limit. It smooths bursts. If your p50 load is above the limit, you need a bigger tier or a cheaper model — no queueing scheme fixes that.

Prerequisites

• Python 3.11+
• Redis 6+ running locally or reachable (we'll use it for the shared ledger)
• An OpenAI API key with a known TPM/RPM limit (check platform.openai.com/account/limits)
• Basic familiarity with asyncio

pip install openai==1.* redis==5.* tiktoken==0.7.* fastapi uvicorn

Step 1: Model your actual limits

Pull your current limits. For gpt-4o-mini on Tier 1 as of writing, you might see something like 200,000 TPM and 500 RPM. Write them down as constants — do not hardcode them in five files.

# limits.py
MODEL = "gpt-4o-mini"
TPM_LIMIT = 200_000        # tokens per minute
RPM_LIMIT = 500            # requests per minute
SAFETY_MARGIN = 0.9        # run at 90% of ceiling
EFFECTIVE_TPM = int(TPM_LIMIT * SAFETY_MARGIN)
EFFECTIVE_RPM = int(RPM_LIMIT * SAFETY_MARGIN)

The safety margin matters because OpenAI's counter and yours will drift. Running at 100% will produce 429s under normal jitter.

Step 2: Estimate token cost before sending

Use tiktoken to count input tokens. For output, take the max_tokens parameter as an upper bound. This is intentionally pessimistic — we'd rather leave budget on the table than overshoot.

# estimator.py
import tiktoken

_enc = tiktoken.encoding_for_model("gpt-4o-mini")

def estimate_tokens(messages: list[dict], max_output_tokens: int) -> int:
    input_tokens = sum(len(_enc.encode(m["content"])) for m in messages)
    # +4 tokens per message for role/formatting overhead
    input_tokens += 4 * len(messages)
    return input_tokens + max_output_tokens

Test it:

>>> estimate_tokens([{"role":"user","content":"Summarize this invoice"}], 500)
511

Step 3: Build the Redis sliding-window ledger

We use two Redis sorted sets — one for tokens, one for requests — keyed by timestamp. Members older than 60 seconds get trimmed on every check. This gives us a true sliding window, not a fixed 60-second bucket that resets.

# ledger.py
import time
import redis.asyncio as redis
from limits import EFFECTIVE_TPM, EFFECTIVE_RPM

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
WINDOW = 60  # seconds

async def try_reserve(tokens: int) -> float:
    """Returns 0.0 if reserved, or seconds to wait if not."""
    now = time.time()
    cutoff = now - WINDOW

    async with r.pipeline(transaction=True) as pipe:
        pipe.zremrangebyscore("tpm", 0, cutoff)
        pipe.zremrangebyscore("rpm", 0, cutoff)
        pipe.zrange("tpm", 0, -1, withscores=True)
        pipe.zcard("rpm")
        _, _, tpm_entries, rpm_count = await pipe.execute()

    tpm_used = sum(int(member.split(":")[0]) for member, _ in tpm_entries)

    if tpm_used + tokens <= EFFECTIVE_TPM and rpm_count < EFFECTIVE_RPM:
        member = f"{tokens}:{now}"
        await r.zadd("tpm", {member: now})
        await r.zadd("rpm", {str(now): now})
        return 0.0

    # Compute wait: time until oldest entry falls out of window
    if tpm_entries:
        oldest_ts = tpm_entries[0][1]
        return max(0.05, (oldest_ts + WINDOW) - now)
    return 0.05

Two things worth noting. First, we store the token count inside the sorted-set member so we can sum actual consumption, not just count entries. Second, when the reservation fails we return an estimated wait — the time until the oldest window entry expires. That estimate drives our queue's next wake-up.

Step 4: The local priority queue and dispatcher

Each app instance runs a single dispatcher coroutine that pulls from an asyncio.PriorityQueue. Priority lets you promote paid-tier users, interactive requests over background ones, etc.

# dispatcher.py
import asyncio
import uuid
from dataclasses import dataclass, field
from typing import Any
from openai import AsyncOpenAI
from ledger import try_reserve
from estimator import estimate_tokens

client = AsyncOpenAI()
queue: asyncio.PriorityQueue = asyncio.PriorityQueue()

@dataclass(order=True)
class Job:
    priority: int
    enqueued_at: float
    id: str = field(compare=False)
    messages: list = field(compare=False)
    max_tokens: int = field(compare=False)
    future: asyncio.Future = field(compare=False)

async def submit(messages, max_tokens=500, priority=5) -> Any:
    loop = asyncio.get_running_loop()
    job = Job(
        priority=priority,
        enqueued_at=loop.time(),
        id=str(uuid.uuid4()),
        messages=messages,
        max_tokens=max_tokens,
        future=loop.create_future(),
    )
    await queue.put(job)
    return await job.future

async def dispatcher():
    while True:
        job = await queue.get()
        tokens_needed = estimate_tokens(job.messages, job.max_tokens)
        wait = await try_reserve(tokens_needed)

        if wait > 0:
            # Put it back and sleep
            await queue.put(job)
            await asyncio.sleep(wait)
            continue

        asyncio.create_task(_execute(job))

async def _execute(job: Job):
    try:
        resp = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=job.messages,
            max_tokens=job.max_tokens,
        )
        job.future.set_result(resp)
    except Exception as e:
        job.future.set_exception(e)

Note that _execute runs as a fire-and-forget task, so the dispatcher immediately loops to admit the next job. This is what keeps throughput high — the dispatcher's only job is admission control, not waiting on HTTP.

Step 5: Wire it into FastAPI

# main.py
from contextlib import asynccontextmanager
import asyncio
from fastapi import FastAPI
from pydantic import BaseModel
from dispatcher import submit, dispatcher

@asynccontextmanager
async def lifespan(app: FastAPI):
    task = asyncio.create_task(dispatcher())
    yield
    task.cancel()

app = FastAPI(lifespan=lifespan)

class ChatRequest(BaseModel):
    prompt: str
    tier: str = "free"  # "paid" gets priority=1, "free" gets 5

@app.post("/chat")
async def chat(req: ChatRequest):
    priority = 1 if req.tier == "paid" else 5
    resp = await submit(
        messages=[{"role": "user", "content": req.prompt}],
        max_tokens=500,
        priority=priority,
    )
    return {"reply": resp.choices[0].message.content}

Run it:

uvicorn main:app --workers 1

Use --workers 1 initially. Multiple workers each run their own dispatcher and queue, which is fine — the Redis ledger is what coordinates them.

Step 6: Load-test against the ledger

Fire 100 concurrent requests with a small script:

# load.py
import asyncio, httpx, time

async def hit(client, i):
    start = time.time()
    r = await client.post("http://localhost:8000/chat",
                          json={"prompt": f"Say hello {i}"})
    return time.time() - start, r.status_code

async def main():
    async with httpx.AsyncClient(timeout=60) as c:
        results = await asyncio.gather(*[hit(c, i) for i in range(100)])
    latencies = [r[0] for r in results]
    errors = [r for r in results if r[1] != 200]
    print(f"p50={sorted(latencies)[50]:.2f}s p95={sorted(latencies)[95]:.2f}s errors={len(errors)}")

asyncio.run(main())

Expected output on a healthy setup:

p50=0.82s p95=3.14s errors=0

The p95 is your queue-wait tail. If it exceeds what your UX can absorb, you have three levers: raise your tier, lower max_tokens, or shed low-priority load (return a "try again in a moment" response for priority >= 8 when queue depth exceeds a threshold).

Step 7: Reconcile estimates with actuals

Your token estimate will be higher than actual usage because max_tokens is a ceiling. Over a day this leaves 10–20% of budget on the floor. Reconcile after each call:

# In _execute, after successful response:
actual = resp.usage.total_tokens
estimated = estimate_tokens(job.messages, job.max_tokens)
delta = estimated - actual
if delta > 0:
    # Refund unused tokens by inserting a negative entry
    await r.zadd("tpm", {f"-{delta}:{time.time()}": time.time()})

You'll need to adjust try_reserve to handle negative members in the sum — trivial change. This reconciliation typically recovers enough headroom to push effective throughput closer to your tier ceiling without changing the safety margin.

Common errors

openai.RateLimitError still appearing occasionally

Two causes. Either your safety margin is too tight (drop to 0.85), or you have background jobs (embeddings, batch scoring) sharing the same tier that aren't going through the queue. Every code path that hits the API must go through submit(). No exceptions.

Queue grows unbounded during sustained overload

Add a max queue depth. When exceeded, reject with 503 and a Retry-After header. This is backpressure — surface it to the client rather than letting requests pile up until timeout.

if queue.qsize() > 500:
    raise HTTPException(503, headers={"Retry-After": "10"})

Redis latency dominates tail p99

Each admission does two pipeline round-trips. If Redis isn't co-located, this hurts. Move Redis into the same AZ/VPC, or switch to a local in-memory ledger with periodic Redis sync if you can tolerate slight over-admission across instances.

Priority inversion — free users starved during paid surge

Add aging: increment priority (lower number) by 1 for every 30 seconds a job has been queued. Prevents indefinite starvation without giving up the priority scheme.

Multiple workers double-count on startup

The Redis ledger is authoritative. Local queue state is per-worker. On restart, the ledger persists — that's the whole point of putting it in Redis rather than in-process.

When this approach is wrong

If your requests are truly background (nightly report generation, bulk embeddings), skip the priority queue entirely and use OpenAI's Batch API — it runs at 50% cost with a 24-hour SLA and has separate, much higher limits. Reserve the pattern above for user-facing synchronous calls where latency matters.

If you're routing across multiple providers (OpenAI, Anthropic, Bedrock), the ledger needs to become per-provider, and you'll want a router that picks the least-loaded backend. At that point you're building a small internal gateway, which is a reasonable place to be for a team past ~500 users. Some of our work in AI product engineering has centered on exactly this transition — knowing when to graduate from a single-provider queue to a proper gateway.

Frequently Asked Questions

Why not just use exponential backoff with jitter as OpenAI recommends?

Backoff is correct for machine-to-machine, non-interactive workloads. For a user waiting on a UI, backoff turns a bounded queue wait into unbounded latency, and multiple retrying clients amplify the surge. Pre-flight admission control gives the user a predictable wait and eliminates retries entirely.

Does this work across multiple app instances or Kubernetes pods?

Yes — the Redis sliding-window ledger is the coordination point. Each pod runs its own local dispatcher and priority queue, but all of them consult the same Redis keys before admitting a request. This is why we don't use in-process token buckets: they cannot coordinate across replicas.

How do I handle streaming responses with this pattern?

Reserve the estimated tokens up front just like non-streaming. The complication is reconciliation — you only know actual usage when the stream ends. Track it in a completion callback and refund the delta to the ledger. Streaming does not change the admission logic, only the accounting.

What if OpenAI's actual limits and my configured limits drift apart?

Log every 429 you do receive along with your ledger's current consumption. If you're getting 429s while your ledger says you're at 70%, lower the safety margin. If you almost never get 429s and your queue is often full, raise it. Treat it as a control loop, not a one-time constant.

Can this pattern handle Anthropic, Bedrock, or other LLM APIs?

Yes, the pattern is provider-agnostic. Swap the token estimator (Anthropic uses a slightly different tokenizer, Bedrock varies by model) and the ledger keys. If you're doing multi-provider routing for redundancy, contact CodeNicely for a personalized assessment of the gateway architecture — the queue is one piece of a larger system at that point.