Rate-Limit LLM API Calls Across Workers Without a Queue

For: A backend engineer at a Series A SaaS startup whose LLM-powered feature just scaled to multiple Celery or Gunicorn workers and is now hitting OpenAI 429 errors in production despite each individual worker staying within its own per-minute limit

If each of your Celery or Gunicorn workers enforces its own token-per-minute counter, your effective rate limit is local_limit × worker_count, which is why OpenAI returns 429s even though every worker looks compliant. The fix is a single shared token bucket in Redis, updated with a Lua script so the check-and-decrement is atomic across processes. No queue, no message broker, no architectural rewrite — about 40 lines of code.

This tutorial walks through building it end to end, including the subtle race condition that makes naive Redis-based limiters silently broken under load.

Why local rate limiting fails the moment you scale horizontally

Most tutorials show a ratelimit or aiolimiter decorator that lives in-process. It works on your laptop. Then you deploy with gunicorn -w 4 or run Celery with a concurrency of 8, and every worker is now independently convinced it has the full 90,000 tokens-per-minute budget. Four workers × 90k = 360k tokens/min heading at OpenAI. You get burst 429s that don't correlate with any single worker's logs.

The instinct is to reach for a queue — push all LLM calls to a single Celery worker, throttle there. That works, but it adds latency, a single point of failure, and turns synchronous request paths into async ones. For a user-facing feature, that's a big surgery for a small problem.

The right primitive is a sliding-window token bucket in Redis. Every worker consults the same bucket. Refill is time-based. The bucket check and the decrement happen inside a Lua script, so Redis executes them as a single atomic operation. No race, no queue.

Why atomicity is non-negotiable

A naive implementation does this:

tokens = redis.get("bucket")
if tokens >= cost:
    redis.decrby("bucket", cost)
    proceed()

Under any real concurrency, two workers read tokens = 500, both decide they have room for a 400-token call, both decrement, and now the bucket is at -300. You've overshot the rate limit and Redis has no idea. Adding WATCH/MULTI works but is verbose and retries badly under contention. A Lua script executes atomically on the Redis server — no round-trips, no races.

Prerequisites

• Python 3.10+
• Redis 6+ running locally or reachable (any managed Redis — Upstash, ElastiCache, Redis Cloud — is fine)
• pip install redis openai tiktoken
• An OpenAI API key in OPENAI_API_KEY
• Basic familiarity with your existing worker setup (Celery, Gunicorn, RQ, whatever)

Step 1: Confirm Redis is reachable

redis-cli ping

Expected output:

PONG

If you're using a managed provider, test with the full URL:

redis-cli -u redis://default:PASSWORD@host:6379 ping

Step 2: Write the Lua script

Create token_bucket.lua. This is the whole limiter. Read it carefully — it's short.

-- KEYS[1]: bucket key (e.g. "llm:openai:gpt-4o")
-- ARGV[1]: capacity (max tokens)
-- ARGV[2]: refill_rate (tokens per second)
-- ARGV[3]: requested tokens
-- ARGV[4]: current timestamp (float seconds)

local capacity     = tonumber(ARGV[1])
local refill_rate  = tonumber(ARGV[2])
local requested    = tonumber(ARGV[3])
local now          = tonumber(ARGV[4])

local state = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(state[1])
local last_ts = tonumber(state[2])

if tokens == nil then
  tokens = capacity
  last_ts = now
end

-- refill based on elapsed time
local elapsed = math.max(0, now - last_ts)
tokens = math.min(capacity, tokens + elapsed * refill_rate)

if tokens >= requested then
  tokens = tokens - requested
  redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
  redis.call("EXPIRE", KEYS[1], 3600)
  return {1, tokens, 0}
else
  local deficit = requested - tokens
  local wait = deficit / refill_rate
  redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
  redis.call("EXPIRE", KEYS[1], 3600)
  return {0, tokens, wait}
end

What it returns: [allowed, remaining_tokens, retry_after_seconds]. If allowed == 1, you proceed. If 0, wait tells you exactly how long until you'd have enough tokens.

Step 3: Wrap it in Python

Create limiter.py:

import time
import redis

class RedisTokenBucket:
    def __init__(self, client: redis.Redis, key: str,
                 capacity: int, refill_per_sec: float):
        self.client = client
        self.key = key
        self.capacity = capacity
        self.refill = refill_per_sec
        with open("token_bucket.lua") as f:
            self.script = client.register_script(f.read())

    def acquire(self, tokens: int):
        allowed, remaining, wait = self.script(
            keys=[self.key],
            args=[self.capacity, self.refill, tokens, time.time()],
        )
        return bool(allowed), int(remaining), float(wait)

    def acquire_blocking(self, tokens: int, max_wait: float = 30.0):
        deadline = time.time() + max_wait
        while True:
            ok, remaining, wait = self.acquire(tokens)
            if ok:
                return remaining
            if time.time() + wait > deadline:
                raise TimeoutError(f"rate limit wait {wait:.2f}s exceeds max_wait")
            time.sleep(min(wait, 1.0))

register_script uses EVALSHA under the hood, so after the first call Redis has the script cached and each check is a single fast round-trip.

Step 4: Estimate tokens before you call OpenAI

OpenAI rate limits are enforced on tokens, not requests. You need to estimate before you spend. Use tiktoken:

import tiktoken

_enc = tiktoken.encoding_for_model("gpt-4o-mini")

def estimate_tokens(messages, max_output_tokens: int) -> int:
    # Rough but works: 4 tokens overhead per message + content tokens
    prompt = sum(len(_enc.encode(m["content"])) + 4 for m in messages)
    return prompt + max_output_tokens

Reserve for the worst case (prompt + max_tokens). After the call, if you want to be precise, you can refund unused output tokens back to the bucket — but for most workloads, over-reserving slightly is fine and much simpler.

Step 5: Wire it into your OpenAI call path

Create client.py:

import os, redis
from openai import OpenAI
from limiter import RedisTokenBucket
from tokens import estimate_tokens

r = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)
openai = OpenAI()

# gpt-4o-mini Tier 1: 200,000 TPM. Set your real limit from the OpenAI dashboard.
TPM = 200_000
bucket = RedisTokenBucket(
    client=r,
    key="llm:openai:gpt-4o-mini:tpm",
    capacity=TPM,
    refill_per_sec=TPM / 60.0,
)

def chat(messages, max_tokens=512):
    cost = estimate_tokens(messages, max_tokens)
    bucket.acquire_blocking(cost, max_wait=20.0)
    return openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        max_tokens=max_tokens,
    )

That's it. Every worker across every machine now shares one bucket. If you have separate limits for requests-per-minute (RPM) and tokens-per-minute (TPM), instantiate two buckets and acquire from both.

Step 6: Verify it works under real concurrency

Write a small stress test:

# stress.py
import concurrent.futures, time
from client import chat

def task(i):
    t0 = time.time()
    r = chat([{"role": "user", "content": "say hi in 5 words"}], max_tokens=20)
    return i, time.time() - t0

with concurrent.futures.ThreadPoolExecutor(max_workers=32) as ex:
    for i, dt in ex.map(task, range(200)):
        print(f"{i:3d} {dt:.2f}s")

Now — this is the important part — run it from multiple processes at once, simulating your Celery workers:

python stress.py & python stress.py & python stress.py & wait

Expected behavior: total throughput stays under your TPM cap regardless of how many processes you run. Individual calls block briefly when the bucket is low, then proceed. You should see zero 429s from OpenAI. If you kept a naive local limiter, you'd see 429s within seconds.

Watch the bucket live in another terminal:

redis-cli --repeat 100 -i 0.5 HGET llm:openai:gpt-4o-mini:tpm tokens

Step 7: Handle 429s that still slip through

Even with a correct limiter, OpenAI can return 429s for reasons outside your control: organization-wide limits, brief provider hiccups, or the reserved-vs-actual token gap. Always wrap the call in a retry with exponential backoff and respect the Retry-After header:

import time
from openai import RateLimitError

def chat_with_retry(messages, max_tokens=512, retries=5):
    for attempt in range(retries):
        try:
            return chat(messages, max_tokens)
        except RateLimitError as e:
            wait = getattr(e, "retry_after", None) or (2 ** attempt)
            time.sleep(wait)
    raise RuntimeError("exhausted retries")

Step 8: What to skip and what to add later

Ship it as-is. Come back for these when you actually need them:

• Per-tenant buckets. Change the key: llm:openai:gpt-4o-mini:tenant:{id}. Same script, different key. Free multi-tenancy.
• Multiple models. One bucket per model. OpenAI enforces limits per model.
• Priority classes. Give background jobs a smaller reserved fraction so user-facing traffic never starves.
• Metrics. Emit remaining to Prometheus/Datadog. When it hits zero for sustained periods, you need a higher OpenAI tier or a queue after all.

Common errors and how to fix them

Still seeing 429s after deploying the limiter

Three usual causes. First, check the OpenAI dashboard for your actual current-tier limit — not the tier you signed up for. New accounts move through tiers based on spend. Second, verify all workers point to the same Redis instance (not a local Redis on each pod). Third, if you're on a self-hosted Redis cluster, make sure the bucket key lives on one node — Lua scripts can't span shards.

NOSCRIPT No matching script errors

Happens after Redis restarts or failovers. The redis-py Script object handles this automatically by falling back from EVALSHA to EVAL, but only if you use register_script. Don't cache the SHA yourself.

Bucket appears to reset at random

The EXPIRE 3600 in the Lua script keeps the key from lingering forever on idle keyspaces. If your traffic is bursty with hour-plus gaps, either raise the TTL or accept that the bucket resets to full — which is actually fine, since a gap that long means you weren't near the limit anyway.

Clock skew between workers causes tokens to appear/disappear

The script uses time.time() from the calling worker. On a fleet with clock drift, you can get weird refill jumps. Fix: pass Redis's own time instead. Replace the timestamp arg with redis.call("TIME") inside the Lua script:

local t = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

Latency spikes when the bucket is near empty

acquire_blocking polls every second. Under sustained saturation, that means added latency. Two options: (a) accept it — you're rate-limited by the provider, this is honest backpressure; or (b) fail fast and return a 503 to your caller if wait > threshold, so the user sees a clear "try again" instead of a hung request.

When you should reach for a queue instead

Be honest about where this pattern stops working:

• Sustained overload. If your steady-state demand exceeds your OpenAI limit, no limiter helps — you're just choosing between queueing and dropping. Move to a proper job queue (Celery, SQS, Temporal) and process async.
• Fairness across tenants matters. A token bucket is first-come-first-served. If tenant A floods, tenant B waits. Per-tenant buckets fix this but only up to the shared cap.
• You need exactly-once semantics. The limiter blocks; it doesn't persist requests. If the worker crashes mid-wait, the request is lost. A durable queue is the right answer.

For most Series A SaaS teams hitting OpenAI 429s for the first time, though, the queue is overkill. A shared Redis token bucket with an atomic Lua script fixes the actual problem — that your workers can't see each other — with roughly the code shown above. Teams we work with at CodeNicely's AI studio often ship this pattern in an afternoon and don't revisit it until they outgrow their OpenAI tier.

Frequently Asked Questions

Why not use a Python library like aiolimiter or ratelimit?

Both are excellent for single-process rate limiting. Neither shares state across processes. The moment you run more than one Gunicorn worker or scale Celery to multiple pods, they enforce the limit N times independently. You need a shared store — Redis is the pragmatic choice.

Can I do this without a Lua script, using WATCH/MULTI?

Yes, but it's worse. Optimistic locking with WATCH retries on conflict, and under high contention (which is exactly when you need the limiter) retry storms hurt latency. Lua executes atomically on the server with no retries. It's also simpler to read.

Does this work with async code (asyncio, FastAPI)?

Yes. Use redis.asyncio instead of the sync client, make acquire and acquire_blocking coroutines, and swap time.sleep for asyncio.sleep. The Lua script is unchanged.

How do I handle OpenAI's separate RPM and TPM limits?

Create two buckets — one keyed on requests, one on tokens — and acquire from both before each call. If either denies, wait for whichever needs longer. The overhead is one extra Redis round-trip.

What if we're calling multiple LLM providers (OpenAI, Anthropic, Bedrock)?

One bucket per provider-model pair. Key them like llm:anthropic:claude-sonnet:tpm and llm:openai:gpt-4o:tpm. Each provider publishes its own limits and enforces them separately. Some teams also add a global "total LLM spend per minute" bucket as a safety cap on top.

How do I decide between fixing this in-app versus talking to someone?

If your team has the Redis and Python chops, the code above is enough — ship it. If your LLM feature is central to the product and you want a review of the broader architecture (fallbacks, per-tenant fairness, observability, prompt caching), contact CodeNicely for a personalized assessment.