01The three things being limited

Almost every model API enforces at least three independent limits. They run in parallel — you can hit any one of them, and "remaining headroom" on the others doesn't help.

On top of those: a daily organisational ceiling (your billing cap), per-model limits (cheap models often have tighter TPM than expensive ones on the same tier), and quietly-enforced per-key limits when admins split traffic. The "I have 60K TPM" you read in the docs is the most generous of these — the one that bit you was something else.

02The tier ladder

New accounts start in Tier 1: deliberately tiny limits, designed to keep abusers from spending themselves into bankruptcy on day one. You climb tiers by spending money and not being flagged. Tier 1 → Tier 5 is typically a 1–3 month journey of consistent legitimate usage.

Approximate; providers adjust these. Confirm in your console.

What gets you flagged backwards: a sudden burst from zero (looks like a stolen key), geographic anomalies (key issued in SF, traffic from three datacentres in different countries), bursts that match known abuse patterns. The mitigations are unglamorous: ramp slowly. Pre-deposit credit (some providers move you a tier up immediately on prepay). Open a support ticket with your use case before launch — provider rate-limit teams will manually bump you in days, not weeks, if you have a real story.

03The retry pattern that works

There is a correct shape for retries and it is everywhere in production code that doesn't fall over:

# pseudocode that maps to anything
attempt = 0
while attempt < MAX_RETRIES:
    resp = call_api(request)
    if resp.ok: return resp

    if resp.status not in {429, 500, 502, 503, 504}:
        raise # not retryable

    retry_after = float(resp.headers.get("retry-after", 0))
    base = 2 ** attempt                    # 1, 2, 4, 8…
    jitter = random.uniform(0.5, 1.5)        # spread the herd
    delay = max(retry_after, base * jitter)
    sleep(min(delay, MAX_BACKOFF))

    attempt += 1

raise RateLimitExhausted

Four things matter, in decreasing order of how much they hurt when missed:

Honour retry-after. Both Anthropic and OpenAI return this header on 429s. Use it as a floor, not a ceiling. Ignoring it just causes another 429 (and provider abuse counters tick up).

Add jitter. Without jitter, every client retrying at the same moment will synchronise. They'll all wake up at t+1s, hammer the API, all get 429, all wait 2s, hammer again. Multiply your backoff by a random 0.5–1.5 factor. Done. The simulator below shows what this single change does.

Cap your retry budget. 5 attempts. 2-minute total. After that, fail loudly with structured logging. Infinite retries turn a 30-second blip into a 4-hour outage and a bill you don't want to explain.

Only retry retryable errors. Most provider errors are your fault and re-sending the exact same broken request will get the exact same error back.

04Idempotency keys

Here is the worst kind of bug: a 504 timeout where the request did reach the server, the server did execute it, but the response packet was lost. Your retry executes it again. Now the user got two emails, was charged twice, has two rows in the database.

Solution: Idempotency-Key header. Generate a UUID per logical operation (not per attempt). Send the same UUID on retries. The server hashes it, looks up the cached response, and returns the original. Newer Anthropic and OpenAI endpoints honour it; for older endpoints you wrap your own service.

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Idempotency-Key: 7f3c2-a91-batch-row-49271" \
  # ^ same key on every retry of this logical op
  -d '{ "model": "claude-sonnet-4-6", "messages": [...] }'

Critical rule: the key must be deterministic for a logical operation, not random per-attempt. If you generate a fresh UUID inside your retry loop, you have idempotency keys that idempotency-protect nothing. Hash the input payload + the row ID + the date, or pass the key down from the queue worker that owns the job.

05Hedged requests for tail latency

For latency-critical interactive apps where p99 matters more than money: fire the request, wait 200ms, then fire a second identical request to a backup region or backup provider. First response wins; cancel the other. Costs roughly 2× in dollars but cuts p99 dramatically — because the slow request usually isn't slow because the model is slow, it's slow because your specific connection landed on a bad host, and the second attempt rolls the dice again.

# sketch — typically lives in your client wrapper
async def hedged_call(req, hedge_after_ms=200):
    primary = asyncio.create_task(call("us-east", req))
    done, _ = await asyncio.wait({primary}, timeout=hedge_after_ms/1000)
    if primary in done:
        return primary.result()
    hedge = asyncio.create_task(call("eu-west", req))
    done, pending = await asyncio.wait({primary, hedge},
                                       return_when=asyncio.FIRST_COMPLETED)
    for t in pending: t.cancel()                  # save money on the loser
    return next(iter(done)).result()

06Dynamic queue management

When you have a fleet of N workers hitting one provider, the failure mode isn't any single worker misbehaving — it's all of them being right but cumulatively wrong. Three patterns survive this:

Token bucket at the queue. A single shared bucket (Redis sorted set, in-process semaphore, whatever) that enforces your TPM locally before any request reaches the API. The API never sees the burst because you ate it. This is the most underused production pattern; it costs you a 5-line Redis Lua script and removes most 429s permanently.

Adaptive concurrency. The same algorithm as TCP congestion control — start with N concurrent workers, halve N when 429 rate spikes, slowly increase when no errors. AIMD (additive increase, multiplicative decrease). Self-tunes to whatever the provider is willing to give you today, including when they silently change it.

Per-tenant fairness. If you're multi-tenant and one customer ships a buggy loop, naive queueing means everyone queues behind their burst. Token-bucket per tenant, with the provider-level bucket on top as a global ceiling. The well-behaved tenants don't notice the abusive one.

07Watch the retry storm in action

Pick a load pattern and a retry strategy. Watch a 60-second timeline play out. The "API endpoint" in the middle has a token bucket: 100 requests/min, refilling at ~1.67 req/sec. Anything above that gets a 429.

08Three pages from the on-call log

09A representative on-call log

This is the shape of a request that survives, from one of our wrappers:

Note what's there: a stable req_id across retries, an idempotency-key on the side-effect-bearing send, the actual retry-after value being respected, jitter explicitly logged, and the remaining-tokens header captured so we can graph proximity to the limit.

10What to do when you see a 429