Rate Limits & Overload Handling

Rate Limits

What you’ll see when limited

• HTTP status: 429 Too Many Requests
• Headers: Standard rate-limiting headers are returned, including RateLimit-* headers when applicable, and a Retry-After header in some cases. Use these to determine when to retry.
• Body: A brief message indicating the request was rate limited.

# Pseudocode logic for client retries
if status == 429:
  wait = response.headers.get('Retry-After') or derive_from_rate_limit_headers()
  sleep(wait_seconds)
  retry()

Provider-level throttling

• HTTP status: 503 Service Unavailable
• Header: Retry-After: <seconds> — indicates when it is safe to retry
• Body (JSON):

{
  "error": {
    "type": "upstream_rate_limited",
    "scope": "provider:<provider>",
    "detail": "Provider budget temporarily exhausted"
  }
}

Overload: HTTP 503

• HTTP status: 503 Service Unavailable
• Header: Retry-After: <seconds> — how long to wait before retrying

1. Read the Retry-After header.
2. Wait the indicated number of seconds.
3. Retry with exponential backoff if needed.

Best Practices

• Prefer exponential backoff with jitter for retries.
• Always honor Retry-After when present.
• Keep requests within reasonable volumes to avoid throttling.
• Consider queuing or batching non-urgent work.