DocsServerless InferenceInference Rate Limits

Inference Rate Limits

Inference limits are applied per user and model. If you use multiple API keys, requests made with those keys share the same limits for each model. Limits are coordinated across API servers.

Limit Types

Code	Description
`request_rpm`	Maximum requests per 60-second window.
`prompt_tpm`	Maximum prompt tokens per 60-second window.
`completion_tpm`	Maximum reserved or generated completion tokens per 60-second window.
`concurrency`	Maximum requests for the model that may run at the same time.

Rate limits apply to authenticated text, image, and video inference requests. Token limits apply only to text input and output; image and video requests use RPM and concurrency limits.

Rate-Limit Responses

When a limit is exhausted, the API returns 429 Too Many Requests. The Retry-After response header contains the number of seconds to wait before retrying the same model.

Request RPM

http

HTTP/1.1 429 Too Many Requests
Retry-After: 42
Content-Type: application/json

json

{
  "success": false,
  "error": "Request rate limit exceeded for this model.",
  "code": "request_rpm",
  "status": 429,
  "current": 100,
  "limit": 100
}

Prompt TPM

json

{
  "success": false,
  "error": "Prompt token rate limit exceeded for this model.",
  "code": "prompt_tpm",
  "status": 429,
  "current": 49990000,
  "limit": 50000000
}

Completion TPM

Completion tokens are reserved before a request is sent to the model. The reservation uses max_completion_tokens, max_output_tokens, or max_tokens. When none is supplied, the default reservation is 4096 tokens. The reservation is reconciled with provider usage after the response completes.

json

{
  "success": false,
  "error": "Completion token rate limit exceeded for this model.",
  "code": "completion_tpm",
  "status": 429,
  "current": 498000,
  "limit": 500000
}

Concurrency

The concurrency slot is released when the response finishes or the client disconnects.

json

{
  "success": false,
  "error": "Concurrent request limit exceeded for this model.",
  "code": "concurrency",
  "status": 429,
  "current": 10,
  "limit": 10
}

Capacity-Service Responses

Capacity enforcement fails closed. If the service cannot safely verify a limit, the inference request is not sent upstream and the API returns 503 Service Unavailable.

json

{
  "success": false,
  "error": "Capacity service unavailable.",
  "code": "redis_unavailable",
  "status": 503
}

An invalid_scope code may be returned with the same status when the authenticated user or resolved model ID is missing internally.

Client Retry Behavior

Read the Retry-After header from a 429 response.
Wait at least that many seconds before retrying the same model.
Use exponential backoff for repeated 429 or 503 responses.
Do not immediately retry concurrent requests in parallel.

View or Request Limits

You can view effective limits and submit a change request from Rate Limits in the console. A limit shown as unlimited has no configured enforcement for that dimension.

Image Generation API

Data Privacy