Open-model limits

Read the header, never a constant

Every 429 carries Retry-After. The per-key window, hosted admission and the model at capacity also send Retry-After-Ms.

Retry-After: <seconds>
Retry-After-Ms: <milliseconds>

Retry-After-Ms is the same instant in milliseconds. Read it first if your client supports it, and the chat API and Messages protocol SDKs both do: it is the only one of the two that can express a wait shorter than a second without rounding up. Retry-After is the RFC 9110 header in whole seconds, always at least 1, always rounded up, so it never advises you to retry early.

Layer 1, the connection gate

Before the API reads your key, a gate bounds how fast unrecognized credentials can be looked up, so a flood of invalid keys cannot crowd out real traffic. It applies per source address and, while the rate-limit store is healthy, across the service as a whole.

There is no warm-up and nothing to request: a new key works on its first call. With a healthy store, a key stops counting against the gate once it succeeds, so steady traffic from a working key never meets it, and rolling a new key out on one request before fanning out avoids it entirely. You are most likely to see it when starting many workers at once with a brand new key, or when a script is retrying a key that is simply wrong.

The budgets are not published, because this is a defensive control and it is tuned. A gate refusal carries no X-RateLimit-* headers, which sets it apart from the per-key window.

If the rate-limit store stops answering, each API server applies the same budgets in memory, without sharing counts with other servers. A key the server already trusted keeps bypassing the gate on that server for up to 15 minutes; other keys get that server's new-key budgets.

When the store is not configured or a request cannot be evaluated, the gate returns 503 limiter_unavailable rather than admitting unbounded work. Its Retry-After is randomized between 1 and 4 seconds, up to 8 seconds under heavy load, so refused clients do not all return at once.

Layer 2, the per-key request window

A rolling 60 second window while the rate-limit store is healthy. A key with no custom limit follows the workspace's current default, so a workspace upgrade applies without rotating the key. A custom per-key limit stays in place, clamped to the workspace maximum.

{
  "error": {
    "message": "Rate limit exceeded. Please retry after the reset time.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  },
  "request_id": "req_example_1000"
}

The window slides, so capacity returns gradually as individual requests age past 60 seconds. There is no clock edge where the whole budget refills at once, which is why pacing evenly beats bursting. During a store outage, each server counts the key's requests in its own fixed 60 second window, and the X-RateLimit-* headers describe that server's count. Going over the limit there returns 429. If the store is not configured, the request returns 503 limiter_unavailable rather than being served unmetered.

Hosted admission and requests with an Idempotency-Key still need the store, so they can return a retryable 503 during an outage. See the 503 codes.

Layer 3, hosted admission

The gates run in the order of the table below. Each one counts per model. A hosted refusal carries limit inside error, plus headers naming the gate that stopped you.

X-Hosted-Limit-Scope: <scope>
X-Hosted-Limit-Value: <limit>
Retry-After: <seconds>
Retry-After-Ms: <milliseconds>

Every hosted refusal uses the same envelope. This is the API-key concurrency case, and the numbers in it are an example:

{
  "error": {
    "limit": 16,
    "message": "This workspace reached its limit of 16 concurrent hosted requests for this model. API keys share this allowance. Retry after the interval in the Retry-After header, or send fewer concurrent requests across the workspace.",
    "type": "rate_limit_error",
    "param": null,
    "code": "hosted_per_key_concurrency_limit"
  },
  "request_id": "req_example_1001"
}

Always read error.limit and X-Hosted-Limit-Value from the response you got, never a number from this page. They change with capacity.

Hosted admission 429 responses also carry X-RateLimit-Limit-Tokens, X-RateLimit-Remaining-Tokens, and X-RateLimit-Reset-Tokens when the limiter has a measured workspace token snapshot. This includes concurrency refusals, so a client can back off with both the refusing concurrency limit and the remaining token budget in view.

Layer 4, the model at capacity

A request can pass every gate above and still find the model's servers full. When the server a request reaches is at capacity, the gateway tries one other server if the model has more than one. If every server it reached was full, the request answers 429 rate_limit_error with code hosted_model_at_capacity and a Retry-After header.

Wait for Retry-After and retry unchanged; the same request succeeds once a slot frees. It is not a limit on your workspace, so it carries no limit field and no X-Hosted-Limit-Scope header, and nothing is charged. An SDK with retries enabled handles it like any other 429.

Handling a 429 in code

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    max_retries=5,
)

response = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Health check"}],
)
print(response.choices[0].message.content)

An SDK with retries enabled does the right thing with no code from you, because every layer sends Retry-After. If you back off yourself, add jitter when many workers share a key so they do not all return at the same instant.

Three cases worth planning for

• A 429 before any request of yours has succeeded, on a key you just created, is the connection gate rather than a quota. Retry once after Retry-After with a single request, then resume normal concurrency.
• One request that alone exceeds the TPM limit is the case where waiting cannot help. Its refusal still carries Retry-After, but retrying it unchanged fails again. Reduce the input or the maximum output tokens before retrying. Hosted token-rate admission uses the same request token estimate as billing, not a byte count.
• Adding API keys does not buy capacity. Every key in a workspace draws on one shared concurrency allowance, so a second key does not raise it. Read the current allowance for a model from max_concurrent_requests_per_api_key in GET /v1/models rather than assuming a fixed figure.

Coding plans and Standby

A coding plan's 5-hour and weekly limits meter usage separately from request and capacity limits. Exhausting plan funding returns a 402 plan limit, with x-should-retry: false and no Retry-After.

Requests still running reserve their share of a limit until they finish. When those reservations alone hold the rest of a 5-hour or weekly limit, the credits cap, or today's Standby, a new request waits on the server for up to 30 seconds. If it is still held, the request gets 429 plan_window_busy with Retry-After and x-should-retry: true. error.details.window is 5h, weekly, credits_cap, or standby_day. The request is not charged and never moves to credits or Standby.

Pro and Team plans live at retirement on 2026-11-03 keep their earned token allowance after the plan ends. Net plan payments contribute to the earned token tier. Read the current limit from max_tokens_per_minute_per_workspace in GET /v1/models.

Plan-funded requests have their own concurrency allowance per model, published as max_concurrent_plan_requests_per_workspace in GET /v1/models. Standby serves chat only and depends on model capacity. A temporary plan or Standby refusal uses these 429 codes and includes retry timing:

Read the applied value from error.limit or X-Hosted-Limit-Value. A daily Standby amount does not guarantee that the model can admit a request now.

Related