Chat completions and Responses

What each model supports

Requesting a capability a model does not have returns 400 with error.code hosted_capability_not_supported and param naming the field.

A model that is temporarily paused is reported in the live list and on its Model Library page.

Give reasoning models room

Reasoning models can spend their output budget before they answer. Reasoning tokens count toward max_tokens before the answer, so set max_tokens to at least the figure in the dated model table below where the live output limit permits it. A smaller budget can be spent entirely on reasoning and return a completion whose content is empty.

Measured on this endpoint on 2026-08-14 by sending the prompt "Write a TypeScript retry helper." and recording how often a complete answer came back. The dated model-specific source reference identifies the models in these two probes; these are historical measurements, not current model limits.

A 2,048-token budget can be fully consumed by reasoning before an answer; 16,384 leaves more room where the live limit permits it. The probes above billed output under the historical contract. Under the current usage contract, a response with no answer settles at zero.

For another model, read the recommended output budget on its page in the Model Library.

The budget is a ceiling, not a spend.

{
  "choices": [
    {
      "finish_reason": "length",
      "message": { "content": null },
      "runinfra": {
        "output_status": {
          "code": "generation_limit_reached_before_answer",
          "message": "The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."
        }
      }
    }
  ],
  "usage": {
    "completion_tokens": 300,
    "runinfra": {
      "output_token_accounting": {
        "visible_answer_tokens": 0,
        "non_answer_completion_tokens": 300,
        "sources": {
          "visible_answer_tokens": "proxy_classified",
          "non_answer_completion_tokens": "provider_reported"
        }
      }
    }
  }
}

The cost of the request, and its cached input, on its usage

Every hosted model response carries the calculated charge for the request. On credits, it uses the settlement formula over the tokens the provider reported. On a coding plan or Standby, the calculated charge is zero and the usage value is reported separately. A response with no billable output settles at zero and prints 0. Beside the cost, every hosted response carries the count of input tokens that were billed at the cached input rate.

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 500,
    "total_tokens": 20500,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "cost": 0.00294,
    "runinfra": {
      "cost_microcents": 294000,
      "cached_input_tokens": 0
    }
  }
}

The cost covers the tokens the provider reported, at the model's prices for uncached input, cached input (when the model discloses a cache hit) and output. During a promotional free window it is 0, present, not absent.

The cached count is the number the cost was computed with. On a billable response, prompt_tokens - cached_tokens is exactly what was billed at the uncached input rate. The cached count is 0 rather than absent whenever nothing was billed as cached: a first send, a prompt too short to match, a response that settled at zero, or a model whose cache is shared across tenants. On a shared cache a hit is neither disclosed nor priced, so 0 there means no cached discount was applied, not that the cache did nothing. /v1/models says which models publish an isolated cache.

Your balance is debited in whole cents, and the fraction of a cent left over carries to your next request, so one debit can differ from one request's cost by less than a cent. Over any period the two agree to within one cent of carry, unless the balance has reached its debt floor, which can limit a debit.

On a stream, the cost and the cached count ride on the usage frame that is sent together with [DONE], so send stream_options: {"include_usage": true} to receive them (the one usage frame forwarded without it, when a stream produced no answer, carries them too). Content chunks carry neither. A stream that ends in an error never quotes a cost. A /v1/responses usage carries the cost in the same two fields, and the cached count as input_tokens_details.cached_tokens and runinfra.cached_input_tokens. The usage of an idempotent replay of a streamed request (X-RunInfra-Idempotent-Replay: true) carries the figures that were settled.

Read the funding headers for the request's admission decision. Plan usage reports current plan limits and policy; Credits and budget reports the credit balance.

If a connection ends before terminal usage arrives, do not infer a final charge from HTTP 200 or from a credit reservation. Recovery uses persisted usage evidence; missing or stale evidence cannot reconstruct every delivered token. Keep the request id and original Idempotency-Key to read the settled receipt; the same key never replays generated text. Getting a new answer requires a new key and starts new paid inference. Follow the retry procedure before retrying.

Turning reasoning off

On a model that supports disabling reasoning, reasoning_effort: "none" turns it off for one request. The completion then carries the answer and no reasoning stream. Because reasoning bills at the output rate, this is the largest per-request cost lever on short tasks.

Measured on this endpoint on 2026-08-16 with an identical prompt at temperature 0, turning reasoning off reduced the completion from 35 tokens to 3, from 335 to 5, and from 67 to 5 on three models. A one-line probe on another model on 2026-08-29 fell from 30 tokens at the default effort to 2 with none. These dated observations, whose models are identified in the source measurements, are not promised savings.

Some models cannot turn reasoning off. On Chat Completions they refuse reasoning_effort: "none" with 400, code hosted_parameter_not_supported and param: "reasoning_effort". The message names the smallest reasoning budget the model accepts: send that value.

On /v1/responses, a model that cannot turn reasoning off does not refuse reasoning: {"effort": "none"}: the request runs at the smallest budget the model supports, and the answer carries the header X-RunInfra-Hint-Warnings with reasoning_effort_clamped. The model still reasons, and those tokens count toward the output budget and bill at the output rate. To ask for that budget directly, send it. A model that cannot turn reasoning off and publishes no reasoning tiers at all (an empty reasoning_efforts) is not lowered: it refuses the request with the same 400 as on Chat Completions.

When you send no reasoning_effort, a model uses its default_reasoning_effort from the model list (null means none is declared, not that reasoning is off). One exception: for some models a request that sends no effort, can call tools (a non-empty tools and a tool_choice that is not "none") and has no JSON response_format runs at a lower effort than that default. Tool calls in the history alone do not trigger it. Whenever the gateway chooses the effort for a request that did not name one, the response header X-RunInfra-Reasoning-Effort-Applied carries the value it applied.

Use the live reasoning_efforts for measured, distinct tiers and default_reasoning_effort for the declared default. Some reasoning models publish no measured tiers; an empty list does not turn reasoning off. The model-specific source reference records the dated tier measurements and accepted synonyms. On some models reasoning_effort: "minimal" is refused with 400 and param: "reasoning_effort"; send the smallest accepted effort named in the refusal. On the source model with that restriction, reasoning: {"effort": "minimal"} maps to low.

The source's effort measurements used repeated runs per level at temperature 0 on 2026-08-16, with later observations dated individually. The following are model-specific observations, not universal reasoning tiers; the source reference identifies each configuration, and the live fields above carry its current measured tiers and declared default.

• One configuration used max by default and a reproducibly distinct medium for a request with tools and no JSON format when effort was omitted. The exact automatic-effort condition and applied-value header are described above.
• Another answered without reasoning at the default, leaving nothing to turn off; its effort levels were not measured.
• A configuration with distinct low, medium and xhigh produced 113, 140 and 89 completion tokens on the probe, with xhigh equal to the omitted default. high was sent as xhigh. It refused none; send the smallest budget named by the refusal.
• Another measurably changed its reasoning at medium against a twice-identical baseline. high and max were sent as xhigh, its maximum and default.
• On a configuration whose effort deltas stayed inside its own run-to-run variance, treat effort as the off switch only. Another reasoned at the default and accepted none to turn it off, without measured levels between them.
• In the 2026-08-27 measurement, low and high were honored as sent. Omitting the field, or sending another accepted value including medium, ran at the model's maximum effort. none was refused; low was its smallest budget.
• In the endpoint measurement on 2026-10-19, low, high and max were distinct tiers, and the omitted field used max. minimal, medium and xhigh were accepted synonyms for max. Chat Completions refused none.
• In the 2026-08-30 measurement, none returned the answer directly; high and max were sent as xhigh, its maximum and default; low and medium were accepted as sent. Its refusal of the top-level minimal spelling, and the reasoning-object alternative, are described above.

For a reasoning object, the source's compatibility mapping compares the requested sequence none / disabled, minimal, low, medium, high, xhigh, max. The model-specific source reference shows how each model maps an explicit effort level. For a model that cannot turn reasoning off, the current Responses behavior and warning header above apply; a model with no published tiers refuses the off request rather than being lowered. Do not infer these exceptions from an empty tier list.

What the gateway does with a field you send

Checked, then sent:

A message's role is system, developer, user, assistant or tool. Any other role returns 400. content may be a string, an array, or null. An assistant message needs content or a non-empty tool_calls. Each tool_calls[].function.arguments must hold a JSON object, such as "{}" for a call with no arguments. A tool message needs content and a non-empty string tool_call_id. Every other role needs content.

Output limits

The body-size and image-count limits count every message in the request, including the images and history a client resends each turn. Base64 adds about a third to every inline image, so a 32MB body carries about 24MB of image files in total; drop older images from the conversation, split them across requests, or use a public image URL on a vision model. Remote image downloads have separate limits.

A text-only request that sets neither max_tokens nor max_completion_tokens defaults to 131,072 output tokens when context permits, or the model's own lower default for JSON output. Set an explicit budget to control the request.

Set an explicit max_tokens budget using the selected model's live max_output_tokens and remaining context. A generation that reaches its output budget reports finish_reason: "length", including when reasoning consumes the budget before an answer. Oversized max_tokens is clamped to the remaining context, with X-RunInfra-Output-Clamped: true and X-RunInfra-Output-Token-Ceiling reporting the value used.

Two bounds apply at once and the smaller one wins. Tokens are bounded by the remaining context above; wall-clock is bounded by the 740 second response limit, which at typical decode rates is the binding limit for very long generations. /v1/models publishes both, max_output_tokens beside response_time_ceiling_seconds, which carries the same 740 second figure, so size a long generation against the pair rather than the token figure alone.

The 504 message names the budget that was exceeded. For a non-streaming request, the error message suggests two remedies: stream: true (response headers arrive with the first token) and a smaller max_tokens. A first-token timeout on a stream suggests a smaller prompt, or retrying when the model has more capacity.

A model's context window is a separate per-model limit, published on that model's page in the Model Library. Served context is a property of how the model is served, not of the weights, so read it on the model page rather than a model card.

Three things a request can be refused for

Each one is refused up front with a 400 naming the field, rather than answered with output you cannot use.

{
  "model": "MODEL_ID",
  "messages": [{ "role": "user", "content": "Classify and rewrite this ticket." }],
  "reasoning_effort": "none",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "triage",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "integer", "enum": [1, 2, 3, 4] },
          "rewrite": { "type": "string" }
        },
        "required": ["category", "rewrite"],
        "additionalProperties": false
      }
    }
  }
}

{
  "type": "image_url",
  "image_url": {
    "url": "https://images.example.com/photo.jpg",
    "detail": "auto"
  }
}

Prefix cache retention

Retention is what GET /v1/models states as cache_tiers and cache_retention. A model reporting cache_retention: "tiered_host_memory" can reuse an idle session's cached prefix beyond immediate memory pressure when the session resumes. Every other chat model reports "best_effort": cached prefixes are retained opportunistically and may be evicted at any time, with no retention promise. Both policies remain best effort. See Session affinity to keep follow-up turns close to a warm cache, and read usage.prompt_tokens_details.cached_tokens for each request's cached input.

Calling a paused model

A hosted model can be temporarily paused. The request returns 503 with code hosted_model_paused, charges nothing, and carries a Retry-After header with the seconds until availability is next rechecked, capped at 3,600, or 60 when no recheck time is known or the known one has already passed. See the 503 codes.

Related

Responses

The open models API serves POST /v1/chat/completions, POST /v1/responses, POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models, GET /v1/models/{model}, GET /v1/credits, and GET /v1/usage. Chat completions and Responses are chat-compatible. Messages and token counting use the Messages-compatible contract.

Not supported anywhere: /v1/completions (legacy, non-chat), /v1/files, /v1/assistants, /v1/threads, and /v1/batches. /v1/responses is a chat-completions compatibility adapter and does not implement state, hosted tools (web search, file search), conversation storage, or background jobs.

On a model whose page lists tool calling, function tools work end to end on /v1/responses:

• Tools, in both spellings: declare tools flat ({"type": "function", "name", "parameters"}, the Responses shape) or nested under function (the chat shape).
• Calls and results: the model's calls come back as function_call output items with call_id, name, and arguments; send results back as function_call_output input items with the matching call_id (string or content-part array), and resend prior output items in input since the endpoint is stateless.
• Streams carry the full item lifecycle (response.created and response.in_progress, response.output_item.added, response.content_part.added, text and function_call_arguments deltas, the matching done events, and response.completed, which ends every stream), so Codex-style clients that build items from output_item.done work unchanged. stream_options: {"include_obfuscation": true} adds random padding to each delta.
• Harness fields: store is accepted and always treated as false, nothing is retrievable later. metadata, include (empty or any documented value; only reasoning.encrypted_content can add output here), truncation ("auto" or "disabled"), parallel_tool_calls, service_tier, client_metadata, safety_identifier, stream_options, max_tool_calls, and prompt_cache_retention are accepted; the hints among them have no effect. Input is never truncated: a prompt longer than the context window returns context_length_exceeded. previous_response_id, conversation, and prompt are accepted only as null, and background only as false or null; other values are refused with the reason.

On /v1/responses, output_text carries the final answer only. The model's thinking arrives as its own reasoning output item ahead of the message (raw text under content as reasoning_text parts, summary empty), and on streams as response.reasoning_text.delta and response.reasoning_text.done events inside a response.content_part.added and response.content_part.done pair. A completion that spends its whole token budget thinking returns an empty output_text with the thinking preserved in the reasoning item, so raise max_output_tokens if you see that.

A reply that max_output_tokens or a content filter cut short is not whole, and the response says so: status is "incomplete" and incomplete_details.reason is max_output_tokens or content_filter. A non-streaming body carries it directly. A stream still ends on response.completed, never on response.incomplete, and the response inside that event carries the same status and reason. Check status before you treat an answer as complete. Item status is advisory: only the item being written when the reply stopped is incomplete, so parse a function call's arguments before you run it.