# Chat completions and Responses

[Open models](https://www.rightnowai.co/docs/open-models) / [Chat completions and Responses](https://www.rightnowai.co/docs/open-models/chat-completions)

Updated 2026-11-11

**Connection**

| Connection | Value |
| --- | --- |
| API base | `https://api.runinfra.ai/v1` |
| Workspace key | `rp_...` |
| Process variable | `RUNINFRA_GATEWAY_KEY` |
| Billing | The existing funding rules of your open-model account. |

Replace `MODEL_ID` with an available id from [Open models](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models); read current capabilities, limits and prices from `GET /v1/models`.

**Text**

```text
POST https://api.runinfra.ai/v1/chat/completions
```

Send a standard chat completion from the SDK you already use, with a live model id from [`GET /v1/models`](https://www.rightnowai.co/docs/open-models/models).

**cURL**

```shell
curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Explain exponential backoff."}],"max_tokens":16384}'
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
response = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Explain exponential backoff."}],
    max_tokens=16384,
)
print(response.choices[0].message.content)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const response = await client.chat.completions.create({
  model: "MODEL_ID",
  messages: [{ role: "user", content: "Explain exponential backoff." }],
  max_tokens: 16384,
});
console.log(response.choices[0]?.message?.content);
```

## What each model supports

Requesting a capability a model does not have returns `400` with `error.code` `hosted_capability_not_supported` and `param` naming the field.

A model that is temporarily paused is reported in the live list and on its Model Library page.

## Give reasoning models room

Reasoning models can spend their output budget before they answer. Reasoning tokens count toward `max_tokens` before the answer, so set `max_tokens` to at least the figure in the dated model table below where the live output limit permits it. A smaller budget can be spent entirely on reasoning and return a completion whose `content` is empty.

Measured on this endpoint on 2026-08-14 by sending the prompt "Write a TypeScript retry helper." and recording how often a complete answer came back. The [dated model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies the models in these two probes; these are historical measurements, not current model limits.

**Dated reasoning-budget measurements**

| Source probe | Use at least | What smaller budgets did |
| --- | --- | --- |
| First reasoning-model probe | 16,384 tokens | At 2,048 and 4,096 the answer was empty every time. At 8,192 it was empty half the time. At 16,384 the model still stopped on its own after roughly 7,500 output tokens; raising the ceiling further did not make it generate more. |
| Second reasoning-model probe | 4,096 tokens | At 2,048 the answer was empty in 2 of 5 runs and cut off mid-sentence in 4 of 5. |

A 2,048-token budget can be fully consumed by reasoning before an answer; 16,384 leaves more room where the live limit permits it. The probes above billed output under the historical contract. Under the current usage contract, a response with no answer settles at zero.

For another model, read the recommended output budget on its page in the [Model Library](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models).

The budget is a ceiling, not a spend.

### How to detect an exhausted budget in code

The open-models service preserves `finish_reason: "length"` and adds `choices[].runinfra.output_status` when a choice reaches its generation limit before producing final answer content.

**JSON**

```json
{
  "choices": [
    {
      "finish_reason": "length",
      "message": { "content": null },
      "runinfra": {
        "output_status": {
          "code": "generation_limit_reached_before_answer",
          "message": "The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."
        }
      }
    }
  ],
  "usage": {
    "completion_tokens": 300,
    "runinfra": {
      "output_token_accounting": {
        "visible_answer_tokens": 0,
        "non_answer_completion_tokens": 300,
        "sources": {
          "visible_answer_tokens": "proxy_classified",
          "non_answer_completion_tokens": "provider_reported"
        }
      }
    }
  }
}
```

Branch on `choices[].runinfra.output_status.code`. `generation_limit_reached_before_answer` means the choice ended for `length`. If another terminal `finish_reason` produced no answer content, the code is `no_answer_content` and you should inspect `finish_reason` before retrying.

The aggregate `usage` annotation appears only when every choice lacks a final answer. The open-models service does not estimate a reasoning-token count: `completion_tokens` stays the model-reported output total, and `non_answer_completion_tokens` records that those tokens produced no answer. The annotation changes no pricing; a response with no billable output settles at zero, as described below.

## The cost of the request, and its cached input, on its usage

Every hosted model response carries the calculated charge for the request. On credits, it uses the settlement formula over the tokens the provider reported. On a coding plan or Standby, the calculated charge is zero and the usage value is reported separately. A response with no billable output settles at zero and prints `0`. Beside the cost, every hosted response carries the count of input tokens that were billed at the cached input rate.

**JSON**

```json
{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 500,
    "total_tokens": 20500,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "cost": 0.00294,
    "runinfra": {
      "cost_microcents": 294000,
      "cached_input_tokens": 0
    }
  }
}
```

**Reference 144**

| Field | Type | Value |
| --- | --- | --- |
| `usage.cost` | number | US dollars, to eight decimals. One microcent is exactly 0.00000001 dollars, so nothing is rounded away. |
| `usage.runinfra.cost_microcents` | integer | Calculated charge in microcents, the canonical figure: one cent is 1,000,000 microcents. Zero when the plan or Standby paid. |
| `usage.runinfra.plan_value_microcents` | integer | Usage value at the applicable per-token rates. Present only when the plan or Standby paid. |
| `usage.runinfra.paid_from` | string | `plan` or `standby`, present only with plan-funded or Standby-funded usage. Credit-funded usage retains the ordinary cost fields. |
| `usage.prompt_tokens_details.cached_tokens` | integer | Input tokens billed at the cached input rate, in the field chat API clients already read. Present on every hosted response; `0` when no input was billed as cached. |
| `usage.runinfra.cached_input_tokens` | integer | The same count, beside the cost, for a reader that only looks at the `runinfra` object. |

The cost covers the tokens the provider reported, at the model's prices for uncached input, cached input (when the model discloses a cache hit) and output. During a promotional free window it is `0`, present, not absent.

The cached count is the number the cost was computed with. On a billable response, `prompt_tokens - cached_tokens` is exactly what was billed at the uncached input rate. The cached count is `0` rather than absent whenever nothing was billed as cached: a first send, a prompt too short to match, a response that settled at zero, or a model whose cache is shared across tenants. On a shared cache a hit is neither disclosed nor priced, so `0` there means no cached discount was applied, not that the cache did nothing. `/v1/models` says which models publish an isolated cache.

Your balance is debited in whole cents, and the fraction of a cent left over carries to your next request, so one debit can differ from one request's cost by less than a cent. Over any period the two agree to within one cent of carry, unless the balance has reached its debt floor, which can limit a debit.

On a stream, the cost and the cached count ride on the usage frame that is sent together with `[DONE]`, so send `stream_options: {"include_usage": true}` to receive them (the one usage frame forwarded without it, when a stream produced no answer, carries them too). Content chunks carry neither. A stream that ends in an error never quotes a cost. A `/v1/responses` usage carries the cost in the same two fields, and the cached count as `input_tokens_details.cached_tokens` and `runinfra.cached_input_tokens`. The usage of an [idempotent replay](https://www.rightnowai.co/docs/open-models/idempotent-retries) of a streamed request (`X-RunInfra-Idempotent-Replay: true`) carries the figures that were settled.

Read the [funding headers](https://www.rightnowai.co/docs/open-models/usage#response-funding-headers) for the request's admission decision. [Plan usage](https://www.rightnowai.co/docs/open-models/usage) reports current plan limits and policy; [Credits and budget](https://www.rightnowai.co/docs/open-models/credits) reports the credit balance.

If a connection ends before terminal usage arrives, do not infer a final charge from HTTP `200` or from a credit reservation. Recovery uses persisted usage evidence; missing or stale evidence cannot reconstruct every delivered token. Keep the request id and original `Idempotency-Key` to read the settled receipt; the same key never replays generated text. Getting a new answer requires a new key and starts new paid inference. Follow the [retry procedure](https://www.rightnowai.co/docs/open-models/idempotent-retries) before retrying.

## Turning reasoning off

On a model that supports disabling reasoning, `reasoning_effort: "none"` turns it off for one request. The completion then carries the answer and no reasoning stream. Because reasoning bills at the output rate, this is the largest per-request cost lever on short tasks.

Measured on this endpoint on 2026-08-16 with an identical prompt at temperature 0, turning reasoning off reduced the completion from 35 tokens to 3, from 335 to 5, and from 67 to 5 on three models. A one-line probe on another model on 2026-08-29 fell from 30 tokens at the default effort to 2 with `none`. These dated observations, whose models are identified in the [source measurements](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models), are not promised savings.

Some models cannot turn reasoning off. On Chat Completions they refuse `reasoning_effort: "none"` with `400`, code `hosted_parameter_not_supported` and `param: "reasoning_effort"`. The message names the smallest reasoning budget the model accepts: send that value.

On `/v1/responses`, a model that cannot turn reasoning off does not refuse `reasoning: {"effort": "none"}`: the request runs at the smallest budget the model supports, and the answer carries the header `X-RunInfra-Hint-Warnings` with `reasoning_effort_clamped`. The model still reasons, and those tokens count toward the output budget and bill at the output rate. To ask for that budget directly, send it. A model that cannot turn reasoning off and publishes no reasoning tiers at all (an empty `reasoning_efforts`) is not lowered: it refuses the request with the same `400` as on Chat Completions.

When you send no `reasoning_effort`, a model uses its `default_reasoning_effort` from the model list (`null` means none is declared, not that reasoning is off). One exception: for some models a request that sends no effort, can call tools (a non-empty `tools` and a `tool_choice` that is not `"none"`) and has no JSON `response_format` runs at a lower effort than that default. Tool calls in the history alone do not trigger it. Whenever the gateway chooses the effort for a request that did not name one, the response header `X-RunInfra-Reasoning-Effort-Applied` carries the value it applied.

Use the live `reasoning_efforts` for measured, distinct tiers and `default_reasoning_effort` for the declared default. Some reasoning models publish no measured tiers; an empty list does not turn reasoning off. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) records the dated tier measurements and accepted synonyms. On some models `reasoning_effort: "minimal"` is refused with `400` and `param: "reasoning_effort"`; send the smallest accepted effort named in the refusal. On the source model with that restriction, `reasoning: {"effort": "minimal"}` maps to `low`.

The source's effort measurements used repeated runs per level at temperature 0 on 2026-08-16, with later observations dated individually. The following are model-specific observations, not universal reasoning tiers; the source reference identifies each configuration, and the live fields above carry its current measured tiers and declared default.

- One configuration used `max` by default and a reproducibly distinct `medium` for a request with tools and no JSON format when effort was omitted. The exact automatic-effort condition and applied-value header are described above.
- Another answered without reasoning at the default, leaving nothing to turn off; its effort levels were not measured.
- A configuration with distinct `low`, `medium` and `xhigh` produced 113, 140 and 89 completion tokens on the probe, with `xhigh` equal to the omitted default. `high` was sent as `xhigh`. It refused `none`; send the smallest budget named by the refusal.
- Another measurably changed its reasoning at `medium` against a twice-identical baseline. `high` and `max` were sent as `xhigh`, its maximum and default.
- On a configuration whose effort deltas stayed inside its own run-to-run variance, treat effort as the off switch only. Another reasoned at the default and accepted `none` to turn it off, without measured levels between them.
- In the 2026-08-27 measurement, `low` and `high` were honored as sent. Omitting the field, or sending another accepted value including `medium`, ran at the model's maximum effort. `none` was refused; `low` was its smallest budget.
- In the endpoint measurement on 2026-10-19, `low`, `high` and `max` were distinct tiers, and the omitted field used `max`. `minimal`, `medium` and `xhigh` were accepted synonyms for `max`. Chat Completions refused `none`.
- In the 2026-08-30 measurement, `none` returned the answer directly; `high` and `max` were sent as `xhigh`, its maximum and default; `low` and `medium` were accepted as sent. Its refusal of the top-level `minimal` spelling, and the reasoning-object alternative, are described above.

For a reasoning object, the source's compatibility mapping compares the requested sequence `none / disabled, minimal, low, medium, high, xhigh, max`. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) shows how each model maps an explicit effort level. For a model that cannot turn reasoning off, the current Responses behavior and warning header above apply; a model with no published tiers refuses the off request rather than being lowered. Do not infer these exceptions from an empty tier list.

### How the effort levels behave between none and the ceiling

The vendor spellings `enable_thinking`, `chat_template_kwargs.enable_thinking`, `thinking_budget` and `thinking` are accepted for request-shape compatibility and have no effect on any listed model: a request relying on them runs at the model's default effort and can still produce billable reasoning. Send `reasoning_effort` or the `reasoning` object instead; both are honored.

### The reasoning object

`reasoning` accepts `effort`, `enabled`, `exclude`: the unified reasoning object some routers send. `include_reasoning` is boolean. Conflicting `reasoning` and `reasoning_effort` return `400`.

`exclude: true` or `include_reasoning: false` hides reasoning; reasoning tokens still bill. With hidden reasoning, `logprobs` cover the answer only. Stream chunks that carry reasoning have `logprobs: null`; a non-streaming answer keeps only the answer's entries, or `null` when they cannot be separated.

Only `none` or `enabled: false` requests reasoning off; other efforts keep reasoning on at the nearest reasoning tier at or below your request, or the model's lowest reasoning tier if none is lower. `default` keeps the model's own reasoning on.

## What the gateway does with a field you send

> Each top-level request field meets one of three fates: validated by the gateway and forwarded, forwarded untouched from the allowlist, or removed before forwarding because it is not on the allowlist.
>
> - What happens to a field you send
> - POST /v1/chat/completions
> - Checked, then sent
> - The gateway validates the type
> - or the range first. A bad value
> - is refused with 400 before the
> - request is served or billed.
> - model, messages, max\_tokens
> - Sent untouched
> - On the forwarding allowlist.
> - Any JSON value, no gateway
> - check. The model can still
> - reject it, as a 400.
> - seed, user, service\_tier
> - Removed
> - Not on the allowlist, so it is
> - dropped before forwarding. The
> - model never sees it and you
> - get no warning.
> - guided\_json, length\_penalty
> - The allowlist is exhaustive. An unknown top-level field passes request validation and is still removed
> - before the request reaches the model. Check the list before depending on a field.

Checked, then sent:

**Reference 221**

| Field | Type | Default | Behavior |
| --- | --- | --- | --- |
| `model` | string | none | Required. Must resolve to a model your key can reach. |
| `messages` | object array | none | Required, at least one message. Extra message fields pass through. Some models need system messages in a set order, see [System message placement](https://www.rightnowai.co/docs/open-models/chat-completions#system-message-placement). |
| `stream` | boolean | `false` | `true` returns an SSE response. |
| `temperature` | number, 0 to 2 | none | Range checked, then forwarded. |
| `top_p` | number, 0 to 1 | none | Range checked, then forwarded. |
| `top_k` | integer | none | `-1` or `0` turns it off. |
| `min_p` | number, 0 to 1 | none | See `logit_bias` restrictions below. |
| `repetition_penalty` | number, greater than 0 | none | Forwarded. |
| `max_tokens` | integer, 1 or more | see [Output limits](https://www.rightnowai.co/docs/open-models/chat-completions#output-limits) | Per-generation output limit, counts toward the total output budget. |
| `max_completion_tokens` | integer, 1 or more | see [Output limits](https://www.rightnowai.co/docs/open-models/chat-completions#output-limits) | Same. If you send both, the larger is used for the budget check. |
| `n` | integer, 1 to 4 | `1` | Generated choices. The total output budget covers all of them. |
| `best_of` | integer, 1 to 4 | `1` | Forwarded when present, included in the choice budget. |
| `stop` | string or string array | none | At most 16 sequences, limited to 4 on some models; each sequence is at most 256 characters. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies the models with the lower count. Exceeding either bound returns `400 hosted_parameter_not_supported`. Remove excess sequences or shorten them, then retry. |

A message's `role` is `system`, `developer`, `user`, `assistant` or `tool`. Any other role returns `400`. `content` may be a string, an array, or `null`. An assistant message needs `content` or a non-empty `tool_calls`. Each `tool_calls[].function.arguments` must hold a JSON object, such as `"{}"` for a call with no arguments. A tool message needs `content` and a non-empty string `tool_call_id`. Every other role needs `content`.

### System message placement

A `developer` message is sent to the model as a `system` message. Most models accept system messages anywhere in the conversation and receive them where you put them.

Some models accept one system message, only as the first message. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies them. For these models, the gateway repairs message placement before forwarding:

- Consecutive system messages at the start are merged into one, in order, separated by a blank line.
- A system message later in the conversation is sent as a `user` message in the same position.

Every other message, tool call, and tool result keeps its position and content. A repaired conversation still extends the previous turn's prompt, so prompt caching keeps working across turns. Coding agents that send more than one system message, such as Claude Code and Codex CLI, run on these models without extra configuration. A leading system message that contains anything other than text is not merged, and the request returns `400` with `param: "messages"`.

### Sent untouched: the forwarding allowlist

The gateway accepts any JSON value on these, applies no default, and forwards it unchanged. The model can still reject the value, which comes back as `400 invalid_request_error`.

`audio`, `frequency_penalty`, `function_call`, `functions`, `logit_bias`, `logprobs`, `metadata`, `modalities`, `moderation`, `parallel_tool_calls`, `prediction`, `presence_penalty`, `safety_identifier`, `seed`, `service_tier`, `store`, `tool_choice`, `top_logprobs`, `user`, `verbosity`, `web_search_options`.

Some allowlisted fields do carry gateway behavior:

**Reference 259**

| Field | Behavior |
| --- | --- |
| `functions`, `function_call` | Forwarded unchanged, except on some models that refuse them with `400 hosted_parameter_not_supported` unless a non-empty `tools` array is also sent. Send `tools` and `tool_choice` instead. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies the exception. |
| `logit_bias`, `min_p` | Some models refuse a non-empty `logit_bias` with `400`, code `hosted_parameter_not_supported` and `param: "logit_bias"`. An empty object is accepted. The same models refuse a `min_p` above 0 with the same code and `param: "min_p"`, unless `temperature` is 0 (a request that sends no `temperature` is refused too). The message says to remove the field. |
| `logprobs`, `top_logprobs` | Forwarded unchanged, except on some models that refuse a logprobs request with `400 hosted_parameter_not_supported`. Remove the logprobs request or choose a model that accepts it; the [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies the exception. |
| `reasoning_effort` | Forwarded unchanged, subject to the model-specific refusals and rewrites in [Turning reasoning off](https://www.rightnowai.co/docs/open-models/chat-completions#turning-reasoning-off). When the gateway chooses an effort that the request did not name, `X-RunInfra-Reasoning-Effort-Applied` reports the value applied. |
| `response_format` | Forwarded to a model that can enforce it; see [Three things a request can be refused for](https://www.rightnowai.co/docs/open-models/chat-completions#three-things-a-request-can-be-refused-for). |
| `stream_options` | The gateway always requests usage so it can settle. Client-visible usage normally needs `include_usage: true`; a stream with no final answer also forwards an annotated usage frame without opt-in. |
| `tool_choice` | Some models refuse `tool_choice: "required"` with `400`, code `hosted_parameter_not_supported` and `param: "tool_choice"`. The message says what to send instead. Send `"auto"` or name one function where the model supports that choice. |
| `tools` | An empty array is refused with `400 hosted_parameter_not_supported`. Anything else is forwarded unchanged, with the two exceptions below. |

The two `tools` exceptions also apply to a `json_schema` response format:

- A bound pair in a tool's parameter schema that no value can satisfy (`minimum` above `maximum` or an exclusive bound that closes the range, `minItems` above `maxItems`, `minLength` above `maxLength`, `minProperties` above `maxProperties`) is removed before the request is forwarded, because a forced tool call enforces the schema during generation and a range no value can satisfy cannot be enforced; satisfiable bounds are kept.
- An `enum` with no values or a `pattern` that is not a valid regular expression cannot be repaired and is refused with `400 hosted_parameter_not_supported` naming the path.

Forwarding does not guarantee the model accepts a particular tool definition.

### Model-specific tool and JSON rules

The [model-specific source reference](https://runinfra.ai/docs/api-reference/models?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) describes a model on which `tool_choice: "auto"` and `"none"`, parallel calls with distinct ids, and tool-result follow-ups are supported. Escaped and nested JSON arguments were verified through the endpoint on 2026-10-19.

For that model, with `tool_choice: "required"` or a named function choice, set `function.strict: true` on every tool. With non-strict tools, the endpoint may answer without a tool call even when you require or name one.

On that model, `response_format` with `json_object` and strict `json_schema` are supported. Enum values, `maxItems` and nested `additionalProperties: false` were verified. The schema applies after reasoning, so `max_tokens` must cover reasoning and the final JSON object. System messages may appear anywhere in the conversation. Stop strings can also match the reasoning stream and leave answer content empty, so inspect `finish_reason` and `content` before using a result.

Keep `tools` in every request whose history carries tool calls, and prefer `tool_choice: "auto"` over `"none"`. Nothing refuses a request that leaves them out, but on some models the answer can then come back empty, with `choices[].runinfra.output_status.code` set to `no_answer_content`.

### Removed before the request leaves the gateway

The allowlist is exhaustive: every other top-level field is dropped. Common examples are `max_output_tokens`, `text`, `include`, `truncation` and `previous_response_id` (Responses API fields on the Chat route); `length_penalty`; any nonstandard guided-decoding or beam-search field such as `guided_json`, `guided_regex`, `guided_choice`, `guided_grammar` and `use_beam_search` (use `response_format` and `tools` for constrained output); and `num_return_sequences`, `num_beams` and `beam_width`, which the gateway reads for the four-choice safety check and then removes.

## Output limits

**Reference 286**

| Limit | Value |
| --- | --- |
| Total output tokens per request | **The model's remaining context**, that is its context window minus your prompt, across all generated choices. Oversized values are clamped, never refused. |
| Generated choices per request | **4** |
| Full response | **740 seconds**, then `504` `hosted_inference_response_timeout` |
| First token on a stream | **180 seconds**, then `504` `hosted_inference_response_timeout` |
| JSON request body | **32MB**, then `413` `payload_too_large`. Bodies over 4MB need a `Content-Length` header, which every SDK sends |
| Images per request | 8 across all messages on vision models, except for the model in the [model-specific image reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) that permits 100. Exceeding the applicable count returns `400 hosted_parameter_not_supported`. Remove older images or split them across requests. |

The body-size and image-count limits count every message in the request, including the images and history a client resends each turn. Base64 adds about a third to every inline image, so a 32MB body carries about 24MB of image files in total; drop older images from the conversation, split them across requests, or use a public image URL on a vision model. Remote image downloads have [separate limits](https://www.rightnowai.co/docs/open-models/chat-completions#three-things-a-request-can-be-refused-for).

A text-only request that sets neither `max_tokens` nor `max_completion_tokens` defaults to 131,072 output tokens when context permits, or the model's own lower default for JSON output. Set an explicit budget to control the request.

Set an explicit `max_tokens` budget using the selected model's live `max_output_tokens` and remaining context. A generation that reaches its output budget reports `finish_reason: "length"`, including when reasoning consumes the budget before an answer. Oversized `max_tokens` is clamped to the remaining context, with `X-RunInfra-Output-Clamped: true` and `X-RunInfra-Output-Token-Ceiling` reporting the value used.

Two bounds apply at once and the smaller one wins. Tokens are bounded by the remaining context above; wall-clock is bounded by the 740 second response limit, which at typical decode rates is the binding limit for very long generations. `/v1/models` publishes both, `max_output_tokens` beside `response_time_ceiling_seconds`, which carries the same 740 second figure, so size a long generation against the pair rather than the token figure alone.

The `504` message names the budget that was exceeded. For a non-streaming request, the error message suggests two remedies: `stream: true` (response headers arrive with the first token) and a smaller `max_tokens`. A first-token timeout on a stream suggests a smaller prompt, or retrying when the model has more capacity.

A model's **context window** is a separate per-model limit, published on that model's page in the [Model Library](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models). Served context is a property of how the model is served, not of the weights, so read it on the model page rather than a model card.

## Three things a request can be refused for

Each one is refused up front with a `400` naming the field, rather than answered with output you cannot use.

### A JSON format the model cannot hold while reasoning

`response_format` of `json_schema` or `json_object` is enforced during generation, and on some models that enforcement cannot start until the model has finished reasoning. The gateway reports an automatically applied value in the `X-RunInfra-Reasoning-Effort-Applied` response header. Send an incompatible effort beside the format and the request is refused with `400`, `code` `hosted_parameter_not_supported` and `param: "response_format"`.

On the models with this restriction, omitting `reasoning_effort` applies an effort measured to return a conforming object: `none` on some, or `low` on another where an explicit `high` also conforms. The [model-specific source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) identifies which case applies; the response header tells you the automatically applied value.

The fix is the effort value sent alongside `response_format`:

**JSON**

```json
{
  "model": "MODEL_ID",
  "messages": [{ "role": "user", "content": "Classify and rewrite this ticket." }],
  "reasoning_effort": "none",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "triage",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "integer", "enum": [1, 2, 3, 4] },
          "rewrite": { "type": "string" }
        },
        "required": ["category", "rewrite"],
        "additionalProperties": false
      }
    }
  }
}
```

Check each model's JSON mode requirements in the [Model Library](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) before changing the model in this request. `response_format: {"type": "text"}` constrains nothing and is never refused. On `/v1/responses` the same remedy is a top-level `reasoning_effort` beside `text.format`. Working snippets are in [Structured output](https://www.rightnowai.co/docs/open-models/guides/structured-output).

### An image, audio, or video content part

Accepted input is declared per model in Open models. On a model with image-input support, `image_url` and `input_image` content parts accept a public `http://` or `https://` image URL, or an inline base64 data URL. Most vision models accept up to 8 images per request; the model in the [model-specific image reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) permits up to 100. The count covers every image in the conversation you send. The images in one request may decode to about 680 MiB of pixels in total, about 99 images at 1920x1080. Remote images must be PNG, JPEG, WebP or a non-animated GIF. Existing inline GIF input is unchanged.

**JSON**

```json
{
  "type": "image_url",
  "image_url": {
    "url": "https://images.example.com/photo.jpg",
    "detail": "auto"
  }
}
```

The open-models service's gateway downloads remote images and sends their bytes inline to the model. A URL must be reachable on the public internet without cookies or authorization headers. The default ports are 80 for HTTP and 443 for HTTPS; an explicit URL may use only ports 80 or 443. Private and local addresses, embedded URL credentials, more than 3 redirects, and redirects from HTTPS to HTTP are refused. Signed URLs work if they meet these rules.

Each remote image is limited to 20 MiB of transferred bytes and 20 MiB after decompression, with at most 16,777,216 pixels. The gateway streams decompression for gzip, deflate and Brotli responses. These are the open-models service's gateway memory bounds; 1 MiB is 1,048,576 bytes.

A request containing remote images may carry at most 50 MiB of expanded image data URLs, including inline images and repeated URLs. Each complete data URL counts once per image part, including its base64 bytes and prefix. Identical URLs are downloaded once per request. The gateway-created JSON body is limited to 54 MiB. An inline-only request keeps its existing [JSON request body limit](https://www.rightnowai.co/docs/open-models/errors#status-and-code-reference).

Each image has a 10 second download limit, including DNS, redirects and its body, with 20 seconds for all images together, or less if the request's deadline comes sooner. Waiting for download capacity counts toward that time. Each gateway process permits 32 active image downloads and 256 queued downloads. A download waits at most 5 seconds for a slot or byte capacity, within its remaining deadline. A capacity refusal returns retryable `429 hosted_admission_congested` with `Retry-After`; it does not mean the image URL is invalid.

A process-wide 256 MiB reservation budget covers cumulative wire and decoded bytes while downloads are in progress. The gateway reserves announced `Content-Length` bytes up front and adds reservations as decoded or unknown-length bytes arrive. Reservations are released when a download ends. This is not a limit on total process memory or completed inline request bodies; the existing per-request image and body limits still apply.

A refused remote image returns `400` with `type: "invalid_request_error"`, `code: "invalid_image_url"` and `param: "messages"`. The message names the part, such as `messages[2].content[1]`, and the reason. It carries no `Retry-After` and sets `x-should-retry: false`. Use a reachable public URL within these limits, or send an inline data URL such as `{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}`. Retrying the same invalid image is not advised.

Image tokens are counted by the server inside `prompt_tokens` and bill at the model's input rate. The gateway's container checks catch known truncation and reject remote animated GIFs. These checks do not prove that every pixel can be decoded. A residual pixel decoder failure keeps the existing upstream `400` refusal instead of being reported as model unavailability.

On a model without declared image-input support, and for audio and video everywhere, a non-text content part inside `messages` returns `400` with `param: "messages"` and `code: "hosted_parameter_not_supported"`, naming the model and the part to remove. The part is refused rather than stripped, so a request carrying media is never billed as if it were text.

### A prompt-cache control field

`prompt_cache_options` and `prompt_cache_retention` are rejected with `400` `hosted_parameter_not_supported`. `prompt_cache_key` is accepted: when no [session header](https://www.rightnowai.co/docs/open-models/session-affinity) supplies an identity, the gateway reads it as a cache-affinity hint and removes it before the request reaches the model. It grants no control over retention. Prefix caching, where a model has it, is automatic server behavior and is not controllable per request.

## Prefix cache retention

Retention is what `GET /v1/models` states as `cache_tiers` and `cache_retention`. A model reporting `cache_retention: "tiered_host_memory"` can reuse an idle session's cached prefix beyond immediate memory pressure when the session resumes. Every other chat model reports `"best_effort"`: cached prefixes are retained opportunistically and may be evicted at any time, with no retention promise. Both policies remain best effort. See [Session affinity](https://www.rightnowai.co/docs/open-models/session-affinity) to keep follow-up turns close to a warm cache, and read `usage.prompt_tokens_details.cached_tokens` for each request's cached input.

## Calling a paused model

A hosted model can be temporarily paused. The request returns `503` with `code` `hosted_model_paused`, charges nothing, and carries a `Retry-After` header with the seconds until availability is next rechecked, capped at 3,600, or 60 when no recheck time is known or the known one has already passed. See [the 503 codes](https://www.rightnowai.co/docs/open-models/errors#the-503-codes).

## Related

[### Streaming Read deltas and ask for the usage frame.](https://www.rightnowai.co/docs/open-models/streaming) [### Rate limits The four layers that can refuse you, and how to back off.](https://www.rightnowai.co/docs/open-models/rate-limits) [### Errors Every status, code, and caller action.](https://www.rightnowai.co/docs/open-models/errors)

## Responses

The open models API serves `POST /v1/chat/completions`, `POST /v1/responses`, `POST /v1/messages`, `POST /v1/messages/count_tokens`, `GET /v1/models`, `GET /v1/models/{model}`, [`GET /v1/credits`](https://www.rightnowai.co/docs/open-models/credits), and [`GET /v1/usage`](https://www.rightnowai.co/docs/open-models/usage). Chat completions and Responses are chat-compatible. Messages and token counting use the [Messages-compatible contract](https://www.rightnowai.co/docs/open-models/messages).

Not supported anywhere: `/v1/completions` (legacy, non-chat), `/v1/files`, `/v1/assistants`, `/v1/threads`, and `/v1/batches`. `/v1/responses` is a chat-completions compatibility adapter and does not implement state, hosted tools (web search, file search), conversation storage, or background jobs.

On a model whose page lists tool calling, function tools work end to end on `/v1/responses`:

- **Tools**, in both spellings: declare tools flat (`{"type": "function", "name", "parameters"}`, the Responses shape) or nested under `function` (the chat shape).
- **Calls and results**: the model's calls come back as `function_call` output items with `call_id`, `name`, and `arguments`; send results back as `function_call_output` input items with the matching `call_id` (string or content-part array), and resend prior output items in `input` since the endpoint is stateless.
- **Streams** carry the full item lifecycle (`response.created` and `response.in_progress`, `response.output_item.added`, `response.content_part.added`, text and `function_call_arguments` deltas, the matching `done` events, and `response.completed`, which ends every stream), so Codex-style clients that build items from `output_item.done` work unchanged. `stream_options: {"include_obfuscation": true}` adds random padding to each delta.
- **Harness fields**: `store` is accepted and always treated as false, nothing is retrievable later. `metadata`, `include` (empty or any documented value; only `reasoning.encrypted_content` can add output here), `truncation` (`"auto"` or `"disabled"`), `parallel_tool_calls`, `service_tier`, `client_metadata`, `safety_identifier`, `stream_options`, `max_tool_calls`, and `prompt_cache_retention` are accepted; the hints among them have no effect. Input is never truncated: a prompt longer than the context window returns `context_length_exceeded`. `previous_response_id`, `conversation`, and `prompt` are accepted only as `null`, and `background` only as `false` or `null`; other values are refused with the reason.

On `/v1/responses`, `output_text` carries the final answer only. The model's thinking arrives as its own `reasoning` output item ahead of the message (raw text under `content` as `reasoning_text` parts, `summary` empty), and on streams as `response.reasoning_text.delta` and `response.reasoning_text.done` events inside a `response.content_part.added` and `response.content_part.done` pair. A completion that spends its whole token budget thinking returns an empty `output_text` with the thinking preserved in the reasoning item, so raise `max_output_tokens` if you see that.

A reply that `max_output_tokens` or a content filter cut short is not whole, and the response says so: `status` is `"incomplete"` and `incomplete_details.reason` is `max_output_tokens` or `content_filter`. A non-streaming body carries it directly. A stream still ends on `response.completed`, never on `response.incomplete`, and the response inside that event carries the same `status` and reason. Check `status` before you treat an answer as complete. Item status is advisory: only the item being written when the reply stopped is `incomplete`, so parse a function call's `arguments` before you run it.

[Source reference](https://runinfra.ai/docs/api-reference/chat-completions?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models)

## Next

[Messages](https://www.rightnowai.co/docs/open-models/messages)