# Open-model compatible clients

[Open models](https://www.rightnowai.co/docs/open-models) / [Open-model compatible clients](https://www.rightnowai.co/docs/sdks/open-models)

Updated 2026-11-11

If your code already speaks the chat API, three values change and nothing else does.

**Connection**

| Connection | Value |
| --- | --- |
| API base | `https://api.runinfra.ai/v1` |
| Workspace key | `rp_...` |
| Process variable | `RUNINFRA_GATEWAY_KEY` |
| Billing | The existing funding rules of your open-model account. |

Replace `MODEL_ID` with an available id from [Open models](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models); read current capabilities, limits and prices from `GET /v1/models`.

**cURL**

```shell
curl https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Hello"}],"max_tokens":16384}'
```

**Python**

```python
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",        # <- change
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],   # <- change
)

response = client.chat.completions.create(
    model="MODEL_ID",               # <- change
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=16384,
)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",   // <- change
  apiKey: process.env.RUNINFRA_GATEWAY_KEY, // <- change
});

const response = await client.chat.completions.create({
  model: "MODEL_ID",              // <- change
  messages: [{ role: "user", content: "Hello" }],
  max_tokens: 16384,
});
```

The model names used by the `OpenAI` API do not alias to model ids in the open-models service. Pass a model id from `GET /v1/models`. Streaming, tools, and structured output use the same request and response shapes where the selected model's page lists that capability. Streams terminate with `data: [DONE]`, tool calls appear on the assistant message, and usage appears on the response.

## Set an output budget

This is the one setting that decides whether a reasoning model has room to answer. Reasoning tokens count toward `max_tokens` and, on a billable response, are billed at the output rate. Set the first examples' `max_tokens` to the minimum of 16,384 and the model's live output limit. Up to 2,048 tokens can be spent on reasoning before any answer.

Where a recommended minimum is published, the model's page in the [Model Library](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models) carries it. You pay for tokens generated, not for the ceiling you set, so a generous budget costs nothing extra. If the answer comes back empty, check `finish_reason`: `"length"` means raise the budget, not that the model produced nothing.

For a model whose own page recommends at least 4,096 output tokens, start at that minimum if the live limit permits it. The source measurements from August 14, 2026 found that a 2,048-token budget could be fully consumed by reasoning, leaving an empty or truncated answer; a 16,384-token budget left more room for the answer. Those historical probes billed output. Under the current usage contract of the open-models service, a response with no answer settles at zero.

## Where the model's thinking arrives

The open-models service emits the model's thinking as **`reasoning`** on the message and on each stream delta. Some chat-compatible clients and providers use the name `reasoning_content` instead, so a client that reads only that name sees the answer but not the thinking. Measured per client:

**Reference 72**

| Client | Visible answer | Model's thinking |
| --- | --- | --- |
| AI SDK (`@ai-sdk/openai-compatible`) | `text` / `text-delta` | `reasoningText`, and `reasoning-delta` parts |
| Chat API SDK, Node and Python | `message.content` / `delta.content` | `message.reasoning` / `delta.reasoning` |
| The routing client | `message.content` | `message.reasoning_content` |
| The chain client (`langchain-openai`) | `content` | **not exposed**, dropped by the integration |
| Plain `fetch` | `delta.content` | `delta.reasoning` |

The answer is correct in every client above. Only access to the thinking differs, so pick accordingly if you want to render it.

## Prompt caching and session affinity

Prefix caching is automatic server behavior on models with a published cached-input price. You do not select cache placement, retention, or reuse per request. On `/v1/chat/completions`, `prompt_cache_options` and `prompt_cache_retention` are rejected with `400` `hosted_parameter_not_supported`. On `/v1/responses`, `prompt_cache_retention` is accepted and has no effect, and `prompt_cache_options` is accepted only as `null`.

For explicit session routing, send `x-session-id`. For multi-turn conversations and agent sessions, give repeat requests a consistent session identity so the requests stay close to their cached prefix:

- Header: `x-session-id: <your-conversation-or-task-id>` (`x-parent-session-id` and the legacy `x-session-affinity` are also accepted)
- Body field: `prompt_cache_key` when no explicit session identity is present. `user` is forwarded to the model and is not a session identity by itself.

These session-routing hints work on `/v1/chat/completions` and `/v1/responses`; they help keep the session's requests and cached prefix together. Session headers stay inside the gateway; `prompt_cache_key` is consumed before forwarding, so neither routing hint reaches the model. Without an explicit identity, requests may be grouped by their opening messages; a request without message text keeps workspace-level grouping. An explicit identity makes grouping exact for your conversation or task. See [Sessions](https://www.rightnowai.co/docs/open-models/session-affinity#prompt_cache_key-user-and-the-opening-messages) for placement precedence and the separate role of `user`. Read cached-token counts and hit rates in [Prompt caching](https://www.rightnowai.co/docs/open-models/guides/prompt-caching#reading-your-own-hit-rate), and any published hit-rate figures on the selected model's page and the pricing page.

## Framework configuration

The source examples selected models listed as available on September 21, 2026. These framework integrations returned complete answers from `https://api.runinfra.ai/v1` in checks on 2026-08-14. These are client patterns from historical checks. Replace MODEL\_ID with a currently available model and verify its required capabilities. Versions used: `ai` 7.0.65, `@ai-sdk/openai-compatible` 3.0.30, `openai` 7.4.0 (Node) and 2.54.0 (Python), `langchain-openai`, `litellm`.

### The chain client

**Python**

```python
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="MODEL_ID",
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    max_tokens=4096,
)

print(llm.invoke("Write a TypeScript retry helper.").content)

for chunk in llm.stream("Write a TypeScript retry helper."):
    print(chunk.content, end="", flush=True)
```

### The routing client

The routing client reaches the open-models service through its `openai/` provider prefix, and exposes the thinking as `reasoning_content`.

**Python**

```python
import os
import litellm

response = litellm.completion(
    model="openai/MODEL_ID",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    messages=[{"role": "user", "content": "Write a TypeScript retry helper."}],
    max_tokens=16384,
)
print(response.choices[0].message.content)
```

Add `stream=True` for deltas, reading `chunk.choices[0].delta.content` and skipping the usage frame, which has no choices.

### The indexing client

Use `OpenAILike` from `llama-index-llms-openai-like`, with `is_chat_model=True`. The `OpenAI` class raises `Unknown model` for any id it does not list.

**Python**

```python
import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="MODEL_ID",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    is_chat_model=True,
)

print(llm.complete("What is the open-models service?").text)
```

### AI SDK

`@ai-sdk/openai-compatible` reads `reasoning_content` and falls back to `reasoning`, which is the name we emit, so both the answer and the thinking come through.

**TypeScript**

```typescript
// Server-side only: never expose RUNINFRA_GATEWAY_KEY in browser code.
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { streamText } from "ai";

const apiKey = process.env.RUNINFRA_GATEWAY_KEY;
if (!apiKey) throw new Error("Set RUNINFRA_GATEWAY_KEY first.");

const runinfra = createOpenAICompatible({
  name: "runinfra",
  baseURL: "https://api.runinfra.ai/v1",
  apiKey,
});

const result = streamText({
  model: runinfra.chatModel("MODEL_ID"),
  maxOutputTokens: 4096,
  prompt: "Write a TypeScript retry helper.",
});

for await (const part of result.fullStream) {
  if (part.type === "reasoning-delta") process.stdout.write(part.text);
  if (part.type === "text-delta") process.stdout.write(part.text);
}
```

`generateText` works the same way, with the thinking on `reasoningText`.

`maxOutputTokens` is the AI SDK's name for the output budget above.

`usage.outputTokenDetails.reasoningTokens` reports `0` even when the model reasoned, because this endpoint does not yet break reasoning out of `completion_tokens`. Total `outputTokens` is correct and is what you are billed for.

### Plain fetch, no SDK

**TypeScript**

```typescript
const response = await fetch("https://api.runinfra.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.RUNINFRA_GATEWAY_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    messages: [{ role: "user", content: "Write a TypeScript retry helper." }],
    max_tokens: 4096,
    stream: true,
    stream_options: { include_usage: true },
  }),
});

if (!response.ok) {
  const { error } = await response.json();
  throw new Error(`${response.status} ${error.code}: ${error.message}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") continue;

    const chunk = JSON.parse(payload);
    const delta = chunk.choices[0]?.delta;
    // The model's thinking arrives as `reasoning`, the answer as `content`.
    if (delta?.reasoning) process.stdout.write(delta.reasoning);
    if (delta?.content) process.stdout.write(delta.content);
  }
}
```

A `data:` line can be split across two network reads, so the trailing partial line stays in `buffer` rather than being parsed. Dropping that carry is the usual cause of intermittent `JSON.parse` failures on a stream that works in testing.

## Compatibility behavior

### Which endpoints exist

The model APIs serve `POST /v1/chat/completions`, `POST /v1/responses`, `POST /v1/messages`, `POST /v1/messages/count_tokens`, `GET /v1/models`, `GET /v1/models/{model}`, `GET /v1/credits`, and `GET /v1/usage`. Chat completions and Responses use the chat-compatible contract. Messages and token counting use the messages-compatible contract.

Not supported anywhere: `/v1/completions` (legacy, non-chat), `/v1/files`, `/v1/assistants`, `/v1/threads`, and `/v1/batches`. `/v1/responses` is a chat-completions compatibility adapter and does not implement state, hosted tools such as web search and file search, conversation storage, or background jobs.

On a model whose page lists tool calling, function tools work end to end on `/v1/responses`. Declare tools flat with `type`, `name` and `parameters` in the Responses shape, or nested under `function` in the chat shape. Calls come back as `function_call` output items with `call_id`, `name`, and `arguments`. Send results back as `function_call_output` input items with the matching `call_id`, as a string or content-part array, and resend prior output items in `input` because the endpoint is stateless.

Streams carry the full item lifecycle: `response.created`, `response.in_progress`, `response.output_item.added`, `response.content_part.added`, text and `function_call_arguments` deltas, the matching `done` events, and `response.completed`, which ends every stream. Clients that build items from `output_item.done` work unchanged. `stream_options: {"include_obfuscation": true}` adds random padding to each delta.

For the open-models service, `store` is accepted and always treated as false; nothing is retrievable later. `metadata`, `include` (empty or any documented value; only `reasoning.encrypted_content` can add output here), `truncation` (`"auto"` or `"disabled"`), `parallel_tool_calls`, `service_tier`, `client_metadata`, `safety_identifier`, `stream_options`, `max_tool_calls`, and `prompt_cache_retention` are accepted; the hints among them have no effect. Input is never truncated: a prompt longer than the context window returns `context_length_exceeded`.

`previous_response_id`, `conversation`, and `prompt` are accepted only as `null`, and `background` only as `false` or `null`; other values are refused with the reason.

On `/v1/responses`, `output_text` carries the final answer only. The model's thinking arrives as its own `reasoning` output item ahead of the message, with raw text under `content` as `reasoning_text` parts and an empty `summary`. On streams it arrives as `response.reasoning_text.delta` and `response.reasoning_text.done` events inside a `response.content_part.added` and `response.content_part.done` pair. A completion that spends its whole token budget thinking returns an empty `output_text` with thinking preserved in the reasoning item, so raise `max_output_tokens` if the model limit permits it.

A reply that `max_output_tokens` or a content filter cut short is not whole: `status` is `"incomplete"` and `incomplete_details.reason` is `max_output_tokens` or `content_filter`. A non-streaming body carries it directly. A stream still ends on `response.completed`, never on `response.incomplete`; the response inside that event carries the same status and reason. Check `status` before treating an answer as complete. Item status is advisory: only the item being written when the reply stopped is `incomplete`, so parse a function call's `arguments` before running it.

See the [Responses reference](https://www.rightnowai.co/docs/open-models/chat-completions#responses) for the complete endpoint contract.

### What happens to a field you send

Fields on the forwarding allowlist reach the model unchanged. Fields that are not on it are dropped before the request leaves the gateway, silently rather than as an error, so a request shape that works today keeps working.

[Chat completions](https://www.rightnowai.co/docs/open-models/chat-completions) lists the allowlist field by field, and names the few that carry gateway behavior. It is the one authoritative list, so this page does not restate it.

### Errors and retries

These chat-compatible routes use the chat API envelope, `{ error: { message, type, code } }`, with six `error.type` values: `invalid_request_error`, `authentication_error`, `permission_error`, `not_found_error`, `rate_limit_error`, and `api_error`. There is no `server_error` type; 5xx responses carry `api_error`. The Messages routes use the Messages protocol envelope. Every status and code is in [Errors](https://www.rightnowai.co/docs/open-models/errors), and the triage is in [Troubleshooting](https://www.rightnowai.co/docs/open-models/troubleshooting).

Retry behavior belongs to your client configuration. Do not blindly retry a charge-bearing request after a partial stream may have reached your app; use an [idempotency key](https://www.rightnowai.co/docs/open-models/idempotent-retries) instead. Quote the `X-Request-Id` response header on any support ticket.

[### The model hub clients `huggingface_hub` and `@huggingface/inference`, verified.](https://www.rightnowai.co/docs/sdks/open-models/inference-clients) [### Tool calling A complete multi-turn loop.](https://www.rightnowai.co/docs/open-models/guides/tool-calling) [### Rate limits Context, output, concurrency, tokens per minute.](https://www.rightnowai.co/docs/open-models/rate-limits)

## Next

[Native TypeScript SDK](https://www.rightnowai.co/docs/sdks/open-models/native-typescript)