Open-model compatible clients

The chain client

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="MODEL_ID",
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    max_tokens=4096,
)

print(llm.invoke("Write a TypeScript retry helper.").content)

for chunk in llm.stream("Write a TypeScript retry helper."):
    print(chunk.content, end="", flush=True)

The routing client

The routing client reaches the open-models service through its openai/ provider prefix, and exposes the thinking as reasoning_content.

import os
import litellm

response = litellm.completion(
    model="openai/MODEL_ID",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    messages=[{"role": "user", "content": "Write a TypeScript retry helper."}],
    max_tokens=16384,
)
print(response.choices[0].message.content)

Add stream=True for deltas, reading chunk.choices[0].delta.content and skipping the usage frame, which has no choices.

The indexing client

Use OpenAILike from llama-index-llms-openai-like, with is_chat_model=True. The OpenAI class raises Unknown model for any id it does not list.

import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="MODEL_ID",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    is_chat_model=True,
)

print(llm.complete("What is the open-models service?").text)

AI SDK

@ai-sdk/openai-compatible reads reasoning_content and falls back to reasoning, which is the name we emit, so both the answer and the thinking come through.

// Server-side only: never expose RUNINFRA_GATEWAY_KEY in browser code.
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { streamText } from "ai";

const apiKey = process.env.RUNINFRA_GATEWAY_KEY;
if (!apiKey) throw new Error("Set RUNINFRA_GATEWAY_KEY first.");

const runinfra = createOpenAICompatible({
  name: "runinfra",
  baseURL: "https://api.runinfra.ai/v1",
  apiKey,
});

const result = streamText({
  model: runinfra.chatModel("MODEL_ID"),
  maxOutputTokens: 4096,
  prompt: "Write a TypeScript retry helper.",
});

for await (const part of result.fullStream) {
  if (part.type === "reasoning-delta") process.stdout.write(part.text);
  if (part.type === "text-delta") process.stdout.write(part.text);
}

generateText works the same way, with the thinking on reasoningText.

maxOutputTokens is the AI SDK's name for the output budget above.

usage.outputTokenDetails.reasoningTokens reports 0 even when the model reasoned, because this endpoint does not yet break reasoning out of completion_tokens. Total outputTokens is correct and is what you are billed for.

Plain fetch, no SDK

const response = await fetch("https://api.runinfra.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.RUNINFRA_GATEWAY_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    messages: [{ role: "user", content: "Write a TypeScript retry helper." }],
    max_tokens: 4096,
    stream: true,
    stream_options: { include_usage: true },
  }),
});

if (!response.ok) {
  const { error } = await response.json();
  throw new Error(`${response.status} ${error.code}: ${error.message}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") continue;

    const chunk = JSON.parse(payload);
    const delta = chunk.choices[0]?.delta;
    // The model's thinking arrives as `reasoning`, the answer as `content`.
    if (delta?.reasoning) process.stdout.write(delta.reasoning);
    if (delta?.content) process.stdout.write(delta.content);
  }
}

A data: line can be split across two network reads, so the trailing partial line stays in buffer rather than being parsed. Dropping that carry is the usual cause of intermittent JSON.parse failures on a stream that works in testing.

Which endpoints exist

The model APIs serve POST /v1/chat/completions, POST /v1/responses, POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models, GET /v1/models/{model}, GET /v1/credits, and GET /v1/usage. Chat completions and Responses use the chat-compatible contract. Messages and token counting use the messages-compatible contract.

Not supported anywhere: /v1/completions (legacy, non-chat), /v1/files, /v1/assistants, /v1/threads, and /v1/batches. /v1/responses is a chat-completions compatibility adapter and does not implement state, hosted tools such as web search and file search, conversation storage, or background jobs.

On a model whose page lists tool calling, function tools work end to end on /v1/responses. Declare tools flat with type, name and parameters in the Responses shape, or nested under function in the chat shape. Calls come back as function_call output items with call_id, name, and arguments. Send results back as function_call_output input items with the matching call_id, as a string or content-part array, and resend prior output items in input because the endpoint is stateless.

Streams carry the full item lifecycle: response.created, response.in_progress, response.output_item.added, response.content_part.added, text and function_call_arguments deltas, the matching done events, and response.completed, which ends every stream. Clients that build items from output_item.done work unchanged. stream_options: {"include_obfuscation": true} adds random padding to each delta.

For the open-models service, store is accepted and always treated as false; nothing is retrievable later. metadata, include (empty or any documented value; only reasoning.encrypted_content can add output here), truncation ("auto" or "disabled"), parallel_tool_calls, service_tier, client_metadata, safety_identifier, stream_options, max_tool_calls, and prompt_cache_retention are accepted; the hints among them have no effect. Input is never truncated: a prompt longer than the context window returns context_length_exceeded.

previous_response_id, conversation, and prompt are accepted only as null, and background only as false or null; other values are refused with the reason.

On /v1/responses, output_text carries the final answer only. The model's thinking arrives as its own reasoning output item ahead of the message, with raw text under content as reasoning_text parts and an empty summary. On streams it arrives as response.reasoning_text.delta and response.reasoning_text.done events inside a response.content_part.added and response.content_part.done pair. A completion that spends its whole token budget thinking returns an empty output_text with thinking preserved in the reasoning item, so raise max_output_tokens if the model limit permits it.

A reply that max_output_tokens or a content filter cut short is not whole: status is "incomplete" and incomplete_details.reason is max_output_tokens or content_filter. A non-streaming body carries it directly. A stream still ends on response.completed, never on response.incomplete; the response inside that event carries the same status and reason. Check status before treating an answer as complete. Item status is advisory: only the item being written when the reply stopped is incomplete, so parse a function call's arguments before running it.

See the Responses reference for the complete endpoint contract.

What happens to a field you send

Fields on the forwarding allowlist reach the model unchanged. Fields that are not on it are dropped before the request leaves the gateway, silently rather than as an error, so a request shape that works today keeps working.

Chat completions lists the allowlist field by field, and names the few that carry gateway behavior. It is the one authoritative list, so this page does not restate it.

Errors and retries

These chat-compatible routes use the chat API envelope, { error: { message, type, code } }, with six error.type values: invalid_request_error, authentication_error, permission_error, not_found_error, rate_limit_error, and api_error. There is no server_error type; 5xx responses carry api_error. The Messages routes use the Messages protocol envelope. Every status and code is in Errors, and the triage is in Troubleshooting.

Retry behavior belongs to your client configuration. Do not blindly retry a charge-bearing request after a partial stream may have reached your app; use an idempotency key instead. Quote the X-Request-Id response header on any support ticket.