# Streaming

[Open models](https://www.rightnowai.co/docs/open-models) / [Streaming](https://www.rightnowai.co/docs/open-models/streaming)

Updated 2026-11-11

For a model whose page lists streaming, set `stream: true` on `POST /v1/chat/completions` to get a Server-Sent Events response. Add `stream_options.include_usage` when your client needs the usage frame. `/v1/messages` uses a different, event-named stream grammar described in [Messages](https://www.rightnowai.co/docs/open-models/messages#streaming).

**Connection**

| Connection | Value |
| --- | --- |
| API base | `https://api.runinfra.ai/v1` |
| Workspace key | `rp_...` |
| Process variable | `RUNINFRA_GATEWAY_KEY` |
| Billing | The existing funding rules of your open-model account. |

Replace `MODEL_ID` with an available id from [Open models](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models); read current capabilities, limits and prices from `GET /v1/models`.

**cURL**

```shell
curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Count from one to three."}],"max_tokens":64,"stream":true,"stream_options":{"include_usage":true}}'
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)
stream = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Count from one to three."}],
    max_tokens=64,
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.usage:
        print(f"\nusage: {chunk.usage}")
        continue
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});
const stream = await client.chat.completions.create({
  model: "MODEL_ID",
  messages: [{ role: "user", content: "Count from one to three." }],
  max_tokens: 64,
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  if (chunk.usage) console.error("usage:", chunk.usage);
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```

## What arrives, and when

> The streaming timeline from request to terminal frame: the 180 second first-token budget, the 740 second response budget, the opt-in usage frame with an exception when no final answer was delivered, and the two terminal frames, data DONE or an error frame on the same HTTP 200.
>
> - Streaming timeline
> - stream: true
> - HTTP 200, event stream opens
> - first token
> - within the 180s TTFT budget
> - delta frames
> - one complete frame at a time
> - opt-in, or no final answer
> - 740s response budget, then 504
> - How a stream ends
> - data: \[DONE\]
> - the normal terminal frame after the last delta
> - a stream that fails mid-generation ends with one final error frame on the same HTTP 200;
> - error.code
> - names why. Handle it in your frame loop.
> - Only tokens actually delivered to you are billed.

## Frame shapes

The response carries `Content-Type: text/event-stream`, `Cache-Control: no-cache` and `Connection: keep-alive`. Content arrives in chat-compatible `data:` frames, and partial output is on `choices[].delta`.

**Text**

```text
data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"}}]}
```

The usage frame has an empty `choices` array. On a hosted model it also carries the request's cost and the count of input tokens billed at the cached rate, both described in [Chat completions](https://www.rightnowai.co/docs/open-models/chat-completions#the-cost-of-the-request-and-its-cached-input-on-its-usage):

**Text**

```text
data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16,"prompt_tokens_details":{"cached_tokens":0},"cost":0.0000028,"runinfra":{"cost_microcents":280,"cached_input_tokens":0}}}
```

Chat completions and Responses streams send `: keep-alive` after 10 seconds of silence. Ignore lines starting with `:`.

A completed stream ends with `data: [DONE]`.

**Text**

```text
data: [DONE]
```

A stream the gateway terminates early ends with an error frame and no `[DONE]`. A transport interruption can close it without an error frame. Treat a missing `[DONE]` as an incomplete response even when the HTTP status was `200`. Retrying with the same `Idempotency-Key` returns the original request's receipt, or `409` while it is still running; it never replays the text. To get an answer, send a new request with a new key, which is new paid inference. See [Retrying a stream](https://www.rightnowai.co/docs/open-models/idempotent-retries#retrying-a-stream).

**Text**

```text
data: {"error":{"message":"The upstream stream ended before signalling completion. Please retry.","type":"api_error","param":null,"code":"upstream_stream_error"},"request_id":"req_example_0000"}
```

`code` is `upstream_error` when the model stream ended, `hosted_inference_response_timeout` when the response deadline did, and `upstream_stream_error` otherwise. See [Errors](https://www.rightnowai.co/docs/open-models/errors).

## Is usage always included?

No. The gateway always asks the model for usage so it can settle the request, but it forwards the usage frame to you only when your request carried `stream_options.include_usage: true`. Omit it and content frames still stream while usage-only frames stay hidden. A usage frame is never synthesized: if the model reports no usage for a request, none is sent.

After an interrupted request, recovery uses the usage evidence that was persisted. A failed checkpoint write can leave that evidence missing or stale, so neither the delivered token count nor the final charge can be inferred from the last content chunk. Keep the request id for support. A new key starts new paid inference and cannot recover the old response.

There is one exception, so you can account for generated tokens even when no answer was delivered. When every choice finishes without final answer content, the gateway forwards one annotated usage frame whether or not you opted in.

## When a stream ends without an answer

A choice that reaches its generation limit before producing answer content carries a machine-readable status on its terminal frame:

**Text**

```text
data: {"choices":[{"index":0,"delta":{},"finish_reason":"length","runinfra":{"output_status":{"code":"generation_limit_reached_before_answer","message":"The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."}}}]}
```

If another terminal `finish_reason` produced no answer, the code is `no_answer_content` instead. Inspect the preserved `finish_reason` before deciding whether to retry.

The annotated usage frame then reports how much of the generated output was not an answer:

**Text**

```text
data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":64,"total_tokens":76,"prompt_tokens_details":{"cached_tokens":0},"cost":0,"runinfra":{"output_token_accounting":{"visible_answer_tokens":0,"non_answer_completion_tokens":64,"sources":{"visible_answer_tokens":"proxy_classified","non_answer_completion_tokens":"provider_reported"}},"cost_microcents":0,"cached_input_tokens":0}}}
```

A stream that produced no answer settles at zero, so this frame says `cost: 0` and `cached_tokens: 0` whatever the model reported: nothing was priced, at either rate.

The usual cause is an output budget too small for a model that reasons first. See [give reasoning models room](https://www.rightnowai.co/docs/open-models/chat-completions#give-reasoning-models-room).

> The same `Idempotency-Key` reads the original request's receipt; it does not replay delivered tokens. Getting a new answer requires a new request and a new key, which starts new paid inference. Read [Retrying a stream](https://www.rightnowai.co/docs/open-models/idempotent-retries#retrying-a-stream) before retrying a stream.

## Related

[### Chat completions The request contract every stream starts from.](https://www.rightnowai.co/docs/open-models/chat-completions) [### Idempotent retries Retry a dropped stream without paying twice.](https://www.rightnowai.co/docs/open-models/idempotent-retries) [### Errors Every status, code, and caller action.](https://www.rightnowai.co/docs/open-models/errors)

[Source reference](https://runinfra.ai/docs/api-reference/streaming?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models)

## Next

[Sessions](https://www.rightnowai.co/docs/open-models/session-affinity)