# Streaming responses

[Open models](https://www.rightnowai.co/docs/open-models) / [Streaming responses](https://www.rightnowai.co/docs/open-models/guides/streaming)

Updated 2026-11-11

On a model whose page lists streaming, stream tokens as they are generated so a user sees the answer appear instead of waiting for all of it. Any chat UI wants this. So does anything long enough that silence looks like a hang. This guide uses the chat-compatible stream. See [Messages](https://www.rightnowai.co/docs/open-models/messages#streaming) for the event-named Messages grammar.

**Connection**

| Connection | Value |
| --- | --- |
| API base | `https://api.runinfra.ai/v1` |
| Workspace key | `rp_...` |
| Process variable | `RUNINFRA_GATEWAY_KEY` |
| Billing | The existing funding rules of your open-model account. |

Replace `MODEL_ID` with an available id from [Open models](https://runinfra.ai/inference-api?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models); read current capabilities, limits and prices from `GET /v1/models`.

## Minimal code

**cURL**

```shell
curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role":"user","content":"Tell me a short story"}],
    "max_tokens": 16384,
    "stream": true
  }'
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
)

stream = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Tell me a short story"}],
    max_tokens=16384,
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue  # the usage frame has no choices
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runinfra.ai/v1",
  apiKey: process.env.RUNINFRA_GATEWAY_KEY,
});

const stream = await client.chat.completions.create({
  model: "MODEL_ID",
  messages: [{ role: "user", content: "Tell me a short story" }],
  max_tokens: 16384,
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```

## What your loop will see

> The streaming timeline from request to terminal frame: the 180 second first-token budget, the 740 second response budget, the opt-in usage frame with an exception when no final answer was delivered, and the two terminal frames, data DONE or an error frame on the same HTTP 200.
>
> - Streaming timeline
> - stream: true
> - HTTP 200, event stream opens
> - first token
> - within the 180s TTFT budget
> - delta frames
> - one complete frame at a time
> - opt-in, or no final answer
> - 740s response budget, then 504
> - How a stream ends
> - data: \[DONE\]
> - the normal terminal frame after the last delta
> - a stream that fails mid-generation ends with one final error frame on the same HTTP 200;
> - error.code
> - names why. Handle it in your frame loop.
> - Only tokens actually delivered to you are billed.

## What to tune

**Reference 80**

| Parameter | Effect |
| --- | --- |
| `stream_options.include_usage` | Adds the final usage frame with token counts. Off by default. |
| `max_tokens` | Hard cap on generated length. Reasoning models spend part of this budget before answering, so keep it generous. |
| `temperature` | Higher is more surprise per token. |
| `stop` | Up to four stop sequences. Check the selected model's capabilities before sending them. The stream ends early on a match. |

## Common mistakes

- **Forgetting `flush=True` in Python.** Without it stdout buffers and the tokens arrive in clumps.
- **Treating an error frame as a network failure.** It arrives on the same HTTP 200, so handle it inside your frame loop.
- **Breaking the loop too early.** The final content frame carries `finish_reason` and an empty `delta.content`. Keep reading until the stream closes.
- **Streaming through a buffering proxy.** Some edges buffer server-sent events. Stream direct, or disable buffering on that layer.
- **Mixing `n > 1` with streaming.** Allowed, but every delta carries its own `choices[].index`. Route by index or the outputs interleave.

## Next steps

[### Stream reference Exact frame shapes, and the usage frame.](https://www.rightnowai.co/docs/open-models/streaming) [### Tool calling Let the model act, then answer.](https://www.rightnowai.co/docs/open-models/guides/tool-calling) [### Structured output Get JSON that matches your schema.](https://www.rightnowai.co/docs/open-models/guides/structured-output)

[Source reference](https://runinfra.ai/docs/cookbook/streaming?utm_source=rightnowai.co&utm_medium=referral&utm_content=docs-open-models)

## Next

[Tool calling](https://www.rightnowai.co/docs/open-models/guides/tool-calling)