Streaming responses

Open models Streaming responses Updated 2026-11-11

On this page

Minimal code
What your loop will see
What to tune
Common mistakes
Next steps

On a model whose page lists streaming, stream tokens as they are generated so a user sees the answer appear instead of waiting for all of it. Any chat UI wants this. So does anything long enough that silence looks like a hang. This guide uses the chat-compatible stream. See Messages for the event-named Messages grammar.

Connection	Value
API base	`https://api.runinfra.ai/v1`
Workspace key	`rp_...`
Process variable	`RUNINFRA_GATEWAY_KEY`
Billing	The existing funding rules of your open-model account.

Replace MODEL_ID with an available id from Open models; read current capabilities, limits and prices from GET /v1/models.

Minimal code

cURL

curl -N https://api.runinfra.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNINFRA_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role":"user","content":"Tell me a short story"}],
    "max_tokens": 16384,
    "stream": true
  }'

What your loop will see

The streaming timeline from request to terminal frame: the 180 second first-token budget, the 740 second response budget, the opt-in usage frame with an exception when no final answer was delivered, and the two terminal frames, data DONE or an error frame on the same HTTP 200.

Streaming timeline
stream: true
HTTP 200, event stream opens
first token
within the 180s TTFT budget
delta frames
one complete frame at a time
opt-in, or no final answer
740s response budget, then 504
How a stream ends
data: [DONE]
the normal terminal frame after the last delta
a stream that fails mid-generation ends with one final error frame on the same HTTP 200;
error.code
names why. Handle it in your frame loop.
Only tokens actually delivered to you are billed.

What to tune

Parameter	Effect
`stream_options.include_usage`	Adds the final usage frame with token counts. Off by default.
`max_tokens`	Hard cap on generated length. Reasoning models spend part of this budget before answering, so keep it generous.
`temperature`	Higher is more surprise per token.
`stop`	Up to four stop sequences. Check the selected model's capabilities before sending them. The stream ends early on a match.

Common mistakes

Forgetting flush=True in Python. Without it stdout buffers and the tokens arrive in clumps.
Treating an error frame as a network failure. It arrives on the same HTTP 200, so handle it inside your frame loop.
Breaking the loop too early. The final content frame carries finish_reason and an empty delta.content. Keep reading until the stream closes.
Streaming through a buffering proxy. Some edges buffer server-sent events. Stream direct, or disable buffering on that layer.
Mixing n > 1 with streaming. Allowed, but every delta carries its own choices[].index. Route by index or the outputs interleave.

Streaming responses

Minimal code

What your loop will see

What to tune

Common mistakes

Next steps

Stream reference

Tool calling

Structured output