Streaming

What arrives, and when

Frame shapes

The response carries Content-Type: text/event-stream, Cache-Control: no-cache and Connection: keep-alive. Content arrives in chat-compatible data: frames, and partial output is on choices[].delta.

data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"}}]}

The usage frame has an empty choices array. On a hosted model it also carries the request's cost and the count of input tokens billed at the cached rate, both described in Chat completions:

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16,"prompt_tokens_details":{"cached_tokens":0},"cost":0.0000028,"runinfra":{"cost_microcents":280,"cached_input_tokens":0}}}

Chat completions and Responses streams send : keep-alive after 10 seconds of silence. Ignore lines starting with :.

A completed stream ends with data: [DONE].

data: [DONE]

A stream the gateway terminates early ends with an error frame and no [DONE]. A transport interruption can close it without an error frame. Treat a missing [DONE] as an incomplete response even when the HTTP status was 200. Retrying with the same Idempotency-Key returns the original request's receipt, or 409 while it is still running; it never replays the text. To get an answer, send a new request with a new key, which is new paid inference. See Retrying a stream.

data: {"error":{"message":"The upstream stream ended before signalling completion. Please retry.","type":"api_error","param":null,"code":"upstream_stream_error"},"request_id":"req_example_0000"}

code is upstream_error when the model stream ended, hosted_inference_response_timeout when the response deadline did, and upstream_stream_error otherwise. See Errors.

Is usage always included?

No. The gateway always asks the model for usage so it can settle the request, but it forwards the usage frame to you only when your request carried stream_options.include_usage: true. Omit it and content frames still stream while usage-only frames stay hidden. A usage frame is never synthesized: if the model reports no usage for a request, none is sent.

After an interrupted request, recovery uses the usage evidence that was persisted. A failed checkpoint write can leave that evidence missing or stale, so neither the delivered token count nor the final charge can be inferred from the last content chunk. Keep the request id for support. A new key starts new paid inference and cannot recover the old response.

There is one exception, so you can account for generated tokens even when no answer was delivered. When every choice finishes without final answer content, the gateway forwards one annotated usage frame whether or not you opted in.

When a stream ends without an answer

A choice that reaches its generation limit before producing answer content carries a machine-readable status on its terminal frame:

data: {"choices":[{"index":0,"delta":{},"finish_reason":"length","runinfra":{"output_status":{"code":"generation_limit_reached_before_answer","message":"The model reached its generation limit before producing answer content. Increase max_tokens or max_completion_tokens if the model limit allows, or shorten the prompt, then retry."}}}]}

If another terminal finish_reason produced no answer, the code is no_answer_content instead. Inspect the preserved finish_reason before deciding whether to retry.

The annotated usage frame then reports how much of the generated output was not an answer:

data: {"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":64,"total_tokens":76,"prompt_tokens_details":{"cached_tokens":0},"cost":0,"runinfra":{"output_token_accounting":{"visible_answer_tokens":0,"non_answer_completion_tokens":64,"sources":{"visible_answer_tokens":"proxy_classified","non_answer_completion_tokens":"provider_reported"}},"cost_microcents":0,"cached_input_tokens":0}}}

A stream that produced no answer settles at zero, so this frame says cost: 0 and cached_tokens: 0 whatever the model reported: nothing was priced, at either rate.

The usual cause is an output budget too small for a model that reasons first. See give reasoning models room.

Related