Open-model retries

What a reused key does

Reuse the same key only with the same HTTP method, route, and request body. A key sent with a different body is a different logical request, and it is refused with 400 idempotency_mismatch rather than answered with the wrong reply.

A non-streamed response up to 6 MiB is held for 24 hours; a larger one answers 422 idempotency_replay_unavailable.

Retrying a stream

A stream is not replayed byte for byte. We do not store the tokens we deliver, so there is nothing to send again. What the key protects on a streaming request is the part that costs money.

The receipt is an ordinary chat completion envelope, so a chat-compatible client can parse it. content is null because the delivered text was never stored. finish_reason is length or content_filter when the original reply was cut short, and null otherwise. The replay facts ride in idempotent_replay.

{
  "id": "chatcmpl-8f0a1c2e-...",
  "object": "chat.completion",
  "created": 1786800000,
  "model": "MODEL_ID",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": null },
      "finish_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 1000,
    "completion_tokens": 500,
    "total_tokens": 1500,
    "prompt_tokens_details": { "cached_tokens": 200 },
    "cost": 0.000282,
    "runinfra": { "cost_microcents": 28200, "cached_input_tokens": 200 }
  },
  "idempotent_replay": {
    "replayed": true,
    "streamed": true,
    "content_replayed": false,
    "regenerated": false,
    "billed_again": false,
    "original_status": 200,
    "stream_outcome": "completed",
    "usage_basis": "provider_reported",
    "cost_microcents": 28200,
    "detail": "The original request with this Idempotency-Key streamed to completion and settled once. Streamed tokens are not stored, so this reply carries the terminal usage and cost instead of the generated text. Nothing was regenerated and nothing was billed a second time."
  }
}

X-RunInfra-Idempotent-Replay is the signal on every route: read it before handing a response to a streaming parser. /v1/chat/completions and /v1/messages return the idempotent_replay object; on /v1/responses the reply is converted to a Responses object and only the header survives. A reply the original cut short keeps status: "incomplete" and its incomplete_details.reason there, as the original stream did.

If you need the generated text after a dropped stream, send a new request with a new key. That is new paid inference. Reusing the old key returns the receipt, not the text.

Only one charge per key

A retried request is charged once, streamed or not: a duplicate settlement for the same key is refused rather than billed twice. The key holds for the whole time a request can run, so a retry sent while a slow non-streaming request is still generating gets 409 idempotency_conflict and never starts a second generation.

When a key cannot be checked, a keyed request is refused rather than run without protection. Every endpoint, chat completions, Responses and messages included, returns 503 idempotency_unavailable with Retry-After before any inference starts or any charge is made; retry with the same key. On /v1/messages the same refusal arrives as Messages protocol's overloaded_error, which the Messages protocol SDKs retry. A request without an Idempotency-Key is never affected.

Client request ids

X-Client-Request-Id is an optional correlation header. The gateway echoes it back on the response. It is not a substitute for Idempotency-Key. Use a non-blank printable ASCII value of 512 characters or fewer. Any other value is refused with 400 invalid_trace_header.

Related