Batch
Updated
- How a batch runs
- Your file
- Create, read and cancel
- What arrives at your address
- What it costs
- What we keep
- One request at the batch price
- Refusals
- Related
Batch runs a file of requests at half of list price, and each answer is posted to your own address. A job runs within 24 hours as room allows, and a line not run by then is not charged.
How a batch runs
A batch starts from your file link. Create: you create the job with your address and your cap, max_cost_cents, and we test your address. The check: we read the file and check every line, and the job then starts by itself. The run: lines run within 24 hours, as room allows, and a line not run by then is not charged. Each answer: one POST to your address, and your address answers 2xx. The bill: each acknowledged line, at half of list price.
Batch uses a model's spare capacity. On a busy model a job waits for quiet moments, so there is no promised start time. If the work cannot wait, use the normal API.
Your file
The file is JSONL in strict UTF-8: one JSON object on each line. Lines end with LF, and CRLF is accepted. A blank line is skipped but still numbered. The last line needs no newline.
MODEL_ID stands for an available id from Open models.
custom_id- A string of 1 to 512 characters, unique in the file.
methodPOST.url/v1/chat/completions.body- A chat completions body. Its
modelis the job's model.
- Line size
- At most 4 MiB (4,194,304 bytes). A larger line is
line_too_large. - File size
- At most 50 MiB (52,428,800 bytes). There is no limit on the number of lines.
- JSON
- A line that is not JSON in strict UTF-8 is
invalid_json. - Members
- A line holds exactly the four members above. Any other top-level member is
invalid_request. - Text only
- An image, audio or other media part is
media_not_supported. - No stream
streamis absent orfalse, andstream_optionsis absent.- Prompt
- A prompt estimated above 32,768 tokens is
prompt_too_large. - Service tier
- A line needs no
service_tier.
The check counts a line that breaks a rule in lines_invalid, and the line never runs. The job's errors gives the count for each code and its first 5 line numbers, counted from 1. The line error codes are invalid_json, line_too_large, invalid_request, media_not_supported, prompt_too_large, link_changed, model_error, delivery_failed, expired, cancelled.
The link is https on port 443 and public, and at most 8,192 bytes long. A signed link must live until the job's 24 hours end. The file host answers a range request with 206 and an ETag, and serves the file as it is.
A link that breaks a rule is refused, or it fails the check: the job then ends failed with end_reason check_failed, and check_error names the rule. See Refusals.
Create, read and cancel
Four calls, each sent with your API key. A key lists and reads every job of its workspace. The ids and amounts in the samples are examples.
A job is not the batch object that chat-compatible clients expect: it has no endpoint, completion_window or auto_start member, and no file ids.
Create a job
POST /v1/batches creates a job. Its body is JSON of at most 24,576 bytes.
| Member | Value |
|---|---|
model | The id of a model that offers Batch. |
input_url | The link to your file. |
destination | An object: type is webhook and url is your address. |
max_cost_cents | Required. Your cap: a whole number of US cents from 1 to 100,000,000 (USD 1,000,000). |
window_hours | Optional. 24 is the one value. |
name | Optional. A label of up to 64 characters. |
The Idempotency-Key header is optional. A repeat with the same header value, the same API key and the same members answers the first job and starts nothing new. A repeat that differs from the job in its model, its cap or its name, or in its link or its address while the job is open, answers 409 invalid_request, and error.param is Idempotency-Key. So does a repeat from another API key or by another person. No repeat sends a test to your address.
The call tests your address before it answers, and the test has 20 seconds. It then stores your address with your cap. The answer holds the new job in validating.
The job starts by itself. The check takes at most 120 seconds. When it passes, the job is queued with no second call. When it fails, the job is failed.
The 24 hours count from the create. The job runs within them as room allows, and a line not run by then is not charged.
Read one job
GET /v1/batches/{id} answers the job with its errors and its quote.
List your jobs
GET /v1/batches answers the jobs of your workspace, newest first. Its query takes four optional members.
limit- From 1 to 50. The default is 20.
cursor- The
next_cursorof the answer before. state- One state. The answer holds the jobs in it.
model- One model id. The answer holds its jobs.
The answer holds batches, the jobs, and next_cursor.
Cancel a job
POST /v1/batches/{id}/cancel takes no body. It answers batch_id and batch_state, which is cancelling or cancelled. A job cancelled before any line ran ends at once.
A key cancels a job that a key started. A job started in the dashboard is cancelled in the dashboard: a key's cancel of it answers 403 batches_scope_violation, and error.param is made_by.
The job
| Member | Value |
|---|---|
id | The batch id, a UUID. |
model_slug | The job's model, or null when the model is gone. |
batch_state | One of the states. |
pause_reason | One of the pause reasons exactly when the state is paused, else null. |
end_reason | null while the job is open, then one of the end reasons. |
check_error | null, or the link's fault that failed the check, one of url_not_public, link_expires_too_soon, file_too_large, link_changed, link_not_supported. |
made_by | An object whose kind is person or key. key means no person is on record: a job made with an API key reads key. |
key_id | The job's key, a UUID, or null when the key is deleted. |
lines_total, lines_invalid, succeeded, failed, expired, cancelled, undelivered | Counts of lines, each a whole number from 0. lines_total counts every line read, the invalid ones too. |
cost_microcents | What the job's lines were charged so far, a whole number of millionths of a US cent. |
max_cost_cents | Your cap, a whole number of US cents, or null. |
created_at, started_at, expires_at, finished_at | ISO 8601 times. expires_at is the end of the 24 hours. started_at and finished_at can be null. |
destination_type | webhook or null. |
notice | batch_ended, batch_paused, batch_expires_soon or null. |
name | Your label, or null. |
errors | Only on GET /v1/batches/{id}. For each line error code: code, count and first_lines, its first 5 line numbers. |
quote | Only on GET /v1/batches/{id}. See What it costs. |
On the API a job's key is the key of the call itself. In the dashboard it is one of your own active keys. A job's cost counts toward the monthly limit of its key.
When the key is rotated, its open jobs started in the dashboard move to the new key. Its open jobs started through the API are cancelled, and each ends with end_reason key_revoked. Start them again with the new key. The replaced key cannot use Batch from the moment it is replaced: every Batch call with it answers 403 batches_scope_violation. When the key is revoked, all its open jobs are cancelled.
States
validating- We are reading the file.
checked- The file is read and the quote is ready. The job has not started.
queued- Started. Its lines run as room allows.
running- Lines are running.
paused- Stopped for now.
pause_reasonsays why. cancelling- A cancel is in progress.
completed- Ended: no line is left to run.
expired- Ended with lines not run. Those lines are not charged.
cancelled- Ended: the job was cancelled.
failed- Ended: the check failed, or the job could not go on.
Pause reasons
funds- The dashboard reads Out of credits. Add funds to run the rest.
destination- The answers we could not deliver passed the bound. Fix the address in the dashboard. A passing test there resumes the job. With your API key you can read the job, cancel it if a key started it, or let its 24 hours end.
source- The link stopped working or the file changed. A working link to the same file resumes the job.
key_limit- The monthly limit of the job's key was reached.
End reasons
all_done- The job went through every line.
max_cost_reached- A line could pass your cap, so the lines left were not run.
cancel_asked- A person or a key cancelled the job.
key_revoked- The job's key was revoked, or the job was started through the API and its key was rotated.
account_deleted- The account was deleted.
window_ended- The 24 hours ended with lines left.
model_withdrawn- The model stopped offering the batch price.
not_flex- The workspace's own price is lower than the batch price.
not_started- The job was checked and not started within 30 minutes.
check_failed- The check failed.
check_errornames a link's fault.
What arrives at your address
Each answered line is one POST to your address. No redirect is followed. A POST has 30 seconds in all, and at most 15 seconds of silence.
The address is https on port 443 and public, and at most 8,192 bytes long. A signed address must live until the job's 24 hours end.
The test
The create posts one signed test to your address. Its body holds test: true and a random value. Your address passes when it answers 2xx and its reply holds that value in its first 64 KiB. A failed test refuses the create with destination_unreachable.
An answered line
id- The same for the same line and attempt. Dedupe on it.
batch_id- The job.
custom_id- The line's own
custom_id. line- The line's number in the file, counted from 1.
attempt- The number of the attempt.
testfalseon a line. Only the test carriestrue.response- The chat completion object.
usageprompt_tokens, completion_tokens, total_tokens.costmicrocents: what the line costs, in millionths of a US cent.
Verify the signature
Every POST is signed in the Standard Webhooks format, so any Standard Webhooks library verifies it.
The signed content is <webhook-id>.. The signature is v1, and then the base64 HMAC-SHA-256 of the signed content. webhook-timestamp is in Unix seconds.
The signing secret is not on the API. The workspace's owner reads it in the dashboard and can read it again. Verify every delivery with it. Each workspace has one secret, and it reads whsec_ and then base64. A rotation raises its version, so the old secret stops working.
Acknowledge
Answer any 2xx. The reply's body is never read. Only then is the line charged, at its final usage.
With no 2xx, the line ends undelivered:
- It costs nothing, and it is never run or sent again.
- After the first one, the job delivers one line at a time until 20 in a row are acknowledged.
- The job pauses for
destinationwhen what its undelivered answers would have cost passes the larger of USD 1 and 2% of what it has paid. The same test runs over the account's last 24 hours.
A failed line
A line that fails in one of these ways gets one POST with error in place of response. error.code says which. A failed line costs nothing. In rare cases a line is charged before it fails: we return that charge to your balance within 3 days.
invalid_requestorprompt_too_large: the line no longer passes the check when it runs, or the model refuses it.model_error: the model failed on 3 attempts.link_changed: the file at the link is not the file that was checked.custom_idisnull.
These lines get no POST:
- A line refused at the check. It is counted in
lines_invalid. - A line we end while it waits.
- The waiting lines of a job that is cancelled, expires or ends at its cap.
What it costs
A line costs half of list price on input, cached input and output. The batch price is never combined with another discount.
max_cost_cents is your cap and is required: a whole number of US cents from 1 to 100,000,000 (USD 1,000,000). The cap is a hard bound. Each waiting line counts against it at its ceiling, and a line that could pass it is not run: it is counted as cancelled, and the job ends with end_reason max_cost_reached. The most a job can cost is your cap.
The check gives the job's quote:
input_cost_cents- The estimated input tokens at the batch input price.
output_cost_ceiling_cents- 8,192 output tokens for each valid line, at the batch output price. A line never outputs more.
most_cents- The two amounts above, added. With a cap below it, some lines may not run.
flex_input_cents_per_million, flex_output_cents_per_million- The batch input and output prices, in US cents per million tokens.
est_input_tokens- The estimated input tokens.
A line is charged only after your address answered 2xx. An undelivered line, a failed line and a line not run within the 24 hours cost nothing.
What we keep
An answer exists only in memory, between the model's reply and your address's 2xx.
While a job is open we keep the link and the address, both encrypted, the file's etag, and for each line its offset in the file, its length, a keyed digest and a token estimate. When the job ends, those are erased. The job keeps its name, its counts and its money.
A line that was charged and then failed keeps its row until the charge is returned and cleared, 3 days at most.
See Data retention.
One request at the batch price
Send service_tier: "flex" on POST /v1/chat/completions, POST /v1/responses or POST /v1/messages to ask for the batch price on one request.
A request that cannot run at the batch price is never refused. It is served at the ordinary price. Every answer says which it was: the header X-RightNow-Capacity reads flex at the batch price and shared at the ordinary price.
At the batch price, Chat completions and Responses also return service_tier: "flex" in the body, and Chat completions on each stream chunk too. On POST /v1/messages the header is the only sign.
A request is served at the ordinary price when:
- The model does not offer the batch price.
- The key is a partner's key.
- The workspace has never bought credits.
- A free period applies.
- Or when the batch price would not lower your price.
- Price
- Half of list price on input, cached input and output, never combined with another discount.
- Output
- At most 8,192 tokens for the whole call, shared across its choices. A larger or absent
max_tokensis set to the cap, and the answer then carriesX-RunInfra-Output-Clamped: trueandX-. A call that reaches the cap ends withRunInfra- Output- Token- Ceiling finish_reason: "length". - Prompt
- At most 32,768 estimated tokens, or the model's large prompt threshold where that is lower.
- Streaming
- Allowed.
- Once it answers
- A request whose answer has started is never ended.
The API refuses a request at the batch price with one of these messages:
| Code | Status | Message |
|---|---|---|
flex_prompt_too_large | 400 | The batch price does not take requests over {tokens} tokens. Send them without the field. |
flex_capacity_unavailable | 429 | No room at the batch price now. Nothing was charged. Send it again after the time in Retry-After. |
flex_capacity_yielded | 429 | This request was ended before its first token to make room. Nothing was charged. Send it again. |
idempotency_answer_not_stored | 422 | This Idempotency-Key was used by a request at the batch price. Its answer is not stored. Send the request again with a new key. |
flex_prompt_too_large:{tokens}stands for the limit.paramisservice_tier, andlimitholds the limit.flex_capacity_unavailable: nothing is queued. The answer carriesRetry-AfterandRetry-After-Ms.idempotency_answer_not_stored: the answer to a request sent again under itsIdempotency-Key.paramisIdempotency-Key.
The ordinary 402 answers apply too.
Refusals
A refusal carries its code in error.code. A link's fault that the check finds is not a refusal: it ends the job, and check_error carries the code.
| Code | Status | When |
|---|---|---|
batch_not_open | 503 | Batch is not ready on this host. Retry after 60 seconds. The answer carries Retry-After: 60. |
invalid_request | 400 | A member is missing, has the wrong type or is out of bounds. error.param names it. |
not_found | 404 | The batch id is unknown, or belongs to another workspace. |
model_not_sold | 404 | The model is unknown or not public. |
flex_not_available | 409 | The batch price is not available. error.reason says why: model when the model does not offer it now, partner when the key is a partner's key, price when the workspace's own price is already lower. |
batches_scope_violation | 403 | The key is limited to one deployment or endpoint, or it cannot use Batch, as a replaced key cannot. A key's cancel of a job started in the dashboard answers it too. |
url_not_public | 400 | The link is not https, not on port 443 or not public, is one of our own addresses, or redirects more than 3 times. |
link_not_supported | 400 | The file host does not answer a range request with 206 and an ETag, or it encodes the content. |
link_expires_too_soon | 400 | A signed link ends before the job's 24 hours end. |
link_changed | 409 | The file at the link is not the file that was checked. |
file_too_large | 413 | The file is above 50 MiB. |
destination_unreachable | 400 | Your address breaks a rule, the test failed, or the test's value did not come back. |
needs_funds | 402 | The account has never bought credits. Batch runs only for accounts that bought credits. |
key_spend_limit_reached | 402 | The monthly limit of the job's key is reached. |
batch_ended | 409 | A cancel on a job that has ended. |
rate_limit_exceeded | 429 | More than 55 batch calls in a minute with one key. |
payload_too_large | 413 | The body of a create is over 24,576 bytes. |