Skip to content
Docs navigation: Open models / Batch

Batch

RightNow AIOpen modelsBatch Updated

On this page

Batch runs a file of requests at half of list price, and each answer is posted to your own address. A job runs within 24 hours as room allows, and a line not run by then is not charged.

How a batch runs

how a batch runs
Text
Illustration of how a batch runs.

A batch starts from your file link. Create: you create the job with your address and your cap, max_cost_cents, and we test your address. The check: we read the file and check every line, and the job then starts by itself. The run: lines run within 24 hours, as room allows, and a line not run by then is not charged. Each answer: one POST to your address, and your address answers 2xx. The bill: each acknowledged line, at half of list price.

Batch uses a model's spare capacity. On a busy model a job waits for quiet moments, so there is no promised start time. If the work cannot wait, use the normal API.

Your file

The file is JSONL in strict UTF-8: one JSON object on each line. Lines end with LF, and CRLF is accepted. A blank line is skipped but still numbered. The last line needs no newline.

requests.jsonl
JSON
{"custom_id":"row-1","method":"POST","url":"/v1/chat/completions","body":{"model":"MODEL_ID","messages":[{"role":"user","content":"Summarize this ticket."}]}}

MODEL_ID stands for an available id from Open models.

custom_id
A string of 1 to 512 characters, unique in the file.
method
POST.
url
/v1/chat/completions.
body
A chat completions body. Its model is the job's model.
Line size
At most 4 MiB (4,194,304 bytes). A larger line is line_too_large.
File size
At most 50 MiB (52,428,800 bytes). There is no limit on the number of lines.
JSON
A line that is not JSON in strict UTF-8 is invalid_json.
Members
A line holds exactly the four members above. Any other top-level member is invalid_request.
Text only
An image, audio or other media part is media_not_supported.
No stream
stream is absent or false, and stream_options is absent.
Prompt
A prompt estimated above 32,768 tokens is prompt_too_large.
Service tier
A line needs no service_tier.

The check counts a line that breaks a rule in lines_invalid, and the line never runs. The job's errors gives the count for each code and its first 5 line numbers, counted from 1. The line error codes are invalid_json, line_too_large, invalid_request, media_not_supported, prompt_too_large, link_changed, model_error, delivery_failed, expired, cancelled.

The link is https on port 443 and public, and at most 8,192 bytes long. A signed link must live until the job's 24 hours end. The file host answers a range request with 206 and an ETag, and serves the file as it is.

A link that breaks a rule is refused, or it fails the check: the job then ends failed with end_reason check_failed, and check_error names the rule. See Refusals.

Create, read and cancel

Four calls, each sent with your API key. A key lists and reads every job of its workspace. The ids and amounts in the samples are examples.

A job is not the batch object that chat-compatible clients expect: it has no endpoint, completion_window or auto_start member, and no file ids.

Create a job

POST /v1/batches creates a job. Its body is JSON of at most 24,576 bytes.

MemberValue
modelThe id of a model that offers Batch.
input_urlThe link to your file.
destinationAn object: type is webhook and url is your address.
max_cost_centsRequired. Your cap: a whole number of US cents from 1 to 100,000,000 (USD 1,000,000).
window_hoursOptional. 24 is the one value.
nameOptional. A label of up to 64 characters.

The Idempotency-Key header is optional. A repeat with the same header value, the same API key and the same members answers the first job and starts nothing new. A repeat that differs from the job in its model, its cap or its name, or in its link or its address while the job is open, answers 409 invalid_request, and error.param is Idempotency-Key. So does a repeat from another API key or by another person. No repeat sends a test to your address.

TypeScript
const res = await fetch("https://api.rightnowai.co/v1/batches", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.RIGHTNOW_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    input_url: "https://files.example.com/requests.jsonl",
    destination: { type: "webhook", url: "https://example.com/batch-answers" },
    max_cost_cents: 1200,
  }),
});
const { batch } = await res.json();

The call tests your address before it answers, and the test has 20 seconds. It then stores your address with your cap. The answer holds the new job in validating.

response.json
JSON
{
  "batch": {
    "id": "0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f",
    "model_slug": "MODEL_ID",
    "batch_state": "validating",
    "pause_reason": null,
    "end_reason": null,
    "check_error": null,
    "made_by": {
      "kind": "key"
    },
    "key_id": "7c1e2f3a-4b5d-4e6f-8a9b-0c1d2e3f4a5b",
    "lines_total": 0,
    "lines_invalid": 0,
    "succeeded": 0,
    "failed": 0,
    "expired": 0,
    "cancelled": 0,
    "undelivered": 0,
    "cost_microcents": 0,
    "max_cost_cents": 1200,
    "created_at": "2026-12-01T09:00:00.000Z",
    "started_at": null,
    "expires_at": "2026-12-02T09:00:00.000Z",
    "finished_at": null,
    "destination_type": "webhook",
    "notice": null,
    "name": null
  }
}

The job starts by itself. The check takes at most 120 seconds. When it passes, the job is queued with no second call. When it fails, the job is failed.

The 24 hours count from the create. The job runs within them as room allows, and a line not run by then is not charged.

Read one job

GET /v1/batches/{id} answers the job with its errors and its quote.

request
Shell
curl https://api.rightnowai.co/v1/batches/BATCH_ID \
  -H "Authorization: Bearer $RIGHTNOW_API_KEY"
response.json
JSON
{
  "batch": {
    "id": "0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f",
    "model_slug": "MODEL_ID",
    "batch_state": "running",
    "pause_reason": null,
    "end_reason": null,
    "check_error": null,
    "made_by": {
      "kind": "key"
    },
    "key_id": "7c1e2f3a-4b5d-4e6f-8a9b-0c1d2e3f4a5b",
    "lines_total": 1002,
    "lines_invalid": 2,
    "succeeded": 640,
    "failed": 1,
    "expired": 0,
    "cancelled": 0,
    "undelivered": 0,
    "cost_microcents": 262400000,
    "max_cost_cents": 1200,
    "created_at": "2026-12-01T09:00:00.000Z",
    "started_at": "2026-12-01T09:02:10.000Z",
    "expires_at": "2026-12-02T09:00:00.000Z",
    "finished_at": null,
    "destination_type": "webhook",
    "notice": null,
    "name": null,
    "errors": [
      {
        "code": "invalid_json",
        "count": 1,
        "first_lines": [
          17
        ]
      },
      {
        "code": "prompt_too_large",
        "count": 1,
        "first_lines": [
          412
        ]
      },
      {
        "code": "model_error",
        "count": 1,
        "first_lines": [
          88
        ]
      }
    ],
    "quote": {
      "input_cost_cents": 100,
      "output_cost_ceiling_cents": 1024,
      "most_cents": 1124,
      "flex_input_cents_per_million": 25,
      "flex_output_cents_per_million": 125,
      "est_input_tokens": 4000000
    }
  }
}

List your jobs

GET /v1/batches answers the jobs of your workspace, newest first. Its query takes four optional members.

limit
From 1 to 50. The default is 20.
cursor
The next_cursor of the answer before.
state
One state. The answer holds the jobs in it.
model
One model id. The answer holds its jobs.
request
Shell
curl "https://api.rightnowai.co/v1/batches?state=running&limit=20" \
  -H "Authorization: Bearer $RIGHTNOW_API_KEY"

The answer holds batches, the jobs, and next_cursor.

Cancel a job

POST /v1/batches/{id}/cancel takes no body. It answers batch_id and batch_state, which is cancelling or cancelled. A job cancelled before any line ran ends at once.

request
Shell
curl -X POST https://api.rightnowai.co/v1/batches/BATCH_ID/cancel \
  -H "Authorization: Bearer $RIGHTNOW_API_KEY"

A key cancels a job that a key started. A job started in the dashboard is cancelled in the dashboard: a key's cancel of it answers 403 batches_scope_violation, and error.param is made_by.

The job

MemberValue
idThe batch id, a UUID.
model_slugThe job's model, or null when the model is gone.
batch_stateOne of the states.
pause_reasonOne of the pause reasons exactly when the state is paused, else null.
end_reasonnull while the job is open, then one of the end reasons.
check_errornull, or the link's fault that failed the check, one of url_not_public, link_expires_too_soon, file_too_large, link_changed, link_not_supported.
made_byAn object whose kind is person or key. key means no person is on record: a job made with an API key reads key.
key_idThe job's key, a UUID, or null when the key is deleted.
lines_total, lines_invalid, succeeded, failed, expired, cancelled, undeliveredCounts of lines, each a whole number from 0. lines_total counts every line read, the invalid ones too.
cost_microcentsWhat the job's lines were charged so far, a whole number of millionths of a US cent.
max_cost_centsYour cap, a whole number of US cents, or null.
created_at, started_at, expires_at, finished_atISO 8601 times. expires_at is the end of the 24 hours. started_at and finished_at can be null.
destination_typewebhook or null.
noticebatch_ended, batch_paused, batch_expires_soon or null.
nameYour label, or null.
errorsOnly on GET /v1/batches/{id}. For each line error code: code, count and first_lines, its first 5 line numbers.
quoteOnly on GET /v1/batches/{id}. See What it costs.

On the API a job's key is the key of the call itself. In the dashboard it is one of your own active keys. A job's cost counts toward the monthly limit of its key.

When the key is rotated, its open jobs started in the dashboard move to the new key. Its open jobs started through the API are cancelled, and each ends with end_reason key_revoked. Start them again with the new key. The replaced key cannot use Batch from the moment it is replaced: every Batch call with it answers 403 batches_scope_violation. When the key is revoked, all its open jobs are cancelled.

States

validating
We are reading the file.
checked
The file is read and the quote is ready. The job has not started.
queued
Started. Its lines run as room allows.
running
Lines are running.
paused
Stopped for now. pause_reason says why.
cancelling
A cancel is in progress.
completed
Ended: no line is left to run.
expired
Ended with lines not run. Those lines are not charged.
cancelled
Ended: the job was cancelled.
failed
Ended: the check failed, or the job could not go on.

Pause reasons

funds
The dashboard reads Out of credits. Add funds to run the rest.
destination
The answers we could not deliver passed the bound. Fix the address in the dashboard. A passing test there resumes the job. With your API key you can read the job, cancel it if a key started it, or let its 24 hours end.
source
The link stopped working or the file changed. A working link to the same file resumes the job.
key_limit
The monthly limit of the job's key was reached.

End reasons

all_done
The job went through every line.
max_cost_reached
A line could pass your cap, so the lines left were not run.
cancel_asked
A person or a key cancelled the job.
key_revoked
The job's key was revoked, or the job was started through the API and its key was rotated.
account_deleted
The account was deleted.
window_ended
The 24 hours ended with lines left.
model_withdrawn
The model stopped offering the batch price.
not_flex
The workspace's own price is lower than the batch price.
not_started
The job was checked and not started within 30 minutes.
check_failed
The check failed. check_error names a link's fault.

What arrives at your address

Each answered line is one POST to your address. No redirect is followed. A POST has 30 seconds in all, and at most 15 seconds of silence.

The address is https on port 443 and public, and at most 8,192 bytes long. A signed address must live until the job's 24 hours end.

The test

The create posts one signed test to your address. Its body holds test: true and a random value. Your address passes when it answers 2xx and its reply holds that value in its first 64 KiB. A failed test refuses the create with destination_unreachable.

An answered line

answer.json
JSON
{
  "id": "bdl_0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f_1_1",
  "batch_id": "0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f",
  "custom_id": "row-1",
  "line": 1,
  "attempt": 1,
  "test": false,
  "response": {
    "id": "chatcmpl_example",
    "object": "chat.completion",
    "created": 1796115900,
    "model": "MODEL_ID",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "The customer asks for a refund."
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 12,
      "completion_tokens": 8,
      "total_tokens": 20
    }
  },
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  },
  "cost": {
    "microcents": 1300
  }
}
id
The same for the same line and attempt. Dedupe on it.
batch_id
The job.
custom_id
The line's own custom_id.
line
The line's number in the file, counted from 1.
attempt
The number of the attempt.
test
false on a line. Only the test carries true.
response
The chat completion object.
usage
prompt_tokens, completion_tokens, total_tokens.
cost
microcents: what the line costs, in millionths of a US cent.

Verify the signature

Every POST is signed in the Standard Webhooks format, so any Standard Webhooks library verifies it.

headers
Text
content-type: application/json
webhook-id: <id>
webhook-timestamp: <Unix seconds>
webhook-signature: v1,<base64 signature>
x-rightnow-signature-version: <version>

The signed content is <webhook-id>.<webhook-timestamp>.<body>. The signature is v1, and then the base64 HMAC-SHA-256 of the signed content. webhook-timestamp is in Unix seconds.

The signing secret is not on the API. The workspace's owner reads it in the dashboard and can read it again. Verify every delivery with it. Each workspace has one secret, and it reads whsec_ and then base64. A rotation raises its version, so the old secret stops working.

Acknowledge

Answer any 2xx. The reply's body is never read. Only then is the line charged, at its final usage.

With no 2xx, the line ends undelivered:

  • It costs nothing, and it is never run or sent again.
  • After the first one, the job delivers one line at a time until 20 in a row are acknowledged.
  • The job pauses for destination when what its undelivered answers would have cost passes the larger of USD 1 and 2% of what it has paid. The same test runs over the account's last 24 hours.

A failed line

A line that fails in one of these ways gets one POST with error in place of response. error.code says which. A failed line costs nothing. In rare cases a line is charged before it fails: we return that charge to your balance within 3 days.

  • invalid_request or prompt_too_large: the line no longer passes the check when it runs, or the model refuses it.
  • model_error: the model failed on 3 attempts.
  • link_changed: the file at the link is not the file that was checked. custom_id is null.
failed.json
JSON
{
  "id": "bdl_0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f_88_3",
  "batch_id": "0b8f6d1e-3c2a-4f7b-9d5e-6a1c2b3d4e5f",
  "custom_id": "row-88",
  "line": 88,
  "attempt": 3,
  "test": false,
  "error": {
    "code": "model_error"
  },
  "usage": null,
  "cost": null
}

These lines get no POST:

  • A line refused at the check. It is counted in lines_invalid.
  • A line we end while it waits.
  • The waiting lines of a job that is cancelled, expires or ends at its cap.

What it costs

A line costs half of list price on input, cached input and output. The batch price is never combined with another discount.

max_cost_cents is your cap and is required: a whole number of US cents from 1 to 100,000,000 (USD 1,000,000). The cap is a hard bound. Each waiting line counts against it at its ceiling, and a line that could pass it is not run: it is counted as cancelled, and the job ends with end_reason max_cost_reached. The most a job can cost is your cap.

The check gives the job's quote:

input_cost_cents
The estimated input tokens at the batch input price.
output_cost_ceiling_cents
8,192 output tokens for each valid line, at the batch output price. A line never outputs more.
most_cents
The two amounts above, added. With a cap below it, some lines may not run.
flex_input_cents_per_million, flex_output_cents_per_million
The batch input and output prices, in US cents per million tokens.
est_input_tokens
The estimated input tokens.

A line is charged only after your address answered 2xx. An undelivered line, a failed line and a line not run within the 24 hours cost nothing.

What we keep

An answer exists only in memory, between the model's reply and your address's 2xx.

While a job is open we keep the link and the address, both encrypted, the file's etag, and for each line its offset in the file, its length, a keyed digest and a token estimate. When the job ends, those are erased. The job keeps its name, its counts and its money.

A line that was charged and then failed keeps its row until the charge is returned and cleared, 3 days at most.

See Data retention.

One request at the batch price

Send service_tier: "flex" on POST /v1/chat/completions, POST /v1/responses or POST /v1/messages to ask for the batch price on one request.

request
Shell
curl https://api.rightnowai.co/v1/chat/completions \
  -H "Authorization: Bearer $RIGHTNOW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role": "user", "content": "Summarize this ticket."}],
    "max_tokens": 512,
    "service_tier": "flex"
  }'

A request that cannot run at the batch price is never refused. It is served at the ordinary price. Every answer says which it was: the header X-RightNow-Capacity reads flex at the batch price and shared at the ordinary price.

At the batch price, Chat completions and Responses also return service_tier: "flex" in the body, and Chat completions on each stream chunk too. On POST /v1/messages the header is the only sign.

A request is served at the ordinary price when:

  • The model does not offer the batch price.
  • The key is a partner's key.
  • The workspace has never bought credits.
  • A free period applies.
  • Or when the batch price would not lower your price.
Price
Half of list price on input, cached input and output, never combined with another discount.
Output
At most 8,192 tokens for the whole call, shared across its choices. A larger or absent max_tokens is set to the cap, and the answer then carries X-RunInfra-Output-Clamped: true and X-RunInfra-Output-Token-Ceiling. A call that reaches the cap ends with finish_reason: "length".
Prompt
At most 32,768 estimated tokens, or the model's large prompt threshold where that is lower.
Streaming
Allowed.
Once it answers
A request whose answer has started is never ended.

The API refuses a request at the batch price with one of these messages:

CodeStatusMessage
flex_prompt_too_large400The batch price does not take requests over {tokens} tokens. Send them without the field.
flex_capacity_unavailable429No room at the batch price now. Nothing was charged. Send it again after the time in Retry-After.
flex_capacity_yielded429This request was ended before its first token to make room. Nothing was charged. Send it again.
idempotency_answer_not_stored422This Idempotency-Key was used by a request at the batch price. Its answer is not stored. Send the request again with a new key.
  • flex_prompt_too_large: {tokens} stands for the limit. param is service_tier, and limit holds the limit.
  • flex_capacity_unavailable: nothing is queued. The answer carries Retry-After and Retry-After-Ms.
  • idempotency_answer_not_stored: the answer to a request sent again under its Idempotency-Key. param is Idempotency-Key.

The ordinary 402 answers apply too.

Refusals

A refusal carries its code in error.code. A link's fault that the check finds is not a refusal: it ends the job, and check_error carries the code.

CodeStatusWhen
batch_not_open503Batch is not ready on this host. Retry after 60 seconds. The answer carries Retry-After: 60.
invalid_request400A member is missing, has the wrong type or is out of bounds. error.param names it.
not_found404The batch id is unknown, or belongs to another workspace.
model_not_sold404The model is unknown or not public.
flex_not_available409The batch price is not available. error.reason says why: model when the model does not offer it now, partner when the key is a partner's key, price when the workspace's own price is already lower.
batches_scope_violation403The key is limited to one deployment or endpoint, or it cannot use Batch, as a replaced key cannot. A key's cancel of a job started in the dashboard answers it too.
url_not_public400The link is not https, not on port 443 or not public, is one of our own addresses, or redirects more than 3 times.
link_not_supported400The file host does not answer a range request with 206 and an ETag, or it encodes the content.
link_expires_too_soon400A signed link ends before the job's 24 hours end.
link_changed409The file at the link is not the file that was checked.
file_too_large413The file is above 50 MiB.
destination_unreachable400Your address breaks a rule, the test failed, or the test's value did not come back.
needs_funds402The account has never bought credits. Batch runs only for accounts that bought credits.
key_spend_limit_reached402The monthly limit of the job's key is reached.
batch_ended409A cancel on a job that has ended.
rate_limit_exceeded429More than 55 batch calls in a minute with one key.
payload_too_large413The body of a create is over 24,576 bytes.