The gemini 3.6 flash batch rates landed at half the standard rates in
94a29e0708, so the hardcoded standard-rate expectation started failing
on staging and red-lit misc / Run tests on every PR.
Substituting a deployment's pricing wholesale billed the token direction it did
not configure at zero: get_model_info fills an absent cost with 0, and any
non-None pricing field suppressed the global fallback. A deployment declaring
only input_cost_per_token therefore billed output at nothing.
Each of the four batch cost fields now falls back to the model's published rate
when the deployment leaves it unset, so a one-sided override applies to the side
it configures and only that side.
Adds a parametrized regression over input-only, output-only, and both-zero, plus
coverage for a deployment whose model has no published entry. Annotates the new
test helpers per the repo's type-coverage rule and drops the narrative banner
comment from the batch tests.
Retrieving a completed batch computed its cost with no model identity: neither the
deployment's model nor its configured pricing reached the batch cost calculation.
For bedrock that left the cost model falling back to the provider's own response
model (e.g. "claude-sonnet-4-6"), which does not resolve under a bedrock provider,
so the lookup missed and cost silently became $0 while usage stayed correct.
Dropping the deployment's model info separately discarded any rates configured on
that deployment, billing a zero-cost deployment at the public rate instead.
Both are the same omission at the call site, so both are fixed by passing the
logging object's own model and the pricing the router registered for the
deployment.
Shape detection and block normalization sat in the generic batch layer, which
let batch and live parsing of the same wire format drift apart. Both now live on
AmazonConverseConfig as is_converse_usage_shape and usage_from_batch_output, so
batch_utils asks the provider adapter rather than knowing Bedrock's field names.
Adds direct coverage for the shape predicate, the completion of an incomplete
block, cache-count inflation, and the streaming usage event that shares the
public transform. Drops the narrative banner from the batch tests.
Every bedrock batch output line went through the Anthropic usage parser, which
reads snake_case input_tokens/output_tokens. Converse-family models (Nova and
friends) report camelCase inputTokens/outputTokens, so their usage came back
0/0/0 and the batch billed $0 despite real token consumption.
Usage is now selected by the shape of the payload: a Converse-shaped block goes
through the same transform the live Converse path uses, so a batch and an
equivalent non-batch call agree on tokens, including cache reads and writes.
Anthropic-shaped bedrock output is unchanged.
A shape neither parser understands (an InvokeModel-native payload from Titan,
Cohere, or Llama, which name their counts differently again) still reads zero,
but now warns with the keys it saw instead of silently billing $0.
Exposes the Converse usage transform as public, since batch parsing is a second
legitimate caller; that also removes the private-member access invoke_handler
was already making.
A third path reads a completed batch's output file, and it could not resolve the
bucket either. When cost is accounted from the retrieve itself rather than from
the poller, the batch success handler calls _handle_completed_batch, which fetches
the output file through _extract_file_access_credentials. That helper forwarded a
whitelist covering Azure and Vertex, gcs_bucket_name included, but nothing for
Bedrock, and retrieve_batch built its litellm_params through get_litellm_params,
whose fixed signature drops the trusted credential snapshot. So the snapshot never
reached the file read and it failed with "S3 bucket_name is required" for a bucket
the deployment had configured, leaving the batch's cost unrecorded.
Adding s3_bucket_name to that whitelist would not have worked. The Bedrock file
config deliberately resolves the bucket only from the immutable server-side
snapshot or the environment, never from a request param, because the bucket is
what managed file ids are validated against. The snapshot is therefore what has to
flow, exactly as it already does for the model-routed and cost-poller paths.
retrieve_batch now re-adds the snapshot after get_litellm_params, the same way the
file operations already do, the whitelist forwards it, and the proxy attaches it
for router-routed managed batches from the deployment behind the unified id.
Verified against a live proxy reading a real completed Bedrock batch: the cost row
appears within seconds of the retrieve carrying the batch's real spend and usage,
where before the read raised and no row was written.
Resolving those credentials is best effort. A batch whose deployment no longer
resolves, which happens when a model group is removed while batches are in
flight, still serves its status instead of failing the request on the lookup.
This matters for the OSS and polling-disabled configurations, where the retrieve
path is the only thing that accounts for a batch at all.
Completed batches that contain only failed requests do not generate an
output file, leaving output_file_id unset while the failures are recorded
through error_file_id instead.
The completion handler attempted to read the output payload regardless of
whether an output file actually existed. During retrieve polling this caused
the logging pipeline to fail with "Output file id is None cannot retrieve
file content", preventing normal completion bookkeeping from running.
Skip output retrieval when no output file is available and return an empty
batch summary (zero usage, zero cost, no model entries). The lower-level
file retrieval helper still reports an error if it is called directly with
an invalid or missing file identifier.
Closes#33987
Coverage jobs collect tests/test_litellm/batches. Move the mixed Responses and chat regression into that suite so CI exercises the normalization branch.
Completed-batch cost tracking parsed the whole output file into a list of
dicts, pretty-printed it into debug strings even with debug logging off, and
walked the list three times (cost, usage, models), so a large batch output
could pin a worker's memory. The output is now folded line by line into small
per-line stats records via _aggregate_batch_cost_usage_models, the eager
json.dumps debug calls are gone, and the raw-vertex path computes cost and
usage in one call instead of two. _get_batch_output_file_content_as_dictionary
becomes _fetch_batch_output_file_content (returns bytes); the superseded
three-pass helpers are deleted and their tests migrated
* feat(batches): track cost for unmanaged Bedrock batches, generalize the flag
CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw
model-invocation-job ARN, the same root cause previously fixed for
unmanaged Vertex batches. Bedrock batches embed the model name in their
s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl),
so the same routing mechanism now derives the model from that layout and
matches it to a configured bedrock deployment.
track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost
since two providers now share this mechanism.
* fix(batches): parse Bedrock batch output and price with deployment model name
Bedrock model-invocation-job results use modelOutput/error rows and short
internal model ids that are not in the cost map, so unmanaged batch cost
tracking logged tokens but $0 spend. Use deployment model name for pricing
and add regression tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
* test(batches): add 1:1 test file scaffold for batches component paths
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add harness test for create batch endpoint
* Add retrieve endpoint harness tests
* Add list endpoint harness tests
* Add cancel endpoint harness tests
* Add cancel endpoint harness tests
* Add test for litellm/batches/main.py
* Add test for litellm/tests/test_litellm/batches/test_batch_utils.py
* Add handler and transformation tests for all providers
* Fix: run batches tests in cicd
* fix(tests): remove azure/__init__.py that shadowed azure namespace package
Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to
insert tests/test_litellm/llms/ into sys.path[0], making our empty
azure/ dir shadow the real azure-identity namespace package. Any test
that patched azure.identity.* would then fail with AttributeError.
* style(tests): apply ruff format to test_batch_utils.py
Base migrated the formatter from black to ruff format (#31317); reformat the
batches scaffold test file to match.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
