54d404ef2c
* fix(e2e): wire batch provider secrets for docker and k8s
Point batch deployments at the credential field names and os.environ refs
the gateway actually resolves from process env (compose .env or EKS secret
mounts). Missing secrets skip instead of failing red so a red run means a
product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback
* fix(e2e): drop batch provider_env unit tests
The batches suite is live e2e only; no monkeypatch or unit-level tests
* fix: batch credentials, provider list, and team db lookup
Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env
Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500
Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"
Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets
* fix: tag reseed, team window spend, and remaining e2e flakes
Reseed spend:tag counters from LiteLLM_TagTable so cold redis still
enforces after the spend writer flushes
When applying post-call cost to team multi-window counters, load the
team from the DB if it is missing from the management cache so window
spend is not dropped on cache misses
Harden cold-counter reseed e2e (namespace-aware keys, burst success,
poll). Give tag budget more headroom. Retry /key/update on redis DNS
blips. Ensure NLTK punkt_tab is present for pipecat realtime audio
* revert: drop product code changes; e2e-only scope
Reverts all litellm/ and unit-test product edits. This branch is limited
to tests/e2e per contributor instruction
* fix(e2e): harden batch list and team member setup races
provider_fallback list falls back when managed batches reject provider
filtering. Team create waits for /team/info and member_add retries on
transient team-not-found so split control-plane lag does not red the suite
* fix(e2e): remove .env.example
Leave local .env and docker-compose env wiring as the secret source
* fix(e2e): wire files_settings and faster budget rescheduler for compose
OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a
short rescheduler window. Drop unsupported bedrock-encoded create_batch cells,
tolerate bedrock file.bytes=0, and surface team-info wait failures instead of
hanging silently
* chore(e2e): strip verbose comments from batch capabilities
* fix(e2e): assert managed list fallback before provider_fallback skip
When provider-scoped list is rejected, still fetch the unfiltered list and
check the envelope. Only skip membership when the id is a raw
provider_fallback batch that managed list cannot index
4.8 KiB
Batches Test Coverage Matrix
Live e2e coverage of the Batches API over a real proxy, real provider keys, and
real cost. Synchronous tier only: a batch's completion window is 24h, so these
tests never wait for completed. They assert the proxy accepts, routes, retrieves,
cancels, and lists a batch; everything created is deleted on teardown.
Provider x operation
Only supported cells are tested. The capability table in capabilities.py holds one
row per supported (provider, scenario) pair, so there are no skipped cells in the
parametrized run. The batches suite never skips: missing provider creds or upstream
failures are hard test failures (see tests/e2e/CLAUDE.md).
| Provider | create | retrieve | cancel | list | file backing |
|---|---|---|---|---|---|
| OpenAI | yes | yes | yes | yes | OpenAI Files |
| Azure | yes | yes | yes | yes | Azure Files |
| Vertex AI | yes | yes | yes | yes | GCS (gcs_bucket_name / GCS_BUCKET_NAME on model) |
| Bedrock | yes (unified only) | yes | no (limited upstream) | no | S3 (s3_bucket_name + aws_* + AWS_BATCH_ROLE_ARN on model) |
Bedrock cancel is unreliable upstream and list is unsupported, so both are gated off
(can_cancel=False, can_list=False) when that provider is enabled in the matrix.
Bedrock file upload requires a model on the request (encoded / unified scenarios only);
model_param and provider_fallback are omitted because POST /bedrock/v1/files has no
model-less passthrough path.
Routing scenarios (per litellm/proxy/batches_endpoints/endpoints.py)
Each create-capable provider runs all four. The test asserts the returned file id
and batch id carry the shape that scenario must produce (matches_id_shape):
| Scenario | How the batch is routed | File id | Batch id |
|---|---|---|---|
encoded |
upload with ?model= -> model-encoded file id -> create with just that id |
model-encoded | model-encoded |
unified |
upload with target_model_names= -> unified managed file id -> create with that id |
managed | managed |
model_param |
raw file (provider-fallback upload) -> create with model in the body |
raw | model-encoded |
provider_fallback |
raw file -> POST /{provider}/v1/batches, env creds, no model |
raw | raw (native provider shape) |
"managed" ids base64-decode to a litellm_proxy marker; "model-encoded" ids keep the
provider prefix and base64-encode litellm:<id>;model,<model>; "raw" ids are the
provider's native ids. Asserting these catches a proxy that returns a raw id where it
should manage it, or vice versa. On top of the id shape, a misroute to the wrong
provider also fails create (the file id / model do not belong there), and the
provider_fallback raw batch id is additionally checked against the provider's native
shape (raw_id_matches_provider).
Key model restriction
test_batch_key_model_access_denied mints a key restricted to one model
(resources.key(models=[...])) and proves the proxy returns 403
key_model_access_denied both when that key uploads a file for a disallowed model
(files endpoint) and when it creates a batch for a disallowed model (batches
endpoint).
Per-endpoint output assertions
Each endpoint's full response is validated, not just the id. File upload asserts
object=="file", purpose=="batch", a positive bytes, a status, and a created-at.
Batch create / retrieve assert object=="batch", endpoint=="/v1/chat/completions",
completion_window=="24h", a non-empty input_file_id, and a created-at; retrieve
additionally cross-checks that id and input_file_id match the created batch.
Cancel asserts the same id, object=="batch", and a cancelling/cancelled status. List
asserts the object=="list" envelope and that the created batch is present as a batch.
File delete asserts object=="file" and deleted==True.
This suite's files
| File | Covers |
|---|---|
batch_client.py |
typed file upload/download + batch create/retrieve/cancel/list/delete over the shared Gateway; runtime batch model registration via /model/new; denial helpers |
capabilities.py |
the provider x scenario matrix + per-provider /model/new params + id-shape classifiers + per-provider raw-id assertion |
conftest.py |
session-scoped batch deployment registration and teardown |
test_batches_e2e.py |
parametrized lifecycle with per-endpoint output assertions, file upload/delete outputs, key-model-access denial |
Out of scope (intentionally)
Driving a batch to completed, cost tracking on completion, and the DB write-back
are not covered here; the 24h window makes them unfit for a synchronous gate. That
logic belongs in a DI-stubbed proxy integration test under tests/test_litellm/proxy/
where the provider client is injected to return completed deterministically.