08fa25042c
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is not a gateway server, it is the client every suite uses to talk to the proxy (keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the module to proxy_client.py and the class to ProxyClient, with build_gateway becoming build_proxy_client and the GatewayProvider protocol becoming ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only identifiers changed; prose and string literals that use the word gateway for the proxy-server concept were left alone. Each suite previously built its own instance through a per-suite build_client() that called build_gateway() inside, duplicating the proxy wiring across suites. There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every suite's client fixture depends on it and injects it, so the wiring lives in one place. claude_code keeps building its own client directly since it has its own harness and does not use the shared fixtures. Behavior is unchanged: shared transport, data-plane/control-plane split routing, poll budget, typed request/response models, and resource cleanup all go through the same object.
4.8 KiB
Batches Test Coverage Matrix
Live e2e coverage of the Batches API over a real proxy, real provider keys, and
real cost. Synchronous tier only: a batch's completion window is 24h, so these
tests never wait for completed. They assert the proxy accepts, routes, retrieves,
cancels, and lists a batch; everything created is deleted on teardown.
Provider x operation
Only supported cells are tested. The capability table in capabilities.py holds one
row per supported (provider, scenario) pair, so there are no skipped cells in the
parametrized run. The batches suite never skips: missing provider creds or upstream
failures are hard test failures (see tests/e2e/CLAUDE.md).
| Provider | create | retrieve | cancel | list | file backing |
|---|---|---|---|---|---|
| OpenAI | yes | yes | yes | yes | OpenAI Files |
| Azure | yes | yes | yes | yes | Azure Files |
| Vertex AI | yes | yes | yes | yes | GCS (gcs_bucket_name / GCS_BUCKET_NAME on model) |
| Bedrock | yes (unified only) | yes | no (limited upstream) | no | S3 (s3_bucket_name + aws_* + AWS_BATCH_ROLE_ARN on model) |
Bedrock cancel is unreliable upstream and list is unsupported, so both are gated off
(can_cancel=False, can_list=False) when that provider is enabled in the matrix.
Bedrock file upload requires a model on the request (encoded / unified scenarios only);
model_param and provider_fallback are omitted because POST /bedrock/v1/files has no
model-less passthrough path.
Routing scenarios (per litellm/proxy/batches_endpoints/endpoints.py)
Each create-capable provider runs all four. The test asserts the returned file id
and batch id carry the shape that scenario must produce (matches_id_shape):
| Scenario | How the batch is routed | File id | Batch id |
|---|---|---|---|
encoded |
upload with ?model= -> model-encoded file id -> create with just that id |
model-encoded | model-encoded |
unified |
upload with target_model_names= -> unified managed file id -> create with that id |
managed | managed |
model_param |
raw file (provider-fallback upload) -> create with model in the body |
raw | model-encoded |
provider_fallback |
raw file -> POST /{provider}/v1/batches, env creds, no model |
raw | raw (native provider shape) |
"managed" ids base64-decode to a litellm_proxy marker; "model-encoded" ids keep the
provider prefix and base64-encode litellm:<id>;model,<model>; "raw" ids are the
provider's native ids. Asserting these catches a proxy that returns a raw id where it
should manage it, or vice versa. On top of the id shape, a misroute to the wrong
provider also fails create (the file id / model do not belong there), and the
provider_fallback raw batch id is additionally checked against the provider's native
shape (raw_id_matches_provider).
Key model restriction
test_batch_key_model_access_denied mints a key restricted to one model
(resources.key(models=[...])) and proves the proxy returns 403
key_model_access_denied both when that key uploads a file for a disallowed model
(files endpoint) and when it creates a batch for a disallowed model (batches
endpoint).
Per-endpoint output assertions
Each endpoint's full response is validated, not just the id. File upload asserts
object=="file", purpose=="batch", a positive bytes, a status, and a created-at.
Batch create / retrieve assert object=="batch", endpoint=="/v1/chat/completions",
completion_window=="24h", a non-empty input_file_id, and a created-at; retrieve
additionally cross-checks that id and input_file_id match the created batch.
Cancel asserts the same id, object=="batch", and a cancelling/cancelled status. List
asserts the object=="list" envelope and that the created batch is present as a batch.
File delete asserts object=="file" and deleted==True.
This suite's files
| File | Covers |
|---|---|
batch_client.py |
typed file upload/download + batch create/retrieve/cancel/list/delete over the shared ProxyClient; runtime batch model registration via /model/new; denial helpers |
capabilities.py |
the provider x scenario matrix + per-provider /model/new params + id-shape classifiers + per-provider raw-id assertion |
conftest.py |
session-scoped batch deployment registration and teardown |
test_batches_e2e.py |
parametrized lifecycle with per-endpoint output assertions, file upload/delete outputs, key-model-access denial |
Out of scope (intentionally)
Driving a batch to completed, cost tracking on completion, and the DB write-back
are not covered here; the 24h window makes them unfit for a synchronous gate. That
logic belongs in a DI-stubbed proxy integration test under tests/test_litellm/proxy/
where the provider client is injected to return completed deterministically.