mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
08fa25042c
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is not a gateway server, it is the client every suite uses to talk to the proxy (keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the module to proxy_client.py and the class to ProxyClient, with build_gateway becoming build_proxy_client and the GatewayProvider protocol becoming ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only identifiers changed; prose and string literals that use the word gateway for the proxy-server concept were left alone. Each suite previously built its own instance through a per-suite build_client() that called build_gateway() inside, duplicating the proxy wiring across suites. There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every suite's client fixture depends on it and injects it, so the wiring lives in one place. claude_code keeps building its own client directly since it has its own harness and does not use the shared fixtures. Behavior is unchanged: shared transport, data-plane/control-plane split routing, poll budget, typed request/response models, and resource cleanup all go through the same object.
81 lines
4.8 KiB
Markdown
81 lines
4.8 KiB
Markdown
# Batches Test Coverage Matrix
|
|
|
|
Live e2e coverage of the Batches API over a real proxy, real provider keys, and
|
|
real cost. Synchronous tier only: a batch's completion window is 24h, so these
|
|
tests never wait for `completed`. They assert the proxy accepts, routes, retrieves,
|
|
cancels, and lists a batch; everything created is deleted on teardown.
|
|
|
|
## Provider x operation
|
|
|
|
Only supported cells are tested. The capability table in `capabilities.py` holds one
|
|
row per supported (provider, scenario) pair, so there are no skipped cells in the
|
|
parametrized run. The batches suite never skips: missing provider creds or upstream
|
|
failures are hard test failures (see `tests/e2e/CLAUDE.md`).
|
|
|
|
| Provider | create | retrieve | cancel | list | file backing |
|
|
|-----------|--------|----------|--------|------|--------------|
|
|
| OpenAI | yes | yes | yes | yes | OpenAI Files |
|
|
| Azure | yes | yes | yes | yes | Azure Files |
|
|
| Vertex AI | yes | yes | yes | yes | GCS (`gcs_bucket_name` / `GCS_BUCKET_NAME` on model) |
|
|
| Bedrock | yes (unified only) | yes | no (limited upstream) | no | S3 (`s3_bucket_name` + `aws_*` + `AWS_BATCH_ROLE_ARN` on model) |
|
|
|
|
Bedrock cancel is unreliable upstream and list is unsupported, so both are gated off
|
|
(`can_cancel=False`, `can_list=False`) when that provider is enabled in the matrix.
|
|
Bedrock file upload requires a model on the request (`encoded` / `unified` scenarios only);
|
|
`model_param` and `provider_fallback` are omitted because `POST /bedrock/v1/files` has no
|
|
model-less passthrough path.
|
|
|
|
## Routing scenarios (per `litellm/proxy/batches_endpoints/endpoints.py`)
|
|
|
|
Each create-capable provider runs all four. The test asserts the returned file id
|
|
and batch id carry the shape that scenario must produce (`matches_id_shape`):
|
|
|
|
| Scenario | How the batch is routed | File id | Batch id |
|
|
|----------|-------------------------|---------|----------|
|
|
| `encoded` | upload with `?model=` -> model-encoded file id -> create with just that id | model-encoded | model-encoded |
|
|
| `unified` | upload with `target_model_names=` -> unified managed file id -> create with that id | managed | managed |
|
|
| `model_param` | raw file (provider-fallback upload) -> create with `model` in the body | raw | model-encoded |
|
|
| `provider_fallback` | raw file -> `POST /{provider}/v1/batches`, env creds, no model | raw | raw (native provider shape) |
|
|
|
|
"managed" ids base64-decode to a `litellm_proxy` marker; "model-encoded" ids keep the
|
|
provider prefix and base64-encode `litellm:<id>;model,<model>`; "raw" ids are the
|
|
provider's native ids. Asserting these catches a proxy that returns a raw id where it
|
|
should manage it, or vice versa. On top of the id shape, a misroute to the wrong
|
|
provider also fails create (the file id / model do not belong there), and the
|
|
`provider_fallback` raw batch id is additionally checked against the provider's native
|
|
shape (`raw_id_matches_provider`).
|
|
|
|
## Key model restriction
|
|
|
|
`test_batch_key_model_access_denied` mints a key restricted to one model
|
|
(`resources.key(models=[...])`) and proves the proxy returns 403
|
|
`key_model_access_denied` both when that key uploads a file for a disallowed model
|
|
(files endpoint) and when it creates a batch for a disallowed model (batches
|
|
endpoint).
|
|
|
|
## Per-endpoint output assertions
|
|
|
|
Each endpoint's full response is validated, not just the id. File upload asserts
|
|
`object=="file"`, `purpose=="batch"`, a positive `bytes`, a status, and a created-at.
|
|
Batch create / retrieve assert `object=="batch"`, `endpoint=="/v1/chat/completions"`,
|
|
`completion_window=="24h"`, a non-empty `input_file_id`, and a created-at; retrieve
|
|
additionally cross-checks that `id` and `input_file_id` match the created batch.
|
|
Cancel asserts the same id, `object=="batch"`, and a cancelling/cancelled status. List
|
|
asserts the `object=="list"` envelope and that the created batch is present as a batch.
|
|
File delete asserts `object=="file"` and `deleted==True`.
|
|
|
|
## This suite's files
|
|
|
|
| File | Covers |
|
|
|------|--------|
|
|
| `batch_client.py` | typed file upload/download + batch create/retrieve/cancel/list/delete over the shared ProxyClient; runtime batch model registration via /model/new; denial helpers |
|
|
| `capabilities.py` | the provider x scenario matrix + per-provider /model/new params + id-shape classifiers + per-provider raw-id assertion |
|
|
| `conftest.py` | session-scoped batch deployment registration and teardown |
|
|
| `test_batches_e2e.py` | parametrized lifecycle with per-endpoint output assertions, file upload/delete outputs, key-model-access denial |
|
|
|
|
## Out of scope (intentionally)
|
|
|
|
Driving a batch to `completed`, cost tracking on completion, and the DB write-back
|
|
are not covered here; the 24h window makes them unfit for a synchronous gate. That
|
|
logic belongs in a DI-stubbed proxy integration test under `tests/test_litellm/proxy/`
|
|
where the provider client is injected to return `completed` deterministically.
|