3bd1854175
* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
|
||
|---|---|---|
| .. | ||
| _support | ||
| authorization | ||
| compatibility | ||
| configuration | ||
| cost_calculation | ||
| database | ||
| management | ||
| mcp | ||
| observability | ||
| pricing | ||
| providers | ||
| routing | ||
| sandbox | ||
| sdk | ||
| spend | ||
| streaming | ||
| translation/decisions | ||
| __init__.py | ||
| AGENTS.md | ||
| conftest.py | ||
| coordination_redis_proxy_config.yaml | ||
| mcp_coverage.toml | ||
| oci_proxy_test_config.yaml | ||
| proxy_config.yaml | ||
| README.md | ||
| run.py | ||
| test_oci_integration.py | ||
| test_oci_proxy_integration.py | ||
Integration contracts
These tests exercise a running gateway, PostgreSQL and Redis with an owned local upstream. CircleCI owns this suite. Tests are grouped by behavior, with no automatic test retries or fallback to paid provider calls
The cost group is driven by cost_tracking_cases.json, which contains the cost map, literal requests, literal provider responses and expected accounting values. Each case has a name, contract ID, cost-map model, optional deployment overrides, request body, tagged response and exact or recount expectations. Request bodies use $MODEL for the registered proxy model, while responses use $REQUEST_ID for the per-run scenario ID. To add a case, add a cost-map entry when the model is new, add the request body and exact provider response data, and add hand-computed expected values. The upstream serves each stored response for any path under /<scenario_id>, while the test-owned cost map is served over loopback through LITELLM_MODEL_COST_MAP_URL
Use tests/integration/run.py management, accounting, database, providers, extensions, mcp, sdk or cost to run a selected group. The group to directory mapping is the GROUPS literal at the top of run.py; a new directory needs a GROUPS entry and an OWNED_DIRECTORIES entry in _support/manifest.py. Set INTEGRATION_WORKERS above 1 to run a group under pytest-xdist; the mcp job does this in CI, so MCP tests must own their resources per scenario. Set INTEGRATION_PROXY_URL, INTEGRATION_UPSTREAM_URL, INTEGRATION_MASTER_KEY and DATABASE_URL to an isolated test deployment. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate
Management also requires INTEGRATION_PEER_URL, REDIS_HOST and REDIS_PORT. CircleCI starts two directly addressed proxy processes sharing only that job's stores. The test-only CLI wrapper supplies enterprise route entitlement, following the existing behavior suite's convention. It does not qualify license validation; run it with one worker and no reload
The generated lifecycle models use 20 examples, eight steps, generation and shrinking, with isolated resources per example. HTTP operation caps include generation and shrinking and exempt cleanup. Local qualification defaults to seed 4106601 and canonical order; CircleCI derives exploration and ordering seeds from the checked-out revision and workflow ID. The ordering seed shuffles the file order and the test order inside each file but keeps each file's tests together, so module fixtures are built once per file. Use --seed and --order-seed to reproduce a run. Actual installed Hypothesis version, settings, seeds and collected order are written beside the execution manifest
Reuse the existing canned provider handlers through _support/upstream.py. It rejects internal request fields and exposes actual received requests for independent assertions. Register every created resource for cleanup immediately, keep expected values independent of production calculations, and assert readback plus the runtime effect of a change
The CircleCI workflow starts its own database and Redis, restricts test-phase egress to its owned services and writes JUnit plus an executed-node manifest. Missing setup, failed cleanup or a selected test with neither a passed call nor a skip fail qualification. Skipped nodes are listed under skipped in execution.json, so the skip reasons double as the open bug list. Existing GitHub Actions jobs do not own these tests
There is no per-node manifest. The runner fails only when pytest fails, when collection errors, or when a selected file collects zero tests. Older tests still carry @pytest.mark.covers(...) decorators; the marker stays registered so they collect, but the IDs are not checked against anything and new tests should not use it. The GitHub Actions coverage census reads the GROUPS literal in run.py and treats every tests/integration/<directory>/test_*.py file in a scheduled group as owned by CircleCI
Provider sentinels currently use the controlled server, not live recordings. The provider shard also runs the existing strict replay controls for changed requests, exhausted interactions, leftover interactions and no provider connection. Future recorded scenarios must use that replay-only implementation; missing recordings cannot fall back to a real provider. The observation endpoint is destructive and the current selection runs serially against one owned upstream
Fixtures must contain synthetic data only. Keep private incident records and source documents out of code, fixtures, logs and PR descriptions
Database cases own their temporary schemas, roles, constraints and proxy processes. They prove reader-versus-writer execution with PostgreSQL lock observations, exercise real transaction wait limits and verify rollback after a reached database failure
Accounting cases compare persisted input and output cost components against literal rates, including zero and default prices. Cache state models assert actual upstream calls, response identity and every persisted charge. Generated accounting tests have a 180-second test limit to accommodate the asynchronous spend writer
Provider contracts exercise actual TCP requests with synthetic credentials and local protocol peers. The S3 verifier uses independently implemented equations, a published known-answer vector, a fixed signing clock and deliberately invalid signed requests. Bedrock cases clear ambient AWS credential sources and check the literal model path, loaded role references, STS requests and bearer-only behavior
Streaming checks send real HTTP transfer chunks, including one-byte partitions, fragmented tools, incomplete transfers and a cancellation barrier. They assert meaningful text, tool arguments, final usage and persisted cost. The Redis recovery case owns a separate database and Redis process, uses the supported one-second circuit-breaker recovery setting, waits for the real subscriber and verifies response data in Redis after restart. CircleCI reuses its existing Redis image for that extra process; it never pulls an image during tests
The sdk shard exercises the SDK's own HTTP clients against local protocol peers with no gateway in the path, so a case here fails only when the client library or its wire behavior changes. The HTTP/2 case runs a hypercorn TLS peer offering h2 and http/1.1 over ALPN, drives the sync and async httpx handlers at it with LITELLM_HTTP2 off and on, and asserts the version both the client and the peer observed on the wire. Put a test here only when it needs no proxy, database or Redis; a case that reaches the gateway belongs in one of the other shards
The extensions shard uses the built-in generic callback and guardrail transports. It checks callback correlation and credential exclusion, guardrail rewriting and denial, retained OpenAI consumers and A2A wire versions. CircleCI runs it on parallel nodes, and each node starts its own database, Redis, upstream and proxy and runs its share of the group's files serially, split by recorded timings with circleci tests split. Tests keep the isolation of a serial run; they still must not assume a particular set of sibling files. run.py <group> --list prints a group's files and run.py <group> <file>... runs a subset of them
The mcp shard runs the MCP gateway against SDK peers owned by each test (_support/mcp.py): streamable HTTP, SSE and stdio peers, an OpenAPI-spec app, and an OAuth 2.1 authorization-server double. Every peer records the requests it receives so a test can assert what reached the peer, not only what the proxy answered. The shard runs with INTEGRATION_WORKERS set and with INTEGRATION_COVERAGE=1, which starts the proxy under coverage run --parallel-mode limited to the MCP modules and stores coverage.txt plus an HTML report with the job artifacts. A test that fails because the product is wrong is skipped with pytest.skip("BUG: <symptom>") so the skip list in execution.json is the open MCP bug list
Browser contracts live in tests/e2e/ui/tests/integrationCritical and run only through tests/e2e/ui/integration.config.ts. The expected browser results are listed in expected.json in that directory and checked by .circleci/scripts/verify_integration_browser.py. The CircleCI browser shard builds the checked-out dashboard, starts the owned proxy with that build, and verifies one exact browser result without retries or skips. The default Playwright selection excludes this directory. The focused project flow asserts the submitted create and clear values, fresh SQL state and actual blocked/restored serving while preserving model restrictions
Two always-on -replica CircleCI jobs (management, database) run their groups in replica mode, where every proxy connects through a real litellm_writer role and a real read-only litellm_reader role against the same PostgreSQL. Nothing is captured there: the job passes when the tests pass, and a write routed to the read-only reader fails the test that issued it. A deeper check runs on demand as the routing_parity workflow, triggered through the CircleCI API v2 pipeline endpoint on the PR branch with {"parameters": {"routing_parity_base": "<40-hex merge-base sha>"}}. The workflow fans out over the seven groups, and each routing-parity-<group> job runs its own group twice against the same test harness, once with litellm/, enterprise/, and litellm-proxy-extras/ checked out from the base revision and once from the head, with a pytest plugin snapshotting pg_stat_statements into routing-observed.json per side. The check step then compares the two observations and writes routing-diff.txt: a statement seen on both sides fails when its role set changed, globally or for the same test (per-test capture is skipped under xdist), unless it is listed in tests/integration/routing/either_role.json, where each entry names the statement and a one-line reason it legitimately runs on whichever role asks for it, printed under == either role ==. Queries seen on only one side are listed, never failed, pg_stat_statements evictions and a role that never ran a statement are failures