7efce7f8e0
* test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|---|---|---|
| .. | ||
| _support | ||
| authorization | ||
| caching | ||
| compatibility | ||
| configuration | ||
| cost_calculation | ||
| database | ||
| management | ||
| mcp | ||
| messages_endpoint | ||
| observability | ||
| pricing | ||
| providers | ||
| routing | ||
| sandbox | ||
| sdk | ||
| security | ||
| spend | ||
| streaming | ||
| translation | ||
| __init__.py | ||
| AGENTS.md | ||
| conftest.py | ||
| coordination_redis_proxy_config.yaml | ||
| mcp_coverage.toml | ||
| oci_proxy_test_config.yaml | ||
| proxy_config.yaml | ||
| README.md | ||
| run.py | ||
| test_oci_integration.py | ||
| test_oci_proxy_integration.py | ||
Integration contracts
These tests exercise a running gateway, PostgreSQL and Redis with an owned local upstream. CircleCI owns this suite. Tests are grouped by behavior, with no automatic test retries or fallback to paid provider calls
The ROI database contracts in database/test_roi_observed.py run as plain pytest outside run_integration.sh in CircleCI's roi-database Postgres job. The job uploads coverage with the roi-postgres flag. These contracts own temporary databases and script only the external provider transport
The Redis cluster IAM contracts in sdk/test_redis_cluster_iam_auth.py start their own redis-server and mock the cloud credential clients, so they run in the GitHub Actions redis-compat matrix against every supported redis-py major rather than in run_integration.sh
The cost group is driven by the CostTrackingTestCase literals under cost_calculation/cases/<provider>.py (gathered by cost_calculation/catalog.py), with the scripted cost map in cost_calculation/cost_map.py. Each case has a name, contract ID, cost-map model, optional deployment overrides, request body, tagged response and exact or recount expectations. Request bodies use $MODEL for the registered proxy model, while responses use $REQUEST_ID for the per-run scenario ID. To add a case, add a cost-map entry when the model is new, add the request body and exact provider response data, and add hand-computed expected values. The upstream serves each stored response for any path under /<scenario_id>, while the test-owned cost map is served over loopback through LITELLM_MODEL_COST_MAP_URL
Use tests/integration/run.py management, accounting, database, providers, extensions, mcp, sdk or cost to run a selected group. The group to directory mapping is the GROUPS literal at the top of run.py; a new directory needs a GROUPS entry and an OWNED_DIRECTORIES entry in _support/manifest.py. Set INTEGRATION_WORKERS above 1 to run a group under pytest-xdist. Every group run by more than one worker gives each worker a private database, Redis, scripted upstream process, fake provider and proxy, with a peer and read-only database role where required. There is no opt-in list and no shared proxy, upstream, database or Redis in that mode: run_integration.sh does not start them and does not hand their addresses to the run, so a test can only reach its own worker's services. The pytest controller owns these services and closes them after worker failures. Their cases use pytest-xdist load groups inside the files selected by CircleCI timing splits. Module and class fixtures are grouped automatically, so authors do not need scheduling markers. Function fixtures remain independently schedulable. Tests that own an independent mutable sink for every case should use function fixtures, so a shared module fixture does not serialize them or share pending observations. Nested harness probes discard the parent run's shard, worker and result settings, then supply their own selections. MCP keeps its explicit load groups. Automatic group suffixes are removed from JUnit and execution records to preserve case identities. Owned proxy helpers wait for a healthy response from every configured, live worker before yielding. Serial runs keep the supplied deployment, including routing parity runs. Set INTEGRATION_PROXY_URL, INTEGRATION_UPSTREAM_URL, INTEGRATION_MASTER_KEY and DATABASE_URL to an isolated test deployment. When that deployment runs more than one proxy worker, set INTEGRATION_PROXY_WORKERS to the count so a test that writes a model and then calls it waits out the config reload interval, the only cross-worker convergence bound the wire exposes. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate
Management and MCP also require INTEGRATION_PEER_URL, REDIS_HOST and REDIS_PORT. Under several workers each worker gets a second directly addressed proxy sharing only that worker's stores, and a single-worker run uses the two proxies run_integration.sh starts. The test-only CLI wrapper supplies enterprise route entitlement, following the existing behavior suite's convention. It does not qualify license validation; run it with one worker and no reload
The generated lifecycle models use 20 examples, eight steps, generation and shrinking, with isolated resources per example. HTTP operation caps include generation and shrinking and exempt cleanup. Local qualification defaults to seed 4106601 and canonical order; CircleCI derives exploration and ordering seeds from the checked-out revision and workflow ID. The ordering seed shuffles files, scoped fixture configurations and cases within each configuration. Ordering keeps shared module and class configurations together where their dependencies permit it. Dependent fixtures with their own module parameters can still rebuild an outer configuration, with every case retained
CircleCI uses its built-in circleci tests split --split-by=timings --timings-type=filename to assign test files using the uploaded JUnit timings. Each file stays on one node, preserving its module and parametrized fixtures. Jobs default to one node. Parallelism should use the smallest node count that keeps the complete job, including setup and cleanup, within twenty minutes. A group that already finishes within that target on one node stays on one node. Management, accounting, providers and management replica currently use four nodes. Extensions uses eight nodes; database and MCP use two, and fast suites use one. No job is configured above eight nodes. A single slow file can still determine a suite's runtime; split such a file along independent fixture boundaries rather than adding more nodes
Every node collects the same candidate files before applying CircleCI's assigned file list. Its execution record retains the candidate inventory and the selected, passed and skipped cases. An empty assigned file list produces an empty shard, not a full-suite fallback. Every parallel group has a dependent census job that rejects an empty candidate inventory, missing shards, omitted cases and duplicate execution. Standard and replica runs keep separate workspace records
Future change-based selection belongs before timing splitting. Pass the selected group files or node IDs to run.py <group> --list to list their files, then give those filenames to CircleCI. Run the same candidate selection on every node with --shard-files <CircleCI-output-path> --shards <count> --shard-index <index>. The census compares execution with the candidate inventory, while full-suite runs remain the default. CircleCI determines scheduling from timing history; timing data never selects or omits candidates. Use --seed and --order-seed to reproduce test generation and order. The installed Hypothesis version, settings, seeds and collected order are recorded beside the execution manifest
A future selector must explicitly skip a group when no candidates apply; omitting positional candidates means the full group. Keep full runs on the default branch and scheduled workflows, and select the full affected groups when shared fixtures, CI configuration or an unknown dependency changes
Reuse the existing canned provider handlers through _support/upstream.py. It rejects internal request fields and exposes actual received requests for independent assertions. Register every created resource for cleanup immediately, keep expected values independent of production calculations, and assert readback plus the runtime effect of a change
Owned proxies disable .env loading by default. A contract testing dotenv behavior can explicitly set PYTHON_DOTENV_DISABLED=0 in its owned environment. An explicit scratch writer database drops an inherited reader with a different database name; pass both database URLs when the contract needs a paired reader. Database relays close listeners, active connections and their event loops on teardown. Upstream startup failures reap the child process before returning the failure. Custom subprocesses, direct SQL and background tasks still need an explicit owner in the test
Owned proxy workers configure a 32-thread executor before application startup, preserving additional worker startup hooks. Held synchronous provider calls must not consume a CPU-dependent executor limit before the rest of a fault burst can start. The proxy entrypoint imports LiteLLM inside main, so spawned workers begin supervision before importing the application. Worker healthchecks have a five-second deadline by default, so uvicorn kills a worker that stays paused longer than that. Tests never signal proxy processes themselves. OwnedProxy.kill_worker SIGKILLs a worker and returns once its replacement answers readiness with its own pid, replacing_worker does the same for a kill that has to land mid-burst, and suspended pauses a worker or the supervisor, which only works on a proxy launched with healthcheck=WorkerHealthcheck.LENIENT. Helper processes that are not the proxy stop through stop_child, stop_processes and stop_process_group and pause through paused. scripts/check_test_quality.py rule TQ010 rejects a raw os.kill, SIGKILL / SIGSTOP / SIGCONT, .kill(), .terminate(), .suspend() or .send_signal() anywhere under tests/integration outside _support, and TQ011 rejects an assertion that caps an elapsed time with a fixed number. Bound timings with _support/timing.py instead, relative to a measured control or to the injected delay. run.py measures how long this box takes to import the proxy application before it starts pytest, records it in boot-calibration.json beside the results and exports it as INTEGRATION_IMPORT_SECONDS. Readiness deadlines, graceful stops and the default per-test timeout are multiples of that measurement (_support/budget.py), so a slower box gets longer budgets and a two-worker proxy gets twice the readiness deadline of a one-worker one. Readiness requires responsive worker identities that are still alive and a positively identified owned listener. Unreadable socket records cannot establish ownership. Crash tests own their client tasks with TaskGroup, release held peers in finally and verify that the intended worker dies. A test requiring traffic on both workers prepares that traffic explicitly instead of assuming OS load balancing, for example by sending each half while the other worker is suspended
The CircleCI workflow starts its own database and Redis, restricts test-phase egress to its owned services and writes JUnit plus an executed-node manifest. Missing setup, failed cleanup or a selected test with neither a passed call nor a skip fail qualification. Skipped nodes are listed under skipped in execution.json, so the skip reasons double as the open bug list. GITHUB_FILES in run.py lists the files excluded from the run_integration.sh selection
There is no hand-maintained list of test cases. A positional argument is a file of the group or a pytest node id inside one (path::test[param]), so one cell of a parametrized file can run alone. The runner fails only when pytest fails, when collection errors, or when a selected file collects zero tests. Older tests still carry @pytest.mark.covers(...) decorators; the marker stays registered so they collect, but the IDs are not checked against anything and new tests should not use it. The GitHub Actions coverage census reads the GROUPS literal in run.py and treats every tests/integration/<directory>/test_*.py file in a scheduled group as owned by CircleCI
Provider sentinels currently use the controlled server, not live recordings. The provider shard also runs the existing strict replay controls for changed requests, exhausted interactions, leftover interactions and no provider connection. Future recorded scenarios must use that replay-only implementation; missing recordings cannot fall back to a real provider. The observation endpoint is destructive and the current selection runs serially against one owned upstream
Translation tests in translation/ compare exact provider requests and LiteLLM responses against a shared fake provider; translation/README.md has their rules
Fixtures must contain synthetic data only. Keep private incident records and source documents out of code, fixtures, logs and PR descriptions
Database cases own their temporary schemas, roles, constraints and proxy processes. They prove reader-versus-writer execution with PostgreSQL lock observations, exercise real transaction wait limits and verify rollback after a reached database failure
Accounting cases compare persisted input and output cost components against literal rates, including zero and default prices. Cache state models assert actual upstream calls, response identity and every persisted charge. Generated accounting tests have a 180-second test limit to accommodate the asynchronous spend writer
Provider contracts exercise actual TCP requests with synthetic credentials and local protocol peers. The S3 verifier uses independently implemented equations, a published known-answer vector, a fixed signing clock and deliberately invalid signed requests. Bedrock cases clear ambient AWS credential sources and check the literal model path, loaded role references, STS requests and bearer-only behavior
CircleCI runs providers on four isolated nodes using its file timing splitter. Each node owns its services. The dependent providers coverage job verifies all four execution records against the candidate case inventory and rejects missing, duplicate, failed or incomplete execution. The node's assigned filenames are saved in node-files.txt beside its execution record
Streaming checks send real HTTP transfer chunks, including one-byte partitions, fragmented tools, incomplete transfers and a cancellation barrier. They assert meaningful text, tool arguments, final usage and persisted cost. The Redis recovery case owns a separate database and Redis process, uses the supported one-second circuit-breaker recovery setting, waits for the real subscriber and verifies response data in Redis after restart. CircleCI reuses its existing Redis image for that extra process; it never pulls an image during tests
The messages_endpoint/ directory holds /v1/messages endpoint contracts: native-provider backends under providers/ (anthropic, bedrock, gemini) and the translation bridges (responses_bridge, chat_bridge) at the top level. It runs in the providers shard; run.py selects test files recursively under each scheduled directory
The sdk shard exercises the SDK's own HTTP clients against local protocol peers with no gateway in the path, so a case here fails only when the client library or its wire behavior changes. The HTTP/2 case runs a hypercorn TLS peer offering h2 and http/1.1 over ALPN, drives the sync and async httpx handlers at it with LITELLM_HTTP2 off and on, and asserts the version both the client and the peer observed on the wire. Put a test here only when it needs no proxy or database. CircleCI starts a local Redis for this shard like the others, so SDK-side caching cases that need a real Redis server belong here too; a case that reaches the gateway belongs in one of the other shards
The extensions shard uses the built-in generic callback and guardrail transports. It checks callback correlation and credential exclusion, guardrail rewriting and denial, retained OpenAI consumers and A2A wire versions. CircleCI runs it on eight xlarge nodes using its file timing splitter. Each node runs its assigned files with four pytest workers, each owning its database, Redis, upstream and proxy. The dependent coverage job verifies all eight execution records against the candidate case inventory. Tests must not assume a particular set of sibling files
The security shard uses four pytest workers on an xlarge machine, each with its own isolated services like every other multi-worker group. Route sweeps use a new HTTP client for every call and share only the TLS verification context, so authentication and response cookies cannot cross callers. The canary rig supplies owned OpenAI and Anthropic endpoints and synthetic credentials and rejects attempted external egress
The two mcp nodes own separate services and use the same case census as providers and extensions. Their coverage data is combined into one MCP report after both nodes finish. Each node runs the MCP gateway against SDK peers owned by each test (_support/mcp.py): streamable HTTP, SSE and stdio peers, an OpenAPI-spec app, and an OAuth 2.1 authorization-server double. Every peer records the requests it receives so a test can assert what reached the peer, not only what the proxy answered. The shard runs with INTEGRATION_WORKERS set and with INTEGRATION_COVERAGE=1, which starts every worker's proxy and peer under coverage run --parallel-mode limited to the MCP modules and stores coverage.txt plus an HTML report with the job artifacts. mcp_coverage.toml measures lines through sys.monitoring, because the CPython 3.12 trace function that branch measurement needs made a covered proxy take about four times as long to become ready. A test that fails because the product is wrong is skipped with pytest.skip("BUG: <symptom>") so the skip list in execution.json is the open MCP bug list
Browser contracts live in tests/e2e/ui/tests/integrationCritical and run only through tests/e2e/ui/integration.config.ts. The expected browser results are listed in expected.json in that directory and checked by .circleci/scripts/verify_integration_browser.py. The CircleCI browser shard builds the checked-out dashboard, starts the owned proxy with that build, and verifies one exact browser result without retries or skips. The default Playwright selection excludes this directory. The focused project flow asserts the submitted create and clear values, fresh SQL state and actual blocked/restored serving while preserving model restrictions
Two always-on -replica CircleCI jobs (management, database) run their groups in replica mode, where every proxy connects through a real litellm_writer role and a real read-only litellm_reader role against the same PostgreSQL. Nothing is captured there: the job passes when the tests pass, and a write routed to the read-only reader fails the test that issued it. A deeper check runs on demand as the routing_parity workflow, triggered through the CircleCI API v2 pipeline endpoint on the PR branch with {"parameters": {"routing_parity_base": "<40-hex merge-base sha>"}}. The workflow fans out over the seven groups, and each routing-parity-<group> job runs its own group twice against the same test harness, once with litellm/, enterprise/, and litellm-proxy-extras/ checked out from the base revision and once from the head, with a pytest plugin snapshotting pg_stat_statements into routing-observed.json per side. The check step then compares the two observations and writes routing-diff.txt: a statement seen on both sides fails when its role set changed, globally or for the same test (per-test capture is skipped under xdist), unless it is listed in tests/integration/routing/either_role.json, where each entry names the statement and a one-line reason it legitimately runs on whichever role asks for it, printed under == either role ==. Queries seen on only one side are listed, never failed, pg_stat_statements evictions and a role that never ran a statement are failures