litellm/tests/integration
2026-09-14 10:58:36 -07:00
..
_support test: isolate management integration coverage from E2E registry 2026-09-14 09:23:00 -07:00
authorization test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
configuration test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
management test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
pricing test: collect pricing observations immutably 2026-09-14 10:58:24 -07:00
providers test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
sandbox feat(sandbox): e2b code execution primitive (#30898) 2026-06-20 16:30:01 -07:00
__init__.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
conftest.py test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
contracts.json test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
oci_proxy_test_config.yaml CI: copy of #25177 (OCI GenAI: embeddings, streaming/reasoning fixes, model catalog) (#28223) 2026-05-23 12:15:41 -07:00
proxy_config.yaml test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
README.md test: isolate management integration coverage from E2E registry 2026-09-14 09:23:00 -07:00
run.py test: cover persisted updates and warmed authorization policies 2026-09-14 04:20:06 -07:00
test_oci_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_oci_proxy_integration.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00

Integration contracts

These tests exercise a running gateway, PostgreSQL and Redis with an owned local upstream. CircleCI owns this suite. Tests are grouped by behavior, with no automatic test retries or fallback to paid provider calls

Use tests/integration/run.py management, accounting or providers to run a selected group. Set INTEGRATION_PROXY_URL, INTEGRATION_UPSTREAM_URL, INTEGRATION_MASTER_KEY and DATABASE_URL to an isolated test deployment. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate

Management also requires INTEGRATION_PEER_URL, REDIS_HOST and REDIS_PORT. CircleCI starts two directly addressed proxy processes sharing only that job's stores. The test-only CLI wrapper supplies enterprise route entitlement, following the existing behavior suite's convention. It does not qualify license validation; run it with one worker and no reload

The generated lifecycle models use 20 examples, eight steps, generation and shrinking, with isolated resources per example. HTTP operation caps include generation and shrinking and exempt cleanup. Local qualification defaults to seed 4106601; CircleCI derives its exploration seed from the checked-out revision. Use --seed to reproduce a run. Actual installed Hypothesis version, settings and seed are written beside the execution manifest

Reuse the existing canned provider handlers through _support/upstream.py. It rejects internal request fields and exposes actual received requests for independent assertions. Register every created resource for cleanup immediately, keep expected values independent of production calculations, and assert readback plus the runtime effect of a change

The CircleCI workflow starts its own database and Redis, restricts test-phase egress to its owned services and writes JUnit plus an executed-node manifest. Missing setup, skipped tests, failed cleanup or a selected test without a passed call fail qualification. Existing GitHub Actions jobs do not own these tests

Define integration contract IDs and their canonical test nodes in contracts.json. Every node must declare the same IDs with covers. The runner checks exact collected and passed selections against that mapping. These IDs belong to this CircleCI suite and must not be added to the separate E2E coverage registry. A manifest declaration alone does not mean a test passed

Provider sentinels currently use the controlled server, not live recordings. The provider shard also runs the existing strict replay controls for changed requests, exhausted interactions, leftover interactions and no provider connection. Future recorded scenarios must use that replay-only implementation; missing recordings cannot fall back to a real provider. The observation endpoint is destructive and the current selection runs serially against one owned upstream

Fixtures must contain synthetic data only. Keep private incident records and source documents out of code, fixtures, logs and PR descriptions