litellm/tests/integration
2026-09-14 09:22:13 -07:00
..
_support test: decouple integration contracts from E2E coverage registry 2026-09-14 09:22:13 -07:00
management test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
pricing test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
providers test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
sandbox feat(sandbox): e2b code execution primitive (#30898) 2026-06-20 16:30:01 -07:00
__init__.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
conftest.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
contracts.json test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
oci_proxy_test_config.yaml CI: copy of #25177 (OCI GenAI: embeddings, streaming/reasoning fixes, model catalog) (#28223) 2026-05-23 12:15:41 -07:00
proxy_config.yaml test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
README.md test: decouple integration contracts from E2E coverage registry 2026-09-14 09:22:13 -07:00
run.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
test_oci_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_oci_proxy_integration.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00

Integration contracts

These tests exercise a running gateway, PostgreSQL and Redis with an owned local upstream. CircleCI owns this suite. Tests are grouped by behavior, with no automatic test retries or fallback to paid provider calls

Use tests/integration/run.py management, accounting or providers to run a selected group. Set INTEGRATION_PROXY_URL, INTEGRATION_UPSTREAM_URL, INTEGRATION_MASTER_KEY and DATABASE_URL to an isolated test deployment. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate

Reuse the existing canned provider handlers through _support/upstream.py. It rejects internal request fields and exposes actual received requests for independent assertions. Register every created resource for cleanup immediately, keep expected values independent of production calculations, and assert readback plus the runtime effect of a change

The CircleCI workflow starts its own database and Redis, restricts test-phase egress to its owned services and writes JUnit plus an executed-node manifest. Missing setup, skipped tests, failed cleanup or a selected test without a passed call fail qualification. Existing GitHub Actions jobs do not own these tests

Define integration contract IDs and their canonical test nodes in contracts.json. Every node must declare the same IDs with covers. The runner checks exact collected and passed selections against that mapping. These IDs belong to this CircleCI suite and must not be added to the separate E2E coverage registry. A manifest declaration alone does not mean a test passed

Provider sentinels currently use the controlled server, not live recordings. The provider shard also runs the existing strict replay controls for changed requests, exhausted interactions, leftover interactions and no provider connection. Future recorded scenarios must use that replay-only implementation; missing recordings cannot fall back to a real provider. The observation endpoint is destructive and the current selection runs serially against one owned upstream

Fixtures must contain synthetic data only. Keep private incident records and source documents out of code, fixtures, logs and PR descriptions