3.6 KiB
tests/integration
Real proxy, Postgres, Redis, scripted upstream. README.md has shards and CI wiring
What good looks like
The root example is from here. The spend row lands async: poll, never sleep
rows = eventually(
lambda: read_rows('SELECT spend FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (body["id"],)),
lambda values: len(values) == 1,
seconds=70,
)
assert float(rows[0]["spend"]) == pytest.approx(10 * 0.001 + 5 * 0.0001 + 7 * 0.002 + 4 * 0.002)
sleep(3) fails on a slow runner and taxes every fast one. Assert the outbound body in the upstream
handler. A leaked field is invisible from the response. monkeypatch.setenv is fine. Patching our own
function in a full stack is not
Where it goes
By the domain a user would name: pricing, spend, routing, mcp. A file only needs to live in a
directory that a GROUPS entry in run.py selects. There is no manifest and no covers marker on new
tests. A product bug the test exposes is pytest.skip("BUG: <symptom>") at the top of the body, not a
fix in the test and not a deletion. Needs no network, proxy, DB or Redis: tests/unit
CircleCI's integration groups run every file here except the OCI tests, which no CI runs, and
the files in GITHUB_FILES in run.py. test-integration.yml runs the same groups through the same
run_integration.sh on GitHub Actions, on main every three hours from 8am to 8pm Pacific Monday to Saturday,
and on a pull request when someone adds the run-ci label. Neither runs every pull request, so a new file here
first runs on main after its PR merges, or earlier if the PR gets run-ci. Wanting a file to run before merge
is no reason to change CI, and a new
dedicated workflow for it is the same mistake as a shared shard. Write the file, leave GITHUB_FILES and
every workflow alone, and tell the user it first runs on main. GITHUB_FILES is only for a file that
needs what CircleCI's groups cannot give it, like the images image-scan.yml builds or the plain pytest
run CircleCI's roi-database job gives test_roi_observed.py. Add a file there only in that case,
together with its dedicated job, and list a new dedicated GitHub workflow with the directory it owns in
DEDICATED_INTEGRATION_WORKFLOWS in .github/scripts/assert_ci_coverage.py. The ci-coverage check
fails a file here that a GitHub workflow runs unless it is in GITHUB_FILES and that workflow owns its
directory (#46780)
The Monty sandbox tests are an explicit exception. test-integration.yml owns sandbox/ and runs
uv run --no-sync pytest tests/integration/sandbox/test_monty_runtime.py -q in its focused Monty job
on pull requests, scheduled runs and manual all or monty runs. It installs the proxy extra and
needs no gateway, database, Redis or credentials. Keep subprocess and timing tests here, not in unit tests
Fixtures that boot a proxy
Scope them to the module or class, or pin a session one to a single worker with an xdist_group mark
as _llm_gateway in mcp/test_mcp_llm_endpoints.py does, and request them as arguments, never through
request.getfixturevalue. The harness puts tests that request the same set of module or class fixtures
on one xdist worker, so the proxy they share boots there once and stops when that worker moves to another
file. A test that requests one more of those fixtures than its neighbors gets its own group and can boot
the proxy again on another worker, so keep that set the same across the tests that share a proxy. An
unpinned session fixture boots a copy on every worker that runs one of its tests and keeps them all until
the node ends, and those copies ran CircleCI integration-extensions nodes out of memory