62b8ee78de
* test: add a manually dispatched integration trial on larger runners Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one * test: run the integration trial on push to this branch * test: give integration trial workers a local redis-server * test: run the providers trial on a 64-core runner with 32 workers The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4 * test: run every integration group in the trial on sized runners * test: add a scan that refuses to upload integration results holding the license * test: drop the member-name pass the archive byte scan already covers * test: use a secret whose base64 and base64url encodings differ * ci: run integration groups on run-ci pull requests and scheduled main runs * ci: scope the integration license to a protected environment
3.2 KiB
tests/integration
Real proxy, Postgres, Redis, scripted upstream. README.md has shards and CI wiring
What good looks like
The root example is from here. The spend row lands async: poll, never sleep
rows = eventually(
lambda: read_rows('SELECT spend FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (body["id"],)),
lambda values: len(values) == 1,
seconds=70,
)
assert float(rows[0]["spend"]) == pytest.approx(10 * 0.001 + 5 * 0.0001 + 7 * 0.002 + 4 * 0.002)
sleep(3) fails on a slow runner and taxes every fast one. Assert the outbound body in the upstream
handler. A leaked field is invisible from the response. monkeypatch.setenv is fine. Patching our own
function in a full stack is not
Where it goes
By the domain a user would name: pricing, spend, routing, mcp. A file only needs to live in a
directory that a GROUPS entry in run.py selects. There is no manifest and no covers marker on new
tests. A product bug the test exposes is pytest.skip("BUG: <symptom>") at the top of the body, not a
fix in the test and not a deletion. Needs no network, proxy, DB or Redis: tests/unit
CircleCI's integration groups run every file here except the OCI and sandbox tests, which no CI runs, and
the files in GITHUB_FILES in run.py. test-integration.yml runs the same groups through the same
run_integration.sh on GitHub Actions, on main every three hours from 8am to 8pm Pacific Monday to Saturday,
and on a pull request when someone adds the run-ci label. Neither runs every pull request, so a new file here
first runs on main after its PR merges, or earlier if the PR gets run-ci. Wanting a file to run before merge
is no reason to change CI, and a new
dedicated workflow for it is the same mistake as a shared shard. Write the file, leave GITHUB_FILES and
every workflow alone, and tell the user it first runs on main. GITHUB_FILES is only for a file that
needs what CircleCI's groups cannot give it, like the images image-scan.yml builds or the plain pytest
run CircleCI's roi-database job gives test_roi_observed.py. Add a file there only in that case,
together with its dedicated job, and list a new dedicated GitHub workflow with the directory it owns in
DEDICATED_INTEGRATION_WORKFLOWS in .github/scripts/assert_ci_coverage.py. The ci-coverage check
fails a file here that a GitHub workflow runs unless it is in GITHUB_FILES and that workflow owns its
directory (#46780)
Fixtures that boot a proxy
Scope them to the module or class, or pin a session one to a single worker with an xdist_group mark
as _llm_gateway in mcp/test_mcp_llm_endpoints.py does, and request them as arguments, never through
request.getfixturevalue. The harness puts tests that request the same set of module or class fixtures
on one xdist worker, so the proxy they share boots there once and stops when that worker moves to another
file. A test that requests one more of those fixtures than its neighbors gets its own group and can boot
the proxy again on another worker, so keep that set the same across the tests that share a proxy. An
unpinned session fixture boots a copy on every worker that runs one of its tests and keeps them all until
the node ends, and those copies ran CircleCI integration-extensions nodes out of memory