42d2909210
* test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module Session scope booted one 2-worker proxy per xdist worker and kept it until the worker's session ended, so a CircleCI node holding two of these files piled up proxies until the VM stopped responding. Module scope lets the harness group each file's rig tests onto one worker and tears the proxies down after them. * docs(tests): keep proxy fixtures module scoped in integration tests * docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
3 KiB
tests/integration
Real proxy, Postgres, Redis, scripted upstream. README.md has shards and CI wiring
What good looks like
The root example is from here. The spend row lands async: poll, never sleep
rows = eventually(
lambda: read_rows('SELECT spend FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (body["id"],)),
lambda values: len(values) == 1,
seconds=70,
)
assert float(rows[0]["spend"]) == pytest.approx(10 * 0.001 + 5 * 0.0001 + 7 * 0.002 + 4 * 0.002)
sleep(3) fails on a slow runner and taxes every fast one. Assert the outbound body in the upstream
handler. A leaked field is invisible from the response. monkeypatch.setenv is fine. Patching our own
function in a full stack is not
Where it goes
By the domain a user would name: pricing, spend, routing, mcp. A file only needs to live in a
directory that a GROUPS entry in run.py selects. There is no manifest and no covers marker on new
tests. A product bug the test exposes is pytest.skip("BUG: <symptom>") at the top of the body, not a
fix in the test and not a deletion. Needs no network, proxy, DB or Redis: tests/unit
CircleCI's integration groups run every file here except the OCI and sandbox tests, which no CI runs, and
the files in GITHUB_FILES in run.py. CircleCI runs on main, not on pull requests, so a new file here
first runs after its PR merges. Wanting a file to run before merge is no reason to change CI, and a new
dedicated workflow for it is the same mistake as a shared shard. Write the file, leave GITHUB_FILES and
every workflow alone, and tell the user it first runs on main. GITHUB_FILES is only for a file that
needs what CircleCI's groups cannot give it, like the images image-scan.yml builds or the plain pytest
run CircleCI's roi-database job gives test_roi_observed.py. Add a file there only in that case,
together with its dedicated job, and list a new dedicated GitHub workflow with the directory it owns in
DEDICATED_INTEGRATION_WORKFLOWS in .github/scripts/assert_ci_coverage.py. The ci-coverage check
fails a file here that a GitHub workflow runs unless it is in GITHUB_FILES and that workflow owns its
directory (#46780)
Fixtures that boot a proxy
Scope them to the module or class, or pin a session one to a single worker with an xdist_group mark
as _llm_gateway in mcp/test_mcp_llm_endpoints.py does, and request them as arguments, never through
request.getfixturevalue. The harness puts tests that request the same set of module or class fixtures
on one xdist worker, so the proxy they share boots there once and stops when that worker moves to another
file. A test that requests one more of those fixtures than its neighbors gets its own group and can boot
the proxy again on another worker, so keep that set the same across the tests that share a proxy. An
unpinned session fixture boots a copy on every worker that runs one of its tests and keeps them all until
the node ends, and those copies ran CircleCI integration-extensions nodes out of memory