Commit graph

2 commits

Author SHA1 Message Date
devin-ai-integration[bot]
e16d456291
test(integration): isolate every multi-worker integration suite by default (#46027)
* test(integration): isolate every multi-worker integration suite by default

Per-worker database, Redis, upstream and proxy were opt-in for management,
accounting, providers and extensions. mcp, security and cost ran 4 to 8
xdist workers against one shared proxy, database and Redis, and those suites
hold most of the integration flakes CircleCI records. Every run with more than
one worker now builds a private stack per worker, run_integration.sh no longer
starts or exports the shared proxy, upstream, database or Redis in that mode,
and each worker's scripted upstream runs in its own process instead of a
thread in the pytest controller.

Worker gateways get what the shared proxy used to provide: a peer for
management and mcp, the scripted cost map for cost, stdio MCP servers, and
coverage for mcp. mcp coverage now measures lines through sys.monitoring,
because branch measurement on CPython 3.12 uses the trace function and that
made a covered proxy take about 4x as long to become ready, which is why the
shared mcp proxy kept missing its 90 s readiness wait. Routing parity
snapshots now also read the worker databases, since its cost and mcp runs use
several workers. The security per-worker fixture is redundant and removed

* test(integration): give the nested probe root the files a worker gateway needs

Two worker probes now run isolated, so the nested controller boots a real per-worker upstream and proxy from the probe root. That upstream imports the fake OpenAI server and the cost map modules, and the proxy reads proxy_config.yaml, none of which the probe copied. The nested run also drops INTEGRATION_COVERAGE_FILE so it never writes into an outer coverage file

* test(integration): budget nested multi-worker runs for their worker stack boots

A nested run.py database run with more than one worker now boots a private database, Redis, upstream and proxy per worker, one after another, before any case runs. The fixed 60 s subprocess timeout left no room for that, so the 2 worker probes sat at 57-60 s on CircleCI. The subprocess budget is now the old 60 s plus 30 s per worker stack, and the outer pytest timeout keeps its previous 30 s of slack on top. 30 s is the slowest worker proxy readiness recorded in startup.json on the integration-cost job of workflow e09b108b (21.6-30.3 s on 4 CPUs, 2026-10-12)

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
2026-10-12 03:46:06 -07:00
devin-ai-integration[bot]
e26a6450c8
test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711)
* test(integration): drop the contracts.json manifest and the covers requirement

Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): run mcp as its own shard with xdist and a peer proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules

The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP transport and access-control matrices

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP credential and OAuth flow coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): assert the REST denied-server listing is refused or empty

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): pin the REST denied-server listing to 403 access_denied

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 07:48:46 -07:00