* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* test(e2e): add the Codex file edit and long session rows
The file edit row has Codex change a random value into config.txt with the patch tool and checks the file on disk. The long session row has it follow a chain of random file names one cat at a time. new_workspace now returns the resolved path, because Codex reports patched files by their real path (/private/var on macOS).
* test(e2e): make the Codex long session prove every earlier command output reached the model
* test(e2e): Codex auto-compaction through LiteLLM (#49389)
* test(e2e): add the Codex auto-compaction row
Codex reads a random value, compacts through the gateway with its remote compaction forced on, and is asked for the value after the file holding it is deleted. The row reads the compaction from the thread's rollout file and marks Claude on Bedrock as a strict expected failure, since the Responses-to-chat bridge ignores the compaction_trigger item
* refactor(e2e): validate Codex rollout lines with the models directly
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row
Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups
* test(e2e): type the Codex self-test event helper as a Mapping
* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error
* test(e2e): record a step when the Codex harness sets up its home and workspace
The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it
* Update run.py
wip
* test(e2e): share the CLI version rule, env lockdown and rate limiter in tests/e2e/cli_harness
Moves the npm release rule (now taking the package name), the env allowlist plus fresh HOME, and the per-upstream rate limiter out of claude_code so the Codex suite can reuse them. Claude Code behavior is unchanged, its CI, cron and Dockerfile paths now call the shared resolver with @anthropic-ai/claude-code
* test(e2e): freeze the packument map and note the rate limiter self-test
* test(e2e): race real processes on one rate limiter bucket so a missing lock fails
Eight spawned processes start together and drain a bucket that never refills. With the flock they get exactly its burst between them, without it every attempt is granted
* test(e2e): keep the old Claude Code version resolver path for the Buildkite runner images
The runner Dockerfiles still run tests/e2e/claude_code/pr_gate_version_resolver.py and silently install @latest when it is missing, which drops the three day review window. The old path now forwards to cli_harness/npm_version_resolver.py
* test(e2e): rename the load suite to perf, one E2E_PERF_TESTS opt-in, and a mixed-provider Locust perf test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop the mixed-provider perf test, keep the rename and the E2E_PERF_TESTS consolidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run every CircleCI check from the integration workflow
* test(ci): keep the startup test CircleCI-owned
* test(ui): find reset log rows by request id so concurrent workers cannot page them out
* test(ui): retry the key budget window spec until the draft refresh fix lands
* ci(ui): set Playwright workers and retries on the CircleCI command only
The shared playwright.config.ts also drives the Buildkite UI suite, where 3 workers
ran out of memory on one of two runs. Buildkite keeps main's 1 worker and 2 retries
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* docs(agents): ban semicolons and colons in human-facing text
Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."
* docs(agents): rewrite semicolons in nested AGENTS.md files
* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines
* docs(agents): point at the PR template path that exists
* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
* ci: restructure lint, unit and smoke workflows
* ci: preserve source formatting in tier workflows
* ci: retire per-shard coverage flags and tighten the tier workflow guards
* test(ci): freeze the workflow startup safety models
* ci: keep the current required check names running until the ruleset moves to the tier collectors
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test(e2e): move the harness self-tests out of tests/e2e
The nightly Buildkite run copies tests/e2e into the runner image and runs
bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit
properties, the stack lock, the load aggregators, the Claude Code driver)
counted as e2e tests on the status page even though none of them reaches a
proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout,
and run in the GitHub Actions lint job and the CircleCI
provider_replay_harness job instead
* fix(ci): point the providers replay controls at tests/e2e_harness
The providers integration job still selected the four replay-control
tests under tests/e2e/test_provider_edge.py, so pytest exited before
they ran. The raw-HTTP check's file walk also drops to one loop per
comprehension
* style(tests): mark the raw-HTTP check's bindings Final
