* test(integration): add read-replica routing harness
* refactor(integration): hoist the maintenance url imports
* fix(integration): keep per-test databases and the witness sequence readable under replica roles
* fix(integration): opt bespoke database and pool tests out of the injected read replica
* test(integration): commit recorded replica routing expectations
* fix(integration): judge routing by role containment so shrinking role sets do not fail
* fix(integration): run the pool-limit shutdown choreography on the superuser database url
* ci(integration): add the mcp group to the replica matrix
* fix(integration): judge routing by exact role sets with a named either-role allowlist
* test(integration): drop containment-era routing expectations for re-recording
* chore(integration): drop docstrings from the replica harness scripts
* docs(integration): describe exact routing matching and the either-role list
* test(integration): record exact replica routing expectations
* test(integration): allow the SELECT 1 health probe on either role
* test(integration): replace committed routing expectations with an on-demand base-vs-head parity run
* test(integration): fix parity env scope, readme wording, and seed-deterministic serialization test
* test(integration): make the sorted-role serialization test deterministic in-process
* test(integration): swap all product code in parity runs and pin role gains
* ci(integration): force tracked-file removal before parity checkout
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test(integration): drop the contracts.json manifest and the covers requirement
Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): run mcp as its own shard with xdist and a peer proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules
The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP transport and access-control matrices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP credential and OAuth flow coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the REST denied-server listing is refused or empty
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the REST denied-server listing to 403 access_denied
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two CI checks were asserting behaviour the proxy no longer has. Neither
was catching anything; both now fail for the right reason.
budget_exceeded (tests/otel_tests/test_e2e_budgeting.py)
bf804f5188 made 422 the default for budget refusals and added
budget_exceeded_status_code to restore 429 for callers that need it. The
e2e budget tests still asserted 429, so all six have been failing on a
status change that was deliberate. Assert 422, the documented default,
rather than reading litellm.budget_exceeded_status_code back — a test
that asks the code what it does would have passed straight through this
change and through the next one.
The helpers also caught bare Exception, so a connection error reached
`e.body` and surfaced as an AttributeError instead of a failed
assertion. Narrow both to openai.APIStatusError, which is what a refusal
actually raises (UnprocessableEntityError for 422, RateLimitError for
429), and let anything else propagate as itself.
test_bad_database_url (.circleci/config.yml)
The check required "Database setup failed after multiple retries", which
only the v1 resolver emits, OR uvicorn's "Application startup failed.
Exiting.". With v2 the default, the first branch is dead and the whole
assertion rests on an incidental uvicorn line that HEAD's run did not
emit at all.
Assert the behaviour instead of the wording: the container exits
non-zero, the log names the unreachable server (P1001), it never reaches
"Application startup complete", and it is not left running. The exit
code was previously discarded by `|| true`, so the one thing the job
most needed to check was never checked.
Verified by running the bad-DATABASE_URL container: exit 3, P1001
present, no startup-complete line, container stopped — the new check
passes and the old one passed only by the uvicorn line's accident.
The migration e2e harness only ever used one image: it seeded the database
with the candidate build and then applied synthetic migrations on top. That
proves the migration machinery (locking, crash recovery, legacy baselining,
pooling) but never executes the real schema of release N against the real
migrations of release N+1, which is the path operators actually run.
Adds a baseline image alongside the candidate, so a test can seed with a
published release and upgrade with the build under test.
Suites:
- test_upgrade.py: the candidate applies the pending release migrations,
keys minted by the baseline release survive, and concurrent replicas
upgrade a baseline database exactly once.
- test_rolling_upgrade.py: a baseline replica keeps serving virtual-key
auth while the candidate migrates underneath it, and both releases serve
and resolve each other's keys during the overlap. This is the reported
failure: a new column on LiteLLM_VerificationToken invalidates prepared
plans on pods still running the old release, which the proxy reads
whole-row, and auth starts failing until those pods leave service.
- test_shaped_database.py: the upgrade completes and preserves rows on a
populated spend log, rather than on the empty database every other
migration test starts from.
Every upgrade assertion is gated on the candidate having actually applied
migrations the baseline had not, so a stale pin fails loudly instead of
passing on an empty delta.
CI adds two jobs to the migration_startup workflow. The baseline defaults
to a committed release pin and is overridable per pipeline, matching how
migration_candidate_image already works; only the upgrade jobs pull it.
Verified against a real v1.101.0 -> v1.102.0 upgrade: 6 passed, with the
baseline seeding 165 migrations and the candidate applying the 6 that
landed between the two releases.
The Postgres-backed smoke job previously exercised one resolver. Running
the default and the legacy variants in it covers v2 now that it is the
default, without losing v1's coverage.
The migrations Job entrypoint (migrations/run.py) has defaulted to v2 with
USE_V2_MIGRATION_RESOLVER=false as the opt-out, and the Helm chart documents
that knob. Proxy startup still defaulted to v1, so the two paths disagreed
about which resolver a deployment runs.
Proxy startup now resolves the same way: v2 unless USE_V2_MIGRATION_RESOLVER
is false or --use_legacy_migration_resolver is passed.
- --use_v2_migration_resolver stays accepted as a no-op that warns, so
existing commands and Helm values do not fail on an unknown option.
- The dedicated Postgres smoke-test job is repointed at the legacy resolver
so v1 keeps real-DB proxy-boot coverage, and the two jobs that deselected
it by name are updated to match the rename.
#39178 reverted an earlier flip because two replicas sharing a database
deadlocked (40P01 / P3018) with neither answering /health/liveliness. That
contention is what #40932 coordinates, which is why this builds on it.
The proxy used to boot with no master key (every request accepted without
authentication) and with sk-1234, the key every example used. It now stops at
startup, before it connects to the database, and prints how to fix it: where the
bad key came from, a copy-pastable command that generates a secure key, and,
when the public key is also encrypting a database, a link to the rotation guide
general_settings.dangerously_allow_unsafe_proxy: true or
LITELLM_DANGEROUSLY_ALLOW_UNSAFE_PROXY=true starts the proxy anyway, for local
development. CI and test boots that rely on sk-1234 or on no key set it
BREAKING CHANGE: deployments with no master key, an empty one, or sk-1234 no
longer start until they set a real key or opt in to the override
Shipped proxy configs now read general_settings.master_key from
os.environ/LITELLM_MASTER_KEY, the .env examples ship a blank value with
the openssl generate command above it, and READMEs, the missing env vars
page and Admin UI code snippets show a generate command or the
<your-master-key> placeholder instead of the literal sk-1234
The two CircleCI docker runs that mount proxy_server_config.yaml and
oai_misc_config.yaml now pass LITELLM_MASTER_KEY so their runtime key is
unchanged
The check spins up a hypercorn TLS peer and drives the SDK's own httpx
handlers at it, so it needs litellm importable, hypercorn installed and a
loopback socket. It lived under tests/e2e, whose Buildkite runner image
installs neither litellm nor hypercorn by design (the suite drives a
remote proxy over HTTP), so every scheduled e2e build since #230 failed
to import the module and pytest reported it as a collection error. The
unit tree bans sockets, so it does not belong there either
tests/integration is the CircleCI tier built for real TCP against local
protocol peers. This adds an sdk shard to it for cases that exercise the
SDK's clients with no gateway in the path, registers the two HTTP/2
nodes in the contracts manifest, and adds the shard to the CircleCI
matrix. The test now flips the feature through LITELLM_HTTP2 (the user
surface) instead of patching module attributes, and asserts the version
the peer observed on the wire next to the one the client reports
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
Both readiness loops broke out on success and fell through on timeout, so a
mock Presidio server that failed to bind left the run going with nothing
serving /analyze. The guardrail then errored at request time and the failure
surfaced as an unrelated Playwright assertion in presidioUserStory.spec.ts
rather than as the missing fixture it actually was.
Fail the local runner with the port in the message, and add the matching wait
step to both CircleCI UI jobs, which had no readiness check at all.
The guardrail section of the release checklist is done by hand every cut:
create a Presidio guardrail through the wizard, send a sentence with PII from
the playground, then open Logs and check the guardrail caught it. Nothing
covered that path, so a break anywhere along it surfaced only when someone
happened to repeat the steps.
Adds a spec that walks it once and turns the eyeball checks into assertions.
The strongest of them is the leak check: it reads the request back and fails if
the stored prompt still carries the raw address or number, which is what the
manual step is really looking for.
The stack gains a Presidio stand-in that answers the two routes the guardrail
calls, detecting a fixed regex set with a Luhn check on card numbers. Real
Presidio's detection quality is Presidio's business, and pinning the UI lane to
it would mean two heavy containers with spaCy models on every CI run for a test
that is about LiteLLM's integration. The real analyzer stays covered in the
Python lane. The stand-in returns the same entities at the same spans as the
real one for the checklist's sentence, and driving the real guardrail against it
produces the same record shape, so a test written against it is written against
the product's real behavior.
Making the stand-in return the text unmasked turns the spec red on the raw
address reaching the spend log, so the leak assertion reads live data.
run_e2e.sh and both CircleCI UI jobs start the stand-in alongside the mock LLM.
Its port is overridable like the others so two checkouts can run at once.
Two routers in one process shared a single handler, because the callback
manager dedupes on the class name plus the handler's public attributes and the
handler had none. The second router's requests were never counted. The handler
now carries the id of the cache it was built on, so routers with different
caches both register while the two selectors one router builds for its routing
groups still collapse into one.
Clamping a negative count back to zero used SET, which drops the key's TTL, so
the next write started the hour over. It uses INCRBY by the negative amount now,
which leaves the expiry alone.
The Lua script had no test that ran it, so tests/local_testing covers both the
sync and async paths against a real Redis, and the file is wired into the
CircleCI job that provides one.
Drops `-x` from all 24 pytest invocations in .circleci/config.yml. With
`-x`, a job stops at its first failure, so a second broken test in the
same suite stays invisible until the first is fixed and CI is re-run.
That turns one round trip into N when a job has several broken tests.
This is exactly what happened in #39770: fixing
test_missing_model_parameter_curl in
tests/store_model_in_db_tests/test_openai_error_handling.py immediately
unmasked test_chat_completion_bad_model_with_spend_logs in the same
file, which had been failing for a long time without ever being
reported.
Only `-x` is removed; -v/-vv/-s/-n/--reruns and every other flag are
untouched.
This reverts merge commit 2b1bd20834 (#31125)
Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:
Error: P3018 A migration failed to apply
Database error code: 40P01, deadlock detected
Process 73 waits for ShareLock on virtual transaction 4/11;
blocked by process 75. Process 75 waits for ExclusiveLock on
advisory lock [16384,0,72707369,1]; blocked by process 73
Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run
Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)
The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
The v2 resolver skips the diff-and-force recovery that caused schema
thrashing when two LiteLLM versions contend for one database during a
rolling deploy. The standalone migration Job already defaulted to v2; this
aligns the proxy-server path.
v1 stays reachable two ways: --use_legacy_migration_resolver on the CLI, and
USE_V2_MIGRATION_RESOLVER=false for containerised deploys, where
prisma_migration.py calls run_server with a fixed argv and the env var is the
only route in. --use_v2_migration_resolver still parses, so existing commands
do not die on an unknown option.
Because v2 fails fast where v1 retried every failed deploy, a database that is
not accepting connections yet, or another instance holding the migration
advisory lock, would now kill a boot that used to ride it out. Those two
failures are retried, with Prisma's stderr logged each round, and still raise
once the attempts are spent.
Moves the resolver tests from litellm-proxy-extras/tests, which no CI job
runs, into tests/litellm-proxy-extras, and repoints the dedicated Postgres
CircleCI job at the legacy path so v1 keeps real-DB and proxy-boot coverage.
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.
#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.
OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.
The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
tests/enterprise is 13 files and 244 tests that only CircleCI runs, and CircleCI
gates nothing: it triggers on PR labeled events, none of its jobs are required,
and red runs get merged past. So the suite that covers the enterprise package's
guardrails, auth and management endpoints has had no say in whether a change
lands.
Measured on 2026-08-21 with every credential stripped from the environment: 240
passed, 4 skipped, nothing failed. It needs no provider key, so it can be a
required shard rather than a scheduled lane, unlike the other CircleCI suites in
this group, which each carry a live-API minority.
The CircleCI job is removed in the same commit so the suite runs once, not twice.
codecov.yaml has carried an `Enterprise` component scoped to `enterprise/**`
since it was written, and it has never received a line of data. Every one of
the 19 coverage invocations across the unit base, the MCP workflow and the
CircleCI config passes `--cov=./litellm` and nothing else, so 11,203 lines of
paid-customer code sat outside the measured universe while the reported number
described only the rest.
litellm-enterprise is a uv workspace member and a direct dependency, so every
job that syncs already has it installed and importable; only the measurement
was missing.
Measured on tests/test_litellm/enterprise, the shard that exercises this code:
0 enterprise files in the report before, 142 after, at `enterprise/...` paths
that match the component's existing glob. That shard alone puts enterprise at
30.8%, which nudged its own total from 24.09% to 24.20% rather than down. The
aggregate direction across every shard is not knowable until they all report,
and a drop there is the instrument working, not a regression.
* ci: port the Postgres suites off CircleCI onto service containers
proxy_behavior_tests, proxy_security_tests and schema_migration_check were
near-identical CircleCI jobs: a Postgres sidecar, a schema seed, and one pytest
tree each. They ran nowhere else, and CircleCI holds none of the branch
ruleset's required checks, so the signal they produced gated nothing.
test-postgres.yml runs the same three trees on a Postgres service container as
one matrix, keeping each suite's own seeding rather than normalising it: the
behavior and security trees keep `prisma db push`, and the migration tree keeps
an empty database, which is what it needs to apply every committed migration
itself.
Their CircleCI definitions and workflow entries go with them, taking the config
from 47 jobs to 44. assert_ci_coverage.py stays green: dropping the new
workflow fails the census on exactly these trees, so the coverage moved rather
than went missing.
auth_ui_unit_tests is deliberately left behind. Ported, two of its
tests fail because prepare_metadata_fields refuses enterprise-only keys without
LITELLM_LICENSE, which exists as a CircleCI project variable and has no
GitHub Actions secret. Creating that secret is a human action, so the job stays
on CircleCI until it exists rather than shipping a red shard or quietly
deselecting the two tests.
* chore(ci): drop the narrative header from test-postgres.yml
The agent job's CircleCI glob collected `tests/agent_tests/**/test_*.py` and then
piped it through `grep -v` to drop `local_only_agent_tests/`. `assert_ci_coverage.py`
reads the glob but not the pipeline, so those two files looked covered and were
invisible to the census. The glob now excludes them structurally and they carry an
allowlist entry instead, which is a decision on the record rather than a hidden
filter. The collected file set is unchanged: `tests/agent_tests/` holds exactly one
CI-runnable test at the top level.
`tests/scim_tests/` held a single JSON fixture and no tests, referenced from nowhere.
`.github/workflows/` is for workflows. Both stray scripts move to `.github/scripts/`
with their callers updated: the price-file updater is invoked by
`auto_update_price_and_context_window.yml`, and the translation-report runner by
`make test-llm-translation`. The audit listed the latter as orphaned, but Makefile
line 317 still runs it, so it moves rather than being deleted.
The rollout heads-up workflow was a deliberate one-shot for the agent-shin rollout.
That rollout is done, the triage and auto-close workflows have been running daily
since June, so the pre-flip warning window is long past. Its script and dedicated
test go with it, and the sibling workflow-invariant test drops its entry.
Both are covered on GitHub Actions. test-litellm-ui-build.yml runs the
dashboard build on every PR, and test-litellm-ui-unit.yml runs the vitest
suite with ui-unit-tests already a required check, so neither CircleCI job
gates anything that GHA does not already gate.
ui_build additionally produced nothing anyone consumed. It persisted
litellm/proxy/_experimental/out to the workspace, and the only job
downstream of it was ui_unit_tests, which never attached the workspace and
reinstalled from source instead. The requires edge was pure sequencing, so
the build output was written and discarded on every client-touching PR.
One real narrowing comes with this, and it is deliberate. ui_unit_tests ran
the full vitest suite on PRs, while the GHA job scopes PR runs to tests
reachable from the diff and keeps the full suite on pushes to staging. That
split was a measured decision in #34175 and it still holds: the suite is
252s and 248s of that is CreateMCPServer.integration.test.tsx alone, so
running everything per PR buys about four minutes to re-run one file.
Note that assert-ci-coverage does not speak to this. It walks
tests/**/test_*.py only, so it is blind to vitest files by construction;
it stays green here because no Python test lost a runner, which is a
narrower claim than the UI side being unaffected.
auth_ui_unit_tests is a different job, a Python suite on a Postgres
sidecar, and is untouched
e2e_ui_testing and e2e_ui_testing_server_root_path run on
cimg/python:3.12-browsers, the one UI executor whose image supplies Node
rather than taking it from a cimg/node tag. That image ships Node 24.14.0,
which bundles npm 11.9.0, so both lanes have failed EBADENGINE against the
engines floor added in #35801. Every Node 24 release through 24.14.0 bundles
an npm below 11.10.0, so engines.node also rises to 24.14.1 (npm 11.11.0),
the first release where the two floors agree
The pinned install goes into /opt/node with /opt/node/bin prepended to PATH
instead of unpacking over /usr/local. On this image /usr/local already holds
npm 11.9.0, and extracting the tarball on top of it merges the two trees into
an npm that reports 11.17.0 and then exits 1 on npm ci printing no error text
at all, which is a worse failure than the one being fixed
The install moves into a reusable install_node command so the version and its
checksum have one home, shared with proxy_pass_through_endpoint_tests, and the
command refuses to run when it disagrees with ui/litellm-dashboard/.nvmrc. A
lane drifting off the version the rest of the toolchain uses is what produced
this failure, so that mismatch now stops the job instead of surfacing later as
an install error
The e2e node_modules cache key moves to v4 because the saved trees were built
by the old npm
