The Postgres-backed smoke job previously exercised one resolver. Running
the default and the legacy variants in it covers v2 now that it is the
default, without losing v1's coverage.
The migrations Job entrypoint (migrations/run.py) has defaulted to v2 with
USE_V2_MIGRATION_RESOLVER=false as the opt-out, and the Helm chart documents
that knob. Proxy startup still defaulted to v1, so the two paths disagreed
about which resolver a deployment runs.
Proxy startup now resolves the same way: v2 unless USE_V2_MIGRATION_RESOLVER
is false or --use_legacy_migration_resolver is passed.
- --use_v2_migration_resolver stays accepted as a no-op that warns, so
existing commands and Helm values do not fail on an unknown option.
- The dedicated Postgres smoke-test job is repointed at the legacy resolver
so v1 keeps real-DB proxy-boot coverage, and the two jobs that deselected
it by name are updated to match the rename.
#39178 reverted an earlier flip because two replicas sharing a database
deadlocked (40P01 / P3018) with neither answering /health/liveliness. That
contention is what #40932 coordinates, which is why this builds on it.
The proxy used to boot with no master key (every request accepted without
authentication) and with sk-1234, the key every example used. It now stops at
startup, before it connects to the database, and prints how to fix it: where the
bad key came from, a copy-pastable command that generates a secure key, and,
when the public key is also encrypting a database, a link to the rotation guide
general_settings.dangerously_allow_unsafe_proxy: true or
LITELLM_DANGEROUSLY_ALLOW_UNSAFE_PROXY=true starts the proxy anyway, for local
development. CI and test boots that rely on sk-1234 or on no key set it
BREAKING CHANGE: deployments with no master key, an empty one, or sk-1234 no
longer start until they set a real key or opt in to the override
Shipped proxy configs now read general_settings.master_key from
os.environ/LITELLM_MASTER_KEY, the .env examples ship a blank value with
the openssl generate command above it, and READMEs, the missing env vars
page and Admin UI code snippets show a generate command or the
<your-master-key> placeholder instead of the literal sk-1234
The two CircleCI docker runs that mount proxy_server_config.yaml and
oai_misc_config.yaml now pass LITELLM_MASTER_KEY so their runtime key is
unchanged
POST /config/update stored keys the config file owns and answered 200
while the file silently kept winning. Run the config-owned check for
general_settings, litellm_settings, and router_settings before the first
database read, the way /config/field/update already does, so a refused
request stores nothing.
The config reload re-loaded the merged settings as yaml settings, which
turned every saved router setting read-only after one tick. Read the
saved router settings row instead so database-owned values stay writable.
The integration harness seeds num_retries through /config/update instead
of the config file, which is what the effective-settings and observed
routing tests need to keep exercising a database-owned value.
The check spins up a hypercorn TLS peer and drives the SDK's own httpx
handlers at it, so it needs litellm importable, hypercorn installed and a
loopback socket. It lived under tests/e2e, whose Buildkite runner image
installs neither litellm nor hypercorn by design (the suite drives a
remote proxy over HTTP), so every scheduled e2e build since #230 failed
to import the module and pytest reported it as a collection error. The
unit tree bans sockets, so it does not belong there either
tests/integration is the CircleCI tier built for real TCP against local
protocol peers. This adds an sdk shard to it for cases that exercise the
SDK's clients with no gateway in the path, registers the two HTTP/2
nodes in the contracts manifest, and adds the shard to the CircleCI
matrix. The test now flips the feature through LITELLM_HTTP2 (the user
surface) instead of patching module attributes, and asserts the version
the peer observed on the wire next to the one the client reports
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
Both readiness loops broke out on success and fell through on timeout, so a
mock Presidio server that failed to bind left the run going with nothing
serving /analyze. The guardrail then errored at request time and the failure
surfaced as an unrelated Playwright assertion in presidioUserStory.spec.ts
rather than as the missing fixture it actually was.
Fail the local runner with the port in the message, and add the matching wait
step to both CircleCI UI jobs, which had no readiness check at all.
The guardrail section of the release checklist is done by hand every cut:
create a Presidio guardrail through the wizard, send a sentence with PII from
the playground, then open Logs and check the guardrail caught it. Nothing
covered that path, so a break anywhere along it surfaced only when someone
happened to repeat the steps.
Adds a spec that walks it once and turns the eyeball checks into assertions.
The strongest of them is the leak check: it reads the request back and fails if
the stored prompt still carries the raw address or number, which is what the
manual step is really looking for.
The stack gains a Presidio stand-in that answers the two routes the guardrail
calls, detecting a fixed regex set with a Luhn check on card numbers. Real
Presidio's detection quality is Presidio's business, and pinning the UI lane to
it would mean two heavy containers with spaCy models on every CI run for a test
that is about LiteLLM's integration. The real analyzer stays covered in the
Python lane. The stand-in returns the same entities at the same spans as the
real one for the checklist's sentence, and driving the real guardrail against it
produces the same record shape, so a test written against it is written against
the product's real behavior.
Making the stand-in return the text unmasked turns the spec red on the raw
address reaching the spend log, so the leak assertion reads live data.
run_e2e.sh and both CircleCI UI jobs start the stand-in alongside the mock LLM.
Its port is overridable like the others so two checkouts can run at once.
Two routers in one process shared a single handler, because the callback
manager dedupes on the class name plus the handler's public attributes and the
handler had none. The second router's requests were never counted. The handler
now carries the id of the cache it was built on, so routers with different
caches both register while the two selectors one router builds for its routing
groups still collapse into one.
Clamping a negative count back to zero used SET, which drops the key's TTL, so
the next write started the hour over. It uses INCRBY by the negative amount now,
which leaves the expiry alone.
The Lua script had no test that ran it, so tests/local_testing covers both the
sync and async paths against a real Redis, and the file is wired into the
CircleCI job that provides one.
