Drop the second pricing pass on interrupted /v1/messages streams: clearing
the stale usage.cost and hidden response_cost is enough for the existing
success and failure logging to price the recovered usage. Add an iterator
test for the upstream-close path the proxy takes on a client disconnect.
Replace the coroutine __qualname__ check with a functional check: queue one event, run with a short flush interval, and wait for async_send_batch to be awaited by the task init scheduled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Redis semantic cache tests wrapped the first import of litellm.caching.redis_semantic_cache in patch.dict("sys.modules", ...), which snapshots and restores all of sys.modules on exit. Every module first imported inside the block, including litellm.proxy.proxy_server, was dropped from sys.modules while staying cached as an attribute on the litellm.proxy package. The next test that patched litellm.proxy.proxy_server.<attr> hit the stale attribute while production code re-imported a fresh module, so the patch never reached it. Replace the whole-dict patch with MonkeyPatch.setitem on the two redisvl keys only
The LangSmith init test globally patched asyncio.get_running_loop while constructing the logger. Any orphaned AsyncHTTPHandler finalized by the cyclic GC during that window also called loop.create_task on the mock, tripping assert_called_once. Run the test under a real event loop and assert on the real task instead of patching asyncio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replicas re-read cooldowns from Redis at most every 10s
(default_redis_batch_cache_expiry), so the 12s window left 2s of slack;
it is now 15s and the benched phase runs from 15s to 26s after the trip.
The reliability rows the new cells cover claimed exercised_on messages
too, but every cell drives /v1/chat/completions, so they now claim
chat_completions only. RouterSettingsOverride.timeout and
RouterCurrentValues.routing_strategy had no reader and are gone.
A slow or unreachable counter store made a pre-relay rejection wait behind
the reservation release before the client saw the error event and the close.
Close first and release in finally, mirroring the relay's own failure path,
so a client that already hung up still gets its reservation released.
The three pre-relay exits of realtime_websocket_endpoint (missing model,
key/model access denied, pre-call rejection such as a rate limit or a
guardrail) returned before the finally that releases the auth-time budget
reservation, so a rejected session pinned the key at the reserved amount
until the counter TTL expired and its next requests got budget_exceeded
while /key/info showed spend 0. A single _reject_realtime_session helper
now releases the reservation before sending the error event and closing,
and release_or_invalidate_budget_reservation shields the release from a
second cancellation and logs, rather than raises, a failing invalidate
fallback so it can never mask the session's own outcome.
The prompt-cache affinity test needs the prompt_caching pre-call check on the
proxy, which the CI stack does not carry until project-releaser #223 lands, so
it now sits behind a prompt_caching_stack marker that is deselected unless
E2E_PROMPT_CACHING_STACK is set, the same shape as managed_files. The
per-directory deselection hooks for weekly and managed_files move into the
parent conftest as one OPT_IN_MARKERS table, so the collector counts a gated
cell only where its env var is set (31/36 today, 32/36 with #223).
The simple-shuffle cell asked for three picks, which a shuffle ignoring the
weights passes one time in eight; it now asks for ten.
The SIGTERM/SIGHUP teardown handlers were installed unconditionally, so a base scan started
under nohup (SIGHUP inherited as SIG_IGN) would start dying on hangups it was told to ignore.
Install them only where the disposition is still the default, and cover the ignored case with a
regression test that hangs up a scan started with SIGHUP ignored and expects it to finish.
The two dict literals behind the refusal event counted against the LIT002 ceiling once the base branch used up its headroom, so the frame is now a ReadOnly TypedDict built in one shot. Importing Literal explicitly also makes the UP037 suppression on the Vertex discovery signature unnecessary, so it goes.
The blanket finally release from the previous commit also zeroed the reservation
of successful sessions. Success settlement is enqueued on the logging worker, not
awaited, so the endpoint's finally ran first and released the reservation the cost
callback still had to reconcile, dropping the real spend from the key/team/user
counters.
The relay now stamps a synchronous marker (REALTIME_SESSION_SUCCESS_LOGGED_KEY) on
the shared logging object at the single success-dispatch site, and the endpoint
releases the reservation only when that marker is absent. Refused or failed
sessions, which never log success, still release; successful sessions leave the
reservation for the cost callback to settle to actual spend. Exactly one settler
touches each reservation, so the idempotent reconcile never double-adjusts.
A refused or failed /v1/realtime session never ran the success cost callback
or a failure hook, so its pre-call budget reservation stayed open and kept the
key/team/user spend counters pinned above real spend, 429ing later requests on
the same key until the counter's TTL expired. The endpoint now reconciles the
reservation in a finally, reusing a shared release_or_invalidate_budget_reservation
helper that mirrors the success/failure paths (release to zero, else invalidate
the reserved counters and finalize).
The relayed upstream close message and reason also go through the proxy's
client-facing redaction, so a credential, internal hostname, private IP, or
server path echoed by the upstream never reaches the client verbatim.
The strip in text_tokens_without_nested_reasoning is capped at the reasoning
share, the reported text, and the over-sum past completion_tokens. Dropping the
caps to a bare over-sum passed every existing test, so this pins each cap with
a parametrized helper test plus one billing test where text over-reports past
the reasoning share and only the nested share may be netted out
The scoped xargs list missed a ruff-tests.toml rule change and skipped ruff on
deletions, so the block now runs test-linting.yml's exact command over tests/.
ruff-tests.toml, test-quality-budget.json, and scripts/check_test_quality.py
trigger the block too, and it sits after the background launches so the
dashboard and gen:api jobs overlap it.
* fix(batches): register ownership for every batch create path
Since the team isolation change, the managed files hook decided whether a
response came from a create by looking for the managed input file id on it,
which only the unified input path sets. Batches created from a model-encoded
input file id, a model param, or a raw provider id with ?provider= never got
an ownership row, so they vanished from GET /v1/batches for the key that
created them.
The create endpoint now stamps a create marker on the response before the
hooks run, and the hook keys ownership registration and the batch-created
metric on that marker instead of on the input id format.
* test(batches): assert ownership registration through the managed files hook
The endpoint tests asserted the private create marker, which is wiring, not
behaviour. They now run the create through the real managed files hook and
assert the ownership row is written for the creating key on every create
path, with the unified path driven by a genuine encoded input file id
instead of patched decoders.
* fix(mcp): let config.yaml MCP servers pin server_id
A config-defined MCP server's id is a hash of server_name|url|transport|
auth_type|alias, recomputed on every config load, so editing any of those
fields mints a new id. Every key and team granted the old id via
object_permission.mcp_servers keeps pointing at an id that no longer exists,
and the server disappears from tools/list for them with nothing logged.
load_servers_from_config now uses an explicit server_id from the server's
config entry when present and falls back to the existing hash otherwise, so
grants survive url/name/alias edits. Rejected at config load: a blank or
non-string server_id, two entries claiming the same id, a pinned id already
held by a database-backed server, and a pinned id that is another entry's
server_name or alias (expand_permission_list matches ids before names, so
that one would capture the other server's grants). Because the database
registry loads after the config on startup, a database row that lands on a
pinned config id is reported as a warning from the database reload instead,
where it is decidable; the warning is latched on the shadowed set so the
config-reload timer does not reprint it every interval.
Deployments that do not set server_id keep the exact id they have today.
* fix(mcp): close two more pinned-id capture paths
A pinned server_id equal to an alias supplied through litellm_settings
mcp_aliases was accepted, because the collision index only held the entry's
own alias field. expand_permission_list matches ids before names, so grants
written for the aliased server resolved to the pinning one. mcp_aliases keys
whose target is a config server are now reserved the same way.
A pinned server_id equal to a database-backed server's name, server_name or
alias had the same effect against the database side, and could not be
rejected at config load because the database registry is not loaded yet. The
database reload now warns about it, latched like the existing shadow warning.
* fix(mcp): reserve only the aliases the loader actually assigns
Reserving every mcp_aliases key targeting a config server was too broad in
two ways: the mapping is ignored when the entry sets its own alias, and only
the first mapping for a server is ever applied. Both cases made a pinned
server_id that could never have collided abort proxy startup. Reserve only
the name load_servers_from_config will really assign.
The database capture warning also fired for a database server whose own id is
the config server_id. There the database row wins the id outright through
get_registry precedence, so the shadow warning above it is the accurate one
and the capture message contradicted it. Skip those rows.
Also mark the two litellm-internal patches in the reload test helper, which
the test-quality gate counts; the database reload has no other seam.
* fix(mcp): match the loader's alias check exactly, is None not falsiness
load_servers_from_config consults mcp_aliases only when the entry has no
alias key at all, so an entry setting alias: "" gets no mapped alias. The
collision index used falsiness and reserved the mapped name anyway, which
failed startup on a pinned server_id that could never have collided with it.
* fix(mcp): skip one identifier, not the whole database row
A database row can shadow one config server_id by id and capture another by
name at the same time. Skipping the entire row when its id shadowed a config
entry dropped the second warning, leaving the operator with half a diagnosis.
Skip only the identifier equal to the row's own id.
* fix(mcp): reject conflicting self-pinned server ids
* fix(mcp): validate config server names before building the identifier index
The collision check reads every entry's body up front, so a malformed
entry under an invalid name surfaced as an AttributeError instead of the
name validation error the loader gave before this change.
CI's required lint job runs ruff with ruff-tests.toml over tests/ and the
test-quality budget gate, but scripts/pre_commit_lint.sh only triggered make
lint on litellm/ files, so a tests-only commit passed make check with a no-op
note and then failed CI (a duplicate test name, ruff F811, did exactly that).
When tests/ Python files are in scope and no litellm/ files are, run ruff
with ruff-tests.toml over the changed test files and make lint-test-quality,
with the matching partial-staging warning, summary line, and no-op condition.
validate_url raises HostResolutionError, a SSRFError subclass, for the two
DNS outcomes (lookup failed, no addresses). The image fetch helper maps
that to a "host could not be resolved" message and keeps the
user_url_allowed_hosts hint for the policy verdicts it can actually fix.
Streaming chat relays on Azure and azure_ai deployments rebuild the response from
the SSE chunks through the OpenAI passthrough assembler, so the spend log carries
usage. The router relays keep the JSON body when the Content-Type carries a
charset, return the upstream status and body instead of a 500 when the deployment
rejects the call, and fall back to the caller's api-version when the deployment
sets none. Lint budgets ratcheted to the measured totals
Generic passthrough calls inferred the provider from the bare model name, so an
azure_ai/gpt-* deployment on an Azure OpenAI host flipped to azure and
get_llm_provider re-prefixed the deployment name into azure_ai/gpt-5.4-mini, a
404 DeploymentNotFound. provider_for_generic_call takes the declared
custom_llm_provider first, then the model's own prefix, and only infers for
unprefixed models
The image edit handler now awaits BaseImageEditConfig.async_transform_image_edit_request, and
Black Forest Labs overrides it so URL images and masks download through async_safe_get instead of
the blocking safe_get on the event loop. Rejected image fetches raise a fixed policy message with
the user_url_allowed_hosts hint rather than echoing the resolver's verdict (resolved IP, DNS
failure) back to the caller. The test fixture also fails any request-path call of the sync
convert_url_to_base64 so a regression cannot pass unnoticed.
GET /router/settings is a management route, so the split transport now
sends it to the control plane instead of the data-plane gateway.
The rpm-1 key behind the 429 cells is spent right before the trip, after
the pair is registered, because the rate limiter's 60s window opens on
that request and the registrations' propagation waits could otherwise
outlast it. Recovery also accepts a 200 served by the benched deployment
itself, since its key's minute can be up by then.
The cooldown recovery deadline now counts from the last failure a stale
replica caused during propagation, because every failure re-arms the
cooldown TTL; the strict bench window stays anchored to the trip.
The handshake error path already runs client-facing error strings through
_redact_string; the relay's _close_client did not, so a secret echoed in an
upstream close reason could reach the client verbatim. Mirror the handshake
path and scrub the close message and reason before relaying them.
Health checks probed every OCR deployment with a PDF, which Cohere Parse
rejects, so /health, background health checks, and the UI Test Connection
button marked Cohere Parse deployments unhealthy. BaseOCRConfig gains a
get_health_check_document hook (PDF by default) that CohereParseConfig
overrides with a 1x1 PNG data URI. cohere also gains ocr in the provider
endpoint matrix
The realtime usage writer passed the provider's output_token_details through as sent, so spend logs and callbacks kept a text_tokens that still contained reasoning_tokens while every other completion_tokens_details producer stores the partitioned share. The writer now applies the same rule the cost calculator uses, moved to litellm/types/utils.py so both read one definition, and the calculator keeps it for usage objects that arrive nested from elsewhere
