* fix(proxy): return the real status code when a credential update is rejected
update_credential ended its except clause with 'return handle_exception_on_proxy(e)'. Returning the exception makes it the response body, so FastAPI answers 200 and every rejection on this route reads as a successful write to any caller that checks the status; the admin dashboard's API client is one. Patching a name that does not exist answered 200 with the real 404 buried in the body.
The sibling handlers in this file already raise. The route had no test coverage, which is why it survived.
* Update tests/test_litellm/proxy/credential_endpoints/test_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Guards _declared_query_params against a regression in the get_flat_params
migration: the flatten step returns path, query, header and cookie params
together, so a dropped ParamTypes.query filter would wrongly treat path or
header names as declared query params and accept unknown ones. Removing the
filter fails these tests.
Streaming /chat/completions and /v1/responses emit nothing, not even response headers, until the upstream yields its first chunk, so an ingress with an idle read timeout (nginx proxy-read-timeout) drops long time-to-first-token streams.
Reuses the existing Anthropic keepalive wrapper with a configurable ping payload, emitting an SSE comment on the OpenAI-shaped routes so conformant clients ignore it. Off unless litellm_settings.sse_keepalive_ping_interval_seconds is set.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
create_websocket_passthrough_route existed but /openai and
/openai_passthrough only registered HTTP methods, so WS upgrades were
rejected at routing. Add catch-all websocket routes mirroring the HTTP
passthrough target construction.
Fixes#36088
create_a2a_client took the raw client off a process-wide cached handler and
called headers.update() on it, then leaned on folding the header set into the
cache key (through the unrelated disable_aiohttp_transport field) to keep one
caller's credentials away from the next.
Per-caller headers now ride with each request through the a2a SDK's call
context, and the agent card fetch gets them through resolver_http_kwargs, so
the shared client is never written to and its cache key no longer varies by
header set. Since the proxy puts a fresh trace id in every request's headers,
that key previously changed on every call, giving each request its own httpx
client and flushing the 200-entry client cache that every other provider
shares. All A2A callers on one timeout now reuse a single pooled client.
Sharing that client also means sharing its httpx cookie jar, which httpx fills
from every Set-Cookie and replays on any later request to a matching domain, so
one agent's session cookie would arrive at another agent on the same host. The
pooled client now carries a cookie policy that stores and sends nothing, which
neither litellm nor the a2a SDK relies on: the SDK's auth interceptor skips
cookie-borne API keys outright.
* fix(proxy): promote caller metadata trace fields into litellm_metadata
Routes in LITELLM_METADATA_ROUTES keep the caller's metadata as a provider
passthrough field and track proxy state in litellm_metadata, which is the dict
the logging integrations read. The caller's trace_id, session_id, trace_user_id
and trace_metadata therefore never reached any callback on /v1/responses,
/v1/messages, /v1/batches or /v1/files, and mask_input / mask_output were
dropped with them so a caller asking for redaction had their prompt logged in
full.
Promote an explicit allow-list of those fields from the requester_metadata
snapshot into litellm_metadata, never overwriting a value already set so
header-derived ids keep precedence. Trace-mutation controls (existing_trace_id,
update_trace_keys) and trace_public are deliberately excluded: langfuse applies
them to an arbitrary caller-chosen trace with no ownership check. tags is
excluded because per-tag budget enforcement runs earlier, at auth time.
This covers providers with a native Responses API config. Providers reaching
/v1/responses through the chat-completions bridge need the companion change to
get_litellm_params.
* ci: retrigger workflows
* warn at startup when a proxy-wide budget is set but no DB is connected
litellm.max_budget is only enforced via DB-loaded global spend, so a DB-less proxy silently ignores it. Log a one-time startup warning.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): inject max_budget into DB-less budget warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover DB-less budget warning startup call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin DB-less budget warning call site
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stabilize budget warning call-site pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The native files and batches routes declare /{provider}/v1/... and their routers are mounted before the passthrough router, so /openai_passthrough/v1/files and /openai_passthrough/v1/batches matched them with provider="openai_passthrough" and 500'd on the LlmProviders lookup instead of reaching openai_proxy_route.
Move the dedicated /openai_passthrough prefix onto its own router mounted ahead of the batches and files routers. /openai/... and every other provider prefix keep their current behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
Replaces the hand-rolled recording double with AsyncMock so the awaited team ids
come from await_args_list instead of a mutated list, and annotates the response
factory now that SearchToolInfoResponse is imported at module level.
The aggregated route was missing from LiteLLMRoutes.self_managed_routes
while its paginated sibling /user/daily/activity was listed, so auth
rejected every internal user with a 401 before the handler ran. That
route backs the default "Your Usage" view in the dashboard, which left
the main Usage page broken for non-admin users.
The handler already self-scopes: it checks admin view first, then falls
back to require_caller_user_id_for_non_admin, defaults a missing user_id
to the caller's own, and returns 403 when a non-admin asks for someone
else's data. Listing the route restores reachability without widening
what a caller can read.
check_route_access matches exactly (plus explicit wildcards), so the
parent entry never covered the /aggregated sub-path.
Every Admin UI session key is stamped with the reserved team id
`litellm-dashboard`, which never has a row in LiteLLM_TeamTable, so the
team lookup in _filter_visible_search_tools raised 404 and the endpoint
returned 500 for every non-admin dashboard session.
Skip the lookup for that sentinel and scope the caller by its key-level
allowlist alone, matching how MCP and agent permission checks already
treat it. A real team id is still resolved, and a genuine lookup failure
now surfaces with its own status instead of being masked as a 500.
* fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day
* fix(proxy): gate the current-UTC-day extension behind an opt-in param sent by the cost optimization dashboard
* fix(ui): label cost optimization savings dates as UTC days
An adversarial review of the previous commit found annotations that
described what the code wished were true rather than what flows through.
A false annotation is worse than the Any it replaced, since it launders a
wrong assumption past the type checker.
- purview: `_resolve_user_id` claimed every request-body value was a
Mapping, contradicting `_resolve_trusted_user_id` one method over, which
types the same argument `Mapping[str, object]`. `_should_block` claimed
every Graph response value was a sequence of str->str mappings and was
not assignable from its own producer's return type.
- cato: `_CatoAnalyzeResponse.required_action` was required and
non-nullable while the API returns null, as seven fixtures in the
guardrail's own suite assert. `analysis_result` had the same problem.
The streaming hook narrowed an override parameter below what
`ProxyLogging` actually passes it.
- marketplace: `_PluginRecord.manifest_json` was `str` against a nullable
column. Making it honest surfaced a latent crash, covered below.
- ownership: two functions took an attribute Protocol while their own
bodies branch on `isinstance(response, dict)`, which no Protocol can
satisfy.
- openapi generator: `paths` claimed every path-item value was an
operation, though path items also carry `parameters`, `summary` and
`$ref`.
- custom openapi spec: a TypedDict asserted a shape that the function
returns raw Pydantic sub-schemas out of. Reverted to Any, which is
imprecise but not false.
`get_marketplace` did an unguarded `json.loads` on the nullable
`manifest_json` inside an `except json.JSONDecodeError`, which cannot
catch the TypeError a NULL raises, so one NULL row 500s the endpoint. It
now skips the plugin like the file's other two read sites already do, with
a regression test that fails without the guard.
Where honesty cost precision, precision lost. `_should_block` went back to
its original signature entirely: the narrowing needed to type it turned a
fail-closed DLP control fail-open, because the TypeError it used to raise
on a malformed response reached `except Exception` and became a 400.
When the batch cost poller found a batch in a terminal failed, expired, or
cancelled state it wrote the provider response straight to the managed object
table, so the stored blob kept raw provider file ids and a raw batch id. Since
the row is final after batch_processed=True and the read paths only resolve
existing managed ids, every later GET /batches/{id} and GET /batches leaked
raw provider output and error file ids that clients cannot fetch through the
proxy. The terminal branch now normalizes the response with
ensure_batch_response_managed_file_ids before persisting, minting managed ids
under the batch owner's identity
POST /batches/{id}/cancel had the same gap: it called update_batch_in_database
without the caller's auth context, so a cancel response that already carried
provider file ids could never mint managed ids. The endpoint now forwards
user_api_key_dict
Terminal batch retrieve could return the raw provider output_file_id, which
skips managed-file ownership checks on /v1/files/{id}/content and lets any key
on the proxy download another user's batch output.
Retrieve now registers the missing managed-file row before responding, and
attributes ownership to the durable batch owner rather than the retrieving
caller, so output and error ids always come back as unified managed ids.
Fixes#33989
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion
* fix: refresh models_by_provider in place so captured references survive reloads
A connection test that redirects the destination already leaves the configured
credentials behind. It kept litellm_credential_name, which names the same stored
secrets and is resolved further down the call, so the reference is now dropped
with them. A request that sets no connection fields of its own is unaffected,
which is how the Admin UI tests a configured model.
The proxy-wide opt-in that already governs callers supplying their own
connection parameters now also governs whether a connection test may pair a
request-supplied endpoint with the configured deployment's credentials. Off by
default, which keeps configured credentials scoped to the endpoint the
configuration names; on, the previous merge behaviour is available unchanged.
A request that supplies its own connection fields describes a connection of its
own, so the configured deployment's credentials are no longer merged underneath
it. Anything the request leaves unset still comes from the configuration, so
naming a configured model and testing it as configured is unchanged, and adding
a second deployment for an already-configured name works as before.
Replaces the earlier outright rejection, which also refused requests that
supplied a complete connection of their own.
When a connection test names a model that resolves to a configured deployment,
that deployment's routing and credential parameters are authoritative. A request
supplying a complete connection of its own is unaffected.
BREAKING CHANGE: /health/test_connection no longer lets a request replace the
routing or credential parameters of a configured model it names. Supply the full
connection parameters instead of naming a configured model.
Multipart callers express nested metadata as flat bracket-notation keys, which
reach the request-body check as literal keys rather than as a metadata dict.
The check now rebuilds them with the same helper the endpoints use, so both
encodings are handled identically and cannot drift apart.
BREAKING CHANGE: a multipart field such as `litellm_metadata[api_base]` is now
subject to the same request-body parameter rules as its JSON equivalent. Set
`general_settings.allow_client_side_credentials`, or the deployment's
`configurable_clientside_auth_params`, to keep passing these.
The URL-destination check previously ran over request-body fields only. The
per-field logic moves into reject_url_valued_destination(field, value) so a
deployment name resolved from the request path runs the same check against the
same admin allowlist.
BREAKING CHANGE: a deployment name supplied in the request path that parses as
an http/https destination is now refused. Add the host to
`provider_url_destination_allowed_hosts` in litellm_settings to keep it working.
An SSE stream that cannot be positively identified as Anthropic (no
parseable message_start event) now blocks instead of passing through
unscanned, closing the bypass where any raw-SSE backend skipped tool
permission checks entirely. Buffered chunks are joined back into one
stream before parsing, so events split across network chunk boundaries
assemble correctly instead of being silently dropped. Rewrite mode now
resets finish_reason to stop when no tool call survives, so the
re-encoded Anthropic stream reports stop_reason end_turn and clients do
not wait for a tool result that never comes
`is_database_connection_error` answered True for any `PrismaError` it did not
recognize, on the reasoning that an unclassified failure might be an outage and
the safer default was to keep serving. That default is inverted for faults that
never resolve. A query engine that is missing or version-skewed, a malformed
generated query, or a misused transaction all satisfied the predicate, so with
`allow_requests_on_db_unavailable` enabled the proxy would absorb one, boot
clean, and keep issuing fallback identities for as long as the process ran.
The predicate is now an allowlist: the httpx transport errors, prisma's
`EngineConnectionError`, and a `no_db_connection` ProxyException. That is what a
real outage produces, since the query engine is a local HTTP server and an
unreachable database surfaces as a transport failure against it, so the
high-availability path is unchanged. Anything unrecognized is now treated as
permanent and surfaces instead of being absorbed.
Deciding whether to serve without a database and deciding what to tell the
caller are different questions, so they no longer share a predicate.
`is_database_infrastructure_error` keeps the previous broad behavior and now
backs the reporting and recovery paths: service-unavailable classification, the
access-group endpoint's status mapping, and the health watchdog's reconnect
trigger. Their behavior is unchanged. Without that split, a permanently faulted
engine would have started reporting as an authentication failure, sending an
operator after a credential problem that does not exist.
Guardrails silently skipped three surfaces on the Anthropic Messages
path, so an agent loop driven by /v1/messages ran unguarded:
- The Anthropic input translation never walked tool_result blocks, so
content returned by a local tool (a curl, a file read, an MCP call)
reached the model unscanned in both the string and list content
shapes, images inside a tool_result included.
- tool_permission only understood ModelResponse, so an Anthropic
non-streaming response or a raw SSE stream carrying tool_use blocks
passed through with no rule ever evaluated.
- ContentFilterGuardrail scanned inputs["texts"] but never
inputs["tool_calls"], so the arguments a model proposes for a tool
call went unchecked.
Tool call arguments are parsed as JSON before filtering so a MASK
action rewrites the value and leaves the payload valid JSON; non-JSON
arguments fall back to scanning the raw string. Denied tool_use blocks
are dropped from the Anthropic content array and replaced with a text
block, and stop_reason resets to end_turn when nothing tool-shaped
survives.
Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
A price data reload replaced litellm.model_cost wholesale, discarding every
runtime registration: the deployment model_info the Router registers from
model_list, and pricing overrides passed to litellm.register_model. Custom
model groups lost max_input_tokens / max_output_tokens in /model_group/info,
and a deployment whose backend model is in the catalog silently reverted to
upstream values. Runtime registrations are now recorded and replayed on top of
the freshly fetched catalog.
Router._pre_call_checks resolved the per-deployment model name only after the
model-info lookup, so an unregistered model left it unset and the supported
params check ran against the bare model group name, raising "LLM Provider NOT
provided" out of deployment selection. The name is now resolved first, and an
unresolvable provider skips that check rather than failing the request.
Resolves LIT-4675
The enterprise gate on the OAuth2 auth path raised a bare `ValueError`,
which the terminal handler in auth_exception_handler.py converts to a 401.
Every sibling enterprise gate answers 403, including `_premium_user_check`
and the SSO gate. A 401 tells the client its credential was wrong and to
retry with a better one, and no credential can satisfy that while the
install is unlicensed, so it invites a retry loop that can never succeed.
It now raises a 403 `ProxyException` shaped like the SSO gate. Two response
fields move with it: the `Authentication Error, ` prefix goes away, since
the catch-all built that around `str(e)` and a `ProxyException` is re-raised
unmodified, and `param` becomes `premium_user`, naming the condition an
operator has to clear.
The gate's own text also gains the sentence break it was missing. The
message concatenated straight onto `CommonProxyErrors.not_premium_user`,
rendering as "premium usersYou must be a LiteLLM Enterprise user".
