* fix(proxy): stop leaking internal exception details to clients
Public error responses could disclose internal details in two places.
A proxy-layer exception with no recognized provider status code (a bug
in a custom callback, a hook, or litellm's own code) forwarded its raw
str() text verbatim on a 5xx, including any embedded credential,
filesystem path, or internal hostname, or a full stack trace; the
same client-facing message now runs through a redaction layer built
on top of the credential redaction that already runs on log output,
so it also drops an embedded traceback and scrubs path-shaped and
hostname-shaped substrings. It intentionally never runs on server-side
logs, which must keep full detail for debugging.
exception_type(), litellm's core exception mapper, is shared by direct
SDK callers (litellm.completion()) and the proxy, and it deliberately
embeds a traceback into an unmapped exception's message as a debugging
aid for library users; a first pass at this fix stripped that
traceback inside exception_type() itself and broke that convention
(caught by tests asserting on the traceback frame). The traceback stays
in exception_type()'s own output; only the proxy's client-facing
response boundary (and the streaming response generator, which never
needs to embed one at all) strips it.
Full generic-message replacement for the unclassified-exception case
was tried first and reverted too: several routes deliberately raise a
bare exception as an informative, secret-free validation message (e.g.
the OCR endpoint's rejection of provider-native file IDs), and
replacing those wholesale broke that convention; targeted redaction
leaves them untouched.
Also stops the default uvicorn-based proxy from sending a Server
response header.
Resolves LIT-6747
* refactor(proxy): drop the unrelated error-message constant and trim redaction comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): honor HTTP(S)_PROXY / NO_PROXY when force_ipv4 uses the httpx transport
Passing an explicit transport to httpx.AsyncClient / httpx.Client disables its
automatic environment proxy mounts, so force_ipv4 on the httpx path sent every
LLM request direct and silently bypassed HTTPS_PROXY. Mount the same env-derived
proxy transports next to the IPv4-pinned direct transport in AsyncHTTPHandler,
HTTPHandler and the OpenAI async client factory.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): carry the client's TLS verify and cert settings onto env proxy mounts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The default password policy from #39381 (12+ chars with upper, lower, number, and special) makes globalSetup's POST /user/update with password "test" fail with a 400, so every role login in the UI e2e suite has been failing since that merge. Seed the roles, the noteam@test.local login, and the second-admin spec with policy-compliant passwords and update the scrypt hash in seed.sql to match.
* fix(ollama_chat): stamp finish_reason tool_calls when tool calls streamed before the done chunk
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ollama_chat): trim finish_reason override comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
* fix(proxy): mark session/SSO/SAML cookies Secure behind a TLS-terminating reverse proxy
litellm only sees a plain-HTTP hop when TLS terminates at a reverse proxy,
so cookie Secure attributes previously derived from (or defaulted without
regard to) the literal request scheme could be dropped in production. The
token session cookie set by every login path never carried Secure/HttpOnly/
SameSite at all.
Adds IPAddressUtils.is_request_https, a single trust-aware resolver used by
every cookie-setting call site: PROXY_BASE_URL, then X-Forwarded-Proto only
from a configured trusted proxy (general_settings.use_x_forwarded_for +
mcp_trusted_proxy_ranges), then the literal scheme. An unconfigured or
untrusted caller cannot spoof the header to force Secure on.
Resolves LIT-6748
* fix(proxy): make the shared session-cookie helper public, type new test helpers
set_session_token_cookie is imported across modules (ui_sso.py ->
proxy_server.py), so the leading underscore was misleading and breached
basedpyright's reportPrivateUsage budget with zero headroom. Also adds
missing parameter/return type annotations to the new test helper functions
per repo convention.
* refactor(python-bridge): split non-streaming bridge modules
* refactor(python-bridge): bring shared function tracing into route layer
* feat(dev): list Python route functions and call sites
* feat(dev): list Rust route functions and call sites
* docs(dev): record OCR parity gaps across Python and Rust
* feat(dev): list executed SDK calls with runtime tracing
* feat(dev): report Python vs Rust SDK pipeline steps in one CLI
* feat(dev): side-by-side pipeline step report in compare CLI
* fix(dev): drop invalid Final annotations in compare cell loop
* feat(dev): blue python-only and yellow rust-only steps in compare CLI
* feat(dev): vertical layout with section spacing in compare CLI
* fix(dev): validate SDK trace stages across sync and async routes
* refactor(rust): align SDK route call structure with Python
* refactor(python-bridge): share sync and async route call wrappers
* refactor(dev): split compare CLI into fixtures, runtime, and report modules
* fix(ci): run SDK trace tests and satisfy test lint
* test(proxy): verify NO_DOCS/NO_REDOC/NO_OPENAPI gate every doc surface
Confirms /docs, /redoc and /openapi.json are reachable by default and
each 404 with no schema in the body when NO_DOCS/NO_REDOC/NO_OPENAPI are
set, and that disabling them does not affect other routes.
* test(proxy): pin the production docs_url construction to the real helpers
Addresses Greptile 4/5 feedback: the disabled-path tests built a fresh
FastAPI() app, which could miss drift in the actual
FastAPI(docs_url=..., ...) call in proxy_server.py. This pins that
exact call site against _get_docs_url/_get_redoc_url/_get_openapi_url.
Coordination Redis (spend counters, budget-window enforcement, and the
reset_spend cache-eviction broadcast) previously only attached when a
deployment set general_settings.coordination_redis or
litellm_settings.cache. Bare REDIS_HOST/REDIS_PORT env vars alone did
nothing, so a multi-replica proxy with no cache block got no cross-pod
coordination at all: a key reset on one pod never cleared another
pod's stale budget enforcement.
The inferred Redis is pinged before being adopted, and a malformed
REDIS_CLUSTER_NODES/REDIS_SENTINEL_NODES value is tolerated too: env
vars can be set for an unrelated reason with nothing reachable there,
and guessing wrong must not turn a previously harmless in-memory-only
proxy into one that fails to boot or raises on its next cache write.
* fix(logging): redact credential query params from the uvicorn access log
Raw virtual keys reached container stdout two ways:
- `GET /key/info?key=sk-...`, `/global/spend/report?api_key=sk-...`,
`/key/spend/report`, `/spend/logs`, `/user/daily/activity` and the Gemini
passthrough routes all put the credential in the request target, and
`uvicorn.access` had no redaction filter (only `uvicorn.error` did).
- the key budget error interpolates `LiteLLM_VerificationToken.key_name`,
a column with no enforced shape, into a message that is both logged and
returned to the caller.
`SecretRedactionFilter` cannot be reused on an access logger: it collapses the
record into `record.msg` and clears `record.args`, and uvicorn's AccessFormatter
unpacks those args at emit time, so every access line would raise TypeError.
`AccessLogRedactionFilter` scrubs the positional args in place instead.
An access line is the one input to the secret regex an unauthenticated caller
controls end to end, so two bounds go with it. The request target is cut back to
a whole query parameter under 512 characters before it is scanned, since a half
parameter is too short to match its own pattern and would be logged raw, and the
dropped tail is not logged at all. The connection-string pattern is bounded too,
because its user half could previously re-scan the rest of the string from every
`://`: a 16 KB URL of `a://` pairs took 314s and now takes 0.12s, with the caps
set high enough that an RDS IAM auth token used as a DSN password still redacts.
Credential query params are terminated by `&` like the existing `key=` and
`sig=` patterns, so redacting one param no longer swallows the rest of the
request line, and a second credential in the same query string is now redacted
on its own instead of surviving once the first one stops the span. `key_name` is
echoed into the budget error only when it still has the masked `sk-...abcd`
shape `abbreviate_api_key` writes, so a value put there by a direct DB write or
a migration falls back to the key alias.
Also point the `/key/info` and spend-report examples at the sha256 hash both
endpoints already accept, so callers stop putting raw keys in URLs that
third-party access logs record.
Resolves LIT-5909
* test(logging): assert on emitted access lines instead of filter registration
The two registration tests checked that an AccessLogRedactionFilter instance
sits in uvicorn.access.filters, which is the shape of the code rather than its
behavior. Handing the logger a real access record and reading what a handler
wrote covers the same wiring and still fails when the registration is removed.
* fix(logging): redact percent-encoded credentials from access logs
?k%65y=sk%2D... is a working credential once the request parser decodes it,
but the redaction patterns match literal text and never see it. Decode the
request target as a detector and drop the query when decoding reveals a
secret. The decoded text is never logged back, so a %0A cannot forge a
following log line
Also accept any four non-space characters in the masked key_name check, since
abbreviate_api_key copies the last four characters of a custom key verbatim
and those can be punctuation or non-ASCII
* fix(auth): keep control codes out of the masked key label
/key/generate accepts a custom key ending in an escape sequence, and
abbreviate_api_key copies those four characters into key_name verbatim, so
the over-budget message carried them to a terminal and a log viewer. Bar
whitespace and C0/C1 control codes from the four, and keep everything else
The mcp_tool_search virtual tool only did substring token matching, so a native MCP client asking for "FX" could not find a tool described as "foreign exchange rates" even though the same catalog is ranked by embeddings on /responses and /chat/completions.
Adds litellm_settings.mcp_tool_search (embedding_model, top_k, similarity_threshold, core_tools). With an embedding model the caller's authorized catalog from _list_mcp_tools is ranked by cosine similarity of name plus description; configured core tools the caller can reach come first and do not consume top_k. Without an embedding model the keyword fallback keeps the old behavior. Settings are hot-reloadable from the DB, exposed on /get and /update mcp_tool_search_settings, and editable from the Admin UI under MCP Servers > Tool Search. The embedding index is shared with agent_search via a new SemanticTextIndex.
Resolves LIT-6751
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get_api_key had no callers. main.py imported it without using it, and
because main.py declares no __all__, the star import in __init__.py
published it as litellm.get_api_key. It duplicated key resolution that
get_llm_provider_logic already performs, which is how a misspelled env
var survived in it unnoticed until #35985. Drop the definition, the
unused import, the test that pinned the ai21 branch, and ratchet the
lint budgets down by the violations it carried.
Resolves LIT-5245
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Permanent Prisma/query-engine faults keep the 503 status and no_db_connection type but stop claiming the database is temporarily unreachable. A permanent fault anywhere in the exception chain outranks the transport error that surfaced it. MCP bridge and DCR flows gain a faulted resolution state with matching wording. Resolves LIT-5208
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
