A file uploaded through Router.acreate_file lands in the account of the
deployment that stored it, so a cross-group fallback silently stores the
file with the wrong provider and every later batch or fine-tuning call
against the returned id permanently fails. Extend the provider-scoped
fallback pin that already covers input_file_id and training_file to file
creation, so the original provider error surfaces instead.
litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
The aggregated daily-activity queries build their WHERE clause as raw SQL, and
an empty entity list rendered as `"team_id" IN ()`, which Postgres rejects with
`syntax error at or near ")"`. Callers reach that state normally: a caller
without admin view and no explicit team_ids has its scope resolved to the teams
it belongs to, so anyone belonging to no teams, an org admin for instance, sent
an empty list and got a 500 back from /team/daily/activity/aggregated.
The paginated endpoint hands the same empty list to Prisma, which renders
`in: []` and matches nothing, so it kept returning 200 with an empty result set.
Emit FALSE for the empty case so the raw-SQL path lands on the same answer,
mirroring what the api_key filter a few lines below already does.
The fix covers both aggregated queries at once because they share one WHERE
clause builder.
true_passthrough and oauth_delegate forward the caller's own bearer and mint
nothing, so their sessions consume no discovered OAuth endpoint. The discovery
completeness gate still failed them closed with a 503 raised before the upstream
was ever contacted, which the tools/list fan-out swallowed into HTTP 200 with an
empty tool list. Any upstream that publishes no RFC 9728 metadata, including
every OpenAPI-backed internal API, was permanently unusable.
A failed discovery is now fatal only to servers whose OAuth flow the gateway
runs itself. Discovery still runs for the forwarding modes, so /authorize,
/token and /register keep resolving their endpoints, and each keeps its own 400
when they are missing.
MCPServer.is_client_forwarded_token now owns the mode pair that five call sites
spelled inline, one of which had already named it is_client_forwarded_mode.
* fix(logging): close three secret-leak paths in verbose logging
The AWS credential pattern was the only key-name matcher in secret_redaction
that skipped optional quotes, so quoted dict-repr values leaked. Fold the three
AWS key names into the shared key-name alternation instead.
SecretRedactionFilter only scrubs str record attributes, so a dict/list/set
passed through extra={...} reached the formatter unredacted. Redact at the
formatter boundary so no value shape can bypass it.
log_raw_request_response wrote the request curl command to metadata["raw_request"]
unredacted, returned an unmasked raw_request_api_base, and fell back to dumping
model_call_details whenever api_base was empty.
* Update litellm/_logging.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(logging): redact JSON log values without breaking the document
JsonFormatter redacted the serialized JSON, so a secret-named member
collapsed from '"api_key": "sk-..."' to a bare REDACTED token and the
line stopped parsing as JSON.
Redact before serialization instead: safe_dumps takes an optional
value_transform hook (default None, so all other callers are unchanged)
and redact_structured_value collapses only the value, leaving the key
and surrounding structure intact.
JsonFormatter now emits "api_key": "REDACTED" where the formatter unit
test expected the already-masked "sk**********". That test bypasses
SecretRedactionFilter, which in production collapses the pair before any
formatter runs, so the assertion is updated to match real behavior.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(vector_stores): stop leaking stored credentials in direct search debug logs
Direct vector store providers (RESP datastores like Valkey) have no HTTP
request to echo, so both search handlers called `logging_obj.pre_call` with
no `api_base`. The logging helper treats an empty `api_base` as "nothing to
render" and falls back to `str(self.model_call_details)`, which carries the
resolved `litellm_params`: the stored `valkey_password` and the embedding
config's `api_key` among them.
The stdout logger's regex redaction hid this, but `pre_call` also writes the
same string to `litellm_params["metadata"]["raw_request"]`, which ships
unredacted to every logging callback (Langfuse, OTel, etc.).
Pass a synthetic `<provider>://<vector_store_id>` endpoint plus an explicit
`request_str` so the debug output describes the call instead of dumping call
details, and fold the duplicated sync/async blocks into one helper so the
sanitized descriptor cannot drift between them.
* fix(vector_stores): type direct search query as Sequence[str]
The new helper's list[str] annotation pushed LIT001 over its
type-discipline ceiling. Sequence is the read-only shape the helper
actually needs, and list[str] still satisfies it at both call sites.
* feat(bedrock): add a config toggle to disable agent-runtime pass-through
The /bedrock pass-through dispatches agents, knowledge bases, flows, rerank,
retrieveAndGenerate, generateQuery and optimize-prompt to bedrock-agent-runtime,
so an operator who only wants to expose model invoke and converse has no way to
narrow that surface
Adds general_settings.disable_bedrock_agent_runtime_passthrough. When set, those
routes are rejected with a 403 before credentials are fetched or the request is
signed. Plain bedrock-runtime model pass-through is unaffected, and the setting
defaults to off, so existing deployments behave exactly as before
The branch is inverted to an early return for the non-agent-runtime case so the
toggle can reject outright instead of falling through to model extraction, which
would surface a confusing 400 about an unparseable model
* style(bedrock): drop redundant docstrings from the agent-runtime toggle
Both surfaces wrote zero bytes for the whole time-to-first-token, so an
intermediary with an idle read timeout drops a healthy connection before the
first token. They reached neither keepalive engine, which is what #37322 left
open.
The streaming assistants run spends that wait inside the awaited call that
produces its response, since create_response buffers the first chunk, so it
takes the same open_sse_before_first_byte seam the native routes use. The A2A
route only contacts the upstream agent once its body iterator is first pulled,
so nothing is awaited before the response exists and the gap has to be filled
from inside the stream instead; wrap_sse_stream_with_keepalive_pings already
does that and now takes the filler as a parameter, so A2A gets an SSE comment
its JSON-RPC clients discard rather than Anthropic's ping event.
Off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds.
* fix(otel): bound and shut down credential-scoped tracer providers
Each credential-scoped TracerProvider owns a BatchSpanProcessor worker thread that
only stops on shutdown, and the v1 cache holding them was an unbounded, unsynchronized
dict that never shut anything down. Every distinct team/key credential set therefore
added a thread for the life of the process, and concurrent first-requests for the same
credential set orphaned duplicate providers outright.
Make the cache a lock-guarded bounded LRU that shuts down whatever it drops, matching
the v2 TenantTracerCache. Providers wrapping a caller-supplied SpanExporter instance
share that exporter with the logger's own provider, so they are dropped without
shutdown; those use SimpleSpanProcessor and own no thread.
* fix(otel): reclaim dropped providers on a dedicated executor
Sustained credential churn queues one blocking shutdown per eviction, so using the
shared logging executor let an unreachable tenant endpoint stall unrelated logging
work behind the OTLP retry budget. Give provider shutdown its own bounded pool; its
threads spawn lazily, so a proxy that never evicts still pays nothing.
* fix(otel): decide provider shutdown from the victim, not the evicting request
Both dynamic entry points share one provider cache, so it can hold providers of
mixed exporter ownership. Reading the ownership flag from the evicting request
therefore stopped a shared caller-supplied exporter in one direction, silencing
telemetry process-wide, and leaked a BatchSpanProcessor thread in the other.
Cache ownership alongside the provider so the drop decision reads the victim's
own flag.
* fix(otel): honor the widened header mapping type instead of dict only
Widening the header parameter to Mapping left the isinstance check on dict, so a
non-dict Mapping silently returned no headers at all, which for the OTLP path means
an unauthenticated exporter and no traces with nothing raised. The dict branch also
returned the caller's own object, and dropping the defensive copy at the call site
let that alias reach a long-lived exporter. Match on Mapping and copy.
* fix(otel): do not give a provider we may never stop an interpreter-exit hook
Every TracerProvider registers an atexit hook by default, and that hook holds a strong
reference. Providers wrapping a caller-supplied exporter are dropped without shutdown,
so they stayed pinned for the life of the process and then stopped the shared exporter
at exit. Tie shutdown_on_exit to ownership: those providers use SimpleSpanProcessor and
buffer nothing, so they lose no flush, while providers that own their exporter keep the
hook and their exit flush.
Also stop the victim the eviction test leaves behind, and trim the added comments.
Key and team router_settings set enable_tag_filtering on the request kwargs,
and get_deployments_for_tag already treats that as authoritative, but
_select_pre_routing_strategy only consulted the router-wide flag, so tagged
auto-router markers still captured untagged requests from keys that enabled
filtering. The e2e auto-router module now enables tag filtering through
key-level router_settings instead of flipping /config/update module-wide,
which was denying concurrently running tagged requests from other suites on
the shared per-build CI proxy.
A terminal HTTP failure partway through chunking now logs the summed usage
and cost of the ApplyGuardrail calls AWS already billed, mirroring the
blocked-chunk path.
Removes one named callback from a team and leaves the team's other callbacks
registered and firing. Before this, the only removal route was
POST /team/{team_id}/disable_logging, which clears every callback at once, so a
tenant sharing a team could not deregister its own integration
The handler filters metadata["logging"], keeps the survivors encrypted, refreshes
the cached team so the removal applies to keys that are already live, and emits a
redacted audit row, matching what the add and disable routes do
Resolves LIT-5161
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
- scale batch output-token reservations by the row's n / best_of candidate count
- parse client-supplied output caps defensively instead of 500ing on unparseable values
- exclude project IO descriptors from the first should_rate_limit pass when TPM
reservation is disabled so their buckets are not double-charged
An internal user who administers an organization saw an empty
Organization Usage dashboard and had to be promoted to proxy admin to
see any of it.
Two independent gates were closed on them. The route layer rejected
GET /organization/daily/activity with 401 before the handler ran, since
the route belonged to no list a non-proxy-admin can reach, and the
handler's own org-admin scoping was therefore dead code. In the
dashboard, viewOrganizationUsage was granted by session role alone, and
an org admin's session role is internal_user, so the Organization Usage
option never rendered and its data fetch stayed disabled.
The route now sits in self_managed_routes, where the handler restricts
results to organizations the caller is ORG_ADMIN of and 403s on any
other org, and viewOrganizationUsage joins the existing per-capability
org-admin allowance that already covers viewDeletedTeams.
A caller who administers no organization resolves to an empty id list
rather than to None, so the organization-alias lookup is scoped by that
same list instead of reading the whole table.
The Usage page falls back to the global view when org-admin membership
is revoked while it is open, so the selector never keeps a value it no
longer offers.
A model with a long time-to-first-token leaves the proxy's response completely
idle, so any hop with an idle read timeout (AWS ALB and nginx both default to
60s) drops a connection that is perfectly healthy and would have delivered its
tokens shortly after.
The keepalive engines LiteLLM already ships wrap the response object, so they
fill a gap once the upstream has answered and then gone quiet. They cannot fill
the gap before it answers at all, and that is where the whole wait is spent:
measured against api.openai.com/v1/chat/completions with gpt-5.6 at
reasoning_effort high, the response headers and the first body byte both arrive
at 37.90s. Nothing has entered the ASGI response phase by then.
The upstream call is now raced against the keepalive interval, and when it
loses, the SSE response is opened immediately and ": ping" comments, which every
conformant SSE client ignores, fill the wire until the real response is ready to
be replayed onto it. One seam per funnel: base_process_llm_request covers every
native route, create_pass_through_route covers every passthrough route.
Committing the status line that early is the cost. A failure discovered after
the first ping reaches the client as an SSE error frame under a 200 rather than
as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not
yet known. keepalive_ping_has_fired already documents the same trade-off for the
existing engines. Both are why this stays off until an operator sets
litellm_settings.sse_keepalive_ping_interval_seconds.
Separately, the passthrough relay reached neither engine even for mid-stream
gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same
treatment, gated on the upstream declaring text/event-stream and only emitted
between complete frames so a binary transport (AWS event streams on /bedrock)
and a stall halfway through a frame are both left alone.
Fixes#34819
On /v1/messages, AnthropicStreamWrapper synthesizes the content_block_start for
the first content block, queues it, then hits a bare `continue` when that same
upstream chunk's translated delta is empty. The queue is only drained at the top
of the next __next__ / __anext__, so the queued content_block_start waits for a
further upstream chunk to arrive.
An empty delta on the opening chunk is the normal tool-call shape: Bedrock
Converse's contentBlockStart carries the tool id and name with no arguments, and
OpenAI-format streams send arguments: "" on the chunk that names the function.
So a client learns a tool call started one upstream event late, and when the
provider delivers argument fragments as a trailing burst it sees nothing at all
after message_start for the whole generation.
Flush the queued event before continuing, in both the sync and async paths. The
sibling block-transition path already returns from the queue, so only the
first-block-open case changed.
The Anthropic messages endpoint's exception handler passed the raw
request body dict to the failure hook, but request setup had already
replaced the processor's dict with one carrying the logging object, so
failure rows for /v1/messages never lifted recovered or estimated usage.
Pass the processor's dict instead.
The input-side estimate only counted the messages list, missing the
Anthropic top-level system prompt (string or text-block list) and the
Responses API instructions field, which live in optional_params. Count
them too.
