* fix(ui): keep Virtual Keys list state in the URL so it survives leaving the page
The search term, sort, pagination and drawer filters lived in component
state, so navigating away from Virtual Keys and back reset the table to an
unfiltered first page. Move them into query state alongside the existing
?key= deep link, which also makes a filtered view shareable.
* fix(ui): namespace the Virtual Keys filter params and bound page inputs
The unprefixed team_id filter hijacked the /api-keys create-key deep link,
which already takes team_id as a prefill, so ?create=true&team_id=X silently
filtered the list underneath the modal. Prefix the four drawer filters.
Now that page and page_size come from the address bar, clamp them to what
/key/list accepts instead of forwarding 0, negatives or an int64-overflowing
page straight through, and trim filter values arriving from a URL the same
way the drawer already trims them.
* fix(ui): fall back to a sortable column when the URL names an unknown one
A hand-edited or stale sort_by reached /key/list, which 400s it, leaving the
Virtual Keys page on its loading skeleton with no error. Validate it against
the fields the table's own headers can produce, and clear sort_by rather than
blanking it when a sort is reset so the URL stays clean.
Also replaces a default-state URL assertion that ran before any query-state
write could land, so it could not fail for the regression it named.
* fix(ui): use TanStack's functionalUpdate instead of a hand-rolled updater resolver
The local helper narrowed typeof updater === "function" against an
unconstrained T, which TypeScript cannot do because T itself may be a function
type, so next build failed to type check. table-core already exports the same
helper.
* Fix hide-secrets guardrail: playground redaction, UI dropdown entry, spend-log telemetry
The hide-secrets guardrail never implemented apply_guardrail, so the UI test
playground echoed secrets verbatim; it was missing from the Add Guardrail
dropdown; and it recorded no guardrail_information, so Spend Logs could not
distinguish a redacted request from a clean one.
- implement apply_guardrail (unified interface) with use_native_lifecycle_hooks
so proxied traffic stays on async_pre_call_hook (per-key opt-out and
data["prompt"] handling live only there)
- record standard_logging_guardrail_information (allow/mask + masked_entity_count)
via _process_response/_process_error; opted-out keys and legacy nameless
callback instances record nothing
- advertise hide-secrets in /guardrails/ui/add_guardrail_settings (pre_call only)
and /guardrails/ui/provider_specific_params with a config model
Resolves LIT-3548
* Fix hide-secrets passthrough telemetry and JSON config input
* fix(guardrails): validate hide-secrets object config before submit
- apply_guardrail treats empty-string-only texts as no input, so no
false allow is recorded
- the UI object field keeps raw text while editing and blocks submission
until it parses to a JSON object, instead of posting a string to an
object-only API
- supported_modes_by_provider keeps its dict[str, list[str]] value type
* fix(guardrails): record no hide-secrets telemetry when nothing was inspected
walk_user_text and the prompt redaction now report how many non-empty
strings they visited; when neither inspected anything (image-only
content, empty strings), the run records no guardrail entry instead of
an 'allow' row that counts a check which never saw any text.
Two branches independently added embedding_executor to the same fake
search handler in this file, #39472 in the middle of the signature and
#39474 at the end. Neither conflicted with the other, so both edits
merged and the function ended up declaring the parameter twice.
Python rejects that at compile time, so the whole module fails to
import and every test in the file is uncollectable, taking the
logging_testing job down on staging.
Keep the earlier of the two, which sits where the real handler declares
the parameter.
* fix(sso): resolve multi-valued role claims to the highest privilege role
A role claim carrying several roles used to resolve to whichever one the IdP
listed first, so a user holding both proxy_admin_viewer and internal_user lost
org-level spend visibility depending on claim ordering alone.
get_litellm_user_role now picks the highest privilege role out of a list-valued
claim, and the Entra app_roles path shares that same resolution instead of
keeping its own copy of the hierarchy. SAML assertions carrying several role
values go through the same path rather than taking the first value.
* test(sso): lock ranked-over-unranked role resolution for mixed claims
org_admin, team and customer sit outside the privilege ladder. Pin the
resolution for a claim that mixes one of them with a ranked role so the
asymmetry is covered rather than implicit.
* fix(sso): label the claim-sequence cast for the type-discipline gate
* fix(sso): resolve claim entries without recursing
The repo's recursive-function gate rejects self-recursion here, and a role
claim is flat anyway. Pull the single-value lookup into its own helper so the
list branch maps over it instead of calling back into itself.
* fix(ui): stop the create team form resetting organization and models
The organization preselect ran in an effect keyed on the organizations
query, so any refetch of that list while the Create Team modal was open
overwrote the user's organization pick, which in turn cleared their
models pick. The models field was also cleared whenever the available
models fetch resolved.
Preselect the organization when the modal opens instead, and clear the
models only when the user picks a different organization. An org admin
whose admin orgs narrow to one while the form is open can still pick,
rather than facing a locked empty field.
* fix(ui): block team create when the picked organization is no longer available
An organization picked in the Create Team form now survives a refetch of
the organization list, so it can outlive the admin's access to it. Refuse
the create with a message on the field rather than letting the request
fail authorization at the proxy.
* fix(ui): keep the team create organization field usable when the pick goes stale
Locking the field on a single admin organization also locked it while it
held a rejected organization, so an admin who lost access could not pick
the one organization left. Lock it only while it holds that organization.
* test(ui): hoist the created team fixture out of the mock call
The inline object pushed the repo past its no-large-inline-object-arg
lint budget, which has no headroom.
The Router executor only routed a query embedding when the vector store
carried extra embedding configuration, so a store registered with no
embedding model at all always went to the Router and 500'd on the
s3_vectors default text-embedding-3-small when no deployment served it.
Route on whether the Router serves the model, which is the rule the
executor had before, and keep the request metadata on the SDK fallback so
the embedding stays attributed either way.
The autoRouterPresets mock imported litellm/proxy/public_endpoints/autorouter_presets.json
as a module. That path sits outside ui/litellm-dashboard, the only directory the UI
Dockerfile copies, so `next build` type-checking inside the image failed with
"Cannot find module" and the ui-image job went red on every PR that touched an
image-scan path. Read the file with fs at runtime instead; vitest still derives
expectations from the real bundled catalog.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HqEPCNLDrssxsuezhAaL4j
Adds chat_completions and transcription as SDK function columns, backed
by the existing rust_bridge test files. Adds a fourth strategy folder,
existing_e2e_test_sdk, that points at already-existing live-API SDK
tests (tests/ocr_tests/ as a whole folder, plus chat completion and
Whisper transcription tests) instead of writing new parity tests.
Extends selector_matches_node with trailing-slash folder selectors so
a whole test folder can back one matrix cell.
* feat(datadog_llm_obs): cost tag dimensions, router decision fields, reasoning token metric, redaction gating
* test(datadog_llm_obs): satisfy test quality gate
* fix: forward integer parent_id as its string form
* fix(datadog): sanitize redacted message roles
* fix(datadog): keep the A2A agent role on redacted spans
* fix(datadog): merge current staging budget
* style(datadog): format redaction tests
* fix(datadog): handle malformed redacted roles
* test(datadog): put the test quality suppression on the reported line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A session pinned to a text-only model failed every image turn with a provider 400, because the modality gate exempts a kept pin by cause. Add modality_pin_override so that exemption is conditional: the image turn is re-placed on a capable model for that request only, reported as cause modality_pin_override, and the stored pin is left untouched so the next text turn replays it.
The pin write on the replay path already happens upstream of the gate and stores the session's own model, so pin survival is structural rather than bookkeeping. The new cause joins the non-pinnable set. Default off at every layer.
S3 Vectors now subclasses BaseQueryEmbeddingVectorStoreConfig, so its query
embedding runs through the Router executor with the request metadata instead
of a private router lookup. embedding_model stays accepted as an alias of
litellm_embedding_model. The router kwarg is gone from the search handler and
every provider transform now that nothing but the executor fallback read it.
* test: add OCR python-to-rust test parity ledger
* feat: add ledger loader for OCR test parity data
* feat: add drift audit for OCR test parity ledger (WIP, untested)
* fix: correct drift in OCR test-parity ledger
Two entries referenced a typo'd Python test name, four duplicated entries already tracked under TestProxySecurityGuard, five real Python tests in test_rust_bridge.py were untracked, and three real Rust custom_logger tests were missing from rust_only_tests. Found by running validate_ledger.py's audit against the live repo.
* test: add regression coverage for the OCR ledger and audit script
Covers schema validation, AST/regex test enumeration, drift detection on both the Python and Rust sides, LedgerDriftError content, and a live-repo clean-audit guard against future drift.
* test: simplify ledger test to one drift-guard assertion
Replace the ledger-internals unit tests with a single test that runs the real audit against the live repo and asserts every OCR test is accounted for (mapped, unmapped-with-reason, or rust_only), printing the exact diff on failure.
* chore: move OCR test-parity ledger to core/ocr
validate_sub_methods/ mixes strategy-catalog metadata with the ledger. Ledger data belongs under a per-function core/<function>/ path instead.
* fix: point LEDGER_PATH at the new core/ocr location
* fix(model_armor): handle Anthropic Messages and Responses streams in post_call
The post_call streaming hook buffered every chunk and fed it to
stream_chunk_builder, which only understands chat-completion deltas.
/v1/messages streams raw Anthropic SSE bytes and /v1/responses streams
typed Responses events, so both raised litellm.APIError and surfaced to
the client as a 500 on every streamed request.
Assemble each surface with its own reader, frame guardrail failures as
terminal items in that surface's wire format, and pass the stream
through unscanned when it cannot be assembled instead of raising.
* fix(model_armor): classify the stream surface and fail closed when it cannot be assembled
Decide the wire format explicitly instead of inferring it from a boolean pair, so an
opaque raw SSE stream (the Google :streamGenerateContent route) is never refused in
Anthropic framing, and a stream that cannot be assembled is blocked rather than
released unscanned unless fail_on_error is disabled.
Also scan Responses tool-call arguments, read the body only off a terminal Responses
event, and record the applied guardrail on the fail-closed path.
* test(model_armor): pin the error-only stream predicate against content-carrying streams
is_sse_error_stream decides whether a buffered stream is forwarded to the client
untouched, so a stream that still carries content must not qualify: the frames-only
join drops typed chunks, an empty stream is not a refusal, and a content event may
carry an empty error field.
* fix(model_armor): let a streamed de-identify match mask instead of blocking
A de-identify template reports MATCH_FOUND for every redaction it makes. The
streaming block check omitted allow_sanitization, so with mask_response_content
enabled that match read as a refusal and the client got a 400 where the
non-streaming sibling returned the redacted text. Pass the flag through, as the
non-streaming hook already does, and stamp the logged status from the same
decision so the spend row agrees with what the client received.
Also drop Any from the chat-completion assembler's parameter; stream_chunk_builder
takes a bare list, so list[object] carries the mutability requirement without
erasing the element type.
* fix(model_armor): fail closed when a streamed de-identify match cannot be applied
Allowing sanitization past the streaming block check is a promise to apply the
redaction Model Armor asked for. Two paths broke that promise and released the
buffered original instead: a match that comes back with no sanitized text, and a
surface with no assembled body to rewrite.
The outcome is now resolved once, before it is recorded, so the status stamped on
request metadata agrees with what the client receives rather than reporting the
success the block check alone would have implied.
* fix: scan the deltas when a Responses stream ends without a body
response.failed and response.incomplete are terminal events like
response.completed, but a turn that broke mid-generation reports an empty
output while the deltas ahead of it already spelled the answer out to the
client. Reading only the terminal body found nothing to scan there, and the
empty-content shortcut then forwarded every buffered delta past the guardrail.
Fall back to the text the delta events carry whenever a Responses stream
assembles to nothing.
* fix: read the Responses delta event types off the event enum
The hand-listed set left out response.mcp_call_arguments.delta, so a turn that
streamed only MCP tool arguments and then reported an empty body still took the
no-content shortcut and forwarded those chunks unscanned.
Deriving the set from ResponsesAPIStreamEvents keeps it complete as the enum
grows, and the str guard in the reader already covers any event whose delta is
not text.
* fix(model_armor): scan responses deltas alongside the terminal body
A /v1/responses stream spells out reasoning summaries and tool-call arguments in
delta events that its terminal body never repeats, so scanning the body alone
handed every summary delta to the client unscanned whenever the body carried text.
* fix(model_armor): scan responses delta fields apart from each other
A Responses turn spells out its reasoning summary, its visible answer and its tool-call
arguments in separate delta events. Joining every delta into one string let a finding form
across the boundary between two fields that each carry nothing to find, so a safe stream
could be blocked. Group the deltas by the field they belong to, join a field's own deltas
as they streamed, and keep the fields apart.
* fix(model_armor): scan each responses field once, not twice
Separating delta fields stopped the terminal body from matching the delta text, so a turn
with two visible fields sent Model Armor both copies. Only the delta fields the body does not
already carry are appended now.
---------
Co-authored-by: yassin <yassin@berri.ai>
* feat(azure): support credential chain for storage
* test(azure): clarify credential seam suppressions
* fix(azure): read chain tokens in a worker thread
The credential chain walk (IMDS probe, CLI subprocess) is blocking I/O,
so reading the provider inline in async set_valid_azure_ad_token stalls
every request on the worker's event loop
* fix(proxy): keep SpendLogs and callback session ids in sync when the request has none
Add general_settings.missing_session_id (generate | reject). In generate mode one id is
stamped into litellm_session_id, litellm_trace_id and metadata.session_id before callbacks
run, so LiteLLM_SpendLogs.session_id and the Langfuse session id match. In reject mode such
requests get a 400. Unset keeps the legacy behavior. MCP routes are not affected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate schema.d.ts and shorten mutable-ok comment for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark generated session ids so affinity consumers do not pin on them
Fireworks x-session-affinity, the router session_affinity pre-call check and the
complexity router session pin all read metadata.session_id as a caller-chosen
stable key. A missing_session_id: generate id is fresh per request, so it now
carries metadata.litellm_session_id_generated and those consumers skip it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): skip streaming guardrail rounds that re-scan cleared output
Streaming guardrails scanned the finished answer twice at end of stream
whenever the chunk count landed on a multiple of the sampling rate, ran
sampled rounds whose payload was identical to the previous one, and on
/v1/messages could scan an empty text before the first content chunk.
Every redundant round is a paid guardrail provider call.
Each endpoint handler now exposes a scan key describing what a round
would hand to apply_guardrail (the text so far, plus tool calls once the
stream has ended), and the unified streaming hook skips a sampled or
end-of-stream round whose key equals the last scanned one or carries
nothing to scan yet. Rounds that carry tool calls are never skipped.
* test(guardrails): expect one end-of-stream scan when the terminal chunk is sampled
Update sampled cadence expectations and use tuple-backed scan state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The dashboard's template picker imported autorouter_presets.json at build time, so every
catalog change needed a dashboard rebuild and artifacts refresh. The catalog now lives in
litellm/proxy/public_endpoints/ and GET /public/autorouter_presets serves it, fetching
litellm.autorouter_presets_url (GitHub raw on main, 1h in-process cache, bundled fallback)
so a merged catalog change propagates to running proxies like the model cost map does.
The dashboard fetches it at runtime via useAutoRouterPresets and keeps no local copy.
Resolves LIT-6764
* fix(ui): paginate request logs by session groups server-side
The logs table server-paginated raw spend logs and then collapsed
multi-call sessions client-side, so a page could render 3 rows while
the footer claimed 25 and sessions straddled pages. Adds an opt-in
group_by_session param to /spend/logs/ui that pages and counts one
representative row per session (DISTINCT ON, newest non-MCP call),
keeps the bounded count contract, enriches whole-session llm/agent
composition counts, and deletes the client-side collapse pipeline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* feat(ui): add a 10 rows-per-page option and default request logs to it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): key session aggregates per api key in the logs enrichment
Grouped pagination splits a reused session id into one row per api key,
but the enrichment still aggregated by session_id alone, so both rows
showed combined spend and counts. The aggregate query now groups by
(session_id, api_key), the count folds into it (the separate group_by
query is deleted), and each row reads its own key's totals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): treat an empty api_key as a real session group value
The spend-log schema defaults api_key to an empty string; truthiness
guards in the enrichment treated it as missing, so keyless multi-call
sessions lost their count and spend. Only None means missing now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
get_or_compute single-flights concurrent misses under a per-key lock, but
invalidate() deletes outside it. A mint already awaiting the IdP when an
invalidation ran wrote its result into the slot afterwards, so a bearer minted
before the invalidation was served after it for its full TTL; the upstream-401
retry then re-presented the bearer the server had just rejected.
The cache now keeps a per-key generation beside the single-flight lock.
invalidate() bumps it and get_or_compute snapshots it inside the lock right
before minting, skipping the cache write when it no longer matches. The token
is still returned to the caller it was minted for, so only the caching is
fenced. Generation and lock share one object that a running mint binds to a
local, which pins the weak map's entry for exactly as long as a write can still
land; a generation held weakly on its own would be collected across the await
and the re-check would silently pass.
* fix(responses): keep provider response headers in streaming logging callbacks
The responses streaming iterator captures the provider's HTTP response headers into
its own _hidden_params, but never puts them on the completed response, and the
model_validate(model_dump()) copy made for logging drops pydantic private attributes.
Success callbacks and StandardLoggingPayload.hidden_params.additional_headers therefore
saw an empty dict for streaming /v1/responses, so Azure's apim-request-id was unreadable
from the callback payload.
Restore the headers on the nested response of the logging copy, preferring any the
provider transform already set (the fake_stream path) and falling back to the ones the
iterator captured from the stream. Skipped when the copy fell back to the original event,
so a serialization failure never leaves logging-only state on the caller's object.
* fix: satisfy LIT002 mutable-collection gate in header restore
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
The pre-flight token exchange resolved its target from the requested path, so a key not entitled to that server still drove an outbound exchange at the IdP and populated the credential cache before the later access check denied it. Resolve the target through the same allowed-server set the rest of the route uses and skip the exchange when the requested server is not in it.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): forward mode and streaming params to crowdstrike_aidr handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): drop stream_chunk_builder patch from crowdstrike cadence test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type test params and cover unsupported crowdstrike mode rejection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop leaking internal exception details to clients
Public error responses could disclose internal details in two places.
A proxy-layer exception with no recognized provider status code (a bug
in a custom callback, a hook, or litellm's own code) forwarded its raw
str() text verbatim on a 5xx, including any embedded credential,
filesystem path, or internal hostname, or a full stack trace; the
same client-facing message now runs through a redaction layer built
on top of the credential redaction that already runs on log output,
so it also drops an embedded traceback and scrubs path-shaped and
hostname-shaped substrings. It intentionally never runs on server-side
logs, which must keep full detail for debugging.
exception_type(), litellm's core exception mapper, is shared by direct
SDK callers (litellm.completion()) and the proxy, and it deliberately
embeds a traceback into an unmapped exception's message as a debugging
aid for library users; a first pass at this fix stripped that
traceback inside exception_type() itself and broke that convention
(caught by tests asserting on the traceback frame). The traceback stays
in exception_type()'s own output; only the proxy's client-facing
response boundary (and the streaming response generator, which never
needs to embed one at all) strips it.
Full generic-message replacement for the unclassified-exception case
was tried first and reverted too: several routes deliberately raise a
bare exception as an informative, secret-free validation message (e.g.
the OCR endpoint's rejection of provider-native file IDs), and
replacing those wholesale broke that convention; targeted redaction
leaves them untouched.
Also stops the default uvicorn-based proxy from sending a Server
response header.
Resolves LIT-6747
* refactor(proxy): drop the unrelated error-message constant and trim redaction comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): honor HTTP(S)_PROXY / NO_PROXY when force_ipv4 uses the httpx transport
Passing an explicit transport to httpx.AsyncClient / httpx.Client disables its
automatic environment proxy mounts, so force_ipv4 on the httpx path sent every
LLM request direct and silently bypassed HTTPS_PROXY. Mount the same env-derived
proxy transports next to the IPv4-pinned direct transport in AsyncHTTPHandler,
HTTPHandler and the OpenAI async client factory.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): carry the client's TLS verify and cert settings onto env proxy mounts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
