Without the auto_router feature in the signed enterprise license a proxy may hold
one complexity router with classifier_type heuristic_v2 across config.yaml and the
DB; with it the limit is lifted. The ceiling is derived once from LicenseCheck and
handed to the Router, which refuses the extra router at registration. config.yaml
over the limit refuses to start, and /model/new, /model/update and
PATCH /model/{id}/update refuse the write with a 403 before touching the DB.
Expiry follows the existing max_users/max_teams pattern: judged when the
license is verified, not on every call, and a verify that rejects the license
(expired or unreadable) leaves no signed payload behind. The rollback after a
failed upsert re-admits state that was already serving, so it is exempt from the
ceiling: an edit that fails, including one refused by a ceiling that has since
tightened, leaves the router serving its previous configuration.
A write that leaves a row on heuristic_v2 under a limited license runs in one
transaction that takes a Postgres advisory lock before counting the DB rows plus
this proxy's config.yaml routers, so concurrent writes on any pod cannot both
claim the sole slot and no surplus row is ever persisted.
Only the row insert runs under that lock: the team model bookkeeping, which
needs a second pool connection, runs after the transaction has committed.
PATCH /model/{id}/update follows the same order as create: the row is written
through the slot first and the team's model list is updated only afterwards, so
a refused write leaves the team as it was.
The slot transaction bypasses the repository's publish-on-write, so it
publishes the config change once after commit, as delete_team_models does.
* fix(vector_stores): only list vector stores the caller was granted
/vector_store/list returned every managed vector store with no team_id to any
key, and let a dashboard session see stores created from the dashboard because
every session shares the litellm-dashboard team id. Non-admin listings now show
a store only when the key or one of the caller's real teams is allowlisted for
it via object_permission.vector_stores, or the team owns it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vector_stores): keep a dashboard session key's own grants when the user has no teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: stop a cleared Organization field from failing key creation
Clearing the Organization combobox in the Create Key modal left organization_id set to an empty string, so /key/generate looked up an organization named "" and failed with "Organization doesn't exist in db. Organization=".
OrganizationDropdown now emits null on clear, and GenerateKeyRequest normalizes an empty organization_id or project_id to None the same way it already does for team_id.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop customer-specific docstring from key request normalization test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* fix(spend_tracking): leave SpendLogs.session_id null when no client session id was established
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): ratchet basedpyright budget after session_id fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): ignore trace ids as session ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): gate null SpendLogs.session_id behind missing_session_id: omit
Unset, generate and reject keep the legacy trace id fallback. omit records only
metadata.session_id, the key Langfuse reads, so a trace id copied into
litellm_session_id by get_litellm_params never becomes a session.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): stamp the omit decision on the request so a config reload cannot fabricate a session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): keep omit covering requests the pre-call stamp never reaches
Router-model provider pass-through calls allm_passthrough_route directly and
skips add_litellm_data_to_request, so those requests never run the pre-call
helper and carry no omit stamp. Reading only the stamp made POST
/anthropic/v1/messages write a fabricated uuid into SpendLogs.session_id under
missing_session_id: omit while its Langfuse trace had no session, the exact
divergence the policy exists to remove.
The stamp now only pins omit on, and an unstamped request falls back to the
configured policy, so a config reload still cannot fabricate a session for a
request that was decided pre-call.
* fix(spend_tracking): make the session-omission marker proxy-owned so clients cannot forge it
* fix(spend_tracking): strip the client-sent omission marker from both metadata buckets before they merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): strip the session-omission marker from both metadata buckets
The pre-call policy ran before litellm_metadata is merged into metadata, so a
client that planted the marker in litellm_metadata had it copied back into the
route's own bucket after the strip and still got a null SpendLogs.session_id.
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): return persisted team memberships from /user/new
new_user attached default teams after building its response from the
pre-membership snapshot, so NewUserResponse.teams was always empty for
users created with default_internal_user_params.teams. The CLI SSO flow
reads that response on a user's first login and minted a teamless JWT,
which skipped the default team's model allowlist.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): return team ids as a tuple to satisfy LIT001
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The logs table already rolled up cost per session but the Tokens column only showed the representative call's usage. The per-session aggregate query now also sums prompt, completion and total tokens, and the Tokens cell switches to those sums for multi-call sessions the same way the Cost cell does.
Claude-Session: https://claude.ai/code/session_01CNasFqyjnLN3Rqman25vde
* fix(proxy/db): translate libpq sslrootcert and verify-* into Prisma's strict TLS params
Prisma silently drops sslrootcert and treats sslmode=verify-ca/verify-full as
prefer, so a DATABASE_URL copied from the RDS docs connected over TLS without
checking the server certificate. The URL handed to Prisma (writer, DIRECT_URL,
read replica, componentized entrypoints) now carries sslmode=require,
sslcert=<bundle> and sslaccept=strict instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy/db): ruff format translate_libpq_ssl_params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): keyset-paginate request logs by session trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep session grouping within type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format session grouping helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only group sessions when group_by_session is an explicit true
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reset session cursor on custom range and live tail toggles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover cursor reset on custom range toggle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): ignore next page clicks while the grouped page is still fetching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): only block next page while grouped placeholder data is shown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
`/team/info` access_group_details now carries mcp_server_ids and agent_ids per
group next to models, so the dashboard can say which group granted a server or
agent. The Object Permissions rows drop the Inherited badge and the row tooltip
reads "Granted via access group <name>. Full ID: <id>", listing every group
when more than one grants the same id and falling back to "an access group"
when the proxy did not say.
Claude-Session: https://claude.ai/code/session_01QvQzYztinxj8ZuD5YxbVdL
* fix(ui): clearing the organization picker no longer sends organization_id="" on key create
* fix(proxy): paginate Request Logs by conversation and aggregate session type counts and models server-side
* fix(proxy): keep access groups in sync when a model is renamed or deleted
* fix(proxy): cap the Request Logs conversation total like the row total
* fix(proxy): judge access group backing by the database for db models
A worker whose router has not polled the database yet still lists a sibling under its old
name, so a delete or rename handled there kept the stale name in every access group. Only
config-sourced deployments count as router backing now; db models are counted in the table.
* fix(ui): keep the conversation badge when an MCP call represents a conversation
A conversation that straddles the bounded page window can be represented by one of its MCP
rows, which showed a plain MCP badge and hid the session counts. The badge now reads the
server aggregates whenever the conversation has more than one call.
* fix(proxy): list every model of a conversation in Request Logs and keep the conversation badge for MCP representatives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type session spend aggregates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): satisfy request logs lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap per-session model aggregation in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet type-discipline budget after staging merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): send an explicit null when the key edit form clears the organization
Clearing the Organization picker in the key edit form wrote undefined into
the form value, and JSON.stringify drops undefined-valued keys, so
/key/update never saw the field and the key kept its old organization.
Writing null instead survives serialization, and the backend's
model_dump(exclude_unset=True) preserves it, so the column is set to NULL.
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): enable Claude Code gateway model discovery by default in lite claude
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cli): build agent env declaratively and document discovery key for lite up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): keep build_agent_env within LIT002 type-discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
prisma's `delete` returns None when the `where` clause matched no row
instead of raising, and the handler never looked at the return value. It
went straight on to filter an in-memory list that never held the name and
answered 200 "Credential deleted successfully", so an operator scripting
credential cleanup could not tell a real deletion from a typo.
Look at what the repository returned and answer 404 with the name, the
same rejection PATCH /credentials/{credential_name} already gives. A
credential that only exists in the config yaml is never written to the
table, so it now answers 404 too, which is honest: reporting success for
it is the same lie, since it comes back on the next proxy boot.
Adds regression tests for the delete 404, the still-working delete, the
config-yaml-only credential, and for the raise-not-return fix on both
DELETE /credentials/{credential_name} and GET /credentials, which
serialized a rejection as the 200 response body.
The config stub shared one list object between save_config and get_config, so the
DB overlay handed the endpoint back the very list it had just appended to and both
tests passed with the product fix reverted. Store the settings as JSON the way the
litellm_config row does, and check the duplicate guard against a list that only
ever existed in the DB.
`POST /v1/agents/{id}/make_public` appended the agent id to
`litellm.public_agent_groups` and only then called `get_config()`, which
re-applies the DB's `litellm_settings` over the module globals and threw the
append away. The config it saved was therefore a no-op: the endpoint answered
200 with an empty `public_agent_groups`, the agent never reached
`GET /public/agent_hub`, and re-publishing never hit the "already public" 400.
Read the config first, derive the new list from the refreshed globals, save it,
then update the global
Also fixes the e2e model hub spec, which is flaky for a second reason: the
"Make Models Public" modal preselects the groups that are already public, so a
blind click on "Select All" cleared them and left "Next" disabled for the full
15s action timeout. Check the box instead of toggling it, and wait for "Next"
to be enabled before clicking
* Fix hide-secrets guardrail: playground redaction, UI dropdown entry, spend-log telemetry
The hide-secrets guardrail never implemented apply_guardrail, so the UI test
playground echoed secrets verbatim; it was missing from the Add Guardrail
dropdown; and it recorded no guardrail_information, so Spend Logs could not
distinguish a redacted request from a clean one.
- implement apply_guardrail (unified interface) with use_native_lifecycle_hooks
so proxied traffic stays on async_pre_call_hook (per-key opt-out and
data["prompt"] handling live only there)
- record standard_logging_guardrail_information (allow/mask + masked_entity_count)
via _process_response/_process_error; opted-out keys and legacy nameless
callback instances record nothing
- advertise hide-secrets in /guardrails/ui/add_guardrail_settings (pre_call only)
and /guardrails/ui/provider_specific_params with a config model
Resolves LIT-3548
* Fix hide-secrets passthrough telemetry and JSON config input
* fix(guardrails): validate hide-secrets object config before submit
- apply_guardrail treats empty-string-only texts as no input, so no
false allow is recorded
- the UI object field keeps raw text while editing and blocks submission
until it parses to a JSON object, instead of posting a string to an
object-only API
- supported_modes_by_provider keeps its dict[str, list[str]] value type
* fix(guardrails): record no hide-secrets telemetry when nothing was inspected
walk_user_text and the prompt redaction now report how many non-empty
strings they visited; when neither inspected anything (image-only
content, empty strings), the run records no guardrail entry instead of
an 'allow' row that counts a check which never saw any text.
* fix(sso): resolve multi-valued role claims to the highest privilege role
A role claim carrying several roles used to resolve to whichever one the IdP
listed first, so a user holding both proxy_admin_viewer and internal_user lost
org-level spend visibility depending on claim ordering alone.
get_litellm_user_role now picks the highest privilege role out of a list-valued
claim, and the Entra app_roles path shares that same resolution instead of
keeping its own copy of the hierarchy. SAML assertions carrying several role
values go through the same path rather than taking the first value.
* test(sso): lock ranked-over-unranked role resolution for mixed claims
org_admin, team and customer sit outside the privilege ladder. Pin the
resolution for a claim that mixes one of them with a ranked role so the
asymmetry is covered rather than implicit.
* fix(sso): label the claim-sequence cast for the type-discipline gate
* fix(sso): resolve claim entries without recursing
The repo's recursive-function gate rejects self-recursion here, and a role
claim is flat anyway. Pull the single-value lookup into its own helper so the
list branch maps over it instead of calling back into itself.
* fix(model_armor): handle Anthropic Messages and Responses streams in post_call
The post_call streaming hook buffered every chunk and fed it to
stream_chunk_builder, which only understands chat-completion deltas.
/v1/messages streams raw Anthropic SSE bytes and /v1/responses streams
typed Responses events, so both raised litellm.APIError and surfaced to
the client as a 500 on every streamed request.
Assemble each surface with its own reader, frame guardrail failures as
terminal items in that surface's wire format, and pass the stream
through unscanned when it cannot be assembled instead of raising.
* fix(model_armor): classify the stream surface and fail closed when it cannot be assembled
Decide the wire format explicitly instead of inferring it from a boolean pair, so an
opaque raw SSE stream (the Google :streamGenerateContent route) is never refused in
Anthropic framing, and a stream that cannot be assembled is blocked rather than
released unscanned unless fail_on_error is disabled.
Also scan Responses tool-call arguments, read the body only off a terminal Responses
event, and record the applied guardrail on the fail-closed path.
* test(model_armor): pin the error-only stream predicate against content-carrying streams
is_sse_error_stream decides whether a buffered stream is forwarded to the client
untouched, so a stream that still carries content must not qualify: the frames-only
join drops typed chunks, an empty stream is not a refusal, and a content event may
carry an empty error field.
* fix(model_armor): let a streamed de-identify match mask instead of blocking
A de-identify template reports MATCH_FOUND for every redaction it makes. The
streaming block check omitted allow_sanitization, so with mask_response_content
enabled that match read as a refusal and the client got a 400 where the
non-streaming sibling returned the redacted text. Pass the flag through, as the
non-streaming hook already does, and stamp the logged status from the same
decision so the spend row agrees with what the client received.
Also drop Any from the chat-completion assembler's parameter; stream_chunk_builder
takes a bare list, so list[object] carries the mutability requirement without
erasing the element type.
* fix(model_armor): fail closed when a streamed de-identify match cannot be applied
Allowing sanitization past the streaming block check is a promise to apply the
redaction Model Armor asked for. Two paths broke that promise and released the
buffered original instead: a match that comes back with no sanitized text, and a
surface with no assembled body to rewrite.
The outcome is now resolved once, before it is recorded, so the status stamped on
request metadata agrees with what the client receives rather than reporting the
success the block check alone would have implied.
* fix: scan the deltas when a Responses stream ends without a body
response.failed and response.incomplete are terminal events like
response.completed, but a turn that broke mid-generation reports an empty
output while the deltas ahead of it already spelled the answer out to the
client. Reading only the terminal body found nothing to scan there, and the
empty-content shortcut then forwarded every buffered delta past the guardrail.
Fall back to the text the delta events carry whenever a Responses stream
assembles to nothing.
* fix: read the Responses delta event types off the event enum
The hand-listed set left out response.mcp_call_arguments.delta, so a turn that
streamed only MCP tool arguments and then reported an empty body still took the
no-content shortcut and forwarded those chunks unscanned.
Deriving the set from ResponsesAPIStreamEvents keeps it complete as the enum
grows, and the str guard in the reader already covers any event whose delta is
not text.
* fix(model_armor): scan responses deltas alongside the terminal body
A /v1/responses stream spells out reasoning summaries and tool-call arguments in
delta events that its terminal body never repeats, so scanning the body alone
handed every summary delta to the client unscanned whenever the body carried text.
* fix(model_armor): scan responses delta fields apart from each other
A Responses turn spells out its reasoning summary, its visible answer and its tool-call
arguments in separate delta events. Joining every delta into one string let a finding form
across the boundary between two fields that each carry nothing to find, so a safe stream
could be blocked. Group the deltas by the field they belong to, join a field's own deltas
as they streamed, and keep the fields apart.
* fix(model_armor): scan each responses field once, not twice
Separating delta fields stopped the terminal body from matching the delta text, so a turn
with two visible fields sent Model Armor both copies. Only the delta fields the body does not
already carry are appended now.
---------
Co-authored-by: yassin <yassin@berri.ai>
* fix(proxy): keep SpendLogs and callback session ids in sync when the request has none
Add general_settings.missing_session_id (generate | reject). In generate mode one id is
stamped into litellm_session_id, litellm_trace_id and metadata.session_id before callbacks
run, so LiteLLM_SpendLogs.session_id and the Langfuse session id match. In reject mode such
requests get a 400. Unset keeps the legacy behavior. MCP routes are not affected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate schema.d.ts and shorten mutable-ok comment for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark generated session ids so affinity consumers do not pin on them
Fireworks x-session-affinity, the router session_affinity pre-call check and the
complexity router session pin all read metadata.session_id as a caller-chosen
stable key. A missing_session_id: generate id is fresh per request, so it now
carries metadata.litellm_session_id_generated and those consumers skip it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): skip streaming guardrail rounds that re-scan cleared output
Streaming guardrails scanned the finished answer twice at end of stream
whenever the chunk count landed on a multiple of the sampling rate, ran
sampled rounds whose payload was identical to the previous one, and on
/v1/messages could scan an empty text before the first content chunk.
Every redundant round is a paid guardrail provider call.
Each endpoint handler now exposes a scan key describing what a round
would hand to apply_guardrail (the text so far, plus tool calls once the
stream has ended), and the unified streaming hook skips a sampled or
end-of-stream round whose key equals the last scanned one or carries
nothing to scan yet. Rounds that carry tool calls are never skipped.
* test(guardrails): expect one end-of-stream scan when the terminal chunk is sampled
Update sampled cadence expectations and use tuple-backed scan state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The dashboard's template picker imported autorouter_presets.json at build time, so every
catalog change needed a dashboard rebuild and artifacts refresh. The catalog now lives in
litellm/proxy/public_endpoints/ and GET /public/autorouter_presets serves it, fetching
litellm.autorouter_presets_url (GitHub raw on main, 1h in-process cache, bundled fallback)
so a merged catalog change propagates to running proxies like the model cost map does.
The dashboard fetches it at runtime via useAutoRouterPresets and keeps no local copy.
Resolves LIT-6764
* fix(ui): paginate request logs by session groups server-side
The logs table server-paginated raw spend logs and then collapsed
multi-call sessions client-side, so a page could render 3 rows while
the footer claimed 25 and sessions straddled pages. Adds an opt-in
group_by_session param to /spend/logs/ui that pages and counts one
representative row per session (DISTINCT ON, newest non-MCP call),
keeps the bounded count contract, enriches whole-session llm/agent
composition counts, and deletes the client-side collapse pipeline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* feat(ui): add a 10 rows-per-page option and default request logs to it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): key session aggregates per api key in the logs enrichment
Grouped pagination splits a reused session id into one row per api key,
but the enrichment still aggregated by session_id alone, so both rows
showed combined spend and counts. The aggregate query now groups by
(session_id, api_key), the count folds into it (the separate group_by
query is deleted), and each row reads its own key's totals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): treat an empty api_key as a real session group value
The spend-log schema defaults api_key to an empty string; truthiness
guards in the enrichment treated it as missing, so keyless multi-call
sessions lost their count and spend. Only None means missing now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
get_or_compute single-flights concurrent misses under a per-key lock, but
invalidate() deletes outside it. A mint already awaiting the IdP when an
invalidation ran wrote its result into the slot afterwards, so a bearer minted
before the invalidation was served after it for its full TTL; the upstream-401
retry then re-presented the bearer the server had just rejected.
The cache now keeps a per-key generation beside the single-flight lock.
invalidate() bumps it and get_or_compute snapshots it inside the lock right
before minting, skipping the cache write when it no longer matches. The token
is still returned to the caller it was minted for, so only the caching is
fenced. Generation and lock share one object that a running mint binds to a
local, which pins the weak map's entry for exactly as long as a write can still
land; a generation held weakly on its own would be collected across the await
and the re-check would silently pass.
The pre-flight token exchange resolved its target from the requested path, so a key not entitled to that server still drove an outbound exchange at the IdP and populated the credential cache before the later access check denied it. Resolve the target through the same allowed-server set the rest of the route uses and skip the exchange when the requested server is not in it.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): forward mode and streaming params to crowdstrike_aidr handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): drop stream_chunk_builder patch from crowdstrike cadence test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type test params and cover unsupported crowdstrike mode rejection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop leaking internal exception details to clients
Public error responses could disclose internal details in two places.
A proxy-layer exception with no recognized provider status code (a bug
in a custom callback, a hook, or litellm's own code) forwarded its raw
str() text verbatim on a 5xx, including any embedded credential,
filesystem path, or internal hostname, or a full stack trace; the
same client-facing message now runs through a redaction layer built
on top of the credential redaction that already runs on log output,
so it also drops an embedded traceback and scrubs path-shaped and
hostname-shaped substrings. It intentionally never runs on server-side
logs, which must keep full detail for debugging.
exception_type(), litellm's core exception mapper, is shared by direct
SDK callers (litellm.completion()) and the proxy, and it deliberately
embeds a traceback into an unmapped exception's message as a debugging
aid for library users; a first pass at this fix stripped that
traceback inside exception_type() itself and broke that convention
(caught by tests asserting on the traceback frame). The traceback stays
in exception_type()'s own output; only the proxy's client-facing
response boundary (and the streaming response generator, which never
needs to embed one at all) strips it.
Full generic-message replacement for the unclassified-exception case
was tried first and reverted too: several routes deliberately raise a
bare exception as an informative, secret-free validation message (e.g.
the OCR endpoint's rejection of provider-native file IDs), and
replacing those wholesale broke that convention; targeted redaction
leaves them untouched.
Also stops the default uvicorn-based proxy from sending a Server
response header.
Resolves LIT-6747
* refactor(proxy): drop the unrelated error-message constant and trim redaction comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
