Commit graph

4929 commits

Author SHA1 Message Date
mubashir1osmani
a1e7293fa9
fix(mcp): apply key and team guardrails to MCP tool calls (#39629)
* fix(mcp): apply key and team guardrails to MCP tool calls

Guardrails attached to a virtual key or team were only enforced on LLM
routes. The synthetic request built for MCP tool call guardrail hooks
carried no guardrails in its metadata, so a guardrail with default_on
false never ran on tools/call even when the key explicitly listed it.
Resolve key, team, and project guardrails onto the synthetic request
with the same helper the chat path uses.

* fix(mcp): pass project metadata through without a mutable default

* fix(mcp): mark the request dict parameter mutable-ok with a reason

* test(mcp): explain the premium_user patch and tighten the helper docstring
2026-09-07 16:45:16 +00:00
Mateo Wang
02522a5441
Merge pull request #39983 from BerriAI/litellm_lit_7081_azure_ai_gpt_6_astra_pricing
feat(cost-map): add azure_ai/gpt-6-astra Foundry pricing
2026-09-06 01:27:22 -07:00
mateo-berri
defd8661f4 refactor(spend): stop queueing a batch's claim row for a writer the proxy never builds
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
2026-09-06 00:04:24 -07:00
mateo-berri
fcb6d2267c fix(spend): keep a batch's claim row out of the logs a proxy was told not to write
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
2026-09-05 23:51:44 -07:00
mateo-berri
a2b21b323a Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_7048_batch_cost_row_once 2026-09-05 23:03:41 -07:00
mateo-berri
24f0be8021 fix(spend): leave a batch uncharged when the database refuses the takeover
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
2026-09-05 22:47:41 -07:00
mateo-berri
0fb3951b2c fix(spend): charge a batch once when an older proxy left its cost row at $0
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
2026-09-05 22:32:06 -07:00
mateo-berri
061c25b5ca fix(spend): let a batch's charge survive an older proxy's $0 poll row
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over

disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
2026-09-05 21:25:03 -07:00
tin-berri
9fd60e4f95
feat(router): gate heuristic v1 tuning (#39952) 2026-09-05 19:24:00 -07:00
mateo-berri
a17fcecf70 refactor(azure_ai): type the Foundry param mapping override and drop test docstrings
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
2026-09-05 19:21:34 -07:00
mateo-berri
15372967c6 fix(azure_ai): read the azure_ai card for gpt-5 series reasoning effort gates
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does

Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
2026-09-05 19:06:38 -07:00
mateo-berri
b067e836f8 fix(batches): claim the batch cost spend row in the database before charging
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path

`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it

The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
2026-09-05 18:58:11 -07:00
tin-berri
60440ee4d3
feat(mcp): add opt-in per-server oauth relay discovery (#39936)
Resolves LIT-7074
2026-09-05 18:04:35 -07:00
tin-berri
0315dd6f58
fix(headroom): inject headroom_retrieve only for service-declared ccr_hashes and keep assistant content blocks intact (#39974)
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.

Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.

Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
2026-09-05 17:58:18 -07:00
ryan-crabbe-berri
a9f8a8d794
Merge pull request #39978 from BerriAI/litellm_remove_migrated_pages_shim
refactor(ui): route the sidebar by pathname and shrink the ?page= shim to a redirect table
2026-09-05 17:17:34 -07:00
yucheng-berri
6e05ac5d97
feat(guardrails): add inspect_embeddings toggle for AIM and Cato (#39918)
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks

`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:

- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
  list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
  redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.

Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.

* feat(guardrails): add inspect_embeddings toggle for AIM and Cato

* fix(guardrails): redact batched embedding input on anonymize

A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.

* fix(guardrails): reject partial embedding redactions

* fix(guardrails): avoid unnecessary batch type check

* style(tests): drop trailing blank line in cato guardrail tests

* fix(guardrails): reject malformed batch redactions

* fix(guardrails): reject malformed batch redactions

* fix(guardrails): reject aim redactions with no text content

The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.

* fix(guardrails): validate all aim redaction paths

Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.

* fix(guardrails): preserve aim output redaction alignment

AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.

* test(guardrails): cover aim output anonymize alignment and malformed redactions

---------

Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
2026-09-05 17:15:46 -07:00
ryan-crabbe-berri
e1fb8affe3
Merge pull request #36841 from BerriAI/litellm_lite_pi
feat(cli): add lite pi to run the pi coding agent through the proxy
2026-09-05 17:14:11 -07:00
mateo-berri
635bb3a209 feat(cost-map): add azure_ai/gpt-6-astra Foundry pricing
A gpt-6-astra deployment on a Foundry project reached through the
azure_ai route had no cost map entry of its own, so it resolved to the
OpenAI gpt-6-astra card: missing from the azure_ai/* wildcard listing,
flex and priority prices and /v1/batch it does not sell, and no none
reasoning effort. Add azure_ai/gpt-6-astra mirroring the
azure/gpt-6-astra Standard Global sheet the way azure_ai/gpt-5.5 mirrors
azure/gpt-5.5, and extend the cost, reasoning-effort, and wildcard
listing tests to the Foundry route.
2026-09-05 17:08:42 -07:00
Mateo Wang
02cbff4918
Merge pull request #39964 from BerriAI/litellm_lit_7050_redact_failure_traceback
fix(proxy): redact provider keys from pass-through failure tracebacks
2026-09-05 16:48:04 -07:00
ryan-crabbe-berri
1258d84221 refactor(ui): route the sidebar by pathname and shrink the ?page= shim to a redirect table
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.

The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.

migratedHref is renamed uiHref, the /ui base-path helper it always was.
2026-09-05 16:44:11 -07:00
mateo-berri
4c00a6e189 fix(batches): account a batch's cost once, from the first retrieve that sees it final
Every retrieve of a batch through the proxy shares one spend row, the batch id
plus the batch cost suffix, and spend log inserts skip duplicates. A poll that
landed while the batch was still validating or in progress wrote that row at
$0 and no later retrieve could overwrite it, and every completed retrieve after
the first added the cost to the key, team, and user counters again with no new
row to show for it.

The cost callback now writes nothing for a batch retrieve until the batch is
final, releasing the poll's budget reservation instead, and once it is final it
charges only when no spend row for that batch is queued for flush or already
stored. Batch cost rows are flushed to the database right away so a second
instance sees them, and the logger prices a batch only once it is final, which
also covers a failed batch that never produced an output file.
2026-09-05 16:35:32 -07:00
mateo-berri
27c55a21d9 test(proxy): type the traceback-recording hook to match CustomLogger
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
2026-09-05 16:19:29 -07:00
mateo-berri
4ec5a6761c fix(proxy): redact provider keys from pass-through failure tracebacks
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
2026-09-05 15:58:33 -07:00
ryan-crabbe-berri
63156a7bd6 test(proxy): explain the proxy_server patches in the cache-hit regression test
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:44:37 -07:00
ryan-crabbe-berri
acddd21860 fix(proxy): keep guardrail cost in spend on cache hits
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:32:02 -07:00
ryan-crabbe-berri
1745d74293
Merge pull request #39853 from BerriAI/litellm_guardrail_usage_cost_ui
feat(ui): show guardrail usage units and cost on the Guardrails Monitor
2026-09-05 14:41:11 -07:00
devin-ai-integration[bot]
a46a076b2a
fix(proxy): reject ambiguous name or alias keys in mcp_tool_permissions on write (#39947)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:22:26 -07:00
Yassin Kortam
9832d6e4a6
fix(mcp): scan and mask MCP tool call arguments in unified guardrails (#35142)
* fix(mcp): scan and mask MCP tool call arguments in unified guardrails

A guardrail configured with mode pre_mcp_call was handed only a synthetic
tool definition (name plus an empty parameters schema), so it never saw the
argument values it was configured to inspect, and any rewrite it returned was
discarded. Detection could not fire and masking could not take effect, while
the applied-guardrails metadata still reported the guardrail as having run.

Pass every string leaf of the tool call arguments as texts, and fold the
guardrail's rewritten leaves back into modified_arguments, which is the channel
the MCP call path reads to decide what to send upstream. The leaf walk reuses
the json_string_leaves / with_json_string_leaves helpers the tool result path
already uses, so both directions share one bounded traversal.

Two guardrails running concurrently under run_in_parallel scan the same payload
snapshot, so each returns a full replacement derived from the original leaf.
Rewrites of the same leaf to different values are rejected rather than silently
losing one redaction; a leaf that already holds this guardrail's own replacement
is convergent and still masks, which is what the bundled content filter does
when it rewrites the arguments itself as well as through texts.

* fix(mcp): annotate guardrail argument rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): isolate MCP guardrail callback state

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet LIT010 budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): remove duplicate Bedrock hook parameter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): fail closed when guardrail rewrites cannot be mapped to MCP arguments

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): patch the guardrail translation mappings cache where staging now keeps it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 20:51:25 +00:00
devin-ai-integration[bot]
80839bb33c
feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port (#39889)
* feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): ruff format prometheus_metrics_server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): fail fast when the separate metrics server cannot start and force the multiproc dir whenever it is enabled

- wait for the child's /health before starting uvicorn; raise a ClickException if it exits first (port in use)
- create PROMETHEUS_MULTIPROC_DIR whenever --prometheus_metrics_port is set, so DB-configured prometheus callbacks work
- honour lowercase prometheus_multiproc_dir; validate the port before spawning
- cover main() entry point, readiness, bind failure and wildcard-host probing in tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): pin metrics-server readiness to the child pid so another service on the port cannot pass the health check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): probe metrics-server readiness through the shared HTTPHandler instead of bare httpx.get

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): serve only /metrics on the prometheus metrics port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): validate metrics server CLI args with pydantic instead of typing.cast

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): satisfy metrics server lint gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:26:09 -07:00
ryan-crabbe-berri
0be8bb98b0 Merge branch 'litellm_internal_staging' into litellm_guardrail_usage_cost_ui 2026-09-05 13:09:45 -07:00
devin-ai-integration[bot]
5df0e12e0f
feat(guardrails): add non-blocking flag() verdict to custom code guardrails (#39728)
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:08:03 -07:00
Yassin Kortam
17e13126cc
feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture (#35394)
* feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture

Only the generic OIDC login path captures the IdP id_token that an oauth2_id_jag MCP
server spends as its RFC 8693 subject token. Under Google, Microsoft, SAML or no SSO at
all, registration succeeds and then every ID-JAG credential resolution fails for every
user, with nothing in the logs, the config or the API response to say why.

Report the mismatch from the two places it is knowable: when an oauth2_id_jag server is
created or updated through the management endpoint, and at SSO callback time when a login
hands the arm nothing while such a server is registered. Provider selection mirrors the
callback's precedence, so a generic client id sitting behind GOOGLE_CLIENT_ID does not
clear the warning.

* test(sso): update merged CLI diagnostic patch target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(mcp): warn about the ID-JAG capture gap for config-declared servers and on the SSO debug page (#39350)

* feat(sso): surface the ID-JAG capture gap on the SSO debug page

/sso/debug/callback is where an operator lands when they are already trying to work out
why ID-JAG is failing, so the reason belongs on it. The annotation appears only when the
active SSO provider captures no identity assertion AND an oauth2_id_jag server is
registered for that gap to break; a deployment without both renders the page it rendered
before, byte for byte. Only the provider name and the remedy are rendered, never a
configured value, and an unreachable MCP table costs the page its annotation rather than
the page itself.

The payload carries the one mutable-ok in this work. Conditionally including a member of a
JSON document has to construct a mapping, and the rejected alternatives are recorded on the
helper so the next reader does not rediscover them.

Held out of the diagnosability PR deliberately: that PR is already reviewed and green, and
this surface ships with the remaining config-load warning as one follow-up.

* feat(mcp): warn at config load when an oauth2_id_jag server outruns the SSO provider's assertion capture

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(sso): trim comments on the ID-JAG debug page diagnostic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): clean up merged imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): satisfy type discipline for diagnostic payload

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(sso): keep the optional ID-JAG payload member on one line for ruff format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): use Python 3.10-compatible assert_never

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): keep the ID-JAG capture-gap diagnostic out of the unauthenticated debug page

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): inject the retention check and log via caplog so the ID-JAG tests pass the test-quality gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): keep the debug-page outage test on the capture-gap path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): annotate the retention check type alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:09 -07:00
Yassin Kortam
110f654f34
feat(mcp): renew the stored SSO identity assertion behind ID-JAG (#35401)
* feat(mcp): renew the stored SSO identity assertion behind ID-JAG

The oauth2_id_jag arm asserts the id_token captured at the user's last
interactive SSO login, and nothing ever renewed it, so an agent holding a
brokered LiteLLM key could act for that user only until that token's exp.
The assertion already carried the IdP refresh token beside it; this
redeems it.

RefreshingSSOAssertionStore wraps the database reader and satisfies the
same protocol, so the egress arm is unchanged. Renewal is lazy and
single-flighted per user through the same RefreshCoordinator the
authorization_code arm uses, since an IdP that rotates refresh tokens
treats two concurrent redemptions as replay. A refusal leaves the expired
assertion in place so the reader still challenges the user; an
unreachable IdP surfaces as a store outage instead.

* fix(mcp): let a cross-replica loser settle the SSO assertion renewal itself

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): satisfy type discipline lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ci): rerun checks after docs main added the missing router setting row

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): answer a cross-replica loser retryable instead of re-electing it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): bypass stale assertion cache during renewal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet type-discipline budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:02 -07:00
moe-berri
fc3da5e830
Merge branch 'litellm_internal_staging' into litellm_auto_router_compression_split 2026-09-05 11:55:05 -07:00
tin-berri
e3b4a82ff9
Merge pull request #39926 from BerriAI/litellm_lit6981_none_url_auth
fix(mcp): reject URL credentials for none auth
2026-09-05 18:44:34 +00:00
devin-ai-integration[bot]
cba3dd5828
fix(proxy): retry deadlocks and requeue spend logs on any DB write error (#39883)
* fix(proxy): retry deadlocks and requeue spend logs on any DB write error

update_spend_logs dequeued the batch and only retried/requeued on transport
errors. A 40P01 deadlock surfaced as a plain prisma DataError and went through
poison-row isolation, which dropped every row it hit; every other DB error was
re-raised with the batch already gone from the queue.

Treat deadlocks as transient (retry, then requeue), keep them out of poison-row
isolation, and requeue the batch at the head of the queue on any other prisma
error so it lands once the DB is healthy.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): drop redundant docstrings and tighten test typing for spend-log requeue

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): assert deadlock retries from mock call history instead of mutable lists

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 11:41:56 -07:00
devin-ai-integration[bot]
4df284e16d
fix(guardrails): record guardrail information for undecorated custom apply_guardrail overrides (#39727)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 11:39:24 -07:00
moe-berri
a0b07b4791 docs(auto-router compression): cut the explanatory comments back
The module, its routing hook and its tests carried long prose rationale where the
repository allows only concise comments for genuinely complex logic. Trimmed to the
non-obvious reasons and dropped the rest; no logic or test behaviour changes.
2026-09-05 11:33:21 -07:00
moe-berri
1c16a5910b fix(tests): undo a stray whole-file reformat and arm a real guardrail
test_router.py is not ruff-formatted on staging and CI's format check only scopes
litellm/*.py, so running ruff format over the whole file rewrote ~900 lines of
unrelated code. That reflow split long single-line patch() calls into multi-line
form, which the test-quality gate counts individually, pushing TQ008 four over its
ceiling. The file is back to staging's formatting with only the compression test
class added.

test_common_request_processing.py armed a model-side guardrail name with no such
guardrail registered, which stopped working once both hops began requiring the name
to resolve to an active compression guardrail.
2026-09-05 10:31:56 -07:00
moe-berri
d143b1954e Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_auto_router_compression_split 2026-09-05 10:19:48 -07:00
ryan-crabbe-berri
f0618394c1
Merge pull request #39680 from BerriAI/litellm_lit_4738_table_pagination
fix(ui): make Admin UI table pagination honor the selected page size
2026-09-05 10:16:56 -07:00
ryan-crabbe-berri
d885104125 chore: merge litellm_internal_staging into litellm_lit_4738_table_pagination
per_user_usage.tsx conflicted with the server pagination that already landed on
staging (default 50 rows, stale-response guard, tag and page-size resets). Took
the staging version and dropped this PR's now-redundant 25-row test for it

Claude-Session: https://claude.ai/code/session_01HkaXiD6gssHnx3kqu1rR8C
2026-09-05 10:08:11 -07:00
moe-berri
f4329d5491 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_auto_router_compression_split
# Conflicts:
#	ui/litellm-dashboard/src/components/edit_auto_router/edit_auto_router_modal.test.tsx
2026-09-05 10:06:01 -07:00
ryan-crabbe-berri
a5a846dd0c
Merge pull request #39672 from BerriAI/litellm_expose_v2_organization_patch_openapi
feat(organization): expose PATCH /v2/organization/{organization_id} in the OpenAPI spec
2026-09-05 10:05:09 -07:00
moe-berri
8284208af2 fix(auto-router compression): restrict both hops to real compression guardrails
The two policy fields are operator-supplied names and nothing else constrained them.
The routing hop calls apply_guardrail directly, which hands the guardrail the
conversation and POSTs it to whatever service backs that guardrail, and the model hop
is added to metadata["guardrails"], which runs it even when it is not default_on. So
naming an ordinary guardrail turned either hop into a way to invoke it and ship prompt
content to it. Both hops now refuse a name that does not resolve to an active
compression guardrail, and say so in the log rather than failing quietly.
2026-09-05 09:51:51 -07:00
tin-berri
d0d09e5343
feat(router): meter auto-router tier and prompt customization against the auto_router license feature (#39674)
Generalizes the heuristic_v2 ceiling from #39468 into a capability table whose
records own their in-process predicate, SQL spelling and refusal wording. The
existing heuristic_v2 capability keeps its own one-router ceiling. A single
customization capability combines operator-defined tier definitions with every
operator-written part of the classifier prompt. The prompt half only applies to
classifier types that call an LLM. The shipped default prompt, classification
rubric presets, tier-label renames and tier model choices remain ungated.

Scope every enforcement point to actual complexity routers. A model-less PATCH
or legacy update now decrypts the stored model before accepting strategy-router
settings, so a regular model cannot acquire a router config or spend a license
slot. Under the existing advisory lock, the cross-pod candidate query returns
only model scalars and the count decrypts and classifies them in process; old
non-router rows carrying a capability-shaped config no longer block a real
complexity router. The signed auto_router license feature makes both ceilings
unlimited.
2026-09-05 09:51:23 -07:00
Mateo Wang
74ebe2974c
Merge pull request #39851 from BerriAI/litellm_fix_realtime_backend_close_hang
fix(realtime): relay the upstream websocket close to the client instead of hanging
2026-09-05 09:51:15 -07:00
mateo-berri
37722eba68 fix(realtime): close a rejected client before releasing its budget reservation
A slow or unreachable counter store made a pre-relay rejection wait behind
the reservation release before the client saw the error event and the close.
Close first and release in finally, mirroring the relay's own failure path,
so a client that already hung up still gets its reservation released.
2026-09-05 02:35:26 -07:00
mateo-berri
5a35e6d41f fix(realtime): release the budget reservation when a session is rejected before the relay starts
The three pre-relay exits of realtime_websocket_endpoint (missing model,
key/model access denied, pre-call rejection such as a rate limit or a
guardrail) returned before the finally that releases the auth-time budget
reservation, so a rejected session pinned the key at the reserved amount
until the counter TTL expired and its next requests got budget_exceeded
while /key/info showed spend 0. A single _reject_realtime_session helper
now releases the reservation before sending the error event and closing,
and release_or_invalidate_budget_reservation shields the release from a
second cancellation and logs, rather than raises, a failing invalidate
fallback so it can never mask the session's own outcome.
2026-09-05 02:18:33 -07:00
mateo-berri
5c80e308cd Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_gate_openai_ws_passthrough 2026-09-05 01:26:47 -07:00