Commit graph

15230 commits

Author SHA1 Message Date
mateo-berri
ecf7e4e766 fix(router): keep the provider's error when the retry skip empties the group
Before excluding the deployment that just refused, the retry-skip guard asked
whether another one could still answer. It asked by re-running a single routing
filter, the order filter, while deployment selection also applies cooldowns, the
context-window pre-call check, tag routing, and routing plugins.

Any filter the guard did not replicate made it answer yes while the real pick was
left with nothing. A group narrowed to one deployment by tag routing turned the
provider's own 400 into a no-deployments 429.

The skip now runs where every filter has already been applied, and it keeps the
deployments untouched when skipping would leave none. The caller gets the
provider's error either way, and a group with one eligible deployment retries in
place as it did before.
2026-09-05 23:36:28 -07:00
mateo-berri
cb1ec76e46 fix(router): keep retry skips on the active order-fallback target
The retry-skip guard checks that some other deployment could still answer
before it excludes the one that just refused, so a single-deployment group
keeps the old retry-in-place behavior. It asked that question at the group's
minimum order, but the router picks the retry's deployment at the order the
request has already escalated to.

So a group with a primary at order 1 and a backup at order 2 answered "yes,
order 1 still has a candidate" while the retry was pinned to order 2, and the
exclusion left order 2 with nothing. The caller got a no-deployments error in
place of the provider's own 400.

The helper now takes the active target order and filters by it, which is the
same value async_get_healthy_deployments reads off the request.
2026-09-05 23:02:44 -07:00
mateo-berri
7c7810df42 fix(router): ignore non-integer status codes when picking retry skips
CI's router_code_coverage gate wants every function in router.py called by
name from a test file with "router" in its name, and the new helper had no
direct caller, so the check-quality job failed on the first tip.

Covering it directly also turned up a hole. litellm._should_retry compares
the status code to 500, so a provider exception carrying a string status code
raises TypeError instead of answering. should_retry_this_error has the same
call, but the retry policy path skips it, which is exactly the path this
change enables, so the helper was the first to touch that value. Narrowing to
int leaves those exceptions on the old retry-in-place behavior.
2026-09-05 22:43:15 -07:00
mateo-berri
2e2fce5e58 fix(router): skip the refusing deployment when retrying a non-transient error
BadRequestErrorRetries and ContentPolicyViolationErrorRetries did let a retry
happen, but the retry re-picked the deployment that had just refused, since a
400 never puts a deployment in cooldown. On a weighted model group the caller
got the same 400 back after every configured retry, and the existing 401/403
"retry on another deployment" rule broke the same way

A retry after a non-transient status now carries the deployments that already
answered this request in the per-request exclusion list weighted failover
already honors, so the next attempt lands on a sibling. Single-deployment
groups still retry in place, and 408/429/5xx retries are untouched

Adds live e2e coverage for reliability.retry.context_window.succeeds_within_retries
and renames the two litellm.utils deployment filters that are now called from
outside the module
2026-09-05 22:25:13 -07:00
Mateo Wang
0aa346cba5
Merge pull request #39972 from BerriAI/litellm_lit_7027_emulated_file_search_scope
fix(file_search): scope emulated file_search to the request's vector stores
2026-09-05 17:07:04 -07:00
Mateo Wang
02cbff4918
Merge pull request #39964 from BerriAI/litellm_lit_7050_redact_failure_traceback
fix(proxy): redact provider keys from pass-through failure tracebacks
2026-09-05 16:48:04 -07:00
mateo-berri
8bf03c10fd fix(file_search): escape the dropped vector_store_id in the warning
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
2026-09-05 16:44:54 -07:00
mateo-berri
55a6132b31 fix(file_search): log when a model-picked vector_store_id is dropped
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
2026-09-05 16:31:24 -07:00
mateo-berri
27c55a21d9 test(proxy): type the traceback-recording hook to match CustomLogger
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
2026-09-05 16:19:29 -07:00
mateo-berri
942e6cb3cd fix(file_search): scope emulated file_search to the request's vector stores
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
2026-09-05 16:12:43 -07:00
yuneng-jiang
45cf1a7ef1
Revert "perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (…"
This reverts commit c091dd4608.
2026-09-05 16:07:09 -07:00
yuneng-jiang
1b25132863
Merge pull request #39953 from BerriAI/litellm_/litellm-e2e-flaky-test-2159ae
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
2026-09-05 16:04:45 -07:00
mateo-berri
4ec5a6761c fix(proxy): redact provider keys from pass-through failure tracebacks
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
2026-09-05 15:58:33 -07:00
ryan-crabbe-berri
63156a7bd6 test(proxy): explain the proxy_server patches in the cache-hit regression test
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:44:37 -07:00
ryan-crabbe-berri
acddd21860 fix(proxy): keep guardrail cost in spend on cache hits
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
2026-09-05 15:32:02 -07:00
Mateo Wang
bf51dea36b
Merge pull request #39862 from BerriAI/litellm_lit_6992_cohere_parse
feat(ocr): add Cohere Parse support for cohere and azure_ai
2026-09-05 15:16:16 -07:00
yuneng-jiang
6a4fb2bbe8
Merge pull request #39938 from BerriAI/litellm_e2e_vertex_cache_first_call
test(e2e): prove Vertex context caching on the first cold call and on the spend row
2026-09-05 15:10:15 -07:00
Yuneng Jiang
cd976624d1
test(e2e): drop the explanatory sentence from the StreamingResponse docstring 2026-09-05 14:53:12 -07:00
Yuneng Jiang
b55a4317a6
test(e2e): annotate new stream-timing locals as Final and trim the docstrings 2026-09-05 14:49:56 -07:00
Yuneng Jiang
d56affa814
test(e2e): judge /v1/messages streaming on the clock, not on the provider's delta count
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.

The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.

Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
2026-09-05 14:44:21 -07:00
ryan-crabbe-berri
1745d74293
Merge pull request #39853 from BerriAI/litellm_guardrail_usage_cost_ui
feat(ui): show guardrail usage units and cost on the Guardrails Monitor
2026-09-05 14:41:11 -07:00
yuneng-jiang
9222a4de2d
Merge pull request #39946 from BerriAI/litellm_e2e_messages_stream_delta_count
test(e2e): stream a longer /v1/messages reply so the delta-count pin has margin
2026-09-05 14:26:55 -07:00
devin-ai-integration[bot]
a46a076b2a
fix(proxy): reject ambiguous name or alias keys in mcp_tool_permissions on write (#39947)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 14:22:26 -07:00
Yassin Kortam
9832d6e4a6
fix(mcp): scan and mask MCP tool call arguments in unified guardrails (#35142)
* fix(mcp): scan and mask MCP tool call arguments in unified guardrails

A guardrail configured with mode pre_mcp_call was handed only a synthetic
tool definition (name plus an empty parameters schema), so it never saw the
argument values it was configured to inspect, and any rewrite it returned was
discarded. Detection could not fire and masking could not take effect, while
the applied-guardrails metadata still reported the guardrail as having run.

Pass every string leaf of the tool call arguments as texts, and fold the
guardrail's rewritten leaves back into modified_arguments, which is the channel
the MCP call path reads to decide what to send upstream. The leaf walk reuses
the json_string_leaves / with_json_string_leaves helpers the tool result path
already uses, so both directions share one bounded traversal.

Two guardrails running concurrently under run_in_parallel scan the same payload
snapshot, so each returns a full replacement derived from the original leaf.
Rewrites of the same leaf to different values are rejected rather than silently
losing one redaction; a leaf that already holds this guardrail's own replacement
is convergent and still masks, which is what the bundled content filter does
when it rewrites the arguments itself as well as through texts.

* fix(mcp): annotate guardrail argument rewrites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): isolate MCP guardrail callback state

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet LIT010 budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): remove duplicate Bedrock hook parameter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): fail closed when guardrail rewrites cannot be mapped to MCP arguments

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tests): patch the guardrail translation mappings cache where staging now keeps it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 20:51:25 +00:00
devin-ai-integration[bot]
80839bb33c
feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port (#39889)
* feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): ruff format prometheus_metrics_server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): fail fast when the separate metrics server cannot start and force the multiproc dir whenever it is enabled

- wait for the child's /health before starting uvicorn; raise a ClickException if it exits first (port in use)
- create PROMETHEUS_MULTIPROC_DIR whenever --prometheus_metrics_port is set, so DB-configured prometheus callbacks work
- honour lowercase prometheus_multiproc_dir; validate the port before spawning
- cover main() entry point, readiness, bind failure and wildcard-host probing in tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): pin metrics-server readiness to the child pid so another service on the port cannot pass the health check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): probe metrics-server readiness through the shared HTTPHandler instead of bare httpx.get

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): serve only /metrics on the prometheus metrics port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): validate metrics server CLI args with pydantic instead of typing.cast

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): satisfy metrics server lint gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:26:09 -07:00
Yuneng Jiang
d6bc8fe289
test(e2e): ask the streamed /v1/messages pin for a reply long enough to span several deltas
Anthropic now returns the 64-token 'count to 20' reply in one to three content_block_delta events, measured directly against api.anthropic.com and through proxies at 7672399 and 49a1145 alike, so the incrementality assertion (at least two deltas) failed in litellm-e2e builds 125, 130 and the 278 rerun with no proxy change behind it. A 'count to 100' reply at max_tokens 400 arrived in five to fifty deltas across every measured run
2026-09-05 13:23:38 -07:00
ryan-crabbe-berri
0be8bb98b0 Merge branch 'litellm_internal_staging' into litellm_guardrail_usage_cost_ui 2026-09-05 13:09:45 -07:00
devin-ai-integration[bot]
5df0e12e0f
feat(guardrails): add non-blocking flag() verdict to custom code guardrails (#39728)
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 13:08:03 -07:00
Yuneng Jiang
b56e4f80a4
test(e2e): make each cold cache call a single-assignment helper so its result stays Final 2026-09-05 13:05:03 -07:00
Yuneng Jiang
1c0172b477
style(e2e): annotate the cold-call locals as Final 2026-09-05 13:04:33 -07:00
yucheng-berri
948e5755eb
test(e2e): cover presidio post_call, tool_permission, and weave logging cells (#39279)
* test(e2e): cover presidio post_call, tool_permission, and weave logging cells

Five registry cells in Logging & Guardrails had no covering test. Each one now
has a live scenario read back from the real destination:

- guardrail.presidio.post_call.masks: an output-scoped Presidio guardrail
  anonymizes the PII the model repeats back. The prompt also asks for the
  address's local part, which Presidio does not mask, so one response proves the
  model saw the raw address (no pre-call masking) while the address itself comes
  back as <EMAIL_ADDRESS>
- guardrail.tool_permission.pre_call.blocks / .allows: an allow-list of one tool.
  A request declaring an unlisted tool is rejected 400 naming it; a request
  declaring the permitted tool is served and carries
  x-litellm-applied-guardrails, so the allow half cannot pass by the guardrail
  never running
- logging.niche_integrations.success.logs_spend / .failure.logs_spend: a
  key-scoped weave_otel callback delivers to the real Weave project, read back
  through Weave's query API. Success asserts exactly one call whose
  llm.response.cost equals the x-litellm-response-cost header; failure asserts
  one ERROR-status call naming the provider exception and carrying no cost

Logging & Guardrails coverage goes 24/59 to 29/59. No registry rows are added.

* test(e2e): make the tool-permission allow case deterministic and scope the Weave read-back

Review follow-ups on the coverage PR.

- the allow scenario forced the outcome to depend on whether the model felt like
  calling an optional tool, and checked for the tool name as a substring of the
  whole body, which a prose mention would satisfy. It now sends
  tool_choice="required" and asserts the parsed response carries exactly one tool
  call, for the permitted tool
- the Weave read-back queried the newest 200 calls of a shared project and
  filtered client-side, so busy traffic could push the target out of the window
  and read as a delivery failure. The query now scopes server-side to the
  litellm_request op and to calls started after the request, and pages through
  the window with offset
- the reader builds its results as tuples instead of accumulating into lists

Also unblocks the lint gate: `basedpyright tests/e2e` runs only on PRs that touch
tests/e2e, and it has been failing on staging for three FakeItem arguments in
test_junit_properties.py. The stand-in now goes through one typed adapter that
says why, so the gate is green without touching junit_properties.py itself.

* test(e2e): scope the presidio post_call guardrail to email and phone

Running the suite three times in a row caught a real flake: Presidio's broader
recognizers sometimes claim the email's local part as an NRP entity, so the
answer came back as `<NRP>\n<EMAIL_ADDRESS>\n<PHONE_NUMBER>` and the assertion
that the raw local part survives failed. That token is what tells output masking
apart from input masking, so it has to survive.

The post_call guardrail now registers pii_entities_config for EMAIL_ADDRESS and
PHONE_NUMBER only, which is also the narrower thing the scenario means. Verified
against the exact marker that failed, plus two others.

* test(e2e): mark weave logging cells stage red

* test(e2e): use per-test stage red skips for the weave logging cells
2026-09-05 13:03:28 -07:00
Yuneng Jiang
b98f8ee2c5
test(e2e): retry a fresh prefix when Vertex rejects the cache create on its minimum-token check 2026-09-05 13:00:24 -07:00
devin-ai-integration[bot]
c091dd4608
perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS (#39121)
* perf: lazy-load SDK symbols so import litellm stays under 60 MB RSS

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: resolve litellm.proxy submodules lazily so litellm.proxy._types stays importable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: correct SlackAlerting lazy mapping and keep eager encoding path importable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: register module-valued public names as module aliases instead of symbol imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: justify module-alias cache write with rebind-ok

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 19:58:35 +00:00
Yuneng Jiang
def734923f
test(e2e): prove Vertex context caching on the first cold call and on the spend row 2026-09-05 12:55:30 -07:00
Yassin Kortam
17e13126cc
feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture (#35394)
* feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture

Only the generic OIDC login path captures the IdP id_token that an oauth2_id_jag MCP
server spends as its RFC 8693 subject token. Under Google, Microsoft, SAML or no SSO at
all, registration succeeds and then every ID-JAG credential resolution fails for every
user, with nothing in the logs, the config or the API response to say why.

Report the mismatch from the two places it is knowable: when an oauth2_id_jag server is
created or updated through the management endpoint, and at SSO callback time when a login
hands the arm nothing while such a server is registered. Provider selection mirrors the
callback's precedence, so a generic client id sitting behind GOOGLE_CLIENT_ID does not
clear the warning.

* test(sso): update merged CLI diagnostic patch target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(mcp): warn about the ID-JAG capture gap for config-declared servers and on the SSO debug page (#39350)

* feat(sso): surface the ID-JAG capture gap on the SSO debug page

/sso/debug/callback is where an operator lands when they are already trying to work out
why ID-JAG is failing, so the reason belongs on it. The annotation appears only when the
active SSO provider captures no identity assertion AND an oauth2_id_jag server is
registered for that gap to break; a deployment without both renders the page it rendered
before, byte for byte. Only the provider name and the remedy are rendered, never a
configured value, and an unreachable MCP table costs the page its annotation rather than
the page itself.

The payload carries the one mutable-ok in this work. Conditionally including a member of a
JSON document has to construct a mapping, and the rejected alternatives are recorded on the
helper so the next reader does not rediscover them.

Held out of the diagnosability PR deliberately: that PR is already reviewed and green, and
this surface ships with the remaining config-load warning as one follow-up.

* feat(mcp): warn at config load when an oauth2_id_jag server outruns the SSO provider's assertion capture

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(sso): trim comments on the ID-JAG debug page diagnostic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): clean up merged imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): satisfy type discipline for diagnostic payload

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(sso): keep the optional ID-JAG payload member on one line for ruff format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): use Python 3.10-compatible assert_never

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): keep the ID-JAG capture-gap diagnostic out of the unauthenticated debug page

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): inject the retention check and log via caplog so the ID-JAG tests pass the test-quality gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(sso): keep the debug-page outage test on the capture-gap path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(sso): annotate the retention check type alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:09 -07:00
Yassin Kortam
110f654f34
feat(mcp): renew the stored SSO identity assertion behind ID-JAG (#35401)
* feat(mcp): renew the stored SSO identity assertion behind ID-JAG

The oauth2_id_jag arm asserts the id_token captured at the user's last
interactive SSO login, and nothing ever renewed it, so an agent holding a
brokered LiteLLM key could act for that user only until that token's exp.
The assertion already carried the IdP refresh token beside it; this
redeems it.

RefreshingSSOAssertionStore wraps the database reader and satisfies the
same protocol, so the egress arm is unchanged. Renewal is lazy and
single-flighted per user through the same RefreshCoordinator the
authorization_code arm uses, since an IdP that rotates refresh tokens
treats two concurrent redemptions as replay. A refusal leaves the expired
assertion in place so the reader still challenges the user; an
unreachable IdP surfaces as a store outage instead.

* fix(mcp): let a cross-replica loser settle the SSO assertion renewal itself

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): satisfy type discipline lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ci): rerun checks after docs main added the missing router setting row

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): answer a cross-replica loser retryable instead of re-electing it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): bypass stale assertion cache during renewal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: ratchet type-discipline budget after merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:43:02 -07:00
moe-berri
b3f28a77d8
Merge pull request #39823 from BerriAI/litellm_auto_router_compression_split
feat(auto-router): decouple compression between the routing decision and the model call
2026-09-05 12:35:37 -07:00
yuneng-jiang
c6399b5728
Merge pull request #39917 from BerriAI/litellm_e2e_key_mgmt_route_group_coverage
test(e2e): cover key spend reset, regenerate grace period, and the llm_api_routes grant
2026-09-05 12:30:59 -07:00
yuneng-jiang
29cf4e8520
Merge pull request #39920 from BerriAI/litellm_e2e_prompt_cache_cohere_passthrough_coverage
test(e2e): cover Anthropic and OpenAI prompt caching, Cohere embeddings, and costed /openai chat passthrough
2026-09-05 12:27:21 -07:00
yuneng-jiang
730d789418
Merge pull request #39932 from BerriAI/litellm_/circleci-pipeline-failure-afe848
test: repair two CI tests broken by intentional changes
2026-09-05 12:22:40 -07:00
yucheng-berri
877197918b
fix(cloudzero): preserve late resource tags (#39873)
* fix(cloudzero): infer daily batch schema from every row

pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.

Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.

* test(cloudzero): cover late tag schema inference

Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.

* fix(cloudzero): preserve late resource tags

* style(cloudzero): remove redundant test comment
2026-09-05 12:10:05 -07:00
yucheng-berri
73e1cfb378
fix(cloudzero): infer daily batch schema from every row (#39871)
* fix(cloudzero): infer daily batch schema from every row

pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.

Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.

* test(cloudzero): cover late tag schema inference

Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
2026-09-05 12:09:53 -07:00
devin-ai-integration[bot]
0ad361a728
fix(router): coordinate async and sync failure handlers at remaining router call sites (#39887)
* fix(router): coordinate async and sync failure handlers at remaining router call sites

Five router failure paths still scheduled logging_obj.async_failure_handler
as a task while starting logging_obj.failure_handler on a raw thread, so
both handlers mutated the same logging object concurrently. Route them
through dispatch_failure_handlers like the streaming paths already do.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): wait on the real logging executor and justify the callbacks global patch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(logging): submit sync failure handler even when the dispatch task is cancelled

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(logging): justify the executor submit patch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 19:02:55 +00:00
Yuneng Jiang
da5af0cb27
test: repair two CI tests broken by intentional changes
test_no_linear_scans_in_router: #39674 renamed heuristic_v2_router_limit_violation
to auto_router_capability_violation, so the allowlist entry stopped matching and the
same admin-only scan tripped the static check. Rename the entry to follow it.

tableScrolling.spec.ts: 9ba6cab889 (LIT-4738) gave the Tags and Model Hub tables
client-side pagination at 25 rows, so the 40 seeded rows no longer render on one
page. Select 50 rows per page before counting, as the Logs case already does.
2026-09-05 12:02:48 -07:00
yuneng-jiang
95a1d301b6
Merge pull request #39916 from BerriAI/litellm_e2e_anthropic_chat_completions
test(e2e): cover Anthropic /chat/completions streaming and tool calls
2026-09-05 11:56:24 -07:00
moe-berri
fc3da5e830
Merge branch 'litellm_internal_staging' into litellm_auto_router_compression_split 2026-09-05 11:55:05 -07:00
Yuneng Jiang
45cc2ed082
test(e2e): require a 200 inside the regenerate grace window and drop the helper docstrings 2026-09-05 11:53:22 -07:00
devin-ai-integration[bot]
a670a4621e
fix(proxy): make the invalid-model 403 path cheap under a burst of rejections (#39892)
* fix(proxy): make the invalid-model 403 path cheap under a burst of rejections

Keep the wildcard pattern registry in specificity order at registration time
so route() no longer re-sorts every pattern per lookup, and reuse the
standardized failure payload across the async and threaded sync failure
handlers regardless of what a callback did to log_event_type. Rejections
are still logged and observable.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(router): wrap the filtered pattern tuple the way ruff format wants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router,logging): assert registry order and callback awaits instead of patching a class

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): inject the pattern sorter so the lookup test observes that route() never sorts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 11:53:04 -07:00
devin-ai-integration[bot]
3c0900b7c5
perf(logging): scan large base64 payloads for log truncation off the event loop (#39890)
* perf(logging): scan large base64 payloads for log truncation off the event loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(logging): make base64 offload threshold a plain constant

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 11:51:15 -07:00
yucheng-berri
a0058ed157
fix(hide-secrets): stop redacting benign identifiers (#39879)
* fix(hide-secrets): stop redacting benign identifiers and make redaction deterministic

The OpenAI key detector matched `sk-` anywhere inside a word, so `<task-notification>`
became `<ta[REDACTED]>`, and the Base64 entropy limit of 3.0 flagged ordinary quoted
identifiers such as `"application/json"` and model ids. Redaction also iterated a
hash-seeded set, so the same request produced different bytes on different workers and
broke prompt caching.

- require a standalone `sk-`/`sk_` token with a digit (still catches sk-proj-/sk-ant-)
- raise Base64HighEntropyString limit from 3.0 to the detect-secrets default 4.5
- redact overlapping matches longest-first in a stable order

Resolves LIT-7049

* fix(hide-secrets): treat separators as key boundaries and defer sk_live_ to the stripe detector

The standalone-token boundary also rejected keys glued to a preceding `_`, `-`
or percent-encoded delimiter (`openai_sk-…`, `key-sk-…`, `Bearer%20sk-…`), which
the old pattern redacted, and `sk_live_…` was counted by both the OpenAI and the
Stripe detector.

* fix(hide-secrets): keep the openai key scan linear on repeated sk separators

The digit requirement was a lookahead, so every `sk` inside a long
`[a-zA-Z0-9_-]` run re-scanned the rest of that run looking for a digit.
100 KB of `-sk-` took over 5s in the worker's event loop and the proxy
closed the connection without a response. The check now runs once per
match in `analyze_string` instead.

* chore(hide-secrets): remove redundant performance test comment

* fix(hide-secrets): consume complete openai key tokens

* chore(hide-secrets): remove redundant fixture comment

* chore(hide-secrets): remove redundant test docstrings

* fix(hide-secrets): redact whole stripe live keys

* style(hide-secrets): wrap secret sorting key
2026-09-05 11:47:36 -07:00