Commit graph

5038 commits

Author SHA1 Message Date
jesus
5a9805c102 fix(auth): refresh lite session token team models from team row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 23:08:49 +00:00
Mateo Wang
5b2b5420af
Merge pull request #39626 from BerriAI/litellm_batch_ui_logs
Some checks failed
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
feat(ui): batch observability on the logs page
2026-09-08 15:19:25 -07:00
Mateo Wang
047b8bef31
Merge pull request #38721 from BerriAI/litellm_fix_post_call_policy_pipeline
fix(policy_engine): execute post_call guardrail pipelines on responses and streams
2026-09-08 15:19:19 -07:00
Mateo Wang
13005cb831
Merge pull request #33738 from BerriAI/litellm_lit_4116_drop_params_string_coerce
fix(utils): honor string drop_params values from config and DB deployments
2026-09-08 15:17:54 -07:00
mateo-berri
2400f1befe Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_4116_drop_params_string_coerce
# Conflicts:
#	tests/test_litellm/test_utils.py
2026-09-08 15:08:38 -07:00
devin-ai-integration[bot]
d963e9fa6e
fix(proxy): keep team member budget enforced at the cap and across Redis counter expiry (#40304)
* fix(proxy): keep team member budget enforced at the cap and across Redis counter expiry

The cached-key auth path admitted a request when the team member counter sat exactly at max_budget, and a Redis counter that expired during a long stream was reconciled against this pod's stale in-memory copy, driving the shared counter negative and reopening the budget. Reject at >= like every other budget check, read Redis before the per-pod copy when judging the reconcile delta, and add the settled request cost after a DB reseed since reserved keys skip the normal increment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): reseeded reservation counter also carries the settled request cost

The reseed test asserted counter == DB floor. The floor is read before the async spend flush writes this request, so the counter now lands at floor plus settled cost, matching the after leg in the PR proof

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 15:06:20 -07:00
mateo-berri
6475443efb fix(policy_engine): run per-chunk hook guardrails whose post_call pipeline cannot stream
The per-chunk streaming hook skipped every guardrail stepped by a post_call pipeline, even when the pipeline is left out of the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that gate the stream, the same way the iterator hook already computes it
2026-09-08 15:00:25 -07:00
tin-berri
0175c7da1c
feat(router): resolve max_tokens to the tier model's ceiling on auto-routed requests (#40209)
A client behind an auto-router sends one max_tokens for every tier, so a value
sized for the smallest tier starves a bigger tier's thinking budget and a value
sized for the biggest is rejected by the smallest. After the complexity router
picks a tier, its per-tier litellm_params now carry max_tokens set to the
smallest max_output_tokens across that tier model's deployments (model_info,
then the cost map), applied the same way a per-tier reasoning_effort already
is, on every routing exit including plan mode, the empty-ask default and the
classifier fallback. The router seam collapses whichever ceiling alias a tier
carries onto the surface's own name, so one tier max_tokens reaches chat,
/v1/messages and /v1/responses alike, drops the caller's other carriers of the
same setting before the merge, and stamps the caller's original once so a
fallback into a group no tier owns gets it back instead of a ceiling sized for
the tier that failed.

Proxy-level reservations were sized from the caller's cap before routing, so a
raised cap left them short. Both owners now re-validate at the deployment hook:
the v3 limiter tops up its combined-TPM and project-OTPM reservations to the
final cap or writes the admitted cap back, and the budget limiter re-estimates
on the chosen deployment and grows the reservation or writes the admitted cap
back. An auto-router alias also reserves budget at its priciest tier model now
instead of pricing to zero.

An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens
still wins, and max_tokens_from_tier_model: false forwards the caller's value
unchanged.
2026-09-08 13:31:28 -07:00
devin-ai-integration[bot]
9e18526887
feat(deploy): metrics sidecar and separate metrics port in Helm and Terraform (#40163)
* feat(deploy): expose SSE keepalive, pre-call checks and a metrics sidecar in Helm and Terraform

Typed reliability values on both Helm charts and the AWS/GCP Terraform
modules, a dedicated ClusterIP Service for the separate Prometheus port,
a /health route on the metrics server and dead-worker pruning so the
aggregate does not keep stale multiprocess samples.

Resolves LIT-7142

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(deploy): drop reliability config from Helm and Terraform, keep only the metrics sidecar

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): cover startup pruning of dead workers' live gauges and unsignalable pids

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 13:31:07 -07:00
devin-ai-integration[bot]
d36e032241
fix(proxy): initialize string success/failure callbacks at startup after config load (#38226)
* fix(proxy): eagerly initialize string callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): materialize string callbacks after load_config so later litellm_settings keys are applied

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop casts when snapshotting string callbacks so LIT006 stays at base

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
2026-09-08 13:00:14 -07:00
mateo-berri
cecd481ae3 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_post_call_policy_pipeline 2026-09-08 12:36:13 -07:00
yujonglee
35d1d40a67
fix(ocr): run post-call logging hooks (#40154) 2026-09-08 12:07:25 -07:00
devin-ai-integration[bot]
ee1a6407cb
fix(guardrails): keep guardrail telemetry when a policy pipeline blocks or modifies the response (#40211)
* fix(guardrails): keep guardrail telemetry when a policy pipeline blocks or modifies the response

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): count every raw-snapshot guardrail evaluation and type the telemetry carry helpers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(policy_engine): type the recording guardrail hooks and telemetry test parameters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 11:45:02 -07:00
Mateo Wang
a85c3152ca
fix(proxy): load db credentials in the model reconcile so a worker never serves a model before its credential (#39876)
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential

* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential

* fix(proxy): read credentials from the writer db before the router update and look a credential up once

* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
2026-09-08 10:08:24 -07:00
Mateo Wang
99824533ff
fix(proxy): kill the whole prisma process group when a boot migration command times out (#39509)
* fix(proxy): kill the whole prisma process group when a boot migration command times out

* fix(proxy): keep boot alive when the installed proxy extras has no prisma runner
2026-09-08 10:06:53 -07:00
devin-ai-integration[bot]
978f93ce9c
fix(tool_permission): log expected skip and deny events below WARNING (#40208)
The no-tools skip path in async_pre_call_hook now logs at DEBUG and the
denied-by-rule messages in async_pre_call_hook and _evaluate_tool_calls
log at INFO. The malformed tool arguments warning is unchanged. Adds
regression tests pinning each level

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 10:05:32 -07:00
tin-berri
1af7a403c6
feat(mcp): start the named server's OAuth directly for a resource-scoped gateway flow (#39933)
An aggregate gateway DCR authorize whose RFC 8707 resource resolves to exactly one
gateway-managed oauth2 server sealed that server into the flow and then sent the browser to
the generic connect grid anyway, so the user had to find the server the client had already
named and click Connect.

The connect URL now carries only the flow handle. GET /authorize/flow classifies the sealed
flow as unscoped, interactive, M2M, or stale, and returns the matching state to the page.
Interactive flows require a live per-user vendor credential before minting and do not burn the
flow on an early submit. M2M flows use the gateway's configured service credential and finish
without an interactive OAuth trip. Stale flows fail closed instead of becoming unscoped.
The existing explicit Finish action and a new Cancel path preserve deliberate user intent.
2026-09-07 23:40:02 -07:00
tin-berri
1a6aa98230
fix(spend): compare auto-router targets by deployment identity (#40206)
Preserve deployment identity through savings calculation, with canonical model fallback only when either ID is absent. Cover negotiated rates, unchanged deployments, alias/base-model cache accounting and missing IDs.

Fixes #38811. Based on the deployment-identity approach proposed by @QuantumBreakz in #38834.

Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-07 23:29:42 -07:00
mateo-berri
08b60c409a refactor(guardrails): drop the unused rewrites_streamed_output hook
Nothing calls it since the streaming pipeline detects rewrites at run time
through the stream observer, so the base method and the content filter's
override were dead code with dead tests
2026-09-07 21:53:26 -07:00
mateo-berri
d08a177bc7 fix(policy_engine): keep a policy-added guardrail's other stages when a pipeline steps it
A policy that both adds a guardrail and steps it in a post_call pipeline
used to drop the guardrail from the request's guardrail list outright, so
its pre_call stage never ran. The per-hook loops already skip guardrails
by pipeline mode, so the mode-agnostic subtraction only lost coverage
2026-09-07 21:51:34 -07:00
mateo-berri
91fc1b2010 fix(policy_engine): record a streaming pipeline step once and in the applied guardrails header
CustomGuardrail.__init_subclass__ wrapped _StreamRewriteObserver.apply_guardrail in log_guardrail_information, so every streaming step recorded a second standard_logging_guardrail_information entry and span next to the inner guardrail's own. The observer's method now carries the marker that skips the wrapper. The step also adds the guardrail to the applied guardrails header the way the non-streaming unified path does, so streamed spend rows name the guardrail that scanned them
2026-09-07 21:48:19 -07:00
mateo-berri
69d2ac1edb fix(policy_engine): run iterator-hook guardrails whose post_call pipeline cannot stream
The streaming loop skipped every guardrail stepped by a post_call pipeline, even when the pipeline was dropped from the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_iterator_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that will gate the stream
2026-09-07 21:46:00 -07:00
mateo-berri
b7c2decb7d fix(drop_params): honor string values in litellm_params and the LITELLM_DROP_PARAMS env var
get_litellm_params normalizes drop_params once, so a client-body string and
router_settings.default_litellm_params reach the anthropic, bedrock, and
azure_ai gates as a bool. LITELLM_DROP_PARAMS=false now means off. A value
that is neither a flag nor a string logs one warning and counts as unset,
both in the deployment validator and in litellm_settings.
2026-09-07 19:00:47 -07:00
yucheng-berri
9bc9104102
fix(proxy): log budget reservation notice once at config load (#40167)
* fix(proxy): log disable_budget_reservation notice once at config load

The disabled-budget-reservation reminder fired as a WARNING inside request
authentication, so every authenticated request on a proxy that deliberately
set the flag produced one warning line. The notice now runs once per worker
when general_settings loads, at INFO, and the request path only skips the
reservation. Reservation skipping and read-time budget checks are unchanged

* fix(proxy): keep budget notice sentinel with constants

* fix(proxy): expose shared budget notice state
2026-09-07 18:18:28 -07:00
mateo-berri
2f397fa128 fix(drop_params): honor string flags in litellm_settings and responses, and fail open on non-flag values 2026-09-07 18:06:29 -07:00
Mateo Wang
26d589cd28
Merge pull request #39234 from BerriAI/litellm_fix_agent_mcp_grants
fix(mcp): clear error when an agent-bound key is denied a scoped MCP server + agent MCP grants in the UI
2026-09-07 18:02:16 -07:00
mateo-berri
192ea9ec80 fix(policy_engine): fail open on streaming shapes post_call pipelines cannot govern yet
A post_call pipeline now releases the original stream instead of refusing the
request on every shape it has no handler for: a background request, a pipeline
guardrail without the unified apply_guardrail interface, a route with no
endpoint translation, a buffered stream no translation resolves, and a rewrite
the translation cannot write back (tool-call edits, text edits on translations
without write-back, n>1 chat, an unended Anthropic stream, a Responses dump
with no event envelope). Each case logs a warning naming the policy and
guardrail. Real blocks and writable text masks are unchanged.
2026-09-07 17:55:37 -07:00
mateo-berri
0d5ea553da Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_post_call_policy_pipeline 2026-09-07 17:24:10 -07:00
yucheng-berri
1009976c49
fix(bedrock): keep x-amzn-RequestId on chat error responses (#40089)
* fix(bedrock): keep x-amzn-RequestId on chat error responses

Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.

- converse and invoke chat handlers pass the real headers and response when
  they turn an httpx.HTTPStatusError into a BedrockError, and read the body
  through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
  invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
  caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
  siblings instead of fabricating a blank one

The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.

* fix(bedrock): keep request-id on text-classified errors

The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.

Also narrows the new BedrockError types and trims its docstrings.

* chore(bedrock): drop the docstrings on the new error helpers

* fix(bedrock): keep request-id on every error path that has one

The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.

Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.

Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.

The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.

* fix(bedrock): forward provider headers from the count tokens route

The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.

get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.

* fix(bedrock): classify every bedrock surface through BedrockError

Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.

* fix(proxy): keep provider headers on the httpx status error branch

_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.

* fix(bedrock): keep the request id on the timeout mappings

Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.

* fix(bedrock): keep the provider response on mapped timeouts

The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.

Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.

* chore(bedrock): drop the explanatory comment on Timeout.response
2026-09-07 17:16:47 -07:00
mateo-berri
4cc0180eab fix(router): keep unresolved drop_params strings so DB rows and env refs survive
The drop_params validator collapsed every string it did not recognize to None. A pre-fix DB row holds the flag as ciphertext, so a partial PATCH rebuilt the deployment without it and dropped the key from the stored row, and /model/new turned an os.environ/ reference into nothing before the loader could resolve it. The validator now returns the raw value when it is not a boolean flag, the field admits strings the way timeout already does, and the flag set follows pydantic's lax bool parsing instead of a hand-rolled true/false pair
2026-09-07 17:06:34 -07:00
mateo-berri
e01bb98960 merge: bring litellm_internal_staging into litellm_fix_agent_mcp_grants again
Staging moved by the auto-router classifier cost change (#40168) between the
first merge and its push; this merge picks it up so the PR merges cleanly
2026-09-07 16:36:55 -07:00
mateo-berri
5e4dec4b88 merge: bring litellm_internal_staging into litellm_fix_agent_mcp_grants
Take staging's test_bedrock_knowledgebase_hook.py, which drops the duplicate
embedding_executor parameter that turned the lint check red, and make the two
cross-module helpers this branch added public (raise_denied_scoped_mcp_access
and routes_through_gateway) so the private-usage budget stays at its base count
2026-09-07 16:35:40 -07:00
tin-berri
9d0c9b9382
feat(ui): itemize auto-router classification spend (#40168)
Resolves LIT-7141

Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-07 16:29:19 -07:00
tin-berri
cd681a573f
fix(mcp): encrypt stored static headers and stdio environment (#40164)
Encrypt secret maps at the shared persistence boundary, preserve plaintext API/runtime views, and extend rotation and migration scanning to legacy rows.

Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-07 16:03:06 -07:00
ryan-crabbe-berri
38683643e0
Merge pull request #40170 from BerriAI/litellm_chatgpt_add_model_provider
feat(ui): list the ChatGPT subscription provider in the Add Model form
2026-09-07 15:21:58 -07:00
Mateo Wang
058d260509
Merge pull request #38914 from BerriAI/litellm_fix_skills_hook_import_side_effect
fix(proxy): register SkillsInjectionHook at proxy startup instead of import time
2026-09-07 15:18:54 -07:00
Mateo Wang
0e9e2c01f3
Merge pull request #38806 from BerriAI/litellm_fix_mcp_test_connection_oauth_bearer
fix(mcp): forward staged credentials on /mcp-rest/test/connection like /test/tools/list
2026-09-07 15:18:35 -07:00
ryan-crabbe-berri
bb2db2d3f8 test(proxy): drop docstrings from the Add Model provider tests
Move the only guidance worth keeping into the parity assertion message so a
failing run tells the contributor to add a catalog entry instead of growing
the frozen unlisted set.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-07 15:12:41 -07:00
ryan-crabbe-berri
8e5a12057a feat(ui): list the ChatGPT subscription provider in the Add Model form
The Add Model provider dropdown is driven entirely by provider_create_fields.json,
and chatgpt had no entry there, so the documented ChatGPT subscription setup was
unreachable from the Admin UI. Add the entry plus the dashboard enum, slug, logo and
placeholder mappings so the provider can be selected and its cost-map models listed.

The entry carries no credential fields on purpose: the chatgpt backend ignores
api_key and api_base and signs in through the device-code auth file on the proxy
host, so any field here would be inert.

Add a parity test that every LlmProviders value is either listed for Add Model or
frozen in an explicit unlisted set, so a new backend provider cannot silently miss
the dropdown again.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-07 14:55:43 -07:00
tin-berri
c5ec2eedc1
fix(spend): price caching savings on the billed request basis (#40160)
Resolves LIT-7137

Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-07 21:41:05 +00:00
devin-ai-integration[bot]
038025ba5e
fix(guardrails): accept on_violation block and alert for mcp_security (#40155)
* fix(guardrails): accept on_violation block and alert for mcp_security

The MCP Security policy template sends on_violation: "block", but the shared
LitellmParams model only allowed the /v1/realtime values "warn" and
"end_session", so POST /guardrails returned 422 before the MCP guardrail was
initialized. Widen the literal to include the MCP actions, map every non-alert
value to MCP's default "block" at init, and regenerate the lazy OpenAPI
snapshot and dashboard API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): restrict on_violation block/alert to mcp_security and keep legacy MCP mapping

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): return 422 when PATCH sets an mcp_security-only on_violation on another guardrail

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-07 14:33:02 -07:00
devin-ai-integration[bot]
b618c7ad86
fix(proxy): let authorized internal users open vector store details (#40150)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-07 13:10:09 -07:00
yucheng-berri
e04e5d7113
fix(router): keep provider response headers on streaming chat completions (#40091)
* fix(router): keep provider response headers on streaming chat completions

The Router re-wraps a deployment's CustomStreamWrapper in FallbackStreamWrapper
(and its sync twin) so a mid-stream failure can fail over. Neither wrapper
forwarded `_response_headers`, so every streaming chat completion handed the
proxy's callbacks and its response-header builder a wrapper with no provider
headers, and a successful mid-stream fallback still published the failed
deployment's identity, `x-request-id` and rate limit counters.

Forward `_response_headers` into both wrappers, repoint the wrapper at the
deployment that served the stream once a fallback takes over, and rebuild the
proxy's response headers from that deployment while `create_response` still has
the first chunk buffered.

* fix(router): follow a nested fallback to the deployment that served the stream

A fallback the router picks is itself a fallback-aware wrapper, and it only
repoints at its own fallback once it yields, so reading its hidden params at
selection time named a deployment that produced no output. Re-read them when
the first fallback item arrives, which is still before the proxy commits
response headers.

Also addresses review feedback: the streaming header builder reads self.data
instead of taking a coarse request_data parameter, and the new test recorder
local is Final.

* test(router): cover the fallback header adoption helper directly

The router_code_coverage gate wants every router.py function named in a
router test, and this also pins the weak-reference behavior: a wrapper
collected mid-stream must not break the generator still draining it.

* refactor(proxy): take a read-only mapping for the model-id lookup

_get_model_id_from_response only reads its request payload, so a Mapping
says what it needs and the two metadata hops are narrowed instead of
assumed to be dicts.

* test: drop mutable recorder locals and routine comments from the new tests

An AsyncMock await_count and an asyncio.Event say the same thing as a
list and a dict that the test mutates.

* chore(router): justify the two rebinds in the fallback loops

Both are the one-shot re-read that follows a nested fallback, so they get
the repo's rebind-ok note like the rest of the file.
2026-09-07 12:59:57 -07:00
ryan-crabbe-berri
11702ae4e1
Merge pull request #38204 from davida-ps/litellm_prompt_security_file_modify_block
fix(guardrails): block Prompt Security file modifications
2026-09-07 12:53:57 -07:00
devin-ai-integration[bot]
6908318c16
feat(keys): allow editing soft budget on existing keys (#39002)
* feat(keys): allow editing soft budget on existing keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(ui): extract KeyBudgetNumberField to keep key_edit_view under max-lines

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(ui): format keyEditFormValues with prettier

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(keys): cover soft budget validation and update adapter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): reject non-finite soft budget values instead of clearing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(keys): assert soft budget validation returns None for valid values

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(keys): write soft budget and key row in one transaction

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
2026-09-07 19:37:44 +00:00
Yassin Kortam
192e38fa7b
feat(skills): semantic search over the LiteLLM-hosted skill registry (#39401)
* feat(skills): semantic search over the LiteLLM-hosted skill registry

Adds GET /v1/skills?query= (custom_llm_provider=litellm_proxy) and a
skill_search MCP virtual tool, ranking the caller's accessible skills by
semantic similarity, mirroring the A2A agent registry search (LIT-6309).

Also fixes a pre-existing bug where create_skill() dropped description and
instructions for the litellm_proxy provider, which left every LiteLLM-hosted
skill with no searchable text.

* fix(mcp): coerce skill_search top_k instead of raising 500 on malformed input

The MCP-REST skill_search dispatch validated raw tool arguments through a
pydantic model directly, so a non-numeric top_k raised a ValidationError
that the endpoint's catch-all turned into an HTTP 500. Mirrors the
agent_search branch's tolerant coerce_top_k handling instead.

* fix(skills): enforce key limits on search embeddings and bound the semantic index

Semantic search embeddings now run the same pre_call_hook the /embeddings
route runs, so key rate limits, budgets and guardrails apply before the
embedding model is called. The shared SemanticTextIndex caps cached vectors
and evicts the least recently searched entries, and each skill's embedded
text is capped so one skill cannot inflate the embedding batch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(skills): surface proxy 429s from search embeddings instead of a 503

ProxyRateLimitError is also an OpenAIError, so the search engine was folding
a key rate limit into skill_search_unavailable. Proxy HTTPExceptions now
propagate so the caller gets the same 429 the /embeddings route returns

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(skills): import assert_never from typing_extensions for Python 3.10

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(skills): embed the request as the pre-call hooks returned it, not the original text

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(skills): keep the litellm_proxy provider check for GET /v1/skills?query= inside llms/

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(skills): move the GET /v1/skills?query= endpoint tests under tests/test_litellm/proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-07 12:28:38 -07:00
tin-berri
eea4c860f7
feat(ui): add key-scoped auto-router usage tab (#39999)
* feat(ui): add key-scoped auto-router usage tab

GET /auto_router/benchmarks takes an optional api_key filter, applied in the
rollup aggregate on the primary key's leading column. Proxy admins get a
separate Auto-router usage tab on key detail pages with spend, baseline,
savings, tier routing, cache metrics and the existing router selector

* fix(ui): share key analytics date range
2026-09-07 12:12:22 -07:00
Mateo Wang
2cfe106f9f
Merge pull request #39660 from BerriAI/litellm_bedrock_passthrough_model_access
fix(proxy): enforce key and team model access on Bedrock passthrough routes (internal copy of #34244)
2026-09-07 12:11:15 -07:00
tin-berri
1ae3216120
fix(router): preserve default heuristic updates (#40007) 2026-09-07 12:05:43 -07:00
Roman D
55fe4a7894
feat(proxy): resolve root_path per request from a configured prefix list (SERVER_ROOT_PATHS) (#35935)
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS

One deployment can encode exactly one client-visible URL path prefix
today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so
a pod fronting several ingress prefixes 404s every prefix but one before
any handler runs, and MCP OAuth discovery can emit only one prefix's
URLs (RFC 9728 section 3 exact-match fails for the rest).

Add an opt-in outermost ASGI middleware that matches the request path
against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on
a segment boundary and sets scope["root_path"] for that request only.
Everything downstream is stock Starlette: route matching strips
root_path so routes stay registered root-relative, and request.base_url
re-includes it, so the discovery documents' resource and the 401
challenges' resource_metadata land under the prefix the client actually
called — with no discovery-builder changes.

LazyFeatureMiddleware now strips the scope root_path (falling back to
the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so
lazily-registered routers — the MCP OAuth discovery router among them —
load under per-request prefixes.

Follow-up to the routing discussion on #35226; composes with, but does
not depend on, #35576.

* fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate)

* review(greptile): trim implementation commentary; fixture-own MCP registry state in tests

Addresses both P2s from the first Greptile pass:
- per_request_root_path_middleware.py (and the related _lazy_features /
  proxy_server comments) cut down to the constraints the code cannot
  express, per repo comment guidance
- the new discovery tests no longer clear/repopulate the shared MCP
  registry inline; a fixture snapshots it, hands the test an empty
  registry, and restores it afterwards so no state leaks between cases

* fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate)

* fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path

The per-request root_path middleware sets scope["root_path"] to the
prefix the client actually called, but the OAuth 401 challenges
(raise_user_oauth_challenge / raise_token_exchange_challenge) still
built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting
several prefixes, the challenge advertised a discovery URL under a
different prefix than the discovery document served — the two
disagreed on where the resource metadata lives, and a strict RFC 9728
client refused the challenge. Route the challenges through a small
ContextVar the middleware populates so they read the same effective
root_path Starlette resolves the request under.

The same accessor fixes get_custom_url: when a request lives under a
SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it,
so appending the SERVER_ROOT_PATH scalar on top produced e.g.
/tenant-a/legacy/sso/callback — a path that does not exist. Reading
the per-request prefix instead (and relying on join_paths's tail-dedup)
keeps SSO login/callback URLs under one prefix — the one the request
actually arrived on.

Fallback: outside a request (module-load-time UI URL builders,
background tasks) the ContextVar is unset and the accessor reads
SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only
deployments are byte-identical.

* fix(mcp): challenge URL under per-request prefix must route, and mock parity

Two follow-ups to the review fix that made the 401 challenge use the
per-request root_path:

1. oauth_protected_resource_path must pick the URL structure that
   actually routes for the mechanism in use:
   - The scalar SERVER_ROOT_PATH deployment registers the well-known
     routes with the prefix INSERTED (via well_known_root_suffix at
     import time), matching RFC 8414 §3. The challenge URL must use the
     same insertion or a client fetching it 404s.
   - The per-request SERVER_ROOT_PATHS deployment can't register routes
     per prefix; PerRequestRootPathMiddleware strips the prefix from
     scope["path"] and the router matches the un-inserted route. The
     URL must place the prefix BEFORE .well-known so the strip leaves a
     matching path.
   The previous fix used the insertion form for both, which 404'd the
   discovery fetch on the per-request path — the discovery doc and the
   challenge would then disagree on where the resource metadata lives,
   the very failure the review flagged. End-to-end verified: the URL
   the challenge advertises routes and the doc's `resource` field
   equals the URL the client originally called (RFC 9728 §3).

2. get_request_root_path now delegates its fallback through
   get_server_root_path() instead of reading the env directly, so every
   existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"`
   test override keeps working. This unstubbed the mock on the /v2/login
   test that failed on the last CI run.

Plus the lint budget: annotate the local accumulator Final, tag the
scope["root_path"] rewrite as an intentional ASGI-contract mutation,
tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and
add reason strings to the two new PLC0415 lazy-import noqas.

* test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL

End-to-end regression test that mounts the discoverable router + the
per-request root_path middleware, hits an MCP endpoint that raises
raise_user_oauth_challenge, fetches the resource_metadata URL the
challenge advertises, and checks the returned document's `resource`
equals the URL the client originally called (RFC 9728 §3 exact match).

Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so
a regression on any prefix — challenge URL 404s, or doc emits a
different prefix than the client called — fails at this test rather
than in a strict MCP client's discovery.

---------

Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com>
2026-09-07 11:54:59 -07:00