Move the only guidance worth keeping into the parity assertion message so a
failing run tells the contributor to add a catalog entry instead of growing
the frozen unlisted set.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
The Add Model provider dropdown is driven entirely by provider_create_fields.json,
and chatgpt had no entry there, so the documented ChatGPT subscription setup was
unreachable from the Admin UI. Add the entry plus the dashboard enum, slug, logo and
placeholder mappings so the provider can be selected and its cost-map models listed.
The entry carries no credential fields on purpose: the chatgpt backend ignores
api_key and api_base and signs in through the device-code auth file on the proxy
host, so any field here would be inert.
Add a parity test that every LlmProviders value is either listed for Add Model or
frozen in an explicit unlisted set, so a new backend provider cannot silently miss
the dropdown again.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
* fix(guardrails): accept on_violation block and alert for mcp_security
The MCP Security policy template sends on_violation: "block", but the shared
LitellmParams model only allowed the /v1/realtime values "warn" and
"end_session", so POST /guardrails returned 422 before the MCP guardrail was
initialized. Widen the literal to include the MCP actions, map every non-alert
value to MCP's default "block" at init, and regenerate the lazy OpenAPI
snapshot and dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): restrict on_violation block/alert to mcp_security and keep legacy MCP mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): return 422 when PATCH sets an mcp_security-only on_violation on another guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep provider response headers on streaming chat completions
The Router re-wraps a deployment's CustomStreamWrapper in FallbackStreamWrapper
(and its sync twin) so a mid-stream failure can fail over. Neither wrapper
forwarded `_response_headers`, so every streaming chat completion handed the
proxy's callbacks and its response-header builder a wrapper with no provider
headers, and a successful mid-stream fallback still published the failed
deployment's identity, `x-request-id` and rate limit counters.
Forward `_response_headers` into both wrappers, repoint the wrapper at the
deployment that served the stream once a fallback takes over, and rebuild the
proxy's response headers from that deployment while `create_response` still has
the first chunk buffered.
* fix(router): follow a nested fallback to the deployment that served the stream
A fallback the router picks is itself a fallback-aware wrapper, and it only
repoints at its own fallback once it yields, so reading its hidden params at
selection time named a deployment that produced no output. Re-read them when
the first fallback item arrives, which is still before the proxy commits
response headers.
Also addresses review feedback: the streaming header builder reads self.data
instead of taking a coarse request_data parameter, and the new test recorder
local is Final.
* test(router): cover the fallback header adoption helper directly
The router_code_coverage gate wants every router.py function named in a
router test, and this also pins the weak-reference behavior: a wrapper
collected mid-stream must not break the generator still draining it.
* refactor(proxy): take a read-only mapping for the model-id lookup
_get_model_id_from_response only reads its request payload, so a Mapping
says what it needs and the two metadata hops are narrowed instead of
assumed to be dicts.
* test: drop mutable recorder locals and routine comments from the new tests
An AsyncMock await_count and an asyncio.Event say the same thing as a
list and a dict that the test mutates.
* chore(router): justify the two rebinds in the fallback loops
Both are the one-shot re-read that follows a nested fallback, so they get
the repo's rebind-ok note like the rest of the file.
* feat(skills): semantic search over the LiteLLM-hosted skill registry
Adds GET /v1/skills?query= (custom_llm_provider=litellm_proxy) and a
skill_search MCP virtual tool, ranking the caller's accessible skills by
semantic similarity, mirroring the A2A agent registry search (LIT-6309).
Also fixes a pre-existing bug where create_skill() dropped description and
instructions for the litellm_proxy provider, which left every LiteLLM-hosted
skill with no searchable text.
* fix(mcp): coerce skill_search top_k instead of raising 500 on malformed input
The MCP-REST skill_search dispatch validated raw tool arguments through a
pydantic model directly, so a non-numeric top_k raised a ValidationError
that the endpoint's catch-all turned into an HTTP 500. Mirrors the
agent_search branch's tolerant coerce_top_k handling instead.
* fix(skills): enforce key limits on search embeddings and bound the semantic index
Semantic search embeddings now run the same pre_call_hook the /embeddings
route runs, so key rate limits, budgets and guardrails apply before the
embedding model is called. The shared SemanticTextIndex caps cached vectors
and evicts the least recently searched entries, and each skill's embedded
text is capped so one skill cannot inflate the embedding batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): surface proxy 429s from search embeddings instead of a 503
ProxyRateLimitError is also an OpenAIError, so the search engine was folding
a key rate limit into skill_search_unavailable. Proxy HTTPExceptions now
propagate so the caller gets the same 429 the /embeddings route returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): import assert_never from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): embed the request as the pre-call hooks returned it, not the original text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(skills): keep the litellm_proxy provider check for GET /v1/skills?query= inside llms/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(skills): move the GET /v1/skills?query= endpoint tests under tests/test_litellm/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add key-scoped auto-router usage tab
GET /auto_router/benchmarks takes an optional api_key filter, applied in the
rollup aggregate on the primary key's leading column. Proxy admins get a
separate Auto-router usage tab on key detail pages with spend, baseline,
savings, tier routing, cache metrics and the existing router selector
* fix(ui): share key analytics date range
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS
One deployment can encode exactly one client-visible URL path prefix
today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so
a pod fronting several ingress prefixes 404s every prefix but one before
any handler runs, and MCP OAuth discovery can emit only one prefix's
URLs (RFC 9728 section 3 exact-match fails for the rest).
Add an opt-in outermost ASGI middleware that matches the request path
against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on
a segment boundary and sets scope["root_path"] for that request only.
Everything downstream is stock Starlette: route matching strips
root_path so routes stay registered root-relative, and request.base_url
re-includes it, so the discovery documents' resource and the 401
challenges' resource_metadata land under the prefix the client actually
called — with no discovery-builder changes.
LazyFeatureMiddleware now strips the scope root_path (falling back to
the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so
lazily-registered routers — the MCP OAuth discovery router among them —
load under per-request prefixes.
Follow-up to the routing discussion on #35226; composes with, but does
not depend on, #35576.
* fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate)
* review(greptile): trim implementation commentary; fixture-own MCP registry state in tests
Addresses both P2s from the first Greptile pass:
- per_request_root_path_middleware.py (and the related _lazy_features /
proxy_server comments) cut down to the constraints the code cannot
express, per repo comment guidance
- the new discovery tests no longer clear/repopulate the shared MCP
registry inline; a fixture snapshots it, hands the test an empty
registry, and restores it afterwards so no state leaks between cases
* fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate)
* fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path
The per-request root_path middleware sets scope["root_path"] to the
prefix the client actually called, but the OAuth 401 challenges
(raise_user_oauth_challenge / raise_token_exchange_challenge) still
built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting
several prefixes, the challenge advertised a discovery URL under a
different prefix than the discovery document served — the two
disagreed on where the resource metadata lives, and a strict RFC 9728
client refused the challenge. Route the challenges through a small
ContextVar the middleware populates so they read the same effective
root_path Starlette resolves the request under.
The same accessor fixes get_custom_url: when a request lives under a
SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it,
so appending the SERVER_ROOT_PATH scalar on top produced e.g.
/tenant-a/legacy/sso/callback — a path that does not exist. Reading
the per-request prefix instead (and relying on join_paths's tail-dedup)
keeps SSO login/callback URLs under one prefix — the one the request
actually arrived on.
Fallback: outside a request (module-load-time UI URL builders,
background tasks) the ContextVar is unset and the accessor reads
SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only
deployments are byte-identical.
* fix(mcp): challenge URL under per-request prefix must route, and mock parity
Two follow-ups to the review fix that made the 401 challenge use the
per-request root_path:
1. oauth_protected_resource_path must pick the URL structure that
actually routes for the mechanism in use:
- The scalar SERVER_ROOT_PATH deployment registers the well-known
routes with the prefix INSERTED (via well_known_root_suffix at
import time), matching RFC 8414 §3. The challenge URL must use the
same insertion or a client fetching it 404s.
- The per-request SERVER_ROOT_PATHS deployment can't register routes
per prefix; PerRequestRootPathMiddleware strips the prefix from
scope["path"] and the router matches the un-inserted route. The
URL must place the prefix BEFORE .well-known so the strip leaves a
matching path.
The previous fix used the insertion form for both, which 404'd the
discovery fetch on the per-request path — the discovery doc and the
challenge would then disagree on where the resource metadata lives,
the very failure the review flagged. End-to-end verified: the URL
the challenge advertises routes and the doc's `resource` field
equals the URL the client originally called (RFC 9728 §3).
2. get_request_root_path now delegates its fallback through
get_server_root_path() instead of reading the env directly, so every
existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"`
test override keeps working. This unstubbed the mock on the /v2/login
test that failed on the last CI run.
Plus the lint budget: annotate the local accumulator Final, tag the
scope["root_path"] rewrite as an intentional ASGI-contract mutation,
tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and
add reason strings to the two new PLC0415 lazy-import noqas.
* test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL
End-to-end regression test that mounts the discoverable router + the
per-request root_path middleware, hits an MCP endpoint that raises
raise_user_oauth_challenge, fetches the resource_metadata URL the
challenge advertises, and checks the returned document's `resource`
equals the URL the client originally called (RFC 9728 §3 exact match).
Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so
a regression on any prefix — challenge URL 404s, or doc emits a
different prefix than the client called — fails at this test rather
than in a strict MCP client's discovery.
---------
Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com>
* fix(spend_logs): keep partition DDL transactions alive for their statement timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format changed files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): avoid dict-literal kwargs and keep cast-ok on the cast line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): cast at the call site instead of widening PrismaClient.tx
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): require partition tx timeout to strictly exceed statement bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): prove the virtual key lifecycle on every replica
Walks one virtual key through create, read, partial update, clear, enforce
and delete against a live proxy and database, reading every write back on
every gateway replica.
The management suite already had single write-then-read tests for keys, but
none of them proved that a partial /key/update leaves the untouched fields
alone, that an explicit null clears a field, or that a write is visible on
more than the one gateway that took it.
Adds read_back_everywhere to the shared ProxyClient: it polls a GET path on
every URL in PROXY_REPLICA_URLS until each replica's parsed body satisfies
the caller's predicate, and fails naming the replica that never converged.
The CLEAR sentinel in the e2e models makes an explicit JSON null expressible
in a body the transport otherwise strips of None fields.
Documents /key/update's merge patch semantics on the endpoint docstring.
* test(e2e): prove key revocation and field preservation on every replica
Applies the findings from an adversarial review of the first commit.
The delete step only checked that chat was refused on the gateway that took
the write, so it would have passed while a sibling gateway kept serving the
deleted key. It now serves one call from every replica first, so each has the
key cached and the delete has something to revoke everywhere, then polls every
replica for the refusal.
The file also carried its own poll loop that tested the deadline before
attempting, so it gave up one attempt early and skipped the attempt landing
exactly on the deadline. It now shares the harness helper, which is generic
over the polled value rather than over a parsed body, so the same loop covers
both the info read-back and the chat refusal.
The model the enforcement step registers now carries a unique marker in its
alias, matching every other deployment this suite creates, so concurrent runs
never share one model group.
The docstring sentence claimed an explicit null clears any field. It does not:
the metadata-backed fields merge into stored metadata, where a null is a silent
no-op, and only the key's own columns clear. Regenerating the dashboard types
picks up the corrected text.
* fix(e2e): delete a deployment that never becomes servable
Registering a model posts /model/new and then waits for every replica to list
it. When that wait timed out the deployment already existed in the database but
its id had never been returned, so no caller could delete it and the row
outlived the run. It is now deleted before the failure propagates.
Found by review on the key lifecycle suite, whose module fixture registers a
deployment this way, but every caller of the shared helper had the same
exposure.
* docs(e2e): drop the duplicated notes from the lifecycle docstrings
The delete method restated what the warm-up helper already explains, and the
module restated the merge patch rule that the endpoint and the request model
both document.
* test(e2e/ui): cover member role and budget edits, member permission delegation, and team guardrail removal
Three Playwright specs for the Teams flows enterprise customers hit most, each
owning its fixtures and proving the mutation through a read-back rather than a
toast.
- teamMemberEdit: an admin edits a member's team role and per-member budget,
and both survive a reload of the Members table
- memberPermissions: a plain member is refused /key/generate for their team,
a team admin grants it on the Member Permissions tab, and the member then
creates a team key that serves a real completion
- teamGuardrailRemoval: clearing a team's only guardrail on the Settings tab
really clears it, and traffic the guardrail refused starts serving again
* test(e2e/ui): make the new team specs safe to run in parallel
Fixture ids came from Date.now(), so two repeats starting in the same
millisecond minted the same user id: one got a 409 and the loser's teardown
deleted the user the other was still signed in as. Ids now carry a random
suffix.
Also move the member-permissions setup inside the cleanup-protected block so a
half-finished setup cannot leak a team, and close both browser contexts the
test opens.
* fix(mcp): apply key and team guardrails to MCP tool calls
Guardrails attached to a virtual key or team were only enforced on LLM
routes. The synthetic request built for MCP tool call guardrail hooks
carried no guardrails in its metadata, so a guardrail with default_on
false never ran on tools/call even when the key explicitly listed it.
Resolve key, team, and project guardrails onto the synthetic request
with the same helper the chat path uses.
* fix(mcp): pass project metadata through without a mutable default
* fix(mcp): mark the request dict parameter mutable-ok with a reason
* test(mcp): explain the premium_user patch and tighten the helper docstring
CI flagged the spec flaky twice more. Both were the same defect in different
places: the Logs drawer renders several nodes per string and the first in DOM
order is often hidden, so .first() waited 20s on an invisible element. The
entity assertions had a second problem on top, since getByText("EMAIL_ADDRESS")
substring-matched the masked prompt div, whose text contains <EMAIL_ADDRESS>,
rather than the entity chip.
Route every drawer assertion through onlyVisible, and match the entity type
and score exactly, which is what the panel renders them as: entity_type and
"Score: N.NN" each get their own span.
Verified on a live stack: 6 of 6 solo runs and the guardrails folder 5 of 5.
Mutating the analyzer to detect nothing turns the spec red on the raw address
reaching the spend log, so the assertions still carry their weight.
The Logs drawer renders the masked prompt in three nodes and the first one
in DOM order is hidden, so the previous commit's .first() traded a strict
mode violation for a locator that waits 20s on an invisible element. Local
runs against a warm stack failed on it every time, resolving the node 22
times and reporting "unexpected value hidden" each time.
Use onlyVisible, the helper this spec already uses for the playground
selectors, which filters to the visible node before taking the first. Solo
runs go 3 for 3 and the guardrails folder passes 5 of 5.
Also drop the 30s wait on the PII step added while chasing this: cold start
was never the cause, and it left the spec sitting on a dead locator longer.
The Logs drawer renders the masked prompt in three places, so matching it
without narrowing raised a strict mode violation instead of asserting
visibility. CI caught it as a flake: the spec failed its first attempt on
b0e53bfbe4 and passed on retry, which is a locator defect rather than a
timing one and would have gone red on any run that saw all three nodes.
Narrow to the first match, matching the guardrail-name assertion above it.
The leak checks below stay on toHaveCount(0), which is unaffected by
multiple matches and is what actually proves nothing raw reached the drawer.
