Three P1 defects from the auth_v2 review, plus the policy-admin flag guard:
- JWKS cache was dead: JWKSProvider was rebuilt per request, so its TTL cache
never survived and every JWT auth refetched the JWKS over the network. Providers
are now cached per jwks_uri at module level.
- Inference with no model bypassed casbin: a body omitting "model" skipped the
call check entirely, so a subject with no grants could reach inference. Now a
model-less inference request is denied up front (before any enrichment/budget).
- Policy-admin endpoints used a v1 _require_admin role check that rejected
JWT/OAuth2 admins casbin had already authorized. Removed it - the routes are in
the casbin route map (policy read/write/delete), so user_api_key_auth enforces
them - and added a per-request flag guard so the unconditionally-registered
router 404s when auth_v2 is off.
Regression tests for each. mypy clean on 23 files, 143 tests green.
The RFC 7662 introspection client secret was read straight from config. Resolve
it through litellm's get_secret_str at introspection time so it can live in a
secret manager (or env ref) instead of plaintext config; a literal value passes
through unchanged. Resolved in _introspect rather than at settings load so the
secret-manager lookup never runs in can_handle for non-OAuth tokens.
mypy clean on 23 files, 141 tests green.
Adds an in-process, dependency-free metrics collector so the authz layer is
observable: decision counts (keyed decision/resource/action), authz latency
(count + sum over the casbin enforce call), and policy-cache hit/miss.
- metrics.py exposes a singleton with observe_decision / observe_latency /
record_cache and a typed snapshot() for a /metrics export at the edge.
- The authorizer counts every allow/deny/loud-open and times the enforce call;
the entry point does the same for the model-call decision; the policy store
records cache hit vs miss. No hot-path dependency - just counters.
Fully typed (no new bare Any), mypy clean on 23 files, 141 tests green, still
imports with casbin/authlib absent.
Every authorization decision now flows through one audit record - the compliance
trail enterprise auth needs and the biggest gap called out in the design review.
- audit.py: a frozen AuthzDecision (decision, subject, domain, obj, action, route,
reason, auth_method) and record(), which logs the decision and fans out to any
registered sinks (DB / SIEM). Sinks are isolated: a failing sink is logged but
never affects the request outcome.
- The authorizer records allow, deny, and loud-open for every governed and
ungoverned control-plane route; the entry point records the model-call (call)
decision on the inference path. auth_method is threaded through so the record
shows how the principal authenticated.
- register_sink / reset_sinks exported for wiring a durable sink.
Fully typed (no new Any), mypy clean on 22 files, 134 tests green, still imports
with casbin/authlib absent.
The decision core used Any where it was over-cautious about coupling. Replace it
with real types (under TYPE_CHECKING, so the framework-free core stays import-
light) and small Protocols:
- identity is UserAPIKeyAuth end to end (authenticators, entry, principal,
context, stages); the authenticator chain returns it, not Any
- SupportsEnforce types the authorizer's engine param; CasbinRuleRow / PolicyDB /
PolicyAdminDB type the dynamic Prisma casbin-rule access (read and write),
narrowed with one explicit cast at the adapter boundary
- AuthContext deps are PrismaClient / UserApiKeyCache / ProxyLogging / Span
- loaders, settings, key sets, role conversion all carry their real types
The only Any left is Dict[str, Any] for genuinely arbitrary JSON (decoded JWT
claims, RFC 7662 introspection responses, request bodies) - that is the honest
type, not a gap. Verified: mypy clean on all 21 files, zero bare Any annotations,
129 tests green, still imports with casbin/authlib absent.
Pure move (git mv, history preserved): the flat 17-file package hid the
architecture behind an alphabetical wall. Group by request phase so the module
is self-documenting:
authn/ authenticators, jwt_claims, jwt_verifier, oauth2_introspection
authz/ enforcer + model.conf, authorizer, route_map, policy_store, policy_admin
stages/ enrichment, end_user, budgets
(top) entry, context, principal, management_endpoints, __init__
No content changes beyond rewired relative imports; tests moved to mirror the
layout. Behavior is identical — the same 129 tests pass, mypy/black/ruff clean,
and the package still imports with casbin/authlib absent.
The enforcer typed its rule params as List[Sequence[str]]; callers pass
list[list[str]], which mypy rejects because List is invariant. Widen to
Sequence[Sequence[str]] (covariant), which accepts list[list[str]]. Type-only
change; clears the 4 mypy errors from make lint-mypy.
Closes the hierarchy-budget gap: team, organization, and global caps live in v1's
common_checks (not the pre-call hooks), which auth_v2 does not run, so they were
unenforced under v2. enforce_hierarchy_budgets calls the exact same functions v1
uses - _team_max_budget_check, _organization_max_budget_check, and
get_global_proxy_spend + _global_proxy_budget_check - so there is one budget
implementation with two callers (true single authority), with the correct
spend-counter conventions and no edit to v1's path. Wired into the inference
branch for all login types; a breach surfaces as the same 429 ProxyException v1
raises.
Verified in-process: a team over budget is blocked (BudgetExceededError), under
budget is allowed, and a teamless identity is a no-op. Real-counter accuracy
across pods remains a live check (see INTEGRATION.md).
Drives the real max_budget_limiter pre-call hook with an enrichment-shaped
identity (teamless JWT/OAuth: no key, user-level budget filled from the user
row), mocking only the spend counter, to confirm the enrichment -> hook chain:
over budget -> 429, under budget -> allowed, no user budget -> no personal
enforcement, and a team member's personal budget is skipped (documenting that
team users are governed by the team budget, the item-4 gap). Real-counter
accuracy is the one piece still needing a live proxy.
In-process integration test with TestClient and an in-memory Prisma stand-in,
exercising the real request path without a running proxy or provider keys (the
allow/deny decision happens at auth time, before any model call):
- flag dispatch -> user_api_key_auth -> v2 entry -> authenticator chain ->
casbin -> 403/200
- master key resolves to proxy_admin and passes a governed route
- a non-admin with no grant is denied; granting a role and assigning it (live
policy CRUD) then unlocks the route, proving the cache reset on write
- the _require_admin gate rejects a non-admin over the wire
- model calls require the `call` permission: denied without a grant, allowed
after granting call on gpt-* and assigning it
This is the wiring that unit tests can't cover, verified in CI.
auth_v2 now seeds request-identity Baggage by calling seed_request_identity at the
auth boundary in each entry branch (model included for inference), the same
SDK-free seeder v1 uses. It propagates team/key/user identity to every downstream
route span and no-ops when the OTel SDK is absent or V2 isn't active, so it's safe
to call unconditionally.
This is the established OTel v2 mechanism, so the earlier identity_span_attributes
helper (and its module/tests) was speculative and is removed. Only live
verification of the exported spans remains; see INTEGRATION.md.
The policy CRUD endpoints were gated by _require_admin but it had no test. Add
deny/allow coverage: only PROXY_ADMIN may edit policies; view-only admins, every
other role, and no role are rejected with 403. This is the guard that stops a
non-admin from editing the casbin policy store, so a silent regression here would
be a privilege escalation.
Two CI failures on the PR:
- code-quality: ensure_async_clients_test forbids constructing httpx.AsyncClient
per request (latency). The OAuth2 introspection call and the JWKS fetch now use
get_async_httpx_client(httpxSpecialProvider.Oauth2Check), which returns the
cached AsyncHTTPHandler (it also raises_for_status internally). RFC 7662 client
auth moves from an auth tuple to an explicit Basic Authorization header since
the shared handler's post() takes headers, not auth.
- documentation: test_env_keys requires every os.getenv key to be documented.
auth_v2 read JWT/OAuth config from both general_settings and AUTH_V2_* env vars;
the env fallbacks were undocumented. Dropped them so config comes from a single
source (general_settings.auth_v2_jwt / auth_v2_oauth2), which is also cleaner.
Virtual keys arrive fully populated via get_key_object; master/JWT/OAuth logins
returned a thin identity, so the existing pre-call budget/limit hooks read None
and enforced nothing for them. enrich_identity copies the user/team budget+limit
fields 1:1 from the user/team rows into the identity's distinct user_*/team_*
slots, filling only unset fields (never overriding an already-resolved value).
Wired into the inference path for non-virtual-key logins, with get_user_object /
get_team_object injected as loaders so the mapping is unit-tested without a DB.
Additive by construction: these logins enforce nothing today, so it cannot
regress existing behavior. The exact enforcement still needs a live rate-limit
check before it is trusted; see INTEGRATION.md.
Importing proxy_server registers the auth_v2 router, which pulled casbin in at
module load via enforcer.py - so the proxy failed to even start in any env
without casbin (CI, minimal installs), regardless of whether auth_v2 was enabled.
An opt-in feature must not be a hard import.
- enforcer.py imports casbin lazily inside CasbinEnforcer.__init__; authlib in
jwt_verifier.py is imported lazily inside the verify/JWKS functions. The package
(and the startup router) now import with both deps absent; they are required
only when auth_v2 actually builds an enforcer or verifies a JWT.
- casbin and authlib were declared in pyproject but missing from uv.lock, so
`uv sync --frozen` never installed them. Regenerated the lock (adds casbin,
authlib, joserfc, simpleeval) so the deps install where the feature runs.
Verified: the v2 package imports with casbin+authlib blocked, and building an
enforcer raises a clear ImportError only when casbin is genuinely absent.
Adds the first two pipeline stages that consume RequestAuthContext instead of
the auth gate doing their work:
- end_user.resolve_end_user: extracts the customer (reusing the existing
request-body/header logic) and records it on the context via attach_end_user.
Validation is dependency-injected so it is testable without a DB. Wired into
the inference path so the context carries the end-user for spend attribution.
- telemetry.identity_span_attributes: returns OTel attributes for the request's
identity, for the route span (OTel v2) to set. Telemetry reads the context
rather than auth seeding the span; absent fields are omitted.
Both are pure/injectable and unit-tested; the contract stays the single source
of truth. Type-safe (mypy), formatted, linted.
Introduces the single, type-safe object every post-auth stage reads instead of
poking at request.state with untyped attribute access. auth_v2 populates it once
(identity, principal, auth_method, route, end_user_id) after authn+authz and
publishes it via set_auth_context; budget/limit hooks, the end-user resolver, and
the telemetry span consume it through get_auth_context / try_get_auth_context.
- RequestAuthContext is a frozen dataclass; attach_end_user replaces rather than
mutates, so one stage can't clobber another's view
- identity is annotated under TYPE_CHECKING so the contract is statically typed
without coupling the decoupled core to the heavy litellm import
- authenticators now advertise an AuthMethod and the chain returns a typed
AuthResult, so how a request authenticated is recorded for telemetry
- context is set in every branch (control, model-call, loud-open)
Verified type-safe with mypy. This is the linchpin for moving budgets, end-user
resolution, and telemetry out of the auth gate into their own stages.
Calling a model is now the `call` action on the `model:<id>` object, decided by
the same casbin role engine as everything else. Grant it via a role (wildcard
objects like model:gpt-*, or g2 groups) or directly to a key/user subject. The
legacy key.models list and access-group expansion are no longer consulted, so
with no grant a key can call nothing (clean-slate default-deny).
Removes the separate data-plane predicate (data_plane.py and its conf/tests);
the inference path now runs the same enforcer the control plane uses, extracted
into a shared _build_enforcer helper. Adds `call` to the valid policy actions.
Note: this puts a casbin check on the inference hot path. It builds the enforcer
per request from the short-TTL policy snapshot; caching a long-lived in-memory
enforcer with cross-pod invalidation is the next step before this is enabled at
real traffic.
The inference gate now allows a model when the key lists an access-group name
the model belongs to, mirroring v1 model_in_access_group. The group lookup is
resolved from the router in the entry point and injected into can_call_model,
which stays a pure predicate. Closes the access-group parity gap flagged when
the data plane moved off casbin; name, wildcard, and sentinel matching are
unchanged.
Credentials are REST-style: the verb is the HTTP method (POST /credentials is
create, GET is list) and the id is in the path, which the verb-in-path RPC
matcher cannot express. Adds method-aware, prefix-capable matching for these
resources and governs the credentials admin surface.
Fixes a latent bypass while wiring this: authorize() re-matched the route
without the method, so any REST route would have resolved to None inside
authorize() and been loud-opened even when the entry point had classified it as
governed. The method is now threaded through. Path-param ids are not extracted
yet, so credential objects stay at "credential:*" (per-id policies a follow-up).
Governs the collection-level management operations (register/list/health/
submissions) for MCP servers and guardrails, mapped to read/write with
wildcard objects since they are not per-id. Runtime guardrail verbs
(apply_guardrail, test_custom_code) remain loud-open as data/runtime concerns,
not management.
Extends the control-plane route map to the remaining clean-CRUD management
resources, following the established read/write/delete/manage pattern with
per-resource ids. These routes were loud-open before, so this only tightens
coverage. Non-uniform surfaces (guardrails, mcp servers, credentials) stay
deferred because their verbs don't map cleanly onto the four actions.
Also corrects two stale comments that still described the data plane as casbin
ABAC after it became a plain predicate.
A token shaped like a JWT (non sk-, two dots) was claimed by the JWT
authenticator even when no jwks_uri was configured, producing a 500 from the
settings loader. It now only claims JWT-shaped tokens when JWT auth is actually
configured, so an unconfigured deployment ends the chain in a clean 401.
The inference-path model check ran through a casbin enforcer whose matcher was a
trivial membership test (unrestricted || requested in allowed). On the hot path
that is pure overhead and indirection; casbin earns its keep on the control
plane (roles, deny-override, domains), not here. Replaces it with a direct
predicate and removes data_plane.conf.
Also closes a parity gap with v1: the gate now honors wildcard patterns
(e.g. bedrock/*, openai/*) via v1's is_model_allowed_by_pattern semantics, where
before it only matched exact names and would over-deny wildcard model lists.
Access-group expansion remains a tracked follow-up.
The JWT verifier used authlib's default decoder, which permits HMAC alongside
RSA/EC. With a JWKS of public keys that is the RS256->HS256 confusion setup: an
attacker signs HS256 using the public key as the HMAC secret. authlib happens to
reject this today via key-type checks, but relying on that is fragile and breaks
the moment a symmetric key enters the key set. Pinning the accepted algorithms
to an asymmetric allowlist closes the class outright.
Adds a regression test that forges an HS256 token from the JWKS public key and
asserts it is rejected.
Adds tenancy via domain-scoped role assignment (casbin g3): a role can be
granted globally (g) or only within a domain (g3), e.g. admin within team:eng
but nowhere else. The matcher accepts either, so existing global assignments are
unchanged. The policy store buckets g/g2/g3 separately and the assignment
endpoint takes an optional domain.
Adds litellm/proxy/auth/v2/README.md documenting configuration, the policy
admin API, and a live verification runbook.
Tests cover domain-scoped vs global role resolution, coexistence of both, the
g3 split in the policy store, and g3 assignment rule construction.
Completes the authn surface from the plan with an RFC 7662 introspection node
for opaque bearer tokens. Dispatched only for tokens that are neither virtual
keys nor JWTs, and only when an introspection endpoint is configured, so
unconfigured deployments fall through to a clean 401 rather than calling out.
The introspection response parsing is a framework-free core: a token is valid
only when explicitly active=true, the subject claim is required, and OAuth
scopes map to a litellm role via an explicit scope->role map (unmapped scopes
grant no role). The HTTP introspection call is isolated from that core.
Tests cover inactive/missing-active rejection, missing subject, and scope
(string and list) to role mapping.
Expands control-plane governance to keys, users, and organizations (read/write/
delete), mirroring models and teams. Membership changes (team/member_add,
organization/member_add, etc.) become the `manage` action on their resource.
Adds casbin resource grouping (g2) so a policy can grant on a named group of
resource ids rather than one row per id; this is where litellm access groups
map in. The matcher prefers a group match and falls back to keyMatch, so
direct/wildcard object matching is unchanged when no g2 rules exist. The policy
store now splits g2 rows out from g groupings and the enforcer loads them under
the g2 role manager.
Tests cover grouped vs ungrouped access, preserved direct matching, the g2/g
split, and governance of the new resources and the manage action.
Brings inference under casbin too, via the hybrid design: control-plane access
stays RBAC policy rows, data-plane access (can this principal call this model)
is a casbin ABAC matcher over an attribute carried on the already-loaded key, so
the hot path reads no policy store and writes no per-key policy rows.
Inference routes (chat/completions, completions, embeddings, responses) now
authenticate and check the requested model against the principal's allowed-model
attribute. Semantics match v1: an empty models list, "*", "all-proxy-models" and
"all-team-models" are unrestricted; otherwise the model must be in the list.
Tests cover the allow/deny/unrestricted matrix and inference-route detection.
Adds endpoints to manage casbin policies and role assignments without raw DB
inserts: add/remove a permission, add/remove a role assignment, and list rules
under /auth/v2/policy/*. Requests go through a validated, normalized core
(actions limited to read/write/delete/manage, effects to allow/deny, role and
object strings normalized) so malformed rules can't reach the policy table.
The policy surface governs itself: those routes are a "policy" resource in the
route map, so only a role permitted to manage policy (the bootstrap proxy_admin
role) can edit them, with an explicit proxy-admin check as defense in depth.
Writes reset the policy snapshot cache so changes take effect immediately.
Tests cover rule validation/normalization and the rule<->row conversion.
Adds a JWT node to the authenticator chain, dispatched on credential shape
(3-part token, not a sk- key). authlib owns the crypto: JWKS fetch with TTL
caching, kid-based key selection, signature verification, and exp/iss/aud
validation. A thin framework-free layer maps verified claims to an identity
using configurable claim names and an explicit upstream-role to litellm-role
map; unmapped role values are dropped rather than trusted.
JWT settings come from general_settings.auth_v2_jwt (or AUTH_V2_JWKS_URI etc.).
Verification failures normalize to 401 so callers never branch on authlib's
internal exception types.
Tests cover signature tampering, expiry, wrong audience/issuer, unknown signer,
and garbage input on the verifier, plus claim extraction and role mapping.
Adds a master-key authenticator ahead of the virtual-key node so the proxy admin
can authenticate under auth_v2 (slice 1 was virtual-key only, which locked the
master key out). Match is a constant-time exact compare; the raw key never
propagates downstream, a stable alias stands in for it.
Extends casbin governance to the team management plane (/team/new, /team/update,
/team/delete, /team/info) mirroring models. Team membership/permission routes
(member_add, etc.) stay loud-open, deferred with the recursive `manage` action.
Tests cover the master-key exact-match deny path (near-match, non-string, empty,
unconfigured) and team route governance.
Introduces auth_v2 as a clean-slate, flag-gated auth path (general_settings
auth_version: v2). When on, the existing auth is bypassed entirely and requests
flow through a new authenticator chain plus a casbin authorization engine.
Slice 1 scope:
- Entry point fork in user_api_key_auth; v1 untouched when the flag is off
- Authenticator chain with a virtual-key node resolving identity via the
existing key store (reuses get_key_object; no parallel identity storage)
- casbin engine: RBAC policy rows for the control plane, role bridged from the
key's existing user_role; per-resource-id objects supported
- Governs the model-deployment management plane only (/model/new, /model/update,
/model/delete, /model/info); every other route is loud-open and logs a warning
so unprotected surfaces are never silent
- Policies/groupings stored in LiteLLM_CasbinRule, loaded on cold routes with a
short snapshot cache; a bootstrap policy keeps proxy_admin fully authorized
- Decision core (enforcer, route map, principal, authorizer, policy store) holds
no framework imports, so it is unit-testable in isolation
Tests cover the allow/deny matrix, deny-override, domain scoping, per-id
granularity, loud-open behavior, and policy loading.
Data plane (inference-time model access) and additional resources/mechanisms are
deferred to later slices. casbin governs everything eventually via ABAC matchers
over cached attributes; this slice lays the control-plane foundation.
In Pydantic v2, Optional[T] without a default is a required field. Any
row with budget_id=null triggered a validation error and returned 401.
Co-authored-by: Florent Chenebault <florent.chenebault@lifen.fr>
* change deployment configs to include a litellm.cache for litellm-backend pod mirroring litellm-gateway pod
* omit backend annotations block when config and podAnnotations are both empty
* reuse gateway config/configmap for backend instead of separate backend config
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Tin Chi Lo <tin@Tins-MBP.localdomain>
Co-authored-by: Tin Chi Lo <tin@Tins-MacBook-Pro.local>
Add the squash-merged SHA of #29622 (style(ui): run prettier --write
across the dashboard) to .git-blame-ignore-revs so the bulk reformat
stops masking the real authors of those lines in git blame and the
GitHub blame UI
* fix(azure): apply api_version fallback chain to image edit URL
`AzureImageEditConfig.get_complete_url` only read `api_version` from
`litellm_params`. When callers configured it via `litellm.api_version`
or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and
Azure responded `404 Resource not found`.
Apply the same fallback chain the Azure chat path already uses in
`common_utils.py`:
litellm_params > litellm.api_version > AZURE_API_VERSION env >
litellm.AZURE_DEFAULT_API_VERSION
Adds 5 unit tests pinning each layer of the chain plus a regression
guard for `api_base` that already carries `?api-version=`.
* feat(mcp): core sampling and elicitation flow with security hardening
- Add sampling_handler.py: full MCP sampling/createMessage flow with
model selection (hint-based + priority-based), auth enforcement,
budget checks, route restriction gates, and tag policy pre-auth
- Add elicitation_handler.py: MCP elicitation/create relay with
downstream client capability detection
- Wire sampling/elicitation callbacks in mcp_server_manager.py
gated behind allow_sampling/allow_elicitation config flags
- Add allow_sampling/allow_elicitation fields to MCPServer type
- Fix session lock deadlock: skip lock for JSON-RPC response POSTs
(elicitation/sampling replies) with truncated-body heuristic
- Extend client.py with sampling_callback and elicitation_callback
- Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for
spoofing fix, Latin-1 header encoding guard
- Add 4 new test modules (model access, priority selection, request
builder, tool conversion) + update existing MCP tests
* fix(security): run pre-call guardrails before MCP sampling acompletion
Without this, an upstream MCP server with allow_sampling enabled could
send prompts that bypass every guardrail (content filtering, PII
redaction, prompt-injection detection) configured on /chat/completions.
- Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before
llm_router.acompletion so guardrails fire for sampling sub-calls
- Add HTTPException to the re-raise list so guardrail rejections
propagate correctly instead of being swallowed as generic errors
* feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490)
* feat(bedrock_mantle): add Responses API transformation config
* test(bedrock_mantle): cover trailing-slash api_base normalization
* feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig
* feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged)
* feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries
* refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing
Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses;
gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding
gpt-oss (which keeps its chat-completions emulation) and defaulting everything else
to the native Responses config, so future frontier models (gpt-6, etc.) route
correctly without a code change. Verified against the live us-east-2 Mantle endpoint:
gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths.
* test(bedrock_mantle): cover supports_native_websocket opt-out
Closes the one uncovered line flagged by codecov on the Responses config.
The assertion documents that Mantle Responses has no realtime/websocket
transport, so realtime routing must not attempt a socket it cannot serve.
* fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle
BedrockMantleResponsesAPIConfig inherited supports_native_file_search()
-> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no
OpenAI vector stores, so a forwarded file_search tool is rejected with a
400 (verified upstream: Tool type 'file_search' is not supported). Opting
out, like the existing supports_native_websocket override, routes the tool
through LiteLLM's file_search emulation instead.
* fix(bedrock_mantle): only route openai.gpt frontier models to Responses
The previous gate excluded gpt-oss and routed every other model to the
native Responses config. But on Mantle only the OpenAI gpt frontier models
(gpt-5.x) are served on /openai/v1/responses; gpt-oss and the non-OpenAI
families (nvidia, mistral, google, zai, ...) are chat-completions only and
400 on that path. Allow-list the openai.gpt- family (excluding gpt-oss)
instead, so chat-only models fall through to the chat-completions emulation.
Verified against the live us-east-2 endpoint: nvidia.nemotron-nano-9b-v2
returns 400 on /openai/v1/responses and 200 on /v1/chat/completions.
* feat(custom_llm): allow streaming/astreaming to yield ModelResponseStream (#27580)
* fix(custom_llm): allow streaming/astreaming to yield ModelResponseStream directly
* fix(streaming): enhance ModelResponseStream handling for custom LLM providers
* fix(streaming): strip finish_reason from content chunks and ensure tool_calls are preserved
* fix(streaming): add type ignore for finish_reason assignment in CustomStreamWrapper
* fix(proxy): strip stack trace from HTTP 503 responses (CWE-209) (#28330)
* fix(proxy/cwe-209): strip Python traceback from HTTP 503 error responses
The /cache/ping endpoint included a full Python traceback in its 503 error
response body (inside the ProxyException message), leaking internal file
paths, line numbers, and call stacks to any caller. Two MCP route handlers
in proxy_server.py similarly interpolated str(e) into "Internal server
error" detail strings.
Fix: log the traceback server-side via verbose_proxy_logger.exception()
and omit it from the ProxyException payload / HTTPException detail returned
to clients. Tests updated to assert no "traceback" keyword or frame paths
appear in the 503 body, with a new dedicated regression test.
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy/cwe-209): apply Greptile P2 fixes and add MCP exception-path tests
Greptile 4/5 review identified two remaining gaps and Codecov reported
0% coverage on the two MCP handler exception branches:
1. caching_routes.py — str(e) in "Service Unhealthy ({str(e)})" could
still leak Redis hostnames/IPs; replaced with static "Service Unhealthy".
HTTPException is now re-raised before the generic handler so the
"cache not initialized" 503 still reaches callers with its detail.
Removed the redundant str(e) arg from verbose_proxy_logger.exception()
(exception() already appends the traceback automatically).
2. tests — two new unit tests cover the exception paths in
dynamic_mcp_route and toolset_mcp_route that were previously at 0%:
- test_dynamic_mcp_route_unexpected_exception_returns_500_without_traceback
- test_toolset_mcp_route_unexpected_exception_returns_500_without_traceback
All 25 tests pass (9 caching + 16 MCP).
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion in test_cache_ping_no_cache_initialized
The assertion was weakened to `"Cache not initialized" in str(data)`, which
matches the raw string of the entire response dict and would pass even if the
error moved to an unexpected field or changed structure.
Restore a targeted check on the parsed response: assert the exact string in
the correct field `data["detail"]`, matching FastAPI's HTTPException
serialisation format {"detail": "<message>"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion and add CWE-209 no-cache path test
The assertion in test_cache_ping_no_cache_initialized was weakened to
`"Cache not initialized" in str(data)`, which matched against the raw string
representation of the entire response dict. This would pass silently even if
the error message moved to an unexpected field or the structure changed.
Restore a targeted assertion on the parsed field:
assert data["detail"] == "Cache not initialized. litellm.cache is None"
matching FastAPI's HTTPException serialisation format exactly.
Add test_cache_ping_no_cache_does_not_expose_internals to show the code path
is still working correctly after the CWE-209 fix: verifies that the HTTPException
is re-raised as-is (no traceback, no source paths), and asserts the complete
response structure is exactly {"detail": "Cache not initialized. litellm.cache is None"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(caching_routes): restore ProxyException envelope for null-cache 503
The except HTTPException: raise guard (added in the CWE-209 fix) caused
the null-cache HTTPException to escape as FastAPI's {"detail": "..."} shape
instead of the {"error": {...}} ProxyException envelope that callers expect.
Move the null-cache guard before the try block and raise ProxyException
directly so the response structure is consistent with all other /cache/ping
503s, and the except HTTPException: raise guard is only reachable by
unexpected downstream HTTPExceptions.
Update the two no-cache tests to assert the correct ProxyException envelope.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update utils.py (#26609)
* feat(pricing): add Snowflake Cortex REST API model pricing (#26612)
* feat(pricing): add Snowflake Cortex REST API model pricing
## Summary
Adds pricing and context window information for 20+ Snowflake Cortex REST API models to `model_prices_and_context_window.json`.
## What's included
- **7 Claude models** (sonnet-4-5, sonnet-4-6, 4-sonnet, 4-opus, haiku-4-5, 3-7-sonnet, 3-5-sonnet) — with prompt caching rates
- **4 OpenAI models** (gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano) — with prompt caching rates
- **5 Llama models** (3.1-8b, 3.1-70b, 3.1-405b, 3.3-70b, 4-maverick)
- **1 DeepSeek model** (deepseek-r1)
- **1 Mistral model** (mistral-large2)
- **1 Snowflake model** (snowflake-llama-3.3-70b)
- **2 Embedding models** (arctic-embed-l-v2.0, arctic-embed-m-v2.0)
Each entry includes `input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost` (where applicable), `max_input_tokens`, `max_output_tokens`, and capability flags (`supports_function_calling`, `supports_vision`, `supports_prompt_caching`, `supports_reasoning`).
## Pricing source
All prices are in USD per token, sourced from the official [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) — Tables 6(b) (REST API with Prompt Caching) and 6(c) (REST API).
## Context
The existing `snowflake/` provider has zero model entries in the pricing JSON, which means LiteLLM cannot track costs for Snowflake Cortex calls. This PR fills that gap.
## Related
- Existing provider: `litellm/llms/snowflake/`
- Cortex REST API docs: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-rest-api
* Update model_prices_and_context_window.json
Fix the JSON parsing error
* Update model_prices_and_context_window.json
Removed the duplicate entry
* fix(utils): copy extra_body before adding unknown params to prevent model config mutation (#29620)
Fixes#29615. In add_provider_specific_params_to_optional_params, the line:
extra_body = passed_params.pop("extra_body", None) or {}
returns the original dict reference when extra_body is non-empty (truthy).
Subsequent writes like extra_body[k] = passed_params[k] then mutate the
shared model config object held by the router, poisoning /model/info and
all subsequent requests for that deployment.
The or {} short-circuit creates a new dict only when extra_body is falsy
(None or {}), which is why the bug does not reproduce with extra_body: {}.
Fix: wrap in dict() so we always work on a fresh shallow copy.
* fix(vertex_ai): Bake tool_choice into Gemini CachedContent body to prevent silent drop (#29097)
* fix(vertex_ai): bake tool_choice into Gemini CachedContent body to prevent silent drop
* address greptile feedback on tool_choice cache test
* adds test that uses ToolConfig(functionCallingConfig=FunctionCallingConfig(mode=ANY)) instead of a dict literal, mirroring what map_tool_choice_values actually produce
* fix(gemini/veo): move image from parameters into instances[0] (#29501)
* fix(gemini/veo): move image from parameters into instances[0]
Veo's predictLongRunning schema puts image (and prompt) on the
instances element; parameters is for aspectRatio/durationSeconds/etc.
The Gemini path was leaving image in params_copy, so it ended up
nested under parameters and the API silently ignored it.
The Vertex path already builds the instance dict explicitly, so this
just aligns the Gemini path with it.
Fixes#29498
* address greptile: unconditional pop + BytesIO test
- Pop `image` from params_copy unconditionally so it never reaches
GeminiVideoGenerationParameters even when None, removing implicit
reliance on Pydantic's extra-field-ignore.
- Add test_transform_video_create_request_image_filelike_goes_to_instance
covering the BytesIO path (_convert_image_to_gemini_format) — round-trips
the base64 to confirm encoding.
- Add test_transform_video_create_request_image_none_is_dropped covering
the new None branch.
* fix(huggingface): handle special token text in embedding usage (#29660)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params (#29655)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params
ToolPermissionGuardrail builds self.rules and the compiled target/pattern
maps only in __init__. The base update_in_memory_litellm_params re-sets raw
attributes via setattr but never rebuilds those maps, so a guardrail updated
in place (PUT /guardrails, or the immediate in-memory sync) keeps enforcing
the construction-time rules until it is reinitialized (PATCH path, periodic
DB poll, or restart).
Extract the compile step into _load_rules and override
update_in_memory_litellm_params to rebuild from it (dict- and model-safe),
re-normalizing default_action / on_disallowed_action. Mirrors the existing
PresidioGuardrail override of the same method. Adds regression tests.
Fixes#29592.
* fix(guardrails): handle dict params in ToolPermissionGuardrail in-memory update
Delegate to super() only for LitellmParams input (the base setattr loop is
model-only); apply the raw-dict case inline. Fixes the mypy arg-type error
and makes the recompile work when the proxy passes the raw DB dict.
* fix(guardrails): preserve tool-permission rules on a partial in-memory update
A partial update (e.g. a LitellmParams whose rules field is None) ran through
the generic setattr, which set self.rules to None, and the recompile was
skipped, leaving the guardrail with no rules. Snapshot the previous rules and
restore them when the update carries no rules; an explicit empty list still
clears them. Adds a regression test for the rules-absent case.
Addresses the Greptile review note on #29655.
* fix(bedrock): stop base_model label from stripping tools/tool_choice (#29621)
* fix(bedrock): stop base_model label from stripping tools/tool_choice
A Router/proxy Bedrock deployment whose model_info.base_model is a friendly
label (e.g. claude-haiku-4-5) silently lost tools/tool_choice: the outgoing
Converse request was built without toolConfig, so the model behaved as if no
tools were provided. Worked in v1.84.0, regressed in v1.85.0, and with
drop_params=true it failed silently.
Two changes compound into the bug. completion() passed model_info.base_model
as the model argument to get_optional_params, so the real Bedrock model id
never reached supported-param resolution; and get_supported_openai_params
resolved the provider config's params from base_model or model, letting the
label fully replace the real model. For Bedrock the label resolves to no tool
support, so tools/tool_choice were dropped before transformation.
completion() now keeps model as the real deployment model and threads the
resolved base_model (kwarg or model_info) through separately, and
get_supported_openai_params treats base_model as additive: it returns the
union of the params supported by model and by base_model. A hint can only add
capabilities, never strip ones the real model already exposes, which also
preserves the original base_model behavior from #27717 and Azure's base_model
driven model-type detection.
Fixes#29618
* test(main): make base_model param test robust to new parametrize cases
Restore an explicit per-case expected_model_param literal instead of
hardcoding the gemini id, so a future case with a different model can't
produce a misleading assertion failure.
* fix(fireworks_ai): pass response_format json_schema through unchanged (#29606)
FireworksAIConfig.map_openai_params was rewriting the OpenAI strict
`{type: json_schema, json_schema: {name, strict, schema}}` shape into
`{type: json_object, schema: ...}` before sending to Fireworks, dropping
`strict` and `name` and changing the `type`. Per Fireworks' docs json_object
means "force any valid JSON output (no specific schema)", so the schema
constraint was effectively dropped and grammar-guided decoding never ran;
model output silently violated the schema.
The rewrite landed in #7085 (Dec 2024) when Fireworks did not yet accept
native json_schema. Fireworks accepts the OpenAI strict shape natively now,
so the rewrite has become a regression.
Removes the rewrite. Passes response_format through unchanged. Updates the
existing test_map_response_format to assert pass-through. Adds focused
regression tests in tests/test_litellm/ covering preservation of type,
strict, name, and schema body, plus that json_object alone still works.
* fix(types): import Required from typing_extensions in gemini types
* style: reformat sampling_handler.py for py312 black compat
* refactor(mcp-sampling): extract helpers to fix PLR0915 too-many-statements in handle_sampling_create_message
* fix(proxy-server): add explicit ProxyLogging type annotation to proxy_logging_obj to fix mypy inference
* fix(mcp-sampling): suppress mypy assignment error on ImportError fallback for proxy_logging_obj
* fix(test): use .value when comparing LlmProviders enum against string in test_default_api_base
* fix(test): iterate LlmProviders enum in test_default_api_base to avoid str pollution from custom provider registration
litellm.provider_list is a mutable global initialized to list(LlmProviders) but custom_llm_setup() appends plain provider strings to it. When a test_custom_llm.py test runs first in the same xdist worker, provider_list contains a str and calling .value on it raises AttributeError. Iterate the immutable LlmProviders enum instead, which is deterministic and what the check intends.
* fix(mcp): depth-aware JSON-RPC response detection and neutral speed-priority fallback
Replace the flat substring check in the truncated-body routing path with a
top-level-key scan so a JSON-RPC response whose result payload nests a
"method" field is still detected as a response and skips the session lock,
removing a deadlock against the in-flight tool call awaiting it.
Drop the inverse max_output_tokens speed proxy when no model exposes
output_tokens_per_second; context-window size does not track latency, so a
neutral score avoids biasing speedPriority toward the smallest-context model.
* fix(guardrails): make ToolPermission rule reload atomic on invalid regex
_load_rules appended each rule to self.rules before compiling its regex, so an
invalid pattern raised mid-loop after the bad rule was already live but without
a _compiled_rule_targets entry. _matches_regex reads a missing compiled target
as a None pattern and returns True, turning the bad rule into a match-all that
silently applies its decision to every tool. Via update_in_memory_litellm_params
(PUT /guardrails) this corrupted the live guardrail.
Build the parsed rules and compiled maps into locals and swap them in only after
every regex compiles, and restore the previous ruleset if a live update is
rejected, so an invalid regex now fails the update without leaving the guardrail
enforcing a broken policy.
* test(mcp): cover sampling conversion, model resolution, and elicitation relay paths
The MCP sampling and elicitation handlers shipped with partial test
coverage, leaving the response-to-MCP conversion, the model resolution
fallback chain, completion-kwargs assembly, guardrail routing, and the
entire elicitation relay untested. That pulled the PR's diff (patch)
coverage below the codecov threshold even though overall project
coverage rose.
Add focused unit tests for _convert_openai_response_to_mcp_result,
_convert_mcp_tools_to_openai, _convert_mcp_tool_choice_to_openai, image
and audio content conversion, the hint-matching and fallback branches of
_resolve_model_from_preferences, _build_completion_kwargs, the router and
guardrail-rejection paths of _run_guardrails_and_call_llm, the
handle_sampling_create_message success and error-propagation flows, the
marker-hoisting fallback for tool content on unexpected roles, and the
elicitation form/url/generic relay together with its decline paths
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: lengkejun <lengkejun@xd.com>
Co-authored-by: Yug <yugborana000@gmail.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: tanmay958 <53569547+tanmay958@users.noreply.github.com>
Co-authored-by: DrishnaTrivedi <142084770+DrishnaTrivedi@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Navnit Shukla <Navnit.shukla25@gmail.com>
Co-authored-by: PRABHU KIRAN VANDRANKI <72809214+VANDRANKI@users.noreply.github.com>
Co-authored-by: Adrian Lopez <109683617+adriangomez24@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com>
Co-authored-by: Dinesh Girbide <85330597+Dinesh-Girbide@users.noreply.github.com>
Co-authored-by: cloudwiz <22098246+andrey-dubnik@users.noreply.github.com>
Co-authored-by: Ahmad Khan <ahmadkhan2508@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini-realtime): use GA event names for Pipecat 1.3.x compatibility
Pipecat v1.3.0 adopted the OpenAI Realtime API GA event naming:
response.audio.delta -> response.output_audio.delta
response.text.delta -> response.output_text.delta
response.audio.done -> response.output_audio.done
response.text.done -> response.output_text.done
The proxy was still emitting the old beta names; Pipecat's
`parse_server_event` raises "Unimplemented server event type" for any
unknown type, which killed the receive task handler and broke audio
playback and tool-call delivery.
Also:
- conversation.item.created -> conversation.item.added (already handled)
- client audio is buffered until backend setupComplete in deferred mode
- call_id fallback UUID when Gemini returns empty id
- status_details / token detail fields added to Pydantic-strict events
The _GA_TO_BETA_EVENT_TYPES map in RealTimeStreaming already translates
GA names back to beta for clients that opt in with the openai-beta
header, so legacy clients are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address greptile review comments
- emit outputTranscription as response.output_audio_transcript.delta
instead of suppressing it; GA_TO_BETA map handles translation for
legacy clients
- cap pre-setup audio buffer at 200 frames to prevent memory exhaustion;
log a warning when the limit is hit and additional frames are dropped
- log remaining dropped message count on flush error
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address veria review comments
- remove unused OpenAIRealtimeConversationItemCreated import
- fix guardrail bypass: semantic_vad early-return now preserves
create_response when set so a guardrail-injected create_response:false
is not silently dropped
- add per-connection 10 MB byte cap alongside the 200-frame count cap
for the pre-setup audio buffer to prevent memory exhaustion
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): fix mypy arg-type on _finalize_gemini_live_setup
setup parameter typed as BidiGenerateContentSetup to match the TypedDict
passed at both call sites; was dict which mypy rejected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): widen _finalize_gemini_live_setup to Dict[str, Any]
BidiGenerateContentSetup (TypedDict) is a subtype of Dict[str,Any] so
both call sites (one passing a plain dict, one passing the TypedDict)
satisfy mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): cast BidiGenerateContentSetup to Dict at _finalize call site
mypy rejects TypedDict as dict[str, Any] argument; cast at the call site
where follow_up_setup is BidiGenerateContentSetup to satisfy the checker.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Gemini realtime beta compatibility
* Fix deferred Gemini setup audio ordering
* fix: preserve Gemini audio transcript ids
* fix(realtime): cap pre-setup client buffer on all append paths
Route every append to the deferred-setup pending buffer through the
per-connection message/byte caps. Previously only the audio-buffer
fast path enforced the caps; once one frame was buffered, a client
that withheld session.update could stream arbitrary frames into
_pending_messages_until_setup unbounded and exhaust proxy memory.
* style(gemini-realtime): apply black formatting to transformation.py
* fix(gemini-realtime): log beta-translation fallback and name native-audio marker
Surface the previously swallowed exception in _send_event_to_client so a
failed GA->beta translation is observable instead of silently forwarding the
untranslated event. Extract the native-audio model substring used by
_finalize_gemini_live_setup into a named constant documenting why speechConfig
is dropped on those setups.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): match passthrough registry routes bare-to-bare with SERVER_ROOT_PATH
After #28547, get_request_route strips the deployment prefix while registry
lookup still re-inflated stored paths via SERVER_ROOT_PATH, causing 404s
under paths like /llmproxy/ml. Compare normalized bare routes in both
is_registered_pass_through_route and get_registered_pass_through_route.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): patch utils.get_server_root_path in passthrough auth tests
After removing get_server_root_path from pass_through_endpoints, route
and JWT tests must mock litellm.proxy.utils where normalization reads it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini): keep googleSearch with server-side tools and googleMaps JSON schema
Wire include_server_side_tool_invocations through completion() so mixed
google_search and function tools are not dropped on Gemini 3+. Rewrite
generationConfig to responseFormat when googleMaps is used with JSON schema.
Fixes#27479Fixes#29451
Co-authored-by: Cursor <cursoragent@cursor.com>
* address greptile review feedback (greploop iteration 1)
* style: fix black formatting in main.py for py312 compat
* Fix Gemini Google Maps extra_body JSON rewrite
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(ui): add frontend-lint job enforcing prettier and eslint on changed files
Lints only the files a PR adds or modifies under ui/litellm-dashboard,
so new and touched code must be prettier-clean and eslint-clean while the
existing tree is grandfathered. Skips cleanly when a PR touches no
lintable UI files. This lets us adopt the formatters incrementally
without a repo-wide reformat
* ci(ui): write frontend-lint file lists to $RUNNER_TEMP
Keep the prettier/eslint changed-file lists out of the checkout dir so
they cannot collide with a future source file of the same name
* lint(ui): baseline existing eslint findings so only new ones block
Capture the current error-level eslint findings (318 across 183 files)
in a committed suppressions baseline via eslint --suppress-all. Every
rule stays at its error severity, so any newly introduced violation
fails the frontend-lint gate, while the existing tree is grandfathered;
touching a legacy file never forces fixing its pre-existing issues. CI
runs eslint with --pass-on-unpruned-suppressions so that fixing a
baselined issue does not fail on a now-stale suppression, and the
generated baseline is prettier-ignored since eslint owns its format.
Burn the baseline down over time with eslint --prune-suppressions
* lint(ui): enforce a count budget for explicit any
Make @typescript-eslint/no-explicit-any a warning and cap the total
instead of hard-blocking each new one. A frontend-lint step counts the
repo-wide explicit any and fails only when it exceeds the committed
budget in eslint-any-budget.json. max starts at 2031, ten above the
current 2021, so the next ten land as warnings and the build fails once
that headroom is gone. Lower max over time toward target to ratchet the
count down. New anys still surface as warnings on changed files via the
normal eslint step
* lint(ui): enable zero-cost rules no-var, no-self-assign, react/no-danger
These have no existing violations, so they need no baseline; turning them
on purely blocks new instances. react/no-danger guards against new
dangerouslySetInnerHTML (XSS), no-var enforces let/const, and
no-self-assign catches self-assignment typos. no-debugger is already
enforced by the recommended preset
* lint(ui): add baselined complexity rules
Enable complexity:20, max-depth:4, max-params:4, max-nested-callbacks:4,
with thresholds set near the codebase p99 so only genuine outliers are
flagged. The 272 existing over-threshold functions are grandfathered in
the suppressions baseline; new over-threshold functions block. Lower the
thresholds over time to ratchet complexity down. max-lines-per-function
is intentionally left off since React components are legitimately long
* lint(ui): ban new raw fetch, standardize on React Query
Add a no-restricted-syntax rule flagging bare fetch() calls, pointing
contributors at React Query (@tanstack/react-query). The rule is not
exempted anywhere, including the already-bloated networking.tsx, so all
331 existing fetch calls are grandfathered but no new ones can be added
there or elsewhere. New data access goes through React Query, and the
networking layer can be migrated out and pruned from the baseline over
time
* lint(ui): ban new @tremor/react imports
Add a no-restricted-imports rule flagging imports from @tremor/react so
tremor is phased out rather than spread further. The 232 existing tremor
imports are grandfathered in the baseline; new ones block and point at
antd. Migrate components off tremor and prune the baseline over time
* lint(ui): widen explicit-any budget headroom to 2040
Raise max from 2031 to 2040, giving ~19 of slack over the current 2021
instead of 10
* style(ui): prettier-format eslint.config.mjs
The frontend-lint gate flagged its own config file. Format it so the
prettier check on this PR's changed files passes
* lint(ui): soften complexity and max-depth to warnings
These two are smell metrics with arbitrary thresholds where a legit new
function can trip them, so make them advisory rather than hard-blocking.
They drop out of the baseline (now 963). max-params, max-nested-callbacks,
and the react-hooks rules stay strict since those are clear-cut
* lint(ui): move complexity and max-depth to the count-budget pattern
Generalize the explicit-any budget into a shared lint-budget mechanism:
eslint-budgets.json maps a rule to {max, target} and check-lint-budgets.mjs
counts each across the repo and fails when a count exceeds its max.
complexity (129, max 140) and max-depth (61, max 70) now use the same
slack-plus-counter model as explicit-any (2021, max 2040): they warn
per-file and the build only fails if the repo-wide total crosses the
ceiling. Lower each max toward its target over time
* docs(ui): note pruning the eslint suppressions baseline when fixing lint debt
Streaming responses from the proxy (/chat/completions, /v1/messages,
/v1/responses, assistants) all return through create_response() but never
sent the headers that tell an intermediary reverse proxy not to buffer the
SSE stream. nginx with the default proxy_buffering, k8s ingress-nginx, and
Envoy/Istio sidecars therefore hold the whole stream and release it in one
batch, which looks like a broken/buffered stream to the client even though
litellm is yielding chunks incrementally.
Add Cache-Control: no-cache and X-Accel-Buffering: no to every
StreamingResponse create_response() returns, matching what the proxy already
does for its own usage/policy SSE endpoints. Fixes#28384.
