The auth object is the httpx client's auth for the whole MCP session; after a
401 recovery it kept sending the rejected token first, burning a 401 round trip
and the single retry on every subsequent call
An expires_in of zero or below computes a ttl of 0; the entry could never be
served but still occupied a slot in the bounded backend, where it could evict
a live token. The mint still serves the current request and the next get
re-fetches under the per-server lock
Greptile P2s: the per-server lock dict now evicts its oldest entry past
max_locks so ephemeral server ids (REST tools preview) cannot grow it
unbounded, and the min-cache floor is capped at the token's actual lifetime so
an expires_in below the skew is never served past expiry
The graft test pinned the pre-migration contract (M2M defers to v1). Replaced
with two tests pinning the new one: a complete-config M2M server resolves via
the v2 arm into ClientCredentialsBearerAuth, and an incomplete-config server
fails closed with a 500 misconfigured naming the missing grant fields
Replaces the not_implemented stub with a live arm: ClientCredentialsTokenSource
mints and caches the M2M token (rotation-aware identity key, expires_in-driven
TTL, audience and token_endpoint_auth_method support) and
ClientCredentialsBearerAuth retries an upstream 401 exactly once with a freshly
minted token. to_server_spec owns oauth2_flow=client_credentials servers and
fails closed on incomplete grant config instead of connecting unauthenticated
Clients exporting large spend-log ranges were forced into 100-row pages,
which meant a bounded COUNT plus an increasingly deep OFFSET scan per
request. Larger pages reduce both the request count and the cumulative
OFFSET cost for the same result set.
The handler already excludes the heavy JSON columns (messages, response,
proxy_server_request) from the paginated SELECT and bounds the COUNT via
SPEND_LOGS_PAGINATION_COUNT_CAP, so per-row cost does not grow with page
size. 1000 matches the ceiling already used by the user and user-agent
analytics list endpoints.
* fix(fireworks_ai): set Content-Type application/json in validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(fireworks_ai): delegate chat validate_environment to OpenAIGPTConfig
Instead of re-adding the JSON Content-Type default inside FireworksAIMixin,
FireworksAIConfig now delegates header construction to OpenAIGPTConfig and only
layers the Fireworks-specific x-session-affinity header on top, so the
Content-Type default can no longer drift away from the OpenAI base and reintroduce
the 415.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fireworks_ai): cover missing api key error path in chat validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
create_model_info_response cast cost-map max_input_tokens / max_output_tokens
with unguarded int(). The surrounding try/except covers only the get_model_info
lookup, so a deployment whose model_info carries a non-numeric limit (e.g.
"128,000" or an empty string) raised inside the per-model listing loop and
failed the entire GET /v1/models and /models response with a 500, taking healthy
deployments down with it. A deployment's model_info is registered into
litellm.model_cost verbatim, so the malformed value reaches the cost map and not
just the router index.
Router.get_configured_token_limits already coerced this safely for the
deployment path; the cost-map path was missed, so the two together still
regressed. Both now share coerce_token_limit in litellm_core_utils, which
returns None for a malformed value so the listing omits that one limit instead
of failing, matching the graceful degradation the endpoint had before the
cost-map switch.
The SERVER_ROOT_PATH fix for the per-server pass-through challenge belongs with its sibling
in exceptions.py (both fabricate a per-server resource_metadata URL and both omit the root
segment), and both are pre-existing paths unrelated to the aggregate discovery this PR adds.
Reverting the server.py change keeps this PR to the aggregate front door and avoids leaving
the two per-server challenge builders inconsistent; the per-server root-path fix lands as its
own change covering both sites.
tests/test_litellm/proxy/test_custom_proxy.py sets SERVER_ROOT_PATH at import time (its app
mounts under a custom path) and never restores it, so in a shared shard the value leaks into the
process. The discovery routes and the 401 challenges now read SERVER_ROOT_PATH to path-insert it
where they previously ignored it, so a leaked value rewrites every resource_metadata URL and the
exact-URL assertions in the delegate, pass-through, and aggregate challenge tests fail depending
on shard order
An autouse fixture clears SERVER_ROOT_PATH for the MCP discovery tests so they deterministically
exercise the default root-mounted deployment; the tests that assert a sub-path deployment set the
value explicitly within their own body. No assertion changed; the leak was invisible before only
because the code ignored the variable
Two RFC 9728 / 8414 discovery fixes on the aggregate front door, both raised by Bugbot on this PR
The aggregate authorization-server document at /.well-known/oauth-authorization-server/mcp used to
defer to a per-server row literally named "mcp", serving issuer {base} while the aggregate
protected-resource document advertises {base}/mcp as its authorization server. A spec client
following that chain fails the RFC 8414 issuer check and cannot sign in. The single segment /mcp is
now reserved for the aggregate so the issuer stays {base}/mcp and matches the protected-resource
document; a server named "mcp" keeps its standard two-segment discovery at
/.well-known/oauth-authorization-server/mcp/mcp
The 401 challenges built the resource_metadata URL as {base}/.well-known/oauth-protected-resource/mcp
with no SERVER_ROOT_PATH segment, but the routes are registered with the path-inserted root segment,
so a proxy mounted under a sub-path pointed DCR clients at a URL that 404s. Both the aggregate
challenge and the pre-existing per-server pass-through challenge now derive the path from one
well_known_root_suffix helper that the route registrations also use, so the advertised URL cannot
drift from the served route
The flag guarded no breaking change: the aggregate discovery lives at new /mcp-suffixed
routes, the challenge only fires at aggregate scope, and the authorize/token/register/admission
arms self-gate on the llm_dcrc_/llm_session_ prefixes. Bare-origin and per-server discovery are
left exactly as they were, and a server literally named mcp keeps its own discovery via
disambiguation, so turning it on for everyone changes nothing about existing flows.
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Move the destructive daily-tag Redis drain back inside the try so a Redis
read failure still releases the pod lock via the finally block, and use a
covariant Mapping for the restore signature. Add regression tests for the
daily-tag requeue-on-failure/no-requeue-on-success paths and the RedisError
swallow branch in restore_transactions_to_redis.
The Redis transaction buffer leader drains the spend buffers with a
destructive lpop before committing to the database. When the DB commit
failed after exhausting retries, the popped transactions were only logged
and then lost, permanently undercounting key/user/team/org/end-user/
team-member/tag/agent and daily spend after a database outage.
Track each popped category and re-push the ones that were not committed
back to their Redis buffers so a later scheduler tick retries them.
Categories that already committed are not re-queued, so their spend is not
double-counted. The daily tag spend path gets the same treatment.
A deployment whose model_info carried a non-numeric max_input_tokens or
max_output_tokens (for example "128,000" or an empty string) made the
bare int() in get_configured_token_limits raise inside the per-model
/v1/models loop, so one misconfigured deployment turned the entire
listing into a 500. Coerce each configured limit safely and treat
malformed values as absent, matching the graceful degradation the
listing had before the cost-map switch
split_concatenated_json_objects re-raised JSONDecodeError on genuinely
malformed (non-concatenated) tool-call arguments, which propagated out of
_convert_to_bedrock_tool_call_invoke and turned every replayed Bedrock
conversation into a 500. Catch the decode error, keep whatever complete
objects parsed, log a warning, and let the caller fall back to input={}
so the conversation continues.
Fixes#18667
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(messages): route Azure Anthropic /messages through Rust behind rust:true
Adds an opt-in Rust path for non-streaming Azure Anthropic Messages. A
deployment sets rust: true in litellm_params to route litellm.messages()
and the proxy /v1/messages endpoint through the native Rust bridge; a
missing flag or rust: false keeps the existing Python path, and non-Azure
providers, streaming, an unavailable bridge, or a None result all fall
back to Python. Rust-backed responses carry an x-litellm-rust: true
response header so callers can see which path served the request.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(docs): exclude LITELLM_USE_RUST_MESSAGES rollout flag from env-doc check
Mirrors the existing LITELLM_USE_RUST_OCR entry; the flag is an internal
rollout toggle that is intentionally not in the public environment settings
docs yet.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(rust_bridge): isolate OCR enable flag and drop dead messages global toggle
use_litellm_rust only mutates the OCR enabled flag when configuring OCR (or
called with no bridge kwargs, preserving the legacy contract), so configuring
only the messages bridge no longer flips OCR state.
Remove the vestigial global enabled/env state from the messages bridge. Routing
is controlled per deployment by rust:true in the shared handler gate, so the
messages module never consulted the global toggle; drop it rather than leave a
no-op switch.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(rust/messages): split Anthropic config into its own provider file and type the request/response contract
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(messages): route eligible Azure Anthropic streaming through Rust via buffered fake-stream
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(messages): fold system-role messages for Azure Anthropic and fall back to Python on Rust bridge errors
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(rust_bridge): use Python::attach for amessages after pyo3 bump
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): mock get_configured_token_limits in model_info tests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* ci: run rust_bridge unit tests in misc shard
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* Revert "ci: run rust_bridge unit tests in misc shard"
This reverts commit c86d861a03.
* test(anthropic): move rust messages bridge tests into misc-shard dir
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(mcp): add ID-JAG egress auth as a v2 outbound-credentials arm
Adds the oauth2_id_jag MCP egress auth mode (draft-ietf-oauth-identity-assertion-authz-grant,
shipped by Okta as "AI agent token exchange") as a first-class arm of the v2
outbound_credentials resolver rather than a standalone v1 handler.
ID-JAG is a two-leg flow: an RFC 8693 token exchange swaps the caller's id_token for an
ID-JAG assertion at the IdP org authorization server, then an RFC 7523 jwt-bearer grant
presents that assertion to the MCP's resource authorization server for the access token
used to call the upstream. The gateway authenticates to both endpoints with a private-key
JWT client_assertion, falling back to client_secret when no key is configured.
The mode is modeled as IdJagConfig in the AuthConfig discriminated union, with client auth
as a ClientAuth tagged union (private_key_jwt or client_secret) so required fields are
enforced at construction and illegal states are unrepresentable. A new token_endpoint
collaborator performs the authenticated OAuth token-endpoint call and caches the result
with per-key single-flight; the resolver's _id_jag arm runs the two legs and returns an
httpx.Auth or a typed CredError. A missing caller identity token fails closed
(precondition_required), so an ID-JAG server never falls back to a static credential. The
v1->v2 adapter maps oauth2_id_jag servers onto IdJagConfig and the existing live v2 path
resolves them, so no standalone handler, has_id_jag_config flag, or resolve_mcp_auth
precedence branch is needed.
The ID-JAG client_private_key is encrypted at rest alongside client_secret.
* fix(mcp): sort token_endpoint imports to satisfy the I001 budget gate
* fix(mcp): give token_endpoint pyright suppressions reasons for the LIT004 budget
The freshly-merged base ratcheted the LIT004 ceiling down, so the six
unexplained pyright suppressions in token_endpoint.py went over budget.
Annotate each with why the boundary is untyped (litellm http handler and
InMemoryCache are untyped; response.json() is validated by
_TokenEndpointResponse in fetch) so the gate counts them as explained.
* fix(mcp): enforce ID-JAG exchange over caller auth overrides and redact token endpoint from client errors
For oauth2_id_jag servers the v2 resolver mints the upstream assertion from the caller's identity token; a caller-supplied x-mcp-auth / x-mcp-<alias>-authorization override or a conflicting injected Authorization must not disable that exchange and forward an arbitrary bearer, so IdJagConfig now joins authorization_code and token_exchange as a resolver-owned mode that keeps the v2 spec and ignores the override.
The token endpoint error branches previously returned the configured endpoint URL in the client-visible 503 detail. The endpoint now stays in server-side logs and clients get a generic token-exchange failure.
* fix(mcp): bind the ID-JAG token cache to the exchange config and map token endpoint network errors to typed CredErrors
* fix(mcp): fail closed when an oauth2_id_jag server is half-configured instead of deferring to v1 static credentials
* fix(mcp): evict the cached ID-JAG bearer on an upstream 401 so the retry re-exchanges
* fix(mcp): map an unsignable client assertion to a typed misconfigured error instead of an unhandled 500
* fix(mcp): redact credential fields from the server-registry debug dump
* fix(router): enforce context-window pre-call checks for Responses API input
* test(router): cover _count_pre_call_check_tokens across API surfaces
* fix(router): count Responses instructions and skip pre-call token count when no input
* fix(router): forward Responses input into deployment selection for context-window checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: add Straiker guardrail integration
Implements LLM security guardrails via Straiker with prompt and response inspection, multi-mode execution (pre_call, post_call), and configurable blocking or redaction of flagged content across providers, streaming, images, and tool calls.
* fix(guardrails): harden straiker source attribution and error-path consistency
Use the operator-configured source for Straiker application attribution instead of a caller-supplied agent_id metadata value, so a caller cannot spoof which application a detection is attributed to. Make _fail reuse _block so a post_call error raises ModifyResponseException like a deliberate post_call block rather than GuardrailRaisedException, and type the blocking helper as NoReturn so the type checker enforces that execution never falls through the BLOCKED branch. Serialize the webhook payload once and send it as raw content to avoid re-serializing on the size check and on every retry.
* fix(guardrails): read straiker config and metadata from all supported shapes
Handle a dict optional_params in _get_config_value so nested guardrail
settings loaded from YAML or the DB (timeout, unreachable_fallback, and
the rest) are applied instead of silently falling back to defaults;
previously only attribute-style access was supported. Build the webhook
metadata bag from the merged metadata so client tags stored under
litellm_metadata on routes like /v1/messages reach Straiker the same way
identity and application fields already do, and widen the internal-key
skip prefix to user_api so proxy-injected budget values are not
forwarded.
* fix(guardrails): fail safe on straiker interventions without redactions
Block instead of passing content through when Straiker returns
GUARDRAIL_INTERVENED without replacement texts, so a positive
intervention verdict can never silently forward the original flagged
content. Fix the streamed-request detection to read the request body
from proxy_server_request.body, where the proxy stores it, instead of a
top-level body key that is never populated; the previous fallback was
dead, so a streamed response whose stream flag was not lifted to the top
level would have been redacted rather than blocked while buffering
replayed the original chunks.
* revert(guardrails): restore straiker caller agent_id application attribution
Restore the original behavior where a request-scoped agent_id in metadata
sets the Straiker application source, falling back to the configured
source. This is the integration's intended per-application attribution;
litellm already resolves a key-owned agent_id ahead of any caller-supplied
value, so a configured key cannot be spoofed.
* revert(guardrails): restore straiker webhook metadata scoping
Restore the original behavior where the Straiker webhook metadata bag is
built from request-scoped metadata only. Forwarding litellm_metadata was
a scope change to what the integration sends to Straiker; keep the
author's intended scoping.
* fix(guardrails): keep proxy key material out of straiker webhook metadata
Widen the internal-key skip prefix from user_api_key_ to user_api so the
proxy-injected user_api_key hash and user_api_end_user_max_budget are not
copied into the Straiker webhook metadata bag. The narrower prefix missed
the bare user_api_key name, leaking the hashed key to the vendor. Keeps
the request-scoped metadata source unchanged.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
The proxy already returns x-litellm-model-id (the deployment id) and x-litellm-model-group (the requested model-group alias), but never surfaces the concrete underlying model that served the request; the router rewrites the response model field to the group alias, so callers had no way to read the actual deployment model like anthropic/claude-haiku-4-5. Expose it as x-litellm-model-name, sourced from the deployment recorded in litellm_params metadata.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info
Resolves the per-model get_model_group_info fan-out on GET /v1/models
(and /models) that pegged the event loop on wildcard listings (#33636).
create_model_info_response now reads max_input_tokens/max_output_tokens
from litellm.get_model_info (the static cost map) rather than the router,
which aggregated and deepcopied every deployment in a group per listed
model.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage
Inject the cost-map lookup (defaulting to litellm.get_model_info) so the
except and max_output_tokens branches are exercised deterministically and
the token-limit tests no longer hardcode mutable cost-map values.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup
Add Router.get_configured_token_limits, an O(1) model-name index lookup that
reads a concrete deployment's configured max_input_tokens/max_output_tokens
without triggering pattern matching or deep copies. create_model_info_response
layers this over the cost map so custom deployments absent from the cost map
still surface their limits, and admin-configured limits override cost-map
defaults, while wildcard-expanded names stay on the fast path.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_get_model_info_reports_realtime_mode resolved gpt-realtime-mini through
litellm.get_model_info, which reads the cost map litellm fetches at import from
raw.githubusercontent.com/BerriAI/litellm/main. The mode=realtime retag from
#33728 is in this repo's json and its bundled backup but has not reached main
yet, so the test failed whenever the fetch succeeded and passed whenever the
runner was rate limited and litellm fell back to the backup, flapping the
Unit Tests: MCP, Secrets, Containers & Misc job on unrelated PRs
Resolve the lookup against the bundled backup instead, the way
tests/test_litellm/test_cost_calculator.py already does: force
LITELLM_LOCAL_MODEL_COST_MAP, rebind litellm.model_cost, and clear the
get_model_info lru cache before asserting so a remote-backed entry cached
earlier in the same worker cannot leak through, then clear it again afterwards
so no locally-backed entry outlives the test
Config.yaml-declared OAuth2 MCP servers using Dynamic Client Registration have no LiteLLM_MCPServerTable row, so the DCR persist path called update_mcp_server, which returns None for a missing row, then update_server(None), which dereferenced .approval_status and raised AttributeError. The exception was swallowed to a warning while /register still returned 200, so the minted client was never stored and every access-token expiry forced a full re-authorization
Persist the acquired DCR client (client_id, client_secret, token_endpoint_auth_method, redirect_uris, encrypted at rest) in a dedicated LiteLLM_MCPServerOAuthClient store keyed by server_id when the server has no row, overlay it onto the in-memory config server so the refresh_token grant can authenticate within the process, and rehydrate it when the registry syncs from the database (which runs after the DB connects, unlike config load) so restarts and other pods pick it up. The store is encrypted at rest and is re-encrypted by the master-key rotation path alongside the server rows, through a shared helper so the two sites cannot diverge. The DB-backed server path is unchanged, and guarding the None return removes the swallowed-crash footgun
Resolves the config.yaml DCR persistence regression introduced in v1.92.0 by #31912
enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are set as live
litellm attributes on the worker that handles the UI save, exactly like
budget_exceeded_throttle_percentage, but they were missing from
LITELLM_SETTINGS_SAFE_DB_OVERRIDES, so a peer worker's config reload merged the DB
value without applying it to the live attribute and stayed stale.
Add both to the allowlist so they behave like the sibling field, and add
test_general_settings_ui_fields_are_db_overridable so the UI registry and the
override allowlist cannot drift again (the exact omission that caused this), plus
a regression test that the flag flips on a simulated peer-worker reload.
