mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
10203 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
49995ba9ce
|
fix(proxy)!: apply request-parameter checks consistently across body, path and form inputs (#36011)
Cherry-pick of the #36011 merge (six commits) as a single unit.
(cherry picked from commit
|
||
|
yucheng-berri
|
8498dc0bb8
|
chore(proxy): clean up request parameter validation and provider destination handling (#34189)
(cherry picked from commit
|
||
|
yucheng-berri
|
08429626f9
|
fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool (#32093)
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
(cherry picked from commit
|
||
|
yucheng-berri
|
c5d2a3664d
|
fix(proxy): restore admin key/team callback_vars.turn_off_message_logging override (LIT-3587) (#31905)
The security fix in |
||
|
Yassin Kortam
|
d115edc388
|
feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655)
* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls
The GenAI semantic conventions record failures of a GenAI client operation as
a log-based event named gen_ai.client.operation.exception, carrying the
exception.type / exception.message / exception.stacktrace trio at severity
WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM
call produced only the deprecated error.* span attributes, a generic exception
span event without a stacktrace, and the stacktrace under the vendor key
litellm.provider.error.stack_trace.
Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring
the metrics plumbing) and record the event behind the enable_events flag, which
until now was defined but consumed nowhere. An operator-configured LoggerProvider
global is reused so the events ride their existing logs pipeline; an explicit
NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all.
The existing span-side error surface (error.type, error.message, the exception
span event, and the litellm.provider.error.* detail keys) is untouched for
backwards compatibility.
* fix(otel): always ride the semconv-required exception pair on the GenAI event
Filtering the event attributes on truthiness conflated "absent" with "empty",
so an empty exception.type or exception.message would have been dropped, leaving
an event with neither semconv-required field. Build the attributes so the pair is
unconditional and only the recommended stacktrace is omitted when the payload
carries none.
* docs(otel): document the events plumbing module in the package README
* test(otel): cover the log exporter selection and logs endpoint normalization
The new logs plumbing had no coverage for exporter-kind selection, the
console fallback for an unrecognized kind, the /v1/logs signal-path rewriting
that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch
processor split.
(cherry picked from commit
|
||
|
yucheng-berri
|
c0814be2c2
|
fix(guardrails): walk Responses-API text taxonomy in shared content helpers (#32542)
* fix(guardrails): walk Responses-API text taxonomy in shared content helpers
Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.
Three defects, all in _content_utils.py:
1. _iter_text_parts_in_content recognised only part.type == "text", but the
Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
Responses input list containing a function_call or function_call_output
item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
reject with a schema error.
Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.
* style: ruff-format changed guardrail files
* test(guardrails): cover function_call_output string form; drop em-dash in new docstring
* fix(guardrails): map function_call_output straight to user role
Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.
* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages
* docs(test): soften AIM-specific claims in LIT-4294 test docstrings
Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.
* refactor(guardrails): move unsupported-role coercion into AIM only
The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).
AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.
function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.
* refactor(guardrails): preserve role fidelity in shared _content_utils
Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).
Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
of the chat-completions tool message shape) instead of role user, so
Responses and chat completions produce symmetric inspection payloads.
A caller-supplied role on the item is still preserved.
AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.
(cherry picked from commit
|
||
|
mateo-berri
|
994d45756c |
fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882)
Backport of #32882 to stable/1.91.x. Cherry-picked from litellm_internal_staging (pending merge there). Adapted for this line: only the four anthropic.claude-fable-5 cost map entries exist on stable/1.91.x (no sonnet-5 or jp opus-4-8 entries), and the fallback-generalizations invariant test is omitted because the feature is absent here. |
||
|
devin-ai-integration[bot]
|
62367705e1 |
test(bedrock): switch image gen live test off EOL Titan to Nova Canvas (#31937)
Backport of #31937 to stable/1.91.x. Cherry-picked from 912ca6255c02e0532e4bd44e93b0eb2107f5bd5c (litellm_internal_staging). Fixes the base-level image_gen_testing failure on this line: amazon.titan-image-generator-v2:0 reached end of life on Bedrock. |
||
|
Mateo Wang
|
951e0ae29b |
fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831)
Backport of #32831 to stable/1.91.x.
Cherry-picked from
|
||
|
Mateo Wang
|
62f239b09b |
fix(bedrock): keep mid-conversation system messages in place for Claude Invoke (#32578)
Backport of #32578 to stable/1.91.x.
Cherry-picked from
|
||
|
yucheng-berri
|
d5d6e87434
|
fix(otel): restore error.* span attributes on v2 error spans (LIT-4179) (#32524)
The v2 emitter has never stamped error.message / error.code /
error.stack_trace / error.llm_provider as span attributes; only error.type
reached the wire. Backends that flatten span attributes into label
indexes (Elastic APM labels.error_*, Datadog span tags) lost these
four fields when v2 became the active integration on v1.90+ for
otel_v2-flagged deployments. The pre-existing exception span event
carrying the full message (LIT-3758) is unchanged; the message now
rides both places at once, matching v1s shape.
SpanError grows three optional detail fields; _parse_error threads
them from StandardLoggingPayloadErrorInformation; the emitters error
branch stamps them via a new module-level helper, guarded per field so
guardrail-shape errors are not polluted with empty attributes. New
semconv constants mirror open_inference.ErrorAttributes byte-for-byte,
so v1 and v2 consumers read the same keys.
Regression tests extend the mapped test files under
tests/test_litellm/integrations/otel/. pytest reports 243 passed.
(cherry picked from commit
|
||
|
yuneng-jiang
|
272c0f895c
|
Merge pull request #32405 from BerriAI/litellm_kraken-remove-envref-gates
fix(proxy): resolve os.environ/ refs universally in DB-sourced models
(cherry picked from commit
|
||
|
yuneng-jiang
|
fec5076c77
|
Merge pull request #32256 from BerriAI/litellm_bedrock_db_env_expansion
fix(proxy): resolve os.environ/ refs for all AWS auth params in DB-sourced models
(cherry picked from commit
|
||
|
devin-ai-integration[bot]
|
fb408608ea
|
revert: undo teamless all-team-models denial from #32022 and #29746 (#32032)
(cherry picked from commit
|
||
|
tin-berri
|
fe9fb0b81a
|
fix(mcp): persist DCR client_id so interactive OAuth token refresh works (#31912)
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix: reuse persisted MCP DCR clients
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit
|
||
|
tin-berri
|
13db8cb140
|
fix(mcp): surface tools/list 401 auth failures as a challenge on single-server routes (#31921)
A 401 while listing tools (a missing or expired per-user OAuth token, or an
upstream 401 for any auth_type) was swallowed to an empty tool list, so a
single-server client got a 200 with no tools and no WWW-Authenticate challenge
instead of a 401 it could re-authenticate against. Only oauth pass-through and
delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the
missing-token case for all of them, masked it.
The surface-vs-absorb decision now keys on the route, not the auth_type. An
upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError
regardless of auth_type, and the per-user OAuth challenge raised during client
creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is
converted to the same type in _get_tools_from_server. The challenge is scoped
to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a
re-auth signal and degrades to an empty list like any other non-auth error, and
the stdio-allowlist 403 (no challenge header) stays absorbed. The existing
routing then does the right thing: single-server routes turn the error into a
401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty
list so one unauthenticated server does not fail the whole listing.
On the UI tools page, an OBO (per-user authorization_code) server now shows the
Authorize gate when the list call returns 401, not only when no credential row
exists. The backend already refreshes a still-refreshable token on the list
call, so a 401 means there is no valid token and none could be minted (expired
with no usable refresh token), which is exactly when the user must reauthorize.
(cherry picked from commit
|
||
|
Shivam Rawat
|
e356ead560
|
fix(bedrock): honor ttl for tool_config cache injection points (#31929)
* fix(bedrock): honor ttl for tool_config cache injection points
Pass cache_control_injection_points control.ttl through to Bedrock
toolConfig cachePoint blocks, matching message/system cache behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex
is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a
manual update for every new Claude release (it already silently missed
Sonnet 5 and Fable 5). Replace it with a lookup against
cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json,
which AWS docs confirm tracks the same 1h-TTL-capable model set.
Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried
that pricing field (their own regional variants didn't have it), which
would have made the JSON-driven check wrongly grant them 1h TTL support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests
test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and
test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id
(...-20250514-v1:0) that never shipped. This passed under the old regex-based
is_claude_4_5_on_bedrock, which matched on substring alone, but fails now
that it looks up cache_creation_input_token_cost_above_1hr in
litellm.model_cost, since the fake id has no pricing entry.
Also force the bundled local cost map in both tests so ttl eligibility reads
this branch's pricing data instead of the network-fetched main copy, which
lacks the fix until merge.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): restore cache and tool config compatibility
* fix(bedrock): preserve Sonnet 5 parallel tool config
* fix(bedrock): decouple parallel tool support from cache ttl
* refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns
Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and
bedrock_converse_supports_strict_tool_schemas (dead code) with a
supports_parallel_tool_use_config key in model_prices_and_context_window.json,
matching how is_claude_4_5_on_bedrock already reads
cache_creation_input_token_cost_above_1hr from the pricing JSON.
New models pick up parallel tool use support automatically when their
pricing entry ships with the key set, with no code change required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test
anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in
model_prices_and_context_window.json, so
bedrock_converse_supports_parallel_tool_use_config returned False and the
test died with KeyError on additionalModelRequestFields. Use
jp.anthropic.claude-opus-4-7, a real entry that carries
supports_parallel_tool_use_config without 1h-TTL cache pricing, which is
exactly the decoupling this test exists to cover
* test(utils): allow supports_parallel_tool_use_config in pricing schema
The misc unit test job validates model_prices_and_context_window.json
against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects
unknown keys. Add the supports_parallel_tool_use_config key this PR
introduced so test_aaamodel_prices_and_context_window_json_is_valid
passes again
* fix(bedrock): preserve ttl for regional claude models
* fix(bedrock): fall back to base model entry when regional pricing lacks capability fields
Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit
cache_creation_input_token_cost_above_1hr shadowed the base entry that has it,
so is_claude_4_5_on_bedrock returned False and requested cache ttl values were
dropped for those deployments. Both capability lookups now consult the full
model id and the region-stripped base entry, matching the coverage of the old
name-pattern list. Also restores ToolBlock keyword construction for the
tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every
supported Python version
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit
|
||
|
Mateo Wang
|
c734ee772f
|
fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8 (#31582) (#31923)
* fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8
Bedrock Converse routes Claude Opus 4.7/4.8 through an Anthropic-compatible
validator that maps toolSpec to the native tool shape and rejects the extra
`strict` key with `tools.N.custom.strict: Extra inputs are not permitted`,
even though Anthropic's native API accepts `strict` as a top-level tool field
for the same models. Sonnet 4.5/4.6 and Opus <=4.6 accept `toolSpec.strict`
unchanged.
The existing gate `get_bedrock_base_model(model).startswith("anthropic")`
(introduced in #29814 to forward `strict` for Claude on Bedrock Converse) is
too broad and regressed Opus 4.7/4.8 callers — see #31582.
Replace the inline check with a small `bedrock_converse_supports_strict_tools`
helper that excludes the Opus 4.7/4.8 family from strict forwarding. All
other Anthropic models on Bedrock keep the existing behavior.
Closes #31582.
* fix(bedrock/converse): move strict-tools regression to a clean test file
The original regression test was added to
test_litellm_core_utils_prompt_templates_factory.py, which has
pre-existing ruff-format violations throughout (multi-line asserts that
fit on one line). The lint workflow runs `ruff format --check` on
changed files only, so touching that file surfaces those pre-existing
violations and fails CI for unrelated reasons.
Move the #31582 regression coverage into a new dedicated test file so
the format check stays green. Also collapses the helper's `not any(...)`
onto a single line to satisfy ruff format.
Covers: #31582
* refactor(bedrock/converse): drive strict-tools gate from model cost map
Replace the hardcoded Opus 4.7/4.8 pattern list with a
bedrock_converse_supports_strict_tools flag on the affected entries in
model_prices_and_context_window.json, resolved via get_model_info with a
local cost map fallback, so future models with the same restriction only
need a JSON update
* chore: revert unrelated credential_migration.py reformat
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
(cherry picked from commit
|
||
|
tin-berri
|
2578d9557b
|
fix(mcp): support client_secret_basic for upstream OAuth token endpoints (#31635)
The MCP gateway authenticated to upstream OAuth token endpoints only with
client_secret_post (client_secret placed in the POST body). Providers that
require HTTP Basic client authentication (client_secret_basic, the OIDC
default) reject that with invalid_client, which surfaced as a 500 on the
/<server>/token exchange and broke both the initial authorization_code
exchange and refresh.
Add a per-server token_endpoint_auth_method ("client_secret_basic" |
"client_secret_post") and a single helper that builds the right headers and
body for the configured method, then route every upstream token-endpoint POST
through it: the inbound exchange and refresh in discoverable_endpoints, the v1
per-user refresh in db, the v2 authorization_code refresher, the M2M
client_credentials fetch, and the RFC 8693 token exchange. The default stays
client_secret_post so existing servers are unaffected; basic sends
Authorization: Basic base64(form-urlencode(client_id):form-urlencode(secret))
per RFC 6749 section 2.3.1 and omits the secret from the body.
client_secret_basic is a confidential-client method, so a server configured for
it with a missing client_id/secret raises rather than silently downgrading to a
body request (no-silent-fallback); the inbound endpoint maps that to a 400 and
the refresh paths to a failed-refresh / needs-reauth. A secretless client_id
under the default method stays valid for public clients authenticating with PKCE.
Resolves LIT-4091
(cherry picked from commit
|
||
|
ryan-crabbe-berri
|
234263fdda
|
fix(router): persist global retry_policy via /config/update (#29540)
* fix(router): persist global retry_policy via /config/update (LIT-3152)
The Admin UI Model Retry Settings tab POSTs
{router_settings: {retry_policy: {...}}} to /config/update, but the
field was dropped on two write-side layers so it never reached the
router. UpdateRouterConfig did not declare retry_policy, so
dict(exclude_none=True) stripped it before the DB upsert. And even when
fed directly, Router.update_settings had no "retry_policy" entry in
_allowed_settings, so the assignment was a silent no-op. The DB row
stayed at {"model_group_alias": {}}, llm_router.retry_policy stayed
None, and the UI fell back to defaultRetry = num_retries = 2 on refresh.
Declare retry_policy on UpdateRouterConfig as a plain dict, and add a
retry_policy branch to update_settings that coerces dict payloads to
RetryPolicy before setattr, mirroring Router.__init__. get_settings
already lists retry_policy, so reads work once writes land.
* fix(router): guard retry_policy type in update_settings
Mirror Router.__init__ semantics in update_settings: only assign
retry_policy when it is None or a RetryPolicy (after dict coercion).
Previously a non-dict, non-RetryPolicy value (e.g. a YAML typo like
retry_policy: 5 flowing through /config/update) was stored verbatim,
deferring the failure to request time in get_num_retries_from_retry_policy
instead of being dropped at write time.
* refactor(ui): harden Model Retry Settings flow and validate retry_policy at the boundary
Types UpdateRouterConfig.retry_policy as RetryPolicy and model_group_retry_policy as Dict[str, RetryPolicy] so /config/update validates the payload and rejects malformed counts instead of silently persisting them; the apply path in update_settings keeps coercing the stored dict back to RetryPolicy
Makes the Model Retry Settings tab the single owner of retry_policy and model_group_retry_policy so the generic Router Settings page no longer renders or writes them, replaces the fire-and-forget save with a react-query mutation that only shows the success toast after the write resolves, surfaces real errors, disables Save while in flight, and re-reads authoritative state on success, and sends both the global and per-group policies atomically so edits in the inactive scope are no longer dropped
Decouples the retry-scope selector from the All Models filter and defaults it to Global, seeds the displayed default from num_retries (falling back to 2), and gives per-group rows real inherit semantics so an empty input shows the global value as a placeholder with a Reset control, keeping 0 ("no retries") distinct from inheriting the global value
* fix(keys): align router_settings examples with typed RetryPolicy and resync UI artifacts
model_group_retry_policy is now Dict[str, RetryPolicy], so the {"max_retries": 5} sample in the key-generate test and the /key/generate and /key/update docstrings no longer validate; they now use a valid {"gpt-4": {"RateLimitErrorRetries": 5}} shape.
Regenerated eslint-metrics.json (no-explicit-any drifted 2027 -> 2026) and schema.d.ts (new RetryPolicy schema, retry_policy field, model_group_retry_policy value type) so the UI build and api-types-sync checks pass
* test(router): pin retry_policy persistence end to end (LIT-3152)
The existing retry_policy tests exercise UpdateRouterConfig and Router.update_settings in isolation, so they would all still pass if a regression flipped ConfigYAML.router_settings back to a loose dict or stopped add_deployment from applying the stored row. This drives the real handler chain an Admin UI save triggers: update_config writes the LiteLLM_Config row, the apply path forwards it to the live router, and get_config serializes it back, pinning retry_policy across persist, apply, and read-back.
* fix(teams): use valid model_group_retry_policy example in router_settings docstring
Same stale {"max_retries": 5} example the key endpoints carried; model_group_retry_policy maps a model group to a RetryPolicy, so the team /team/new and /team/update docs now show {"gpt-4": {"RateLimitErrorRetries": 5}}. Regenerated schema.d.ts to match.
* fix(ui): load retry settings via deferred fetch to satisfy set-state-in-effect
The Model Retry Settings effect called loadRetrySettings synchronously; eslint-plugin-react-hooks (react-hooks/set-state-in-effect) traces into it and flags the setState calls, failing frontend-lint. Split the loader into fetchRouterSettings + applyRouterSettings and run the fetch in an inline async IIFE with a cancellation flag, so state is applied in the post-await callback rather than on the effect's synchronous path. Behavior is unchanged and onSuccess still refreshes via loadRetrySettings.
* fix(ui): match CI rendering of RateLimitError 429 docstring in generated schema
gen:api run on a dev env (python 3.13 / newer fastapi) rendered the RateLimitError response description with 4-space indentation, but CI regenerates it with 8-space under its frozen python 3.12 toolchain, which is the canonical committed form. The Check UI API Types Sync job regenerates and diffs, so restore that block to the CI rendering; verified byte-identical to the pre-existing committed version.
* fix(ui): pin RateLimitError 429 docstring to CI's frozen schema rendering
Base #29619 regenerated schema.d.ts on a newer FastAPI that renders the RateLimitError response description at 4-space indent, but the Check UI API Types Sync job regenerates under the frozen python 3.12 toolchain, which renders 8-space. Merging base pulled in the 4-space form; restore the 8-space rendering so the generated types match what CI produces (verified byte-identical to the pre-#29619 committed form), which also corrects the base drift once this PR merges.
|
||
|
ryan-crabbe-berri
|
ac56320f26
|
fix(agents): show an agent's attached virtual key in the UI (#29619)
* fix(agents): show an agent's attached virtual key in the UI
The A2A agent detail view never surfaced which virtual key was attached to
an agent, so after assigning a key during agent creation there was no way to
see it again. Surface the attached key(s) in the agent detail view, derived
from the key table's agent_id foreign key the same way spend is already
joined into the agent response.
Backend adds an agent_id filter to /key/list (mirrors team_id) and enriches
GET /v1/agents and GET /v1/agents/{id} with a non-secret key summary (alias,
masked key_name, hashed token id). The frontend renders a Virtual Keys
section in the agent detail view that lists the agent's keys and links
through to the key detail, and the list view drops its fetch-500-keys-and-
filter-client-side workaround in favor of the enriched response. The orphaned
AgentCard and AgentCardGrid components, left behind when the agent list
switched from a card grid to a table, are removed
* fix(agents): redact attached virtual keys for non-admins
_attach_keys_to_agents joins keys onto the agent response by agent_id with
no caller scoping, but _redact_sensitive_agent_fields never cleared the new
keys field. A non-admin able to view an agent therefore received the alias,
masked name, and hashed token of every key attached to it, including keys
owned by other users or teams; the old client-side path used the scoped
key list, so this was a visibility regression. Clear keys in the redaction
path so only admins see attached-key metadata.
Adds an endpoint-level regression test asserting keys is populated for admins
and null for non-admins, and a list-view test covering the Active vs Needs
Setup badge that lost coverage when the agent card tests were removed.
* fix(agents): satisfy strict lint and resync key/list types
- use builtin list/dict generics in the new agent key helpers to stay
under the UP006 strict-rule ceiling
- swap @tremor/react for antd Typography in agent_virtual_keys (tremor is
being phased out; the new component was the only unsuppressed import)
- regenerate schema.d.ts so the /key/list agent_id query param is typed
* style(agents): prettier-format key hook test and agent_info
|
||
|
Mateo Wang
|
ef3dcf91a2
|
chore: remove unused keys from model cost map (#31528) | ||
|
tin-berri
|
5963b9320f
|
feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31493)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5) The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss. * feat(mcp): DualCache-backed token cache backend (step 1b §1.5) The cross-replica TokenCacheBackend implementation that plugs into the foundation's CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected; a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss. * feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5) The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis. * feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5) The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh), release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired / not-held so a cache blip causes an extra refresh, never a crash on the resolve path. * feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5) Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh coordinator when Redis is wired, falling back to the foundation's in-process defaults on a single replica. Layers the cross-replica path on top of the single-replica dispatch store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): refresh on lock-backend error instead of serving a stale token The cross-replica refresh coordinator elected refreshers with a boolean acquire: a Redis transport error was caught and returned as False, which is indistinguishable from "another worker holds the lock". On a total Redis outage every worker therefore took the wait-then-reread branch and served the still-expired token upstream (the upstream then 401s), even though the lock and coordinator docstrings claimed a Redis blip "degrades to an extra refresh". Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the coordinator can tell a busy holder from a dead backend, and refresh anyway on ERROR. This single-flight lock is a load optimization, not a correctness mutex, so failing open is correct: it degrades a lock-backend outage to the no-coordinator behavior (an extra refresh), never a stale bearer. Add a regression test asserting an acquire error refreshes rather than re-reading the expired token, and update the docstrings to match. * style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format * fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed The cross-replica coordinator's losers re-read the token the winner persisted. If the winner's refresh failed, the store still holds the expired token, so the loser re-read it and RefreshingTokenStore handed that expired bearer to the caller (the upstream then 401s) instead of the re-auth challenge the winner returned via None. Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a re-read that is still expired surfaces None so the arm challenges. This only affects the loser path; the winner's freshly refreshed token is returned directly by the coordinator and is unaffected. * fix(mcp): log per-user token decrypt failures at debug, matching v1 When a cached blob cannot be decrypted (e.g. after a salt or master-key rotation) the codec logged a full traceback at error level, since decrypt_value_helper defaults to exception_type=error. v1's MCPPerUserTokenCache passed exception_type=debug on the same path. The blob is ciphertext so this is log noise only, but matching v1 avoids error-level traceback spam on stale entries after a key rotation * fix(mcp): namespace the refresh lock key and fence its release with a token The Redis lock wrote its key through the raw client from init_async_client(), bypassing RedisCache's namespace, so two deployments sharing one Redis collided on mcp:refresh_lock:<user>:<server> for any overlapping (user, server) and a colliding deployment skipped the refresh and challenged its own users. The lock now runs every key through an injected namespace_key wired to RedisCache.check_and_fix_namespace, matching the namespace its token cache already uses release() also deleted the key unconditionally, so a holder whose lock PX-expired and was re-acquired by another worker could delete the new holder's lock and let a third worker run a duplicate refresh, recreating the rotating refresh_token race. acquire now writes a unique per-acquisition token generated by the coordinator and release deletes only when the key still holds that token, via a compare-and-delete Lua script Adds regression tests: release with a stale token is a no-op while the owner's release deletes; keys are namespaced before reaching Redis; the coordinator acquires and releases with the same token * fix(mcp): fail open when the per-user token cache delete errors DualCache swallows get/set errors internally but not delete, and the Redis layer underneath re-raises through its circuit breaker. So a Redis outage on the delete() path escaped CachedOAuthTokenStore.fetch()'s unauthorized branch (which deletes before returning None) and invalidate(), turning a cache blip into a 500 instead of the v1-style fallback. Catch in the backend so delete degrades to the TTL-bounded stale entry like get/set already do. * style(mcp): reformat outbound-credentials files to line-length 120 The merge from staging brought in ruff's line-length 120, but these two PR-authored files were still wrapped at the old width, so the diff-scoped ruff format --check in CI flagged them. Pure reformatting; no behavior change. * fix: harden mcp oauth redis refresh coordination * fix(mcp): make the per-user token cache backend airtight on boundary failures get/set now degrade a cache or codec failure to the safe value (miss / no-op) in the backend itself rather than relying on DualCache and decrypt_value_helper happening to swallow internally, matching delete() and v1's MCPPerUserTokenCache. This upholds the layer's boundary-failure-is-a-miss contract regardless of the injected collaborators, so a Redis outage or an undecryptable entry reads as a cache miss that re-reads the DB instead of a 500. Adds contract tests for the cache raising on get/set/delete and the codec raising on encode. * test(mcp): pin per-user cache get() to a miss when decrypt raises Greptile's out-of-diff repro had the decrypt reject a blob with ValueError (bad ciphertext after key rotation); cover that exact raise path, not just the decrypt-returns-None case, so get() is regression-locked to read it as a miss. * refactor(mcp): use frozen dataclasses for the trivial DI constructors Replace the hand-written self._<arg> = arg constructors on OAuthTokenCacheCodec, RedisRefreshCoordinator, RedisDistributedLock, and DualCacheTokenCacheBackend with frozen slotted dataclasses, matching the rest of this layer. Fields take the former parameter names so the constructor API (and the tests' keyword args) are unchanged; KW_ONLY preserves the keyword-only collaborators. * fix: serialize lazy per-user oauth store rebuild * fix(mcp): stop losers challenging mid-refresh by decoupling wait from lease TTL wait_timeout_seconds defaulted to the same 10s as lock_ttl_seconds, but the holder renews its lease while a slow token endpoint runs, so a loser waiting past 10s bailed and re-read the still-expired DB token, challenging the user even though a valid refresh was in flight. Bound the holder's renewal with a refresh budget so its lock-hold is finite, and set the loser's wait to outlast that budget (refresh_budget_seconds + one lease tail) so a loser only re-reads once the holder has finished or its bounded lease has lapsed, never mid-refresh. * fix: allow concurrent lazy OAuth fetches without Redis --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
d515e5bf05
|
fix(vertex_ai): append rawPredict suffix for custom api_base on /v1/messages (#31529) | ||
|
Yassin Kortam
|
88c7755283
|
fix(redis): loop-scope async Lua script registration (#31501)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(redis): loop-scope async Lua script registration async_register_script registered the Lua script eagerly and returned a callable bound to the Redis client of the event loop running at registration time. The v3 parallel request limiter registers its three scripts once in __init__ at proxy startup and stores them, so a request or logging callback on another loop awaited a script bound to the startup loop and hit "got Future attached to a different loop". The limiter then fell back to a pipeline that reset the window TTL every increment, so counters never expired and an 80M TPM model rate-limited around 40M. Defer registration to call time and cache the per-loop executor in in_memory_llm_clients_cache (which already keys on the running loop), so each loop runs the script against its own client. Covers all five consumers of the primitive. Resolves LIT-3298 * fix(redis): await evalsha on the cluster Lua script path The cluster branch returned the evalsha coroutine without awaiting it, so callers received a coroutine instead of the script result. Await it, which also addresses the cluster path called out in review. |
||
|
Yassin Kortam
|
b2e708d5ae
|
feat(prometheus): add per-team litellm_team_members_metric gauge (#31506)
Emit litellm_team_members_metric on every team member add and delete, labelled by team and team_alias and set to the team's authoritative member count. Because it is set from the current membership rather than incremented or decremented, it tracks the count up and down, never goes negative, and self-corrects on the next change after a proxy restart. Bulk member add is covered for free since it delegates to team_member_add, and the helper no-ops when the Prometheus callback is not registered. Resolves LIT-3082 |
||
|
Yassin Kortam
|
1883f975e2
|
fix(proxy/auth): honor user_api_key_cache_ttl for management-object cache writes (#31504)
general_settings.user_api_key_cache_ttl was ignored for every management-object write into user_api_key_cache. The configured value is propagated to the cache's default_in_memory_ttl at startup, but DualCache only applies that default when no explicit ttl kwarg is passed, and every management-object writer passed ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL (60s), which always won. So keys, teams, users, budgets, object permissions, vector stores, JWT user syncs and MCP caches all expired after 60s regardless of the setting. Adds get_management_object_ttl(cache) in user_api_key_cache.py, which returns the configured default_in_memory_ttl and falls back to the 60s constant only when no default is set, and routes every management-object writer through it. The helper takes a DualCache so it works at the many call sites that are typed UserApiKeyCache but exercised with a bare DualCache. Also covers the spend-update writeback in update_cache (async_set_cache_pipeline), which hardcoded ttl=60 on the same key/user/team objects and reset an active key's cache entry back to 60s on every priced request, so the configured TTL was never observed for keys receiving traffic. Resolves LIT-3338 |
||
|
Yassin Kortam
|
63490655ad
|
fix(pass_through): log pre-call guardrail blocks at WARNING, not ERROR with a traceback (#31500)
A pre-call guardrail block on a pass-through endpoint (e.g. OpenAI moderation flagging disallowed content) was logged at ERROR level with a full stack trace, even though the guardrail is working as designed and the client correctly receives the 4xx. The generic except in pass_through_request logged every exception via verbose_proxy_logger.exception(), so an intentional block produced scary traceback noise for operators tailing logs. Branch on the existing CustomGuardrail._is_guardrail_intervention classifier (the same predicate pipeline_executor already uses) so guardrail interventions log once at WARNING without a traceback while genuine failures keep their ERROR and traceback. This covers every guardrail that signals a block through the shared typed exceptions or an HTTPException 400, not just OpenAI moderation, and leaves the client-facing response unchanged. Resolves LIT-3538 |
||
|
Yassin Kortam
|
c33a7f8757
|
fix(proxy): cancel upstream LLM stream when client disconnects during time-to-first-token (#31499)
create_response buffers the first streamed chunk (to detect error-only streams) before handing the StreamingResponse to Starlette. Starlette only starts listening for client disconnects once it is serving that response, so a disconnect during a long time-to-first-token left the upstream LLM call running until the request timeout. This races the first-chunk fetch against an http.disconnect monitor; on disconnect it cancels the fetch, which propagates into async_streaming_data_generator's cleanup (records the 499 and closes the upstream stream), and returns a 499. Resolves LIT-3568 |
||
|
Yassin Kortam
|
437acc9b09
|
perf(proxy): bound event-loop blocking from oversized requests (#31497)
Skip token counting in Router._pre_call_checks when no deployment in the group declares max_input_tokens, and skip the full-body surrogate-repair regex in _read_request_body above a configurable size, raising the existing 400 immediately. Resolves LIT-3541 |
||
|
Mateo Wang
|
64d8d7f8cb
|
fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364)
* fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke
* style(bedrock): use builtin generics in new Invoke helpers to clear UP006 gate
* fix(bedrock): honor explicit thinking budget_tokens=0 in clear_thinking conversion
The clear_thinking_20251015 -> adaptive conversion resolved the thinking
budget with `thinking.get("budget_tokens") or BEDROCK_MIN_THINKING_BUDGET_TOKENS`,
which treats a caller-supplied `budget_tokens=0` as missing and silently
substitutes the Bedrock minimum. Resolve the budget with an explicit
`is not None` check so an explicit 0 is honored.
* fix(bedrock): gate Fable 5 into clear_thinking adaptive injection on Invoke
_ensure_thinking_for_clear_thinking_context_management returns early when
_supports_extended_thinking_on_bedrock(model) is False, so the adaptive-thinking
injection never runs for models absent from that gate. Opus 4.8 slips through on
the incidental "opus-4" substring, but Fable 5 had no matching pattern, so a
clear_thinking_20251015 request on Fable 5 reached Bedrock with an unsupported
context-management edit and no thinking field; the exact 400 this path exists to
prevent. Add the fable-5 patterns to the gate so Fable 5 (mapped ids and unmapped
aliases) gets thinking.type=adaptive + output_config.effort like the other
adaptive models.
Extend the adaptive-injection regression test to cover Fable 5 (a mapped id and
an unmapped alias) so it fails without the gate entry, and add focused coverage
for the budget->effort tiers, the disabled/enabled/adaptive thinking branches,
output_config.effort preservation, and list/dict system-role normalization.
Also normalize the Invoke transformation module and its test to line-length 88
so ruff format --check (CI format-check) passes.
* refactor(anthropic): make supports_adaptive_thinking flag authoritative for thinking detection
Replace the per-version name helpers (_is_claude_4_6/4_7/4_8_model,
_is_claude_fable_5_model) with cost-map-flag-first detection. _is_adaptive_thinking_model
now reads supports_adaptive_thinking from the model cost map and falls back to a single
generalized family-version regex (_claude_version_at_least(model, 4, 6)) only when a model
is unmapped, instead of hard-coding each new Claude release.
Wire supports_adaptive_thinking through ProviderSpecificModelInfo and ModelInfo so the cost
map flag actually surfaces at lookup time. Reroute the Bedrock Invoke extended-thinking gate
and the two anthropic/chat/transformation.py call sites through _is_adaptive_thinking_model.
Known gap left to the fallback_generalizations work (#29718): unmapped Fable 5 aliases have
no parseable minor version, so they defer to the cost map and are not detected until a mapped
entry or a generalization rule exists. Covered by an explicit regression test.
* refactor(anthropic): drop name-based version fallback; resolve adaptive thinking from cost map only
The prior commit kept a regex (_claude_version_at_least) as a fallback when an id
resolved to no cost-map entry. Remove it: _is_adaptive_thinking_model now reads
supports_adaptive_thinking and nothing else, so "which Claude versions think
adaptively" lives entirely in the model cost map, and a new adaptive release is a
JSON edit rather than a Python edit.
To keep the flag authoritative across the id forms the Bedrock Invoke and anthropic
paths actually see, backfill supports_adaptive_thinking=true on every adaptive Claude
entry that was missing it (Opus 4.6/4.7 and Sonnet 4.6 across region/provider aliases)
in both the root and bundled cost maps, and generalize _model_map_lookup_candidates to
normalize an id to its base cost-map key: strip a Bedrock version suffix (-v1:0 fully,
or just the :0 inference-profile minor so the -v1-keyed 4.6 entries resolve), strip a
dated-release suffix (-20260219), and rewrite a dotted family version (4.6 -> 4-6).
This is id normalization feeding the lookup, not capability-by-name.
Tests load the PR-local cost map (the flags are not on main until merge) and cover each
normalization path plus the unmapped-alias deferral to fallback_generalizations (#29718).
* refactor(reasoning_effort): single-source effort<->thinking-budget mappings
Route every reasoning_effort <-> thinking-budget conversion through the DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants so the numbers stay in sync across providers. The five constants are now 2000/5000/10000/20000/40000
Add reasoning_effort_from_thinking_budget() in litellm_core_utils/reasoning_effort_utils.py and route the three OpenAI-style forward maps (anthropic adapters, responses adapters, hosted_vllm) through it. The bedrock invoke and experimental messages adaptive maps now reference the constants directly; the only behavior change is the xhigh threshold moving from 24000 to 20000. Reverse maps and the cross-provider test grid read the same constants
* test(reasoning_effort): lift budget-mode max_tokens above the new high budget
The single-sourced DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET thresholds moved
high from 4096 to 10000. The live reasoning_effort grid sends budget-mode
requests with max_tokens=8192, so reasoning_effort=high now produces
budget_tokens=10000 > max_tokens and every provider returns 'max_tokens must be
greater than thinking.budget_tokens'. Derive a shared BUDGET_MODE_MAX_TOKENS
(2x the high budget) for the spec and the request builder so the ceiling always
clears the largest 200-expected tier. Also resolve the inherited base
test_reasoning_effort assertion off the same high-budget constant instead of the
stale 4096 literal so it tracks the source of truth.
* fix(reasoning_effort): keep effort<->budget thresholds at pre-PR values
The single-sourcing refactor moved the shared effort<->budget thresholds up
(low 1024->2000, medium 2048->5000, high 4096->10000, xhigh 8192->20000,
max 16384->40000). That silently changes the effort->budget direction: a caller
who sets reasoning_effort together with a max_tokens that used to sit above the
old per-tier budget but below the new one now trips the provider's
"max_tokens must be greater than thinking.budget_tokens" 400. It spans every
backend that derives a budget from an effort (Anthropic, Gemini/Vertex,
hosted vLLM), not just Bedrock.
Restore the constants to their pre-PR values while keeping every backend reading
from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, so the
mapping stays single-sourced without the behavior change. Tests that pinned the
raised thresholds now derive their boundaries from the same constants.
* test(reasoning_effort): derive high effort->budget assertions from the shared constant
The cross-provider translation tests pinned reasoning_effort="high" to a literal
budget_tokens=10000, the raised value. Point them at
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET so they track the single source
instead of a magic number.
* fix(anthropic): resolve adaptive flag for combined dated+versioned Bedrock ids
The model-map candidate normalization applied each suffix strip independently to
the original id, so the real Bedrock shape "<base>-<YYYYMMDD>-v1:0" never reduced
to its base cost-map key: stripping the version left the date, and the
dated-suffix regex is anchored to the end so it could not fire while the version
was still present. An adaptive Claude model invoked by its full dated+versioned
id (e.g. us.anthropic.claude-sonnet-4-6-20251101-v1:0) therefore resolved to
supports_adaptive_thinking=null and was treated as non-adaptive, reaching Bedrock
with the rejected thinking.type=enabled shape, the exact 400 this path prevents.
Add a composed normalization that rewrites the dotted family version, then peels
the -vN:rev version suffix, then the -YYYYMMDD dated suffix, so the combined form
resolves to its base key. Regression tests pin the combined suffix on sonnet-4-6
and opus-4-8 across provider/region prefixes.
* fix(reasoning_effort): align budget<->effort tests with reverted constants and format common_utils
The constant revert restored the effort<->budget thresholds to their pre-PR
values (1024/2048/4096/8192/16384) and single-sourced the reverse
budget->effort ladder through reasoning_effort_from_thinking_budget, but
several tests still pinned the briefly-raised values and the old hardcoded
reverse buckets, so the "All Other Providers" shard failed
Derive the anthropic chat effort->budget assertions from the shared
DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, and update the
experimental pass-through and responses adapter expectations to the
single-sourced reverse ladder (budget 1024 -> low, 5000 -> high)
Also run ruff format --line-length 88 over anthropic/common_utils.py so the
CI format-check, which checks the whole changed file, passes
|
||
|
Yassin Kortam
|
80d3b69d9c
|
fix(pass-through): remove stale routes by key so the registry stops growing every reload (#31314)
The 30s add_deployment_job re-runs initialize_pass_through_endpoints, which
re-registers every config/DB pass-through endpoint. Endpoints without a
persisted id get a fresh uuid each cycle, so their route key
("{id}:{type}:{path}:{methods}") changes every reload. The stale-route cleanup
called remove_endpoint_routes(route_key), but that helper matches entries by
endpoint_id, so it never matched a route key and never deleted anything. The
registry grew by one entry per route per reload, turning the per-cycle cleanup
and the per-request is_registered_pass_through_route scan into a CPU sink that
eventually pins a core and slows every endpoint.
Pop the stale key from the registry directly in O(1). openai_routes is left
alone: its append is path-deduped and the path is still owned by the live
endpoint re-registered under a new id in the same cycle.
Resolves PERF-13
|
||
|
tin-berri
|
4b398ef6d4
|
feat(mcp): migrate authorization_code MCP to the v2 resolver (single-replica) [1/2] (#31473)
* feat(mcp): implement the authorization_code resolver arm
Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).
* feat(mcp): v1-backed OAuth token source for authorization_code
V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.
* style(mcp): modern type annotations in the authorization_code arm and source
* refactor(mcp): share v1's OAuth egress core; make V1PerUserTokenStore refresh-capable
Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).
* feat(mcp): route oauth2 per-user (authorization_code) servers through the v2 resolver
to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.
* feat(mcp): per-server fail-closed OAuth challenge at the v2 egress
When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.
* feat(mcp): cut the call_tool egress over to v2 for authorization_code servers
_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.
* feat(mcp): cut the tools/list connection over to v2 for authorization_code servers
The listing connection's per-user OAuth header is no longer built by v1 for migrated servers; the
v2 resolver drives it at connect time, ending the double-resolution where v1 built the token into
extra_headers and the v2 graft then deferred to it. Safe because the preemptive 401 (in the
streamable-http and SSE handlers) already challenges a missing token before the listing connection
runs, so the connection is only reached with a token present. Non-migrated oauth2 (delegate) and
the rest still build their header on v1. With this, resolve_credentials' result is honored on every
authorization_code upstream path: tool calls and listing.
* feat(mcp): route the preemptive 401 existence check through the v2 resolver
The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.
* refactor(mcp): extract the authorization_code arm into a helper
Mirror the api_key arm's structure: the inline AuthorizationCodeConfig body moves into
_authorization_code(subject, server), keeping resolve_credentials a flat one-line-per-arm dispatch.
The helper is annotated with the concrete StaticHeaderAuth it returns rather than the abstract
httpx.Auth (which api_key uses) because a new method carrying the unresolved httpx.Auth return
would add reportUnknownMemberType; the concrete type is both precise and budget-neutral.
* fix(mcp): emit the canonical WWW-Authenticate header name in the OAuth challenge
raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.
* feat(mcp): v2-native per-user token read store (step 1b inner store)
Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.
* feat(mcp): v2-native authorization_code token refresher (step 1b)
The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.
* feat(mcp): wire the v2-native per-user OAuth store into the resolver (step 1b piece 4)
Assemble Cached(Refreshing(V2PerUserTokenStore)) at the composition root and replace
V1PerUserTokenStore in mcp_server_manager. The chain is built lazily on first fetch (its cache/DB/
Redis collaborators are LiteLLM globals not ready at import); when Redis is wired it uses the
cross-replica path (DualCache cache + SET NX PX coordinator), else the in-process defaults. The DB
read, refresh-grant POST, and persist acquire their globals per call like v1. authorization_code
resolution now reads/refreshes through the v2-native lifecycle, not v1's core.
* refactor(mcp): delete the unwired V1PerUserTokenStore adapter (step 1b piece 5)
Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.
* fix(mcp): green CI for authz_code dispatch (format + UTC expiry + v2-seam tests)
- ruff format per_user_oauth_store.py (clears the lint check)
- v2_token_store._iso_to_epoch: anchor a tz-naive expiry to UTC before
.timestamp(), matching v1's db.py _remaining_token_seconds (Greptile P1) so a
non-UTC host doesn't read the expiry as local time and skew refresh timing
- test_mcp_stale_session: repoint the 3 discovery tests off the removed v1
_get_user_oauth_extra_headers_from_db onto the v2 has_user_oauth_token seam;
the delegate test now asserts the existence check is never consulted (delegate
short-circuits to the resource_metadata 401 before any token lookup)
- test_mcp_server_manager: repoint test_deferred_mode_uses_v1_auth_value at M2M
(oauth2 client_credentials), which is still a deferred mode, since per-user
oauth2 (authorization_code) now routes to the v2 resolver
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): caller Authorization must not override the stored per-user OAuth token
A caller with a valid x-litellm-api-key could include their own
"Authorization: Bearer <chosen>" header and have the proxy execute tools against
that bearer instead of the user's stored OAuth credential. For a v2-migrated
authorization_code server the caller's Authorization was seeded into
extra_headers, and the graft's apply-if-absent then dropped the resolved
per-user token in its favor. v1 prevented this by overwriting a stale client
Authorization with the stored token; this restores that precedence on both
egress paths (connect + call_tool).
- _should_strip_caller_authorization: also strip for migrated per-user OAuth
(authorization_code) servers - the v2 resolver injects the stored token, so a
caller-forwarded Authorization must not be forwarded upstream. Delegate /
pass-through (to_server_spec is None) keep forwarding the caller's bearer.
- both seed sites (_prepare_mcp_server_headers, _call_regular_mcp_tool) drop only
the Authorization from the caller's oauth2_headers (via _without_authorization),
keeping any other forwarded header and any hook/static Authorization (which
still wins, as in v1).
- regression test for the call_tool path; updated the two tests that asserted the
old (vulnerable) forwarding to assert the secure behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): preserve recorded OAuth scopes across authorization_code refresh
When a refresh response omits `scope` (RFC 6749 §5.1, where omission means unchanged), the v2 refresher persisted scopes=None and overwrote the user's recorded grant. v1 carried the prior scopes forward via `or cred.get("scopes")`; the v2 path lost that because OAuthToken did not model scopes
OAuthToken now carries scopes, V2PerUserTokenStore populates them on read, and AuthorizationCodeRefresher carries them forward for both the persisted write and the returned/cached token, so repeated refreshes do not erode them. A present `scope` in the response still replaces the prior grant
Adds regression tests: a refresh omitting `scope` preserves the prior scopes, and a present `scope` overrides them
* fix(mcp): keep user token in authorization_code tools preview
After to_server_spec maps oauth2 onto the v2 resolver, the interactive tools preview for an unsaved authorization_code server read the per-user token store, found nothing, and fail-closed with a 401, so the create/test tab could no longer list tools
The preview now routes the just-authorized token (forwarded in oauth2_headers) through mcp_auth_header, so _create_mcp_client takes the per-request-override v1 path and uses it directly, matching v1's preview. Gated to the v2-mapped oauth2 case; M2M, delegate/passthrough, and token-exchange keep their existing preview path
Adds tests: interactive oauth routes the forwarded token to mcp_auth_header, M2M and token-exchange do not
* fix(mcp): stop caller-supplied auth from overriding stored authorization_code tokens
A caller-supplied per-request override (mcp_auth_header / x-mcp-auth / x-mcp-<alias>-authorization) disabled the v2 resolver in _create_mcp_client for any spec, so an authenticated user with a stored authorization_code token could force an arbitrary upstream bearer and bypass the stored credential and its save-time validation. _create_mcp_client now keeps the v2 spec for authorization_code and ignores the override; other modes keep the client-side-credentials override
The create/test tools preview no longer relies on that override path. It resolves the just-authorized, not-yet-persisted token through the v2 resolver via a one-shot PresentedOAuthTokenStore passed as cred_provider - the same path runtime uses for the stored token - so preview and runtime resolve identically. This replaces the mcp_auth_header routing added earlier
Adds tests: a caller override cannot bypass the v2 resolver for authorization_code; the interactive preview resolves via the presented store rather than a caller header; M2M and token-exchange build no presented provider
* feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31474)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5)
The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.
* feat(mcp): DualCache-backed token cache backend (step 1b §1.5)
The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.
* feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5)
The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.
* feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5)
The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.
* feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5)
Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh
coordinator when Redis is wired, falling back to the foundation's in-process defaults on a
single replica. Layers the cross-replica path on top of the single-replica dispatch store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): refresh on lock-backend error instead of serving a stale token
The cross-replica refresh coordinator elected refreshers with a boolean acquire:
a Redis transport error was caught and returned as False, which is
indistinguishable from "another worker holds the lock". On a total Redis
outage every worker therefore took the wait-then-reread branch and served the
still-expired token upstream (the upstream then 401s), even though the lock and
coordinator docstrings claimed a Redis blip "degrades to an extra refresh".
Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the
coordinator can tell a busy holder from a dead backend, and refresh anyway on
ERROR. This single-flight lock is a load optimization, not a correctness mutex,
so failing open is correct: it degrades a lock-backend outage to the
no-coordinator behavior (an extra refresh), never a stale bearer.
Add a regression test asserting an acquire error refreshes rather than
re-reading the expired token, and update the docstrings to match.
* style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format
* fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed
The cross-replica coordinator's losers re-read the token the winner persisted.
If the winner's refresh failed, the store still holds the expired token, so the
loser re-read it and RefreshingTokenStore handed that expired bearer to the
caller (the upstream then 401s) instead of the re-auth challenge the winner
returned via None.
Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a
re-read that is still expired surfaces None so the arm challenges. This only
affects the loser path; the winner's freshly refreshed token is returned
directly by the coordinator and is unaffected.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "feat(mcp): cross-replica single-flight refresh for the v2 per-user OA…" (#31492)
This reverts commit
|
||
|
yucheng-berri
|
0216c969b8
|
fix(otel): point AgentOps OTLP exporter at otlp.agentops.ai (#31490)
The AgentOps preset hardcoded https://otlp.agentops.cloud/v1/traces, a domain that no longer resolves (NXDOMAIN), so every span silently failed to export with a NameResolutionError in the BatchSpanProcessor worker. The live ingest host is otlp.agentops.ai (the auth host api.agentops.ai was already correct). Pin the endpoint to the resolvable host and add a regression test on the constant. |
||
|
Mateo Wang
|
ef66620223
|
fix(cost): preserve Anthropic server_tool_use web search usage in cost tracking (#31355)
* fix(cost): preserve Anthropic server_tool_use web search usage in cost tracking Anthropic /v1/messages responses report built-in web search usage under usage.server_tool_use.web_search_requests, but the sync cost path reconstructs an OpenAI-shape Usage that drops server_tool_use and validates the response through AnthropicResponse, which previously stripped the field. Either path could leave the web-search fee uncounted. AnthropicResponseUsageBlock now allows extra fields so model_validate/model_dump keeps server_tool_use, and the built-in tool cost tracker reads the web search count straight off the raw Anthropic response dict when the reconstructed Usage lacks it, synthesizing a ServerToolUse without mutating the caller's Usage. * fix(lint): use PEP 604 unions in anthropic web search probes to satisfy strict-rule budget * refactor(cost): move Anthropic web search response parsing into llms/anthropic Relocate the raw /v1/messages web-search-count probe out of the shared built-in tool cost tracker into litellm/llms/anthropic/cost_calculation.py, next to get_cost_for_anthropic_web_search, so provider-specific response parsing lives under llms/. The core cost tracker now delegates to get_anthropic_web_search_requests_from_response and keeps only the generic Usage/ServerToolUse orchestration. * fix(cost): price Anthropic web search when only the raw response carries the count response_object_includes_web_search_call enters the web search branch as soon as the raw Anthropic dict reports usage.server_tool_use.web_search_requests, but _usage_with_anthropic_web_search bailed when the caller did not also pass a Usage object. _handle_web_search_cost then skipped the per-request anthropic path and fell back to the flat search_context_size_medium tier, charging a fixed fee instead of per_query x count (or zero when the count is zero). Synthesize a Usage from the raw dict when no Usage is supplied so count-based pricing runs uniformly regardless of how the response reaches the tracker. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
yuneng-jiang
|
e4aedb0342
|
Merge pull request #31391 from BerriAI/litellm_multipart_file_upload
fix(passthrough): forward all multipart files with repeated field names |
||
|
Shivam Rawat
|
de82f78e5b
|
fix(websearch): sync tool_choice when converting web_search tools (#31375)
failing test is not related to the pr * fix(websearch): sync tool_choice when converting web_search tools Claude Code forces native web search via tool_choice pointing at web_search while websearch_interception renames the tool to litellm_web_search, causing Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced tool_choice to match the converted tool name. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(websearch): re-wrap agentic loop responses as SSE for streaming clients When websearch interception converts stream=true to false for the agentic loop, dict responses from the loop were returned as application/json even though the client requested SSE. Wrap those responses in FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers (e.g. Claude Code) receive text/event-stream after search completes. Fixes #27721 Co-authored-by: Cursor <cursoragent@cursor.com> * test(websearch): cover tool_choice sync and post-loop SSE wrap; fix UP006 Add regression tests for both websearch interception fixes: _sync_forced_tool_choice repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and _maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for streaming clients (#27721). Switch the new helper annotations to builtin dict/list so the ruff UP006 strict-rule ceiling stays within budget. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(websearch): resolve merge conflict and unify fake stream wrapping Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MBP.localdomain> |
||
|
Krrish Dholakia
|
99b1a323c1
|
feat(guardrails): add headroom guardrail for message compression (#31407)
* feat(guardrails): add headroom guardrail for message compression Adds a headroom guardrail that compresses request messages via POST /v1/compress before they reach the LLM. The guardrail implements apply_guardrail so it runs on the unified guardrail path; it receives pre-built structured_messages (OpenAI format) from the translation layer, calls the headroom compression service, and returns the compressed messages as structured_messages. Set x-headroom-bypass: true on the request to skip compression. Also adds structured_messages write-back support to the OpenAI and Anthropic translation handlers: when apply_guardrail returns structured_messages, those are written to data["messages"] directly (OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic) instead of falling through to the existing text-patch path. This is a prerequisite for any guardrail that needs to replace the full message list rather than patch individual text spans. * fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs * style: fix ruff format violations * fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037) * fix(guardrails): only write back structured_messages when guardrail actually changed them * fix(guardrails/headroom): raise 502 when compression returns empty message list * fix(guardrails/headroom): catch transport errors and fix stale debug log * fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation * fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back * debug(headroom): add INFO logging to trace guardrail execution * debug(headroom): use print() for immediate visibility * debug(headroom): print request_data keys to diagnose metadata dict mismatch * fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log * fix: use model_call_details litellm_params metadata on Logging object * fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy * fix: read slg_info from litellm_metadata when metadata key absent * fix: write slg_info to both litellm_params attr and model_call_details copy * chore: remove debug prints; fix now verified end-to-end * refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py - Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from both async and sync wrappers in @log_guardrail_information, fixing guardrail_information=null in spend logs for all passthrough routes (/v1/messages, /v1/responses, etc.) in one place - Remove the 35-line inline sync block from the anthropic translation handler - Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses - Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase - Add regression tests for _sync_guardrail_info_to_logging_obj * fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold * fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity * fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity * fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity |
||
|
Mateo Wang
|
b9765458ac
|
fix(websearch): wrap agentic loop response in fake stream for streaming requests (#31484)
* fix(websearch): wrap agentic loop response in fake stream for streaming requests When websearch_interception converts stream=True to stream=False internally, the agentic loop returns a plain dict. Previously this dict was returned directly to the client expecting SSE events, resulting in empty streams. Added _maybe_wrap_in_fake_stream() which checks the websearch_interception_converted_stream flag and wraps dict responses in FakeAnthropicMessagesStreamIterator. Applied to all return paths in _call_agentic_completion_hooks: - async_run_agentic_loop (legacy path) - _execute_anthropic_agentic_plan (plan-based path) - plan.response_override - plan.terminate Includes unit tests for _maybe_wrap_in_fake_stream(). * test(websearch): cover agentic-loop wrap paths; gate fake-stream on anthropic_messages surface Guard _maybe_wrap_in_fake_stream on api_surface == anthropic_messages so the responses API surface is never wrapped in an Anthropic SSE iterator, and type logging_obj as Optional to match the None call sites. Adds regression tests that drive the legacy, response_override, and terminate return paths of _call_agentic_completion_hooks end to end. * test(websearch): cover _execute_anthropic_agentic_plan and tail wrap paths Drives the remaining two fake-stream return paths of _call_agentic_completion_hooks (the _execute_anthropic_agentic_plan branch via a stubbed handler, and the tail path when no agentic loop runs) so every converted-stream return path is regression-tested. --------- Co-authored-by: Clawd <fffff.c@gmail.com> |
||
|
Mateo Wang
|
4157f3b580
|
fix(passthrough): schedule spend logging via durable logging worker (#31485)
Pass-through success logging was scheduled with a bare asyncio.create_task whose return value was discarded, for non-streaming HTTP, streaming, and the vertex live websocket paths. The event loop keeps only a weak reference to such tasks, so under GC or load the task can be collected before it finishes writing the SpendLogs row; a request then returns 2xx to the caller yet never produces a costed spend log. This is the most likely cause of the flaky vertex passthrough e2e test and a rare real source of unbilled pass-through spend. Route these coroutines through GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue instead, matching how the SDK completion path already enqueues async logging. The worker holds a strong reference in its _running_tasks set and drains on shutdown via flush/stop/clear_queue and the atexit handler, so the write can no longer be dropped mid-flight. |
||
|
tin-berri
|
2e69708ef8
|
feat(mcp): shared OAuth token foundation - challenge, store seam, expiry-aware cache, single-flight refresh (#31275)
* feat(mcp): let CredError.of_unauthorized carry a 401 challenge The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate header + optional structured body) instead of a bare string, and raise_public emits the header and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's provisioning prompt) through the generic resolver edge. of_unauthorized's new params are keyword-only and default to None, so existing callers and the summary string are unchanged. * fix(mcp): make Unauthorized a frozen dataclass to keep the type budget flat CredError's unauthorized payload was a pydantic BaseModel, whose base resolves as unknown in this repo's basedpyright (every model in the file trips reportUntypedBaseClass plus an unknown model_config), so the tagged-union case read as unknown and the public edge's challenge access added reportUnknownMemberType errors over the per-rule ceiling. A frozen dataclass is fully typed here, so error.unauthorized resolves directly with no cast or accessor and the per-rule basedpyright counts match base. * feat(mcp): OAuth token store seam + expiry-aware cache for authorization_code Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token, expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and distributed single-flight are deferred to the hardening step. * feat(mcp): proactive token refresh with self-cleaning single-flight Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it) and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while concurrent callers await the same in-flight task and share its result, so the IdP is not stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore. OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica single-flight (Redis) and reactive-401 refresh are the later distributed hardening. * style(mcp): modern type annotations (dict/tuple/X | None) + sorted imports in the token modules * refactor(mcp): FIFO cache eviction, fix stale single-flight comment + refresh_token docstring * refactor(mcp): cache positive tokens only, matching v1 (no negative caching) CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow is visible on the next request without an invalidation hook, and uniformly across replicas since the in-process cache holds no stale None to clear. invalidate() now only covers rotation or revocation of a cached token. Negative caching (with distributed invalidation) can return later if a slow DB-backed v2-native source makes per-miss reads expensive. * fix(mcp): default OAuth expiry skew to 60s, the industry standard The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is an outlier among OAuth clients. Spring Security uses 60s as both its JWT clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's "a few minutes" leeway while preserving nearly all of a typical token's life; 30s was untested, so pin the default with two boundary-probe regression tests. * refactor(mcp): thread user_id/server_id through the TokenRefresher seam The refresh seam took only the OAuthToken, but a refresher needs the server's config (token endpoint, client credentials, scopes) to run the grant and the (user_id, server_id) key to persist the minted token, neither of which is derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id, token) and pass them through from RefreshingTokenStore so each stacked mode PR plugs into the final seam rather than forcing a later signature change across the stack. * feat(mcp): inject cache-backend and refresh-coordinator seams (cross-replica token caching) Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be). The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a reread callback for the cross-replica case (losers re-read the persisted token) that the in-process default ignores. * fix: reread oauth token before refresh --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
7acc0157df
|
fix(mcp): stop logging tool-call input in MCP client (#31393)
The MCP client logged the full tool arguments (and prompt arguments) at INFO on every call, so caller input such as user queries, model names, and instructions landed in the proxy application logs and any downstream log aggregator Log only the tool or prompt name and drop the arguments from these INFO lines |
||
|
yucheng-berri
|
ec4e0146c7
|
feat(prometheus): add requested_model label to spend and requests metrics (#31410)
litellm_spend_metric_total and litellm_requests_metric_total previously exposed only the resolved backend model_id and friendly model name, so operators could not group spend or request counts by the model alias the caller actually asked for when a router fronts multiple deployments behind one name. This adds the existing UserAPIKeyLabelNames.REQUESTED_MODEL to both labelname lists; the value is already populated upstream from standard_logging_payload["model_group"] and flows through the shared _increment_top_level_request_and_spend_metrics call site. The sibling token metrics (input/output/total) already carry the label, so this also restores cross-metric consistency. Resolves LIT-3796 |
||
|
Yassin Kortam
|
c14329128b
|
fix(guardrails): match policy-pipeline block response to direct guardrail attachment (#31421)
When a guardrail blocked a request through a flow-builder policy pipeline, the proxy discarded the guardrail's own exception and synthesized a generic guardrail_pipeline_error response, so the same guardrail produced a different HTTP response and trace span depending on whether it was attached directly or via a policy. The pipeline now carries the guardrail's original exception and re-raises it verbatim on block, enriching it with the blocking guardrail's name and mode exactly as the direct path does, so the two attachment methods are indistinguishable to clients and tracing. The generic pipeline error remains only as a fallback for blocks with no underlying exception (e.g. a guardrail that could not be found). Resolves LIT-4041 |
||
|
Yassin Kortam
|
ce658367a4
|
fix(auth): cache auth-path team object under canonical team_id key (#31418)
The auth builder cached the team object under the raw `valid_token.team_id`,
while `get_team_object`, `_cache_team_object`, and `_update_team_cache` all read
and write under `team_id:{id}`. The raw-key write was therefore never served
back, and on a non-team (personal) key, whose team_id is None, the original
unguarded version passed a None key straight to the cache layer; the in-memory
cache tolerates None keys but Redis rejects them with a NoneType key error, so
with `enable_redis_auth_cache: true` the team object never reached the L2 cache
and every request fell back to Postgres.
Write under the canonical `team_id:{id}` key, keeping the existing guard that
skips the write when team_id is None. Add a regression test that drives the real
auth builder for a team-scoped key against an in-memory cache and asserts the
team object is served back under `team_id:{id}` and never under the raw team_id
or a None key.
Resolves LIT-4000
|
||
|
Yassin Kortam
|
f2fa23b0ec
|
fix(guardrails): instrument during-call and post-call guardrail latency (#31414)
litellm_guardrail_latency_seconds was only emitted for pre-call guardrails. during_call_hook and post_call_success_hook ran guardrails without recording any latency, so during-call and post-call guardrail time was invisible in the metric and leaked into litellm_overhead_latency_metric, making the documented "subtract guardrail latency from overhead" workaround under-report total guardrail time. Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics and add _run_guardrail_with_metrics, a single wrapper that times a guardrail coroutine, classifies its outcome (success / intervened / error), enriches any raised HTTPException, and records the latency under the given hook_type. Route the pre-call emit, during_call_hook, and post_call_success_hook through it so every guardrail phase contributes to the metric the same way. Resolves LIT-3999 |
||
|
Yassin Kortam
|
bd5e046464
|
fix(bedrock): surface web identity token aud/iss on InvalidIdentityToken (#31412)
When STS rejects a web identity token with InvalidIdentityToken (the "Incorrect token audience" case), litellm propagated the raw botocore error, which never names the aud LiteLLM actually sent. Diagnosing an audience mismatch then required enabling LITELLM_LOG=DEBUG on the prod instance, which degrades performance. _auth_with_web_identity_token now catches InvalidIdentityTokenException, decodes the public aud/iss claims of the resolved JWT without verifying its signature (no secret is read), and raises an AwsAuthError that preserves the STS reason and names the token audience and issuer, so the mismatch is visible from the error alone. Resolves LIT-4026 |
||
|
Yassin Kortam
|
7209e139d6
|
fix(spend): fold logs-tab total into the page query to avoid a separate COUNT(*) (#31423)
The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) ran a standalone SELECT COUNT(*) before the page query to compute total_pages. On sharded engines like YugabyteDB a COUNT(*) is a distributed RPC that contacts every tablet leader and aggregates partial results regardless of row count, so it hits the distributed RPC timeout and the logs tab 500s even on a one-minute window with a couple of rows. The startTime range cannot prune tablets because rows hash to tablets on request_id, not startTime. Fold the count into the same scan as the page data with COUNT(*) OVER () and read total off the returned rows, dropping the helper column before serialisation. One distributed scan per page load instead of two; the response shape is unchanged. An empty page carries no count row, in which case the total is zero. Resolves LIT-4027 |
||
|
Yassin Kortam
|
f55d13ebba
|
fix(team): persist budget_duration on /team/member_add member budgets (#31443)
/team/member_add could not set budget_duration on an individual member budget. add_new_member created the budget row with only max_budget and allowed_models, and TeamMemberAddRequest had no budget_duration field, so a member added with an explicit per-member budget while the team ran a recurring member budget got a lifetime cap instead of a recurring allowance. Thread budget_duration from TeamMemberAddRequest through _process_team_members into add_new_member, and pull the member-budget resolution into a helper that writes budget_duration plus a computed budget_reset_at. When only a budget_duration is supplied and the team has a default member budget, the default is cloned and its reset window overridden so the member keeps the default's max_budget rather than becoming uncapped; a duration with no team default creates a window-only budget. Invalid durations are rejected with a 400 before any DB write, symmetric with /team/member_update. The available-team self-join bypass only grants the ability to join, so reject per-member budget and model controls (max_budget_in_team, budget_duration, allowed_models) for non-admin self-join callers in _validate_team_member_add_permissions, before any DB write. Otherwise a self-joining non-admin could set their own cap, reset window, or model scope past the team default; admins, team admins, and org admins are unaffected and a clean self-join still inherits the team default budget. Resolves LIT-4052 |
||
|
yuneng-jiang
|
63cf835b14
|
Merge pull request #31420 from BerriAI/litellm_/lucid-wilson-408605
test(pass-through): fix langfuse auth=true test broken by allowed_passthrough_routes gate |