Commit graph

10721 commits

Author SHA1 Message Date
Devin AI
11cd1d9203 fix(sap): forward extra_headers on Orchestration provider requests 2026-07-15 10:10:12 +00:00
yucheng-berri
d6f498ff5c
test(e2e): failed request error span carries the full untruncated message and status (LIT-4179) (#33304)
* test(e2e): failed request error span carries the full untruncated message and status

Covers logging.otel.failure.exports_metric on chat_completions: a request that
fails at the provider (invalid upstream key deployment) must export one
complete trace whose gen-AI span carries the LIT-4179 error contract, declared
as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated
error.message proven by parsing the embedded provider error JSON back out of
the attribute. The root SERVER span must record the 401 the client received.
Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which
the suite's model-registering tests already assume

* test(e2e): clean failure diagnostics on the error-span contract per review

A truncated error.message with missing braces now fails with a readable
assertion instead of an unhandled ValueError, an unparseable embedded JSON
fails via pytest.fail with the truncation context, and the retry loop now
asserts the upstream provider failure was actually observed so a fresh-key
propagation deadline cannot masquerade as a trace-export failure

* test(e2e): pin the full error attribute set including the litellm.provider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message

* test(e2e): author the error-span test docstring
2026-07-14 22:13:13 -07:00
Krrish Dholakia
b96460608d
feat(router): resolve auto-router routing plugins from proxy YAML config (#33251)
* feat(router): resolve auto-router routing plugins from proxy YAML config

Router(plugins=[...]) was Python-SDK constructor only, so proxy/YAML users
had no way to configure it, and the merged pipeline narrowed candidates
from the outer model alias rather than the auto-router's actual tier pool,
making it a no-op for auto_router deployments.

Add complexity_router_config.plugins (dotted-path strings resolved via
get_instance_fn, the same convention litellm_settings.callbacks uses) and
run the resolved plugins against ComplexityRouter's tier pool at every
model-pick site, so a policy plugin narrows what get_model_for_tier
actually returns instead of the outer alias list. adaptive=True with
plugins set now raises at config validation instead of silently ignoring
the plugins, since the bandit selector doesn't consume narrowed pools yet.

Also fixes a latent bug in Router._generate_model_id: it json.dumps every
litellm_params dict value to build a deployment hash id, which crashed
once a live plugin object could land inside complexity_router_config.

* fix(router): use stable class name, not object repr, in model-id json fallback

json.dumps(v, default=str) on a litellm_params dict containing a live
RoutingPlugin instance fell back to object.__repr__'s default
<module.Class object at 0x...>, embedding the instance's memory address.
_generate_model_id's hash (and therefore the deployment id) changed on
every process restart/hot-reload for any deployment with
complexity_router_config.plugins configured, defeating the function's own
"consistently generate the same id" contract and orphaning anything keyed
on that id across restarts (e.g. Redis-backed per-deployment state).

Use the plugin's fully-qualified class name instead, which is stable
across restarts.

* test(router): cover _json_default_stable_id for router_code_coverage gate

router_code_coverage.py's AST scanner requires every router.py function be
called by name somewhere in tests/, and flagged the new
_json_default_stable_id helper from the previous commit.

* fix(router): close two routing-plugin policy-bypass gaps flagged by Veria AI

Session-affinity pin shortcut: async_pre_routing_hook returned a session's
first-turn pinned model on every later turn without ever re-running it
through the plugin pipeline, so a policy plugin (e.g. a budget cap crossed
mid-session) was only enforced on turn one. Now the pin shortcut is
disabled whenever plugins are configured, so every turn re-runs
_classify_and_route (and therefore the plugins).

Plugin resolution validation: get_instance_fn accepts any dotted path and
returns whatever object it finds there, so a misconfigured
complexity_router_config.plugins entry passed proxy startup silently and
only surfaced as a confusing AttributeError on the first request that
reached the plugin pipeline. Extracted the resolution logic into
resolve_complexity_router_plugins() and added an isinstance(...,
RoutingPlugin) check that fails proxy startup immediately with a clear
error instead.

* fix(router): raise instead of falling back to default_model on empty plugin-narrowed tier

default_model was never checked against the configured plugins, so it
functioned as an unconditional escape hatch around whatever policy a
plugin enforces -- a tenant/budget plugin narrowing a tier to zero
candidates could still be bypassed by the fallback. Drop the fallback
entirely for this path; a plugin narrowing to zero is a policy decision,
not something to route around, matching the fail-closed behavior the
Router-level plugin pipeline already uses for the same situation.

Flagged by Veria AI on PR #33251.

* style: ruff format complexity_router.py

* style(proxy): use modern str | None instead of Optional[str] in resolve_complexity_router_plugins

* fix(router): stop default_model short-circuit from skipping plugins on no-user-message path

self.config.default_model or await self._pick_model_for_tier(...) -- Python's
`or` short-circuits on a truthy default_model, so _pick_model_for_tier (and
therefore the plugin pipeline) never ran at all for the no-user-message path
whenever default_model was configured. A tenant/budget plugin's decision was
silently bypassable this way even after the other two policy-bypass fixes,
since this call site had a different shape from the other three pick sites.

Removed the short-circuit; falls through to _pick_model_for_tier ->
get_model_for_tier, which already checks the MEDIUM tier before default_model
-- the same priority every other call site uses.

Flagged by Veria AI on PR #33251.

* fix(router): address Greptile findings on the plugin-bypass fixes

Preserve default_model-first priority in the no-user-message path when no
plugins are configured, instead of unconditionally flipping to the MEDIUM
tier -- the plugin-bypass fix must not silently change model selection for
the (much larger) population of users who don't use plugins at all. Gated
on self.config.plugins, matching the pattern already used elsewhere in
this PR, per CLAUDE.md's guidance against backwards-compat flags when a
plain conditional does the job.

Also close a gap in the plugin validation added earlier:
@runtime_checkable only checks that `run` exists as an attribute, not that
it's a coroutine function, so a synchronous `def run(self, context)`
passed isinstance(resolved_plugin, RoutingPlugin) at startup and only
failed at request time with a confusing TypeError. Added an
inspect.iscoroutinefunction check.

Both flagged by Greptile on PR #33251.
2026-07-14 21:27:14 -07:00
yucheng-berri
817582e697
test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
tin-berri
9cca6c3ef1
Merge pull request #33286 from BerriAI/litellm_mcp_oauth_discovery_persist
fix(mcp): persist discovered OAuth endpoints and keep last known good on failed re-discovery
2026-07-14 19:43:15 -07:00
Mateo Wang
03ef18a9ea
Merge pull request #33315 from BerriAI/litellm_fix_empty_delta_thinking_block
fix(anthropic-adapter): drop empty content_block_delta events
2026-07-14 19:40:39 -07:00
Mateo Wang
f974d1d489
Merge pull request #33129 from BerriAI/litellm_websearch_responses_interception
fix(websearch): intercept web search on the Responses API
2026-07-14 19:40:08 -07:00
yucheng-berri
e3546c20af
feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode (#33299)
* feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode

Adopted from #30830 by OS-joaocastilho; the original PR was merged into
litellm_oss_staging_230626, which never landed, so this re-lands it on
litellm_internal_staging

Beyond the original diff, this fold includes the review fixups that were
made on the staging branch (warn on unrecognized check keys, keep empty
known checks as enable-with-defaults, fail fast when the checks block has
no usable keys, tz-aware datetimes, stricter typing) and adapts the block
path to the ModifyResponseException contract from LIT-4186, which replaced
GuardrailInterventionNormalStringError after the original PR was written

* fix(bedrock guardrails): only evaluate configured checks in violation collection

An unsolicited score in the InvokeGuardrailChecks response (e.g. a future
API revision returning checks the user never requested) previously fell
through to the default 0.5 threshold and could block a request the user
only asked to scan with other checks. Violation collection now skips any
check absent from the configured checks block

* fix(bedrock guardrails): fail closed on truncated PII results and tighten checks-path typing

Truncated sensitiveInformation results now count as a violation when the
PII check is configured: Bedrock omitted detections that were never
scored, so sub-threshold visible entries no longer let the request pass.
Also blocks on score == threshold per the documented contract (regression
test added), rejects checks combined with guardrailVersion, turns a
malformed 200 body into a logged guardrail_failed_to_respond 500 instead
of a raw ValidationError, types the checks parameter and violations
(BedrockChecksConfigModel, BedrockChecksViolation) instead of dict/object,
types _sign_and_post against AWSPreparedRequest, hoists stdlib imports,
and builds checks messages without intermediate mutation

* fix(bedrock guardrails): tag all InvokeGuardrailChecks INPUT content as user

Bedrock excludes system content from prompt-attack evaluation (per the
AWS guardrails docs), so mapping a caller-supplied system/developer
message onto the system role let a caller hide a prompt injection from
the promptAttack check by self-labeling its role. At the proxy every
INPUT message is caller-controlled, so all of it is now tagged as
untrusted user input, which also matches AWS guidance to tag untrusted
content as user input. OUTPUT stays assistant. Removes the now-unused
role map; the input-message test asserts the new tagging as a regression

* fix(bedrock guardrails): pass prepared request headers to httpx without dict coercion

httpx accepts botocore's HTTPHeaders mapping directly, and wrapping it in
dict() broke the existing test_bedrock_guardrail_make_api_request_passes_api_key
which supplies a bare Mock as the prepared request (dict(Mock) calls
Mock.keys())

---------

Co-authored-by: OS-joaocastilho <144790013+OS-joaocastilho@users.noreply.github.com>
2026-07-14 19:31:17 -07:00
devin-ai-integration[bot]
56f4dbf60a
test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548)
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: restore the upload-coverage job dropped by mistake with the compat gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments

* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-07-14 19:19:03 -07:00
mateo-berri
0f9d593d29 fix(anthropic-adapter): drop empty content_block_delta events
An empty upstream delta (e.g. Bedrock Converse's empty reasoning delta
mid-thinking-block) falls through the translate fallback as
text_delta {"text": ""} at the open thinking block's index, crashing
Anthropic SDK clients like Claude Code with "Content block is not a
text block". Payload-less deltas carry no information, so never emit
them.
2026-07-14 18:16:20 -07:00
Tin Chi Lo
a0d5df21da fix(mcp): do not persist authorization_url from the DCR path; the build hook is the provenance-guarded writer 2026-07-14 18:08:55 -07:00
Mateo Wang
65ca095d4d
Merge pull request #32963 from BerriAI/litellm_e2e_bedrock_mid_system_cache
test(e2e): cover model-aware mid-conversation system handling on Bedrock Invoke /v1/messages
2026-07-14 18:06:39 -07:00
Devin AI
75ccb4f416 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_websearch_responses_interception 2026-07-15 00:53:57 +00:00
Krrish Dholakia
7f598c6a9b fix(websearch): address Responses review findings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-15 00:52:16 +00:00
yucheng-berri
b2202cb1aa
feat(guardrails): streaming text transformation in generic_guardrail_api (#33110)
* feat(guardrails): support streaming text transformation in generic_guardrail_api

* chore(guardrails): address PR review feedback

* fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform

* fix(guardrails): address Bugbot review on streaming transform correctness

* fix(guardrails): coerce holdback in handler for in-process guardrails

* fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason)

* test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes

* fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases

* test: move ComplianceChecker mode tests to the compliance PR

* fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform

* fix(guardrails): four correctness fixes for incremental_diff streaming path

Four bug fixes on top of the OSS PR's incremental_diff streaming text
transformation, all inside the incremental_diff code paths only. No
existing block_only, non-streaming, or pre_call behavior is touched.

Fix #1 — Mixed content+tool_call finish_reason ordering
  _tool_call_passthrough_chunk now takes an optional finish_reason_per_choice
  map. For a choice carrying both delta.content and delta.tool_calls,
  finish_reason is stripped from the passthrough and recorded on the map so
  the final synthetic text chunk delivers it. Without this, SSE-compliant
  clients stopping at finish_reason drop the guardrailed text — defeating
  the redaction the whole feature exists for. (Greptile P1 twice, Veria.)

Fix #2 — Choice index sort in _process_streaming_transform
  indices/texts_to_check were derived from dict insertion order. For n>1
  streams where choice 1 emits before choice 0, guardrail-returned texts
  aligned to the input order mapped back to the wrong choice indices on
  write-back — wrong text goes to wrong choice. Sort raw_by_index.keys()
  up front so realignment is deterministic. (Bugbot Medium.)

Fix #3 — Cross-chunk pre-tool-call text flush
  With default streaming_sampling_rate=5, text chunks followed by a pure
  tool-call chunk carrying finish_reason='tool_calls' would emit the
  passthrough with finish_reason before any transformed text delta had
  fired. Same failure mode as fix #1 but cross-chunk. Now we flush any
  accumulated text via _round(is_final=False) BEFORE yielding the
  tool-call passthrough. (Greptile P1.)

Fix #4 — Terminator chunk for deferred finish_reason on empty mutated_text
  _build_transform_chunk returned None early when mutated_text_per_choice
  was empty. If a mixed content+tool_call chunk had deferred its
  finish_reason (via fix #1) and the guardrail then suppressed the text
  (empty return), the deferred finish_reason was never delivered. Now on
  is_final=True with empty mutated_text_per_choice, we emit a terminator
  carrying finish_reason per choice from finish_reason_per_choice.
  (Bugbot High.)

Also normalized Optional[X] → X | None across the OSS PR's added surface
via ruff UP045 autofix to keep the strict-rule gate within budget. Pure
mechanical typing style change, no semantic effect.

Regression tests for all four fixes:
- test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1)
- test_text_flush_precedes_tool_call_passthrough (#3)
- test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4)
- test_transform_sends_texts_sorted_by_choice_index (#2)

All fixes reachable only when streaming_transform_mode == 'incremental_diff'
is configured (via _run_incremental_transform_stream) or when a
StreamTransformSink is present (via _process_streaming_transform). Verified
scope-clean: no changes to block_only, non-streaming, pre_call, moderation,
or sibling guardrails.

---------

Co-authored-by: Marton Schneider <marton@schneider.co.nl>
2026-07-14 17:38:11 -07:00
yucheng-berri
32af83d63a
fix(s3): sanitize slashes in response-id-derived object key file name (#33271) 2026-07-14 17:26:00 -07:00
mateo-berri
a0c4e4684a test(e2e): scope virtual keys to the deployment under test 2026-07-14 17:12:15 -07:00
mubashir1osmani
f7849f9e91
fix(auth): scope the JWT enterprise gate to actual JWTs (#33296)
With enable_jwt_auth enabled but no enterprise license (premium_user
False), the JWT premium check fired on every request before the token
was inspected, so the master key, sk- virtual keys, and the encrypted
CLI/UI SSO session token that `lite login` issues all 401'd with "JWT
Auth is an enterprise only feature" and were never decoded. That broke
`lite login`, `lite claude`, and the proxy master key on any deployment
that turned JWT auth on without a license.

Move the premium check inside the is_jwt branch so it gates only real
JWTs. Non-JWT credentials fall through to their own auth paths
regardless of license; actual JWTs still require premium, so the
enterprise gate is unchanged for the feature it protects.
2026-07-14 17:05:05 -07:00
Mateo Wang
8c776605d8
Merge pull request #33274 from BerriAI/litellm_gemini_omni_flash_preview_pricing
feat(pricing): add gemini-omni-flash-preview with video output token pricing
2026-07-14 16:58:04 -07:00
yucheng-berri
f31dacbcd4
fix(proxy): never log raw virtual keys in key insertion debug output (#33268)
* fix(proxy): never log raw virtual keys in key insertion debug output

* fix(proxy): tolerate None token in insert_data debug log redaction
2026-07-14 16:32:26 -07:00
mateo-berri
04193649ee Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_e2e_bedrock_mid_system_cache 2026-07-14 15:33:19 -07:00
Tin Chi Lo
25ada3ae11 fix(mcp): persist discovered OAuth endpoints and keep last known good on failed re-discovery 2026-07-14 15:32:52 -07:00
mateo-berri
0f28b1114e refactor(e2e): share anthropic cache-control shapes in endpoints_client 2026-07-14 15:11:34 -07:00
mateo-berri
6874271db4 docs(e2e): add cache_hit to the naming grammar assertion vocabulary 2026-07-14 15:00:31 -07:00
Krrish Dholakia
477ef3a7e2
fix(anthropic): use native output capability (#33235)
Some checks are pending
CodSpeed Benchmarks / benchmarks (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(anthropic): route native structured output

Use model capability metadata so new native structured-output models do not require transformation allowlist changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): pass provider to capability

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(anthropic): cover dotted model IDs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): handle remote capability lag

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-14 14:23:49 -07:00
Mateo Wang
6ad2f85e0c
Merge pull request #32914 from BerriAI/litellm_e2e_key_rate_limit_coverage
test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers
2026-07-14 14:12:28 -07:00
mateo-berri
ce3bf2d839 fix(gemini): map video response modality instead of MODALITY_UNSPECIFIED 2026-07-14 14:11:52 -07:00
mateo-berri
598fa9d64d feat(pricing): add gemini-omni-flash-preview with video output token pricing 2026-07-14 14:04:51 -07:00
mateo-berri
a21669aaef refactor: make the code easier to read 2026-07-14 13:58:23 -07:00
mubashir1osmani
edd3bce0ec
fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265)
The rate-limited batch spend test snapshotted unattributed rows via the
unpaginated /spend/logs whole-table read, which grows with the environment
(58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every
scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an
explicit date window instead, and SpendLogsParams now rejects a filterless
read so the whole-table call cannot come back
2026-07-14 13:32:10 -07:00
devin-ai-integration[bot]
ffe0c4c185
fix(proxy)!: enforce user budget on team keys (read-time + reservation) with UI opt-out (#32005)
* fix: enforce user budget on team keys

User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.

Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: update test to expect user budget enforcement on team keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI

Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: assert budget_exceeded ProxyException in personal budget test

Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style

* fix: revert to bool | None to stay under UP045 strict budget

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
2026-07-14 13:21:32 -07:00
yucheng-berri
939117bb8d
fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook (#33136)
* fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook

* fix(guardrails): keep request-body dispatch predicate unchanged

* fix(guardrails): fail closed when proxy extras are missing at deployment hook
2026-07-14 12:38:27 -07:00
devin-ai-integration[bot]
71dffc1e9a
fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244)
* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models

* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models

Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-14 12:31:28 -07:00
Tin Chi Lo
a30c25a121 refactor(mcp): extract the dcr_bridge token flow into bridge_token_flow.py
discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it

Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply
2026-07-13 23:12:54 -07:00
yucheng-berri
6a213de9f4
test(e2e): otel trace completeness on /v1/messages (#33133)
* test(e2e): OTEL trace completeness on /v1/messages

Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.

* test(e2e): reuse the shared AnthropicMessagesBody per review

Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature

* test(e2e): author the messages trace test docstring

* test(e2e): declare the messages surface on the covers marker

* test(e2e): otel trace completeness on /v1/responses (#33134)

* test(e2e): OTEL trace completeness on /v1/responses

Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.

* test(e2e): author the responses trace test docstring

* test(e2e): declare the responses and chat surfaces on the covers markers
2026-07-13 20:19:06 -07:00
Mateo Wang
ab87ebe26d
Merge branch 'litellm_internal_staging' into litellm_e2e_key_rate_limit_coverage 2026-07-13 19:44:57 -07:00
ryan-crabbe-berri
bf501c38a5
fix(sso): paginate through all pages when fetching service principal group assignments (#33149)
get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.

Ported from #32792 by @saisurya237 so CI can run.

Fixes #32790

Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
2026-07-13 19:21:53 -07:00
yucheng-berri
948a43cd64
test(e2e): otel trace completeness on /chat/completions (#33132)
* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination

Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage:
a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix
preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export
through a preset-owned provider - the code path where trace splits happen), a
typed Jaeger query read-back client, and the first test: one successful
non-streaming /chat/completions call exports ONE complete trace (root SERVER
span + auth/db/cost children + gen-AI CLIENT span, no dangling parents).

* test(e2e): harden the otel trace read-back per review

Jaeger reads now query server-side by the litellm.call_id span tag instead of
paging recent traces and filtering client-side; the compose stack's background
jobs alone can push a request trace past the page. A failed query hard-fails
instead of reading as an empty result, the settle predicate now also waits for
the prefix-matched db span the assertion demands, parent-chain walking follows
CHILD_OF references only, the zero-trace and split-trace failures get distinct
messages, jaeger gets a healthcheck so the depends_on condition is accurate,
and the chat docstring names the route the code actually asserts

* test(e2e): author the chat trace test docstring

* Update logging section in CLAUDE.md

Removed mention of OTEL trace-tree completeness from logging integration section.
2026-07-13 19:06:12 -07:00
yucheng-berri
07ea4b3e14
feat(prometheus): expose video duration and image count consumption metrics (#33138) 2026-07-13 18:51:13 -07:00
Krrish Dholakia
397c84678b
feat(router): opt-in session affinity for complexity router (#33126)
* feat(router): opt-in session affinity for complexity router

Complexity router reclassified every turn, which could flip the routed
model group mid-session and break provider-side prompt caching. Add a
session_affinity config flag: when a session_id is resolvable, pin the
model chosen on the first turn and reuse it for the rest of the session,
skipping reclassification. Pinned turns still stamp the adaptive
bandit's chosen-model metadata so reward feedback keeps working when
adaptive=True.

* fix(router): refresh session-affinity TTL on hit, scope pin by API key

Two issues from review: the TTL was only set on the first classification,
so an active session outliving session_affinity_ttl_seconds silently lost
its pin instead of refreshing as documented. And the cache key was scoped
only by session_id, which is client-supplied and unauthenticated, so two
different callers reusing the same session_id could poison each other's
routing pin. Refresh the TTL on every cache hit, and namespace the cache
key by the proxy-derived API key hash when available.
2026-07-13 18:28:13 -07:00
devin-ai-integration[bot]
001457af8b
fix(keys): persist key_type so the UI shows correct key scope instead of "All Proxy Models" (#33115)
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'

key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.

Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(ui): move key_scope helper to components root

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(keys): persist key_type on virtual keys so the UI reads scope directly

Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken

The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-13 18:08:43 -07:00
Tin Chi Lo
b70de76df3 Merge origin/litellm_internal_staging (#32980 bridge refresh envelope) into litellm_mcp_oauth_error_relay
The refresh-envelope helpers merge cleanly alongside the faults package. The upstream invalid_grant
special case for bridge refreshes moves inside the post-call except branch (its old location after a
second raise_for_status would be unreachable under the call-time-raise structure this branch
introduced) and now keys off the classified fault; _upstream_oauth_error is dropped since the
classifier already parses the RFC 6749 error field with total accessors
2026-07-13 17:31:17 -07:00
tin-berri
53aaabba5e
Merge pull request #32980 from BerriAI/litellm_bridge_refresh_envelope
feat(mcp): client-held refresh envelope for the dcr_bridge oauth_delegate flow
2026-07-13 17:18:07 -07:00
Tin Chi Lo
9335edeb85 fix(mcp): keep upstream self-blame codes and gateway capability gaps off the caller
Extends the fault matrix per review: server_error and temporarily_unavailable are codes by which the
upstream blames itself, so they classify as a new UpstreamReportedFault arm rendering 502/503 with a
matching wire code instead of a 400 that blames the caller; invalid_target is a gateway capability
gap (RFC 8707 resource indicators, LIT-4339) and is gateway-blamed regardless of whose credentials
were presented; the DCR classifier shares the same blame assignment. The gateway-fault arm is renamed
GatewayRejected since it now covers capability gaps as well as stored-credential rejections
2026-07-13 16:37:49 -07:00
Tin Chi Lo
9dbebf27a6 fix(mcp): detect upstream invalid_grant by the RFC 6749 error field, not a body substring
The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run

Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match
2026-07-13 16:22:16 -07:00
Tin Chi Lo
9e94f2be14 refactor(mcp): classify upstream OAuth faults once and derive status, code, and prose from the value
Replaces the accreted relay helpers with a faults package (types, classify, render_oauth): every
upstream token/DCR rejection is classified into exactly one fault value and the response status,
wire error code, and prose are all derived from that value, so a caller-fault code can never ship
on a server-fault status (the bugbot finding on invalid_grant over a 500). Classification takes the
credential source into account: invalid_client and friends against the server's stored credentials
are the operator's fault and render as 502 server_error with gateway-authored prose while the IdP's
prose stays in server logs; the same codes against caller-supplied credentials relay on the status
the code implies. Classifiers are total, so an unreadable rejection body (lying content-encoding,
unconsumed stream) yields the same 502 fault instead of resurrecting the opaque 500 (the second
bugbot finding); DCR rejections normalize to 400 per RFC 7591 regardless of the upstream's status
2026-07-13 16:11:12 -07:00
Tin Chi Lo
a9fac3c483 fix(mcp): carry the requested scope forward when the upstream omits it on a bridge refresh
The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token

When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked
2026-07-13 16:04:23 -07:00
Krrish Dholakia
aa7b480f4c fix(websearch): add Responses API surface to websearch interception
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-13 22:57:15 +00:00
Tin Chi Lo
2eb37d7948 fix(mcp): re-request the sealed scope on a bridge refresh when the client omits it
The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope

Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback
2026-07-13 15:28:53 -07:00
Tin Chi Lo
0d9c3cf97b fix(mcp): keep out-of-contract upstream error bodies out of client responses
The token and DCR relays serve unauthenticated OAuth clients, so only the RFC 6749/7591 error fields may cross the trust boundary. A rejection body outside those contracts (HTML error page, proxy banner, stack trace) is now logged server-side, bounded, and the client response names only the upstream status. Addresses the Veria information-exposure finding
2026-07-13 15:13:39 -07:00