Commit graph

41215 commits

Author SHA1 Message Date
Tin Chi Lo
bcc5fadbb1 test(e2e): fix EndpointsClient gateway attribute left behind by the ProxyClient rename 2026-07-21 00:05:59 -07:00
yuneng-jiang
c1b6c4062e
Merge pull request #34070 from BerriAI/litellm_/remaining-table-complexity-053477
refactor(ui): migrate available teams table onto shared DataTable
2026-07-20 21:52:34 -07:00
Mateo Wang
7e66f00fca
Merge pull request #34068 from BerriAI/litellm_codex_auto_drop_params 2026-07-20 23:45:29 -04:00
Mateo Wang
e3f7019fee
Merge pull request #34058 from BerriAI/litellm_mantle_codex_additional_tools
fix(bedrock_mantle): gate unsupported service_tier on drop_params for the Responses API
2026-07-20 23:41:53 -04:00
Yuneng Jiang
484cc12ab6
fix(ui): ignore stale available-teams fetch on unmount or token change 2026-07-20 20:35:51 -07:00
mateo-berri
fadec17a0f Merge origin/litellm_internal_staging into litellm_mantle_codex_additional_tools (resolve overlap with #33228 hoist) 2026-07-20 20:30:34 -07:00
Mateo Wang
07a355e867
Merge pull request #33228 from lyb0307/litellm_bedrock_mantle_codex_additional_tools
fix(bedrock_mantle): hoist Codex additional_tools input items to top-level tools
2026-07-20 23:14:47 -04:00
Yuneng Jiang
fe57eedb5c
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/remaining-table-complexity-053477 2026-07-20 20:08:50 -07:00
Yuneng Jiang
99f215df68
refactor(ui): migrate available teams table onto shared DataTable 2026-07-20 20:08:42 -07:00
tin-berri
c9b50120ae
Merge pull request #34059 from BerriAI/litellm_lit4629_google_registry
fix(mcp): add Google Sheets, Drive, Calendar, and Docs to the OpenAPI registry
2026-07-20 20:08:21 -07:00
Mateo Wang
464537a12d
Merge pull request #34047 from BerriAI/litellm_shared_key_capability_flags
fix(router): propagate capability flags to shared backend cost map key
2026-07-20 23:05:58 -04:00
mateo-berri
d2b573c37d feat(proxy): auto-enable drop_params for Codex user agents 2026-07-20 19:58:47 -07:00
mateo-berri
354f3971a9 fix(bedrock_mantle): gate unsupported service_tier on drop_params for the Responses API 2026-07-20 19:49:42 -07:00
mateo-berri
8745f355a4 fix(bedrock_mantle): log additional_tools hoist at debug level 2026-07-20 19:33:31 -07:00
yucheng-berri
9ad8698aab
feat: add deepkeep as custom guardrail (#33844)
* adding deepkeep as custom guardrail

* adding deepkeep as a custom guardrail

* adding deepkeep as a custom guardrail (hooks)

* adding litellm/proxy/_experimental/out/ to .gitignore

* adding deepkeep as custom guardrail in litellm

* removing sentinel_fortress

* comparing schema.prisma files

* fix(deepkeep): address greptile review comments

- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
  merge them into _build_request_headers() so user-configured headers
  reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
  explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
  return value so downstream callers don't lose that content

Adds tests for all four fixes.

* fix(deepkeep): address greptile review comments

- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
  merge them into _build_request_headers() so user-configured headers
  reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
  explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
  return value so downstream callers don't lose that content

Adds tests for all four fixes.

* fix: add missing __init__.py and allowlist entries for upstream merge

- tests/test_litellm/proxy/client/__init__.py: fixes pytest collection
  collision with tests/test_litellm/models/test_models.py (same basename)
- tests/test_litellm/models/__init__.py: same fix
- backend/routes/allowlist.py: add /config_overrides/ and /v1/unified_access_group
  prefixes for new routes added by upstream

* fix(ui/tests): resolve frontend-lint failures in new test files

- useLogDetails.test.ts: add Wrapper.displayName, replace 'null as any'
  with null, type resolveCall promise resolver properly
- usePaginatedDailyActivity.test.ts: remove unused waitFor import,
  add Wrapper.displayName, change Record<string,any> to Record<string,unknown>
- UsageViewSelect.adminFiltering.test.tsx: replace all props:any with
  explicit SelectProps/BadgeProps/SelectOption types, replace (X as any).displayName
  with direct X.displayName assignment

no-explicit-any count: 2034 (budget: 2040). Prettier check: clean.

* fix(ui): sync proxy/_experimental/out/ exactly to upstream

245 stale JS chunk files from earlier merges were left in the out/
directory but had been deleted in upstream. The Docker image in CI is
built by copying this directory verbatim, so the stale artifacts caused
the SERVER_ROOT_PATH redirect E2E to fail.

Synced by: git checkout upstream/litellm_internal_staging -- out/ (adds
new files) + git rm on every file present in HEAD but absent from
upstream.

* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py

Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>

* fix(makefile): fall back to upstream/litellm_internal_staging for strict-budget gate

origin/litellm_internal_staging exists on BerriAI's CI but not on forks
that use a different remote name (e.g. Azure DevOps as origin).  Fall
back to upstream/litellm_internal_staging when the origin ref is absent.

* linter reformat

* fix(deepkeep): apply guardrail tool/tool_call redactions from API response

When DeepKeep returns GUARDRAIL_INTERVENED with redacted tools or
tool_calls, the previous code ignored those redactions and forwarded
the original (potentially sensitive) values to the model — a guardrail
bypass for content embedded in tool schemas or function arguments.

Fix: prefer response_json["tools"] / response_json["tool_calls"] when
present, falling back to the originals only when the guardrail did not
return replacements — consistent with the existing pattern for texts and
images.

Refactor _build_return_inputs() into a private static helper to keep
apply_guardrail() under the PLR0915 statement limit (50).

Adds test_apply_guardrail_applies_tool_redactions_from_response to
assert that redacted tool payloads from the API response are used.

* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py

Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>

* fix(lint): move base-ref fallback into ruff_strict_gate.py; revert Makefile

The previous Makefile fix had a shell bug: 'git rev-parse --verify'
writes the resolved SHA to stdout, so the $$(...) substitution captured
both the SHA and the echo output, handing '--base <sha>\norigin/...' as
two tokens to the Python script, causing exit code 1 in CI.

Fix: revert Makefile to its original single-line invocation and add
_resolve_base() to ruff_strict_gate.py. The function checks whether the
requested ref resolves; if not, it tries the 'upstream/' equivalent
before falling back to the original ref (letting git emit a clear error).

Behaviour in BerriAI CI: origin/litellm_internal_staging resolves → used
as before, no change.
Behaviour on forks with a different 'origin': falls back to
upstream/litellm_internal_staging transparently.

* fix(lint): fix UP006/UP045/F401 in changed files; add depth guard to check_any_discipline

- Replace Dict/List/Optional/Tuple typing imports with built-in equivalents
  (UP006, UP045) across files touched in this PR diff, then clean up
  the now-unused typing imports (F401).
- Add _MAX_CONTAINS_ANY_DEPTH guard to check_any_discipline.contains_any()
  to prevent RecursionError on deeply-nested mypy types.

* fix(lint): resolve all three CI lint job failures

1. lint (ruff_strict_gate) — UP006/UP045/F401 violations introduced on
   changed lines. Fixed Dict/List/Optional/Tuple → built-in equivalents
   across every file in the PR diff; cleaned up now-unused typing imports.

2. any-discipline — RecursionError in check_any_discipline.contains_any()
   on deeply-nested mypy types. Upstream fixed this by converting to an
   iterative stack-based algorithm (merged). Also added deepkeep.py to
   any-discipline-budget.json via 'make lint-any-budget-update' so the
   new file's Any count is baselined instead of failing against the
   zero-baseline default.

3. basedpyright reportMissingParameterType — **kwargs in DeepKeepGuardrail
   __init__ lacked a type annotation. Added **kwargs: Any.

* Update litellm/deepkeep_tilt_config.yaml

Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>

* fix(lint): black reformat after merge

* fix(deepkeep): honour empty-list replacements in _build_return_inputs

When DeepKeep returns GUARDRAIL_INTERVENED with an intentional empty
replacement (e.g. texts:[], tool_calls:[]) the previous truthiness check
treated [] as absent and forwarded the original content downstream —
a guardrail bypass for any case where the firewall wants to fully clear
a field.

Fix: replace all response_json.get(field) truthiness checks with
'is not None' comparisons so that an empty list is respected as a
deliberate replacement. Applies to texts, images, tools, tool_calls,
and the original-input fallback guards.

Adds test_apply_guardrail_honours_empty_list_replacements.

* fix(test): replace live httpbin.org call with mocked transport in test_pass_through_with_httpbin_redirect

Root cause of OOM: the test made a real HTTP request to https://httpbin.org
inside a pytest-xdist worker. Under memory pressure the worker's httpx client
and redirect-following logic allocated enough virtual memory to trip the OOM
killer (confirmed by ulimit -v 16GB reproducing the crash with 'node down: Not
properly terminated' on this exact test).

Fix: replace the real network call with a custom httpx.AsyncBaseTransport that
returns a pre-built 302 -> 200 response sequence in-memory. The test now runs
hermetically with no network dependency and no excess memory allocation.

ulimit -v 16GB: 24,284 passed (0 crashes) after this fix.

* fix: merge upstream/litellm_internal_staging (197 commits), resolve conflicts

7 conflicts resolved:
- 6 Python files: upstream added new code with old-style typing (Optional,
  Dict, List) on lines where we had ruff-fixed modern syntax (str | None,
  dict, list). Took upstream's version then re-ran ruff UP006/UP045/F401
  --fix to keep both the new content and ruff compliance.
- test_openapi_compliance.py: upstream replaced 'role' with 'steps' in
  output_fields and updated the spec comment. Took upstream's version.

Also: added _resolve_base() fallback to type_check_gate.py and removed
the hard 'git fetch origin litellm_internal_staging' from the Makefile's
lint-basedpyright target (same pattern as ruff_strict_gate.py fix).

* fix: merge upstream (41 commits), resolve .gitignore conflict, fix BLE001

- .gitignore: upstream removed package.json/out/ ignore entries; took theirs
- deepkeep.py: added '# noqa: BLE001' on catch-all Exception handler
  (BLE001 rule newly enforced in ruff-strict-budget)
- type_check_gate.py: added _resolve_base() fallback for basedpyright gate
- Makefile: removed hard 'git fetch origin' from lint-basedpyright target

* fix: merge upstream (57 commits), resolve conflicts

- Makefile: upstream added lint-fetch-base target; made it tolerant of
  missing origin/litellm_internal_staging (git fetch || true)
- test_websearch_chat_completion.py: took upstream's new assertions and
  skipif marker
- anthropic_cache_control_hook.py: upstream added new code using List/Dict/Tuple
  which were undefined after our earlier UP006 cleanup; replaced with
  built-in list/dict/tuple

* fix(coverage): revert ruff UP006/UP045 changes on upstream files

The previous ruff fixes (Dict→dict, Optional→X|None) on 7 upstream files
added ~500 changed lines of pure type-annotation no-ops to our PR diff.
codecov/patch penalised these uncovered lines, dropping patch coverage
to 51.35% (target 61.83%).

Fix: revert these files to exactly match upstream/litellm_internal_staging.
The ruff_strict_gate still passes because the violations exist equally in
both the base and HEAD (total == base_count → no breach).

* fix: merge upstream (130 commits), resolve Makefile + base_email conflicts

- Makefile: upstream changed lint deps to $(LINT_DEP_INSTALL)/$(LINT_DEP_BASE);
  kept our --base removal (handled by _resolve_base in Python scripts)
- base_email.py: took upstream's dedup cache addition
- deepkeep.py: ruff format after merge

* chore: remove lint/format-only changes and non-feature files

Revert all lint-infra and black/ruff-reformat-only changes back to
upstream/litellm_internal_staging so the PR diff shows only the DeepKeep
guardrail feature:
- Makefile, scripts/ruff_strict_gate.py, scripts/type_check_gate.py
  (lint-gate infra)
- credential_migration.py + enterprise/* + assorted test files
  (black-reformat / xdist test-isolation drift)
- backend/routes/allowlist.py (merge glue)
Remove non-feature local artifacts: build-and-push.sh,
deepkeep_tilt_config.yaml, stray __init__.py collision shims, and
unrelated UI test files.

* fix(lint): add reason to BLE001 noqa to satisfy type-discipline gate (LIT003)

The type-discipline budget ratcheted LIT003's ceiling to 292 as upstream
fixed reasonless suppressions, so our '# noqa: BLE001' (code but no
reason) tipped the total to 293 and failed CI. Add a reason per the
required '# noqa: CODE  # <reason>' shape.

* fix(deepkeep): apply structured_messages redactions returned by the guardrail API

_build_return_inputs dropped any structured_messages the DeepKeep API returned and
always forwarded the original input, so redactions on that field never took effect.
Check the response first, same as texts/images/tools/tool_calls

* chore(ui): drop redundant preserve prop from the guardrail form

preserve defaults to true in rc-field-form (isMergedPreserve falls back to true when
unset), so the explicit prop changed nothing and only widened this PR's blast radius
to every guardrail provider in the shared form

* fix(deepkeep): stop extra_headers list from crashing the guardrail call and name the real firewall id config key

litellm_params.extra_headers is a list of header names to forward, so passing it
straight into dict.update raised ValueError and, under fail_closed, took the request
down with it. Only merge mapping values and warn otherwise

The docstring example and the missing-secret error both said firewall_id, but
initialize_guardrail only reads deepkeep_firewall_id, so anyone following them
had their value silently ignored

* refactor(proxy): drop normalize_callback change; split to its own PR (#33905)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yaniv Israel <yaniv@deepkeep.ai>
Co-authored-by: DK-yaniv <164404355+DK-yaniv@users.noreply.github.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-20 19:27:40 -07:00
bingbing
8307e3ee32 fix(bedrock_mantle): hoist Codex additional_tools input items to top-level tools 2026-07-20 19:27:33 -07:00
Tin Chi Lo
39cdfbdd28 test(mcp): pin all four Google registry entries in the shape test 2026-07-20 19:18:42 -07:00
tin-berri
7df76fc0d8
Merge pull request #33756 from BerriAI/litellm_mcp_dcr_client_redirect_33699
fix(mcp): return the DCR client's own redirect_uris to stop the /callback self-redirect loop
2026-07-20 19:16:45 -07:00
Tin Chi Lo
3334ee9b13 fix(mcp): add Google Sheets, Drive, Calendar, and Docs to the OpenAPI registry 2026-07-20 19:02:00 -07:00
Tin Chi Lo
20dd0e6a39 fix(mcp): return the DCR client's own redirect_uris to stop the /callback self-redirect loop
The MCP OAuth DCR relay's client-facing /register response handed back LiteLLM's own
<base>/callback as the client's redirect_uris on every arm except the true bridge relay.
A spec-compliant OAuth 2.1 DCR client (e.g. Open WebUI over Streamable HTTP) adopts that
value for its subsequent /authorize calls, so /callback redirects to itself; the second
hit carries the client's opaque state, fails to decrypt, and surfaces as the LIT-4197
"oauth_state ... Incorrect padding" error, making pass-through MCP OAuth unusable against
real DCR-capable upstreams.

The short-circuit arm keeps registering the gateway callback upstream (unchanged) and now
echoes the client's own redirect_uris back to the client across all register arms. Since
the client then authorizes with its own separate-origin redirect, the /authorize rejection
hint now points operators to MCP_TRUSTED_REDIRECT_ORIGINS, the mechanism a legitimate
cross-origin OAuth client needs (auto-trusting a DCR-registered redirect would reintroduce
the VERIA-57 open-redirect vector, since dynamic registration is unauthenticated).
2026-07-20 18:56:47 -07:00
tin-berri
2ea09a9969
Merge pull request #34026 from BerriAI/litellm_lit4608_autoroute_fixed_port_key
fix(cli): stable port and persisted master key for lite autoroute up
2026-07-20 18:54:17 -07:00
yuneng-jiang
731efafbca
Merge pull request #34053 from BerriAI/litellm_/migrate-simple-table-631c9b
refactor(ui): migrate credentials table onto shared DataTable
2026-07-20 18:18:28 -07:00
yuneng-jiang
ea045300bc
Merge pull request #34056 from BerriAI/litellm_/osv-dependency-vulnerabilities-662a89
build(deps-dev): bump js-yaml to 4.3.0 and brace-expansion to 5.0.7
2026-07-20 18:13:44 -07:00
devin-ai-integration[bot]
9d6d6c4fe0
fix(agents): allow optional securityScheme fields so /public/agent_hub does not 500 (#33897)
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-20 18:13:09 -07:00
devin-ai-integration[bot]
089de50d20
fix(auth): apply temp_budget_increase for cache-hit keys (#33841)
temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget.

Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies.

Fixes #25760

Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-20 18:12:15 -07:00
Yuneng Jiang
c93ccefb4a
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/osv-dependency-vulnerabilities-662a89 2026-07-20 17:49:26 -07:00
Mateo Wang
1e741094fe
Merge pull request #33807 from BerriAI/litellm_vertex_azure_midsys
fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages
2026-07-20 20:48:47 -04:00
Yuneng Jiang
f2ca0a149b
build(deps-dev): bump js-yaml to 4.3.0 and brace-expansion to 5.0.7
Both are dev-only build and lint tooling in the dashboard, not part of the
browser bundle. The bumps pull in upstream maintenance releases that address
inefficient handling of certain inputs

js-yaml is force-pinned through the overrides block because
@redocly/openapi-core exact-pins an older copy; a plain lockfile change would
not hold since the tree re-spawns a nested stale version on re-resolution, so
the override moves from 4.2.0 to 4.3.0. brace-expansion is added to overrides
at 5.0.7 so npm install does not leave the previously resolved 5.0.6 in place.
Both resolve to a single deduped copy after the change
2026-07-20 17:46:20 -07:00
devin-ai-integration[bot]
10d2a27d87
fix(proxy/auth): handle tz-aware temp_budget_expiry (#33840)
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-20 17:43:57 -07:00
Yuneng Jiang
368bfe19bf
fix(ui): surface add/update credential failures with an error toast
The add and update handlers had no try/catch (carried over from the legacy
panel), so a failed credentialCreateCall / credentialUpdateCall became an
unhandled rejection: no error notification and the modal left open with no
feedback. Bring them in line with the co-located delete handler by catching
and calling NotificationsManager.error, keeping the modal open on failure so
the user can retry. Add panel tests for the success (modal closes, refetch,
success toast) and failure (error toast, modal stays open) paths.
2026-07-20 17:37:24 -07:00
ryan-crabbe-berri
46a80e1ef5
fix(auth): set budget_reset_at when JWT upsert seeds a budget_duration (#34050)
The JWT first-login upsert in get_user_object creates the user row by
merging default_internal_user_params straight into table.create, so a
configured budget_duration landed with budget_reset_at NULL. The reset
sweep now heals such rows (PR #33623), but until the next sweep the row
shows a null reset time and its first window starts at the sweep instead
of one full duration after creation. Compute budget_reset_at at creation
like every other write path (/user/new, UI SSO, /key/generate, /team/new)
already does
2026-07-20 17:23:44 -07:00
Yuneng Jiang
25c37f2320
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/migrate-simple-table-631c9b 2026-07-20 17:09:59 -07:00
Yuneng Jiang
c06c16b0bf
refactor(ui): migrate credentials table onto shared DataTable
Move the Credentials panel off its hand-rolled tremor table onto the shared
DataTable and cell library, matching the SimpleTable design and the sibling
Vector Stores / Guardrails tables. Split the panel into a modal-owning parent
(CredentialsPanel), a thin client-mode DataTable consumer (CredentialsTable),
and a CredentialsTableColumns factory.

Credential Name and Provider render as shared cells (IdentityCell + provider
logo via getProviderLogoAndName); the per-row edit/delete icons become a right
-aligned overflow menu (Edit, Copy credential name, Delete). Admin-viewer read
parity is preserved: viewers still see the list but get no actions column.
Detail/edit and delete modals stay in the parent, so the public prop stays
uploadProps only.

Drops the now-unused tremor import (pruning its eslint suppression) and the
dead antd Form handle.
2026-07-20 17:09:55 -07:00
Yassin Kortam
2c99b7804e
perf(core): fast-path SafeAttributeModel.__delattr__ for declared fields (#33993)
Continues #29762. Response models (Message, Choices, Usage) delete unset
optional fields in __init__ so model_dump matches the OpenAI spec. Each delete
routed through pydantic's BaseModel.__delattr__, whose per-call
ModelMetaclass.__getattr__ lookup and _check_frozen dominate construction. When
the target is a declared field already present in __dict__ on a non-frozen
model, delete it with object.__delattr__ directly; that is exactly what pydantic
2.13 does for that case, minus the metaclass getattr and the frozen check. It
falls back to the previous super().__delattr__ path for extras, private
attributes, cached properties and missing names, so behavior is unchanged.

Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
2026-07-21 00:06:56 +00:00
Joseph Yeung
ffe56cf5a2
fix(budget): reset users/teams with NULL budget_reset_at (#33623)
Internal users seeded from default_internal_user_params (SSO/JWT first-login
upsert, or /user/new without an explicit budget_reset_at) get budget_duration
set but budget_reset_at = NULL. The ResetBudgetJob user/team queries filter on
{"budget_reset_at": {"lt": now}}, which never matches NULL, so these rows are
never reset: their spend accumulates for the lifetime of the row and silently
exceeds max_budget with no periodic reset.

The budget-table query already handles this by OR-ing in a
{budget_reset_at IS NULL AND budget_duration IS NOT NULL} branch. Apply the
same pattern to the user and team reset queries in PrismaClient.get_data.

Adds a regression test asserting both the user and team reset queries select
NULL-budget_reset_at rows that have a budget_duration.

Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 16:55:38 -07:00
mateo-berri
8b1a19fb02 test: give cost-map guard next() a default so a renamed rule fails with a clear assertion 2026-07-20 16:50:15 -07:00
mateo-berri
335b79d635 test(e2e): mark vertex mid-conversation system rows proven after live QA
Ran the before/after proof live against Vertex Claude (global endpoint,
project vertex-check-481318): base transform 400s an unflagged model
(claude-opus-4-7) on a mid-conversation role:system reminder, the fix
hoists it to a 200, and a flagged model (claude-opus-4-8) keeps the
reminder in messages with cache_read held at 15615 across the reminder
turn. Flip both vertex.mid_conversation_system rows to fail_before_fix:
proven.
2026-07-20 16:48:30 -07:00
mateo-berri
23b5b7d199 fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted

Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400

Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag

Closes the Vertex/Azure gap from the customer RCA
2026-07-20 16:48:29 -07:00
mateo-berri
1e6a34850d fix(router): propagate capability flags to shared backend cost map key 2026-07-20 16:46:30 -07:00
tin-berri
b086cd32a6
Merge pull request #33183 from BerriAI/litellm_mcp_walker_consolidation
refactor(mcp): consolidate exception-tree walkers into one shared faults traversal
2026-07-20 16:43:26 -07:00
Yassin Kortam
e906dbe4a1
perf(streaming): build per-chunk Delta directly instead of setattr/delattr churn (#33992)
Continues #29761. Delta.__init__ set roughly ten attributes through pydantic's
__setattr__ and then deleted the five OpenAI omits on every chunk. Those keys
are extra fields (extra='allow'), so this builds __pydantic_extra__ and
__pydantic_fields_set__ directly after the parent init instead of round-tripping
each field through __setattr__/__delattr__. The resulting __dict__,
__pydantic_extra__, __pydantic_fields_set__ and model_dump output (including
exclude_unset, which the streaming path relies on) are byte-identical to the
previous behavior; a serialization-contract test locks that. A TYPE_CHECKING
block re-declares the extra attributes with their concrete types so type
checkers still see delta.content and friends.

Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
2026-07-20 16:40:43 -07:00
Tin Chi Lo
8e0d3af81b Merge origin/litellm_internal_staging into litellm_mcp_walker_consolidation
Staging now contains #33153, whose final rounds made _extract_upstream_auth_failure a thin delegate
to upstream_auth_challenge and introduced the response-level iterator this branch predates. The
resolution completes the consolidation both branches were converging on: iter_exception_tree
(faults/traversal.py) is the one tree walk, _iter_upstream_responses is rebuilt on top of it instead
of carrying a second copy of the traversal, the manager keeps the delegate, and the semantic filter
port from this branch stands. Test conflicts were append-append and both sides are kept
2026-07-20 16:29:24 -07:00
Mateo Wang
fa59cfa6da
Merge pull request #34041 from BerriAI/litellm_lit4544_shared_key_model_info_leak
fix(router): stop custom model_info leaking onto shared backend cost map key
2026-07-20 19:22:57 -04:00
tin-berri
f61e16c734
Merge pull request #32259 from BerriAI/litellm_mcp_v2_client_credentials
feat(mcp): migrate client_credentials (M2M) onto the v2 resolver arm
2026-07-20 16:22:14 -07:00
tin-berri
43e4af73f0
Merge pull request #33631 from BerriAI/litellm_lit4517_messages_mcp_gateway
feat(mcp): support MCP servers on the Anthropic /v1/messages API
2026-07-20 16:22:03 -07:00
tin-berri
f9b10eb296
Merge pull request #33886 from BerriAI/litellm_lit4582_cache_control_present
fix(anthropic): only inject cache_control when the request carries none
2026-07-20 16:20:55 -07:00
mubashir1osmani
28f012bb52
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping

Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.

* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse

Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.

* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes

Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.

* test(e2e): drop Langfuse spend suite; feature is being deprecated

Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.

* test(e2e): fold provider/batch feature tests into their endpoint suites

Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature

Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py

Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant

Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes

Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path

* test(e2e): address review feedback and re-home virtual-key coverage

Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN

Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check

Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions

Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
2026-07-20 16:15:55 -07:00
Yassin Kortam
3810130105
test(e2e): add reliability suite covering fallback, timeout, and cache behavior (#34023)
* test(e2e): add reliability suite covering fallback, timeout, and cache behavior

* test(e2e): move reliability suite under router and drive it with real deployments

* test(e2e): make the router complexity fixture opt-in so reliability tests can coexist
2026-07-20 23:06:46 +00:00
yucheng-berri
432954a2ab
fix(cache): make in-memory and disk cache increments atomic (#34013)
Some checks failed
CodSpeed Benchmarks / benchmarks (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
* fix(cache): make in-memory and disk increments atomic

* refactor(cache): narrow in-memory increment lock scope

* fix(cache): address follow-up review on increment tests/types

* fix(cache): refresh atomic increment coverage

* test(cache): widen increment race window with non-zero _SlowInt seed

The zero seed was falsy, so InMemoryCache.increment_cache's `get_cache(...) or 0`
and DiskCache.get_cache's truthiness guard both discarded the _SlowInt before
__add__ could run, leaving the sleep-based window-widening inert. Seed a non-zero
value and return _SlowInt from __add__ so the sleep fires on every read-modify-write
in both backends, making the concurrency regression deterministic.

* test(cache): cover InMemoryCache.async_increment delegation

Add a focused async test asserting async_increment accumulates through the
locked sync path, exercising the previously uncovered delegation line.

---------

Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
2026-07-20 15:51:01 -07:00
mateo-berri
3f712e3fde fix(router): stop custom model_info leaking onto shared backend cost map key 2026-07-20 15:49:19 -07:00