Commit graph

2000 commits

Author SHA1 Message Date
Krrish Dholakia
f775fa10d0
Merge branch 'litellm_internal_staging' into litellm_memory_endpoints 2026-04-24 17:45:51 -07:00
yuneng-jiang
7723a54478
Merge pull request #25677 from BerriAI/litellm_migration_projects
[Refactor] Proxy: move projects management to enterprise package
2026-04-24 17:40:33 -07:00
ishaan-berri
8a9faa81b2
feat(guardrails): LLM-as-a-Judge guardrail (#26360)
* feat(guardrails): add LLM_AS_A_JUDGE to SupportedGuardrailIntegrations

* feat(types): add EvalVerdict, StandardLoggingEvalInformation; wire eval_information into SpendLogsMetadata

* feat(guardrails): add self-contained llm_as_a_judge guardrail hook

* fix(a2a): filter agent-only litellm_params from acompletion kwargs; pass agent_id into body

* feat(ui): add LLMJudgeFields criteria builder component

* feat(ui): wire LLM-as-a-Judge into add guardrail form

* feat(ui): update EvalViewer — title 'LLM Judge Results', weighted score column, summary row

* fix(ui): wire EvalViewer into LogDetailContent to show LLM judge results on logs page

* fix(guardrails-ui): route llm_as_a_judge to criteria builder step; rename to LiteLLM LLM as a Judge; add litellm logo

* fix(guardrail-viewer): stack lifecycle + eval details vertically to avoid badge overflow in narrow drawer

* fix(guardrail-create): surface config validation errors on create instead of silently orphaning guardrail in DB

* fix(guardrail-registry): hardcode llm_as_a_judge in initializer registry so it loads regardless of package install path

* fix(llm-as-a-judge): fix P1 code quality issues - validate weights/on_failure, guard pre_call, handle multimodal, move imports to module level, fix spurious finally logging

* fix(guardrail_endpoints): use correct PK field in rollback delete and log rollback failure

* fix(llm_as_a_judge): support Pydantic object in _get_litellm_param fallback chain

* fix(LLMJudgeFields): replace @tremor/react Button with antd Button

* fix(llm_as_a_judge): remove dead registry dicts, fix KeyError in prompt builder, set correct status on judge failure

* test(llm_as_a_judge): add unit tests for guardrail hook

* fix(llm_as_a_judge): remove @log_guardrail_information decorator to fix duplicate guardrail_information entries

The decorator and the manual finally block both called add_standard_logging_guardrail_information_to_request_data, producing two entries per request. The decorator also misclassified HTTPException(422) blocks as guardrail_failed_to_respond (it checks for 400). The finally block correctly tracks status throughout, so removing the decorator is sufficient.

* fix(test_gcs_pub_sub): ignore metadata.eval_information in comparison

* fix(test_spend_management): ignore metadata.eval_information in payload comparison

* fix(types/guardrails): add input_type and messages to ApplyGuardrailRequest

* fix(guardrail_endpoints): pass input_type and messages through apply_guardrail endpoint

* fix(guardrail_endpoints): auto-detect post_call guardrails and use input_type=response

* fix(a2a_endpoints): merge agent litellm_params guardrails into data before post_call hooks

* fix(llm_as_a_judge): use float sum with tolerance for weight validation

* fix(guardrail_registry): split long import line for black formatting

* fix(llm_as_a_judge): guard guardrail_name Optional for mypy

* fix(llm_as_a_judge): set guardrail_status=guardrail_intervened when score fails, regardless of on_failure mode

* fix(a2a_endpoints): use try/finally so deferred spend log fires even when guardrail blocks with 422

* fix(litellm_logging): declare _defer_async_logging and _enqueue_deferred_logging on Logging class for mypy

* fix(logging_worker): restore queue.join() in flush() to wait for in-flight callbacks
2026-04-24 17:15:32 -07:00
Krrish Dholakia
765804aa69 fix(memory): require team admin to modify pure team rows
Tightens the write-authorization rule for "pure team rows" (rows with
no user_id stamped, only team_id) to match the pattern used by
team-management endpoints (`_is_user_team_admin` + `_is_user_org_admin_for_team`):

- Plain team members can READ team rows via the OR visibility filter
  (intentional, unchanged).
- Only PROXY_ADMIN, team admins of the row's team_id, or org admins
  for the team's organization may MODIFY them. Plain members get 403.

`_assert_write_access` is now async and takes the prisma_client so it
can fetch the team and run the existing `_is_user_team_admin` /
`_is_user_org_admin_for_team` helpers from
`litellm.proxy.management_endpoints.common_utils`. The org-admin path
is best-effort: it calls `get_user_object`, which depends on the
proxy_server module being initialized, so any exception there is
treated as "not an org admin" rather than crashing the request.

Tests:
- team admin can modify pure team row → 200
- plain team member cannot modify pure team row → 403
- plain team member cannot delete pure team row → 403

Updates the test fake to add a tiny `litellm_teamtable.find_unique`
implementation and a `_make_team(team_id, admin_user_ids=[...])`
helper.

27/27 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 16:45:15 -07:00
Krrish Dholakia
36c5174c99 fix(memory): close cross-user write gap + sanitize 500 errors (Veria)
Addresses two Veria findings:

**High — cross-user memory tampering via team membership.** The
visibility filter uses an OR (`user_id == caller OR team_id == caller`)
so team members can SEE each other's team-scoped rows. That's
intentional for list/get. But because PUT/DELETE used the same filter
to find the target row, any team member could overwrite or delete a
teammate's *personal* row whenever both `user_id` and `team_id` were
stamped on it — broader visibility was being silently treated as
broader authority.

New `_assert_write_access(row, caller)` enforces ownership for
mutations. Non-admin rules:

- The row's `user_id` must match the caller (personal ownership), OR
- The row has no `user_id` and its `team_id` matches the caller's
  team (a "pure team row" intended for shared writes).

Admins bypass the check. The same gate runs in PUT (both regular
and post-race-recovery branches) and DELETE.

**Medium — DB internals leaked through 500 detail.** Every `except`
block was raising `HTTPException(500, detail=str(e))`, which surfaces
Prisma error strings (table/column names, host:port, error class
names) to API callers. New `_internal_error()` helper logs the real
exception server-side and returns a generic, caller-safe `detail`.
Applied to create, list, upsert (general fallthrough), and delete.

Also tightened the race-recovery 409 message to drop the "in a
different scope" wording — the caller never needs to know whose
scope it lives in.

Tests (+5):
- teammate cannot overwrite personal row → 403
- teammate cannot delete personal row → 403
- teammate CAN modify pure team row (no user_id stamped) → 200
- admin bypasses write-auth → 200
- 500 response never echoes Prisma internals (table/host/class names)

25/25 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 16:38:39 -07:00
ryan-crabbe-berri
c91a22a001
Merge pull request #26438 from BerriAI/litellm_fix-jwt-admin-bypass
fix(jwt-auth): apply team TPM/RPM + attribution for admins using x-litellm-team-id
2026-04-24 16:07:50 -07:00
ishaan-berri
7cf6a95b62
fix(vertex passthrough): log :embedContent and :batchEmbedContents responses (#26146)
* fix(vertex passthrough): log :embedContent and :batchEmbedContents responses

* test(vertex passthrough): add unit tests for :embedContent and :batchEmbedContents logging

* fix(vertex passthrough): extract input text from request body for embedContent token counting

* fix(vertex passthrough): add embedContent and batchEmbedContents to TRACKED_VERTEX_ROUTES

* fix(vertex passthrough): detect Google AI Studio URLs in embedContent handler

* test(vertex passthrough): add unit test for Google AI Studio URL embedContent provider detection

* style: black format vertex_passthrough_logging_handler
2026-04-24 16:07:11 -07:00
Yuneng Jiang
b300dc3e54
Preserve UISettings docstring on effective class for GET schema
create_model does not inherit the base class docstring, so once an
extension registered a field the effective class had no description.
The UI renders schema.description as a header paragraph — losing it
broke the 'Configuration for UI-specific flags' text. Pass __doc__
through explicitly and add a regression test.
2026-04-24 13:56:59 -07:00
Ryan Crabbe
e1bb542556
chore: fix linting (ruff PLR0915, black) on admin team-header fix
Extract the admin team-header attachment into a helper so
auth_builder stays under the 50-statement lint threshold; apply
black formatting to the two files flagged on the prior commit.
No behavior change.
2026-04-24 13:38:28 -07:00
yuneng-jiang
09f0a3380f
Merge pull request #26362 from BerriAI/litellm_fix_proxy_test_master_key_leak
[Fix] Tests - Proxy: Isolate master_key/prisma_client module globals between tests
2026-04-24 10:04:09 -07:00
yuneng-jiang
d73b790cae
Merge pull request #26248 from BerriAI/litellm_anthropic_messages_call_type_fix
fix(proxy): preserve anthropic_messages call type for /v1/messages logging
2026-04-24 09:42:36 -07:00
Ryan Crabbe
6ea95a6379
fix(jwt-auth): apply team TPM/RPM + attribution for admins using x-litellm-team-id
Scope the header-driven team fetch to LLM API routes so admin
management routes keep the pre-existing bypass behavior (no
phantom teams, no 404s on mgmt calls). Team context is threaded
onto UserAPIKeyAuth so spend logs, rate limits, and team_models
attribution are correctly applied when admins act on behalf of
a team via x-litellm-team-id.
2026-04-24 09:40:59 -07:00
Shivam Rawat
9dcb2bd528
fix(proxy): respect object-level permissions for managed vector store endpoints (#26351)
* fix(proxy): honor object_permission for managed vector store access

* perf(proxy): preload team object_permission on UserAPIKeyAuth

Populate team_object_permission during virtual-key and JWT auth when the
team is loaded, so can_user_access_vector_store uses it in memory first
and only falls back to get_object_permission by id when missing.

Made-with: Cursor
2026-04-24 09:21:13 -07:00
Krrish Dholakia
c5fafe56bb fix(memory): compose visibility + key filters via explicit AND
Greptile P1 (filter-fragility): `where.update(vis)` was semantically
correct today, but dict-merging by key meant any future visibility
filter that grew a new top-level "OR" would silently clobber the
existing key filter.

Compose explicitly instead:

    where = {"AND": [key_filter, vis]}

Applied to both `list_memory` and `_find_memory_for_caller`. When
either side is empty (admin has no visibility filter; list has no
key filter), skip the wrapper and use the non-empty side directly
to keep the generated SQL clean.

Test fake's `_matches` now understands top-level `AND` too.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 09:08:53 -07:00
ishaan-berri
863f922be8
fix(team_endpoints): auto-add SSO team members to org on move (proxy admin only) (#26377)
* fix(team_endpoints): auto-add SSO team members to org for proxy admins

* test: proxy_admin vs team_admin security boundary for team→org move

* screenshots: before/after for team-org SSO fix

* fix(team_endpoints): restore staging security features dropped in SSO commit

Co-Authored-By: Ishaan Jaff <ishaan@berri.ai>

* style: black formatting for team_endpoints
2026-04-24 08:36:25 -07:00
Krrish Dholakia
be7fe9b5f5 fix(memory): typed Prisma error + explicit-null metadata on PUT
Two more greptile threads from the last review:

- Unique-violation detection was string-matching "Unique"/"UniqueViolation"
  in the exception message, fragile across Prisma/driver versions. Now
  check the typed error `code == "P2002"` first, with string fallback.

- PUT could not distinguish "metadata omitted" from "metadata: null" —
  both parsed as `None`, so callers had no way to clear stored metadata.
  Switch to Pydantic v2's `model_fields_set` to tell which fields the
  caller actually sent; explicit null now clears the column.

New tests:
- explicit null clears metadata
- omitted metadata preserves existing value

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 22:14:11 -07:00
Krrish Dholakia
55f7655c2a fix(memory): address greptile review feedback
Addresses 5 greptile findings (3/5 → higher confidence target):

1. Identity-less orphan rows (P1): non-admin callers with no user_id AND
   no team_id could create rows that the visibility filter would never
   match again. Now rejected up front with 400 — caller must authenticate
   with a scoped key or act as PROXY_ADMIN.

2. Upsert race returning 500 (P1): PUT's check-then-create isn't atomic;
   a concurrent writer could slip a row in between the 404-check and the
   create call. Now catch unique-violation on create, re-read, and fall
   through to update — PUT stays idempotent. If the conflicting row
   belongs to a different scope, surface a 409 instead of 500.

3. PUT-create scope inconsistency (P2): PUT's create branch always used
   the caller's own user_id/team_id, so admins couldn't bootstrap rows
   scoped elsewhere via PUT (only POST). Now PUT-create calls the shared
   `_resolve_scope()` helper, matching POST semantics.

4. Stale schema comment (P2): schema said "Keyed by (key, user_id,
   team_id)" but `key` is globally unique. Updated all three schema
   copies to reflect the actual design.

5. UI silently truncated at 200 (P2): MemoryView fetched pageSize=200
   with no load-more. Swapped to real server-side pagination driven by
   `data.total`; page size is now 50 and the pager is a real AntD
   control.

Also extracts a shared `_resolve_scope()` helper and `_is_unique_violation()`
from create_memory so POST and PUT don't drift on the scope/error logic.

Tests: +3 new (identity-less 400, PUT admin bootstrap, PUT race →
update), 18/18 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 22:11:50 -07:00
ryan-crabbe-berri
0992bf2271
Merge pull request #26367 from BerriAI/litellm_/split-mcp-routes-management-vs-inference
Split MCP routes into inference vs management (unblock Admin UI on DISABLE_LLM_API_ENDPOINTS nodes)
2026-04-23 22:05:48 -07:00
Yuneng Jiang
e68c60a66e
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_proxy_test_master_key_leak
# Conflicts:
#	tests/test_litellm/proxy/realtime_endpoints/test_realtime_webrtc_endpoints.py
2026-04-23 18:20:01 -07:00
shin-berri
8e652d129d
Merge pull request #26356 from BerriAI/litellm_cci_gha_dedup_and_shard
Some checks are pending
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
[Infra] Remove CCI/GHA test duplication and semantically shard proxy DB tests
2026-04-23 18:17:56 -07:00
Ryan Crabbe
4d2acafa43
Split MCP routes into inference vs management categories
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.

Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.

Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
2026-04-23 16:52:45 -07:00
yuneng-jiang
87e120d958
Merge pull request #26346 from BerriAI/litellm_reset_budget_is_not_null
[Fix] Reset budget windows failing due to Prisma Json? null filter
2026-04-23 16:37:09 -07:00
milan-berri
2001d91b27
fix(mcp): share temporary MCP OAuth sessions across instances via Redis (#26162) (#26318)
Temporary MCP OAuth sessions were kept in process-local memory, so on
multi-instance/LB proxy deployments a session created on instance A could
not be found when the follow-up /server/oauth/{server_id}/... request
landed on instance B.

Persist temporary session records to Redis (encrypted with the existing
proxy encryption helpers) as a best-effort L2 cache alongside the current
in-memory L1. Convert get_cached_temporary_mcp_server to async and await
it from the authorize/token/register OAuth endpoints.

Made-with: Cursor
2026-04-23 16:21:27 -07:00
shin-berri
7c69262279
Merge pull request #26349 from BerriAI/litellm_deflakeSpendTests
[Fix] Deflake spend tracking tests
2026-04-23 16:12:19 -07:00
Yuneng Jiang
1481b125ed
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_proxy_test_master_key_leak 2026-04-23 16:06:41 -07:00
milan-berri
b6d0f6b649
fix(vertex_ai): use aiplatform.{geo}.rep.googleapis.com for multi-region locations (#26281)
Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.

common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.

Tests cover us/eu for the common_utils helper.

Made-with: Cursor
2026-04-23 15:58:02 -07:00
Yuneng Jiang
5b9c7be558
test: isolate proxy master_key/prisma_client module globals between tests
Sibling tests were mutating litellm.proxy.proxy_server.master_key and
prisma_client with raw setattr. Values leaked across tests in the same
xdist worker, flipping the auth short-circuit in user_api_key_auth and
causing unrelated tests (e.g. test_ui_view_session_spend_logs_pagination)
to return 401 instead of 200.

Replace raw setattr with monkeypatch in the two offending files and add
an autouse conftest fixture that snapshots/restores the known-leaky
module globals for every proxy test.
2026-04-23 15:31:16 -07:00
Yuneng Jiang
32c390a0f6
fix(tests): restore proxy_server.master_key in realtime fixture; add shard-coverage guard
Two fixes to proxy-db CI:

1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the
   module-global `proxy_server.master_key` without restoring it, leaking
   state into any test that shared the same xdist worker. Under
   --dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the
   google_endpoints tests to fail with "No api key passed in." because
   user_api_key_auth saw a set master_key and a missing API key on the
   test request. The fixture now saves and restores the original value.

2. Address the Greptile note that the semantic shard design has no
   catch-all, so a new test file added to tests/proxy_unit_tests/ without
   a matrix entry would silently skip CI. Adds an assert-shard-coverage
   job that enumerates test_*.py files and fails the workflow if any are
   not referenced by a matrix entry, with a clear message telling the
   author which semantic shard to place it in. All proxy-db shards now
   depend on this guard.
2026-04-23 15:01:25 -07:00
Michael-RZ-Berri
c81342e3c2
Merge pull request #26204 from BerriAI/litellm_budgetLimitFix
Fix bugs that bypasses per-team member budget limit
2026-04-23 14:59:07 -07:00
Yuneng Jiang
e37d1b0cb6
[Fix] Deflake spend tracking tests
Two independent deflakes:

1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
   of 401/403 when earlier tests in the file left proxy-auth globals
   (prisma_client, master_key, user_custom_auth, general_settings,
   user_api_key_cache) in a state that let invalid tokens pass auth and
   fall through to the endpoint's own start_date/end_date validation.
   Add an autouse fixture that pins those globals to their import-time
   defaults for every test in the file. Harden the assertion to include
   response body so future flakes are diagnosable.

2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
   on the Redis transaction buffer flushing spend to Postgres. The buffer
   uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
   and a single global buffer list key. Pointing the proxy at the shared
   remote Redis means concurrent CI pipelines contend for the same lock
   and can drain each other's buffer into the wrong database. Add a
   start_redis reusable command that boots a per-job redis:7-alpine
   container (digest-pinned), and switch proxy_spend_accuracy_tests to
   REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
   isolated per CI run.
2026-04-23 14:13:55 -07:00
Yuneng Jiang
c41567eaa0
fix(budget_reset): use raw SQL for IS NOT NULL filter on Json? columns
The periodic budget-window reset job filtered keys/teams with
`where={"budget_limits": {"not": None}}`. The prisma-client-python
library does not support null-filtering on `Json?` columns (no
DbNull/JsonNull sentinel — upstream issue #714). The client drops the
`None` value during serialization and the engine rejects the query with
`MissingRequiredValueError: where.budget_limits.not: A value is
required but not set`, so neither the key nor team reset path runs.

Switch those two `find_many` calls to `query_raw` with
`WHERE budget_limits IS NOT NULL`, selecting only the PK and the
`budget_limits` column. Writes still go through the ORM. Add unit tests
covering the expired/unexpired paths for keys and teams, string-encoded
JSON payloads, empty payloads, error isolation between the two paths,
and a regression guard asserting the query still uses `IS NOT NULL`.
2026-04-23 12:26:25 -07:00
Krrish Dholakia
3b9445eeb2 fix(memory): make key globally unique, 409 on any duplicate
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
Switches from the compound unique `(key, user_id, team_id)` to a simple
`key @unique`. The compound form silently allowed duplicates when
user_id or team_id was NULL (Postgres treats each NULL as distinct), so
callers could POST the same key repeatedly. Globally-unique key means
one row per key, period — any duplicate create → 409.

- schema.prisma (×3): `key String @unique`, drop `@@unique(...)`.
- initial add_memory_table migration: unique index on (key) only.
- Remove the now-unused follow-up NULLS NOT DISTINCT migration.
- Endpoint error message simplified ("already exists" — no "for this scope").
- Test fake's create() now enforces global key uniqueness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 11:15:19 -07:00
Krrish Dholakia
cc1f0d4cf3 feat(memory): add key_prefix filter + promote Memory to AI GATEWAY nav
Backend:
- GET /v1/memory now accepts `key_prefix` for Redis-style namespace
  scans (e.g. `?key_prefix=user:`). When both `key` and `key_prefix`
  are passed, `key_prefix` wins.
- Prefix filter sits under the visibility filter in the Prisma where
  clause, so it can never leak rows across user/team scopes.
- New tests: prefix match, and cross-scope isolation (another user's
  `user:*` rows must not appear in the caller's results).

UI:
- Memory moved from a Tools submenu to a top-level AI GATEWAY item
  (alongside Agents, MCP Servers, Skills) — it's an API primitive,
  not a tool-management surface.
- Search box now drives prefix search, matching the Redis mental
  model ("type the namespace, see everything under it").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 08:24:07 -07:00
Sameer Kankute
d5449f5b1a
Merge pull request #26300 from BerriAI/litellm_oss_staging_04_22_2026
Litellm oss staging 04 22 2026
2026-04-23 18:53:58 +05:30
sakenuGOD
034f4fdef2
fix(mcp_semantic_tool_filter): match tools with client-side namespace prefix (#26078) (#26117)
* fix(mcp_semantic_tool_filter): match canonical tools that arrive with
a client-side namespace prefix.

`SemanticMCPToolFilter._get_tools_by_names` matched by exact equality
between the canonical name stored in the router
(`<server><MCP_TOOL_PREFIX_SEPARATOR><tool>`) and the name in the
incoming `tools[]` list. MCP clients such as opencode wrap every tool
name with their own additive alias prefix
(`<client_alias>_<canonical>`), so the two never matched, the filter
dropped every tool to zero, and the proxy forwarded `tools: []` with
`tool_choice: auto` — which strict upstream providers reject with a 400.

The fix adds anchored suffix matching with a separator check: the
canonical must form the complete tail of the incoming name and be
preceded by `_` or `-`. Exact matches still win over suffix matches,
incoming tools are returned at most once, and the original tool object
is passed through unchanged so the client-facing name survives for
tool-call round-trips.

Seven unit tests in a new TestGetToolsByNames class cover exact
match, underscore- and dash-prefixed variants, non-separator-anchored
suffixes (which must not match), exact-wins-over-prefixed precedence,
deduplication when two canonicals suffix-match the same incoming tool,
and ordering-follows-router-output.

Fixes #26078

* review: strengthen the suffix-fallback tie-breaker and the
deduplication regression test (Greptile comments on #26117)

- test_same_tool_not_returned_twice now passes two distinct canonicals
  ("read_file" and "file") that both suffix-match the same incoming
  tool, rather than the same canonical twice, so the assertion
  actually exercises the used_ids dedup path instead of the
  duplicate-input-list path.

- The suffix fallback in _get_tools_by_names now prefers the shortest
  incoming name that still qualifies under the separator-anchored
  match. In the one-prefix-per-client opencode scenario this is a
  no-op, but in multi-namespace configurations the shortest qualifying
  name is the least-wrapped one and is the most defensible deterministic
  choice, replacing the dict-insertion-order fallback.

- Adds test_suffix_fallback_prefers_shortest_candidate covering the
  new tie-breaker directly.

Still 15 tests passing locally (was 14).

* review(#26117): gate suffix-matching on canonical containing MCP_TOOL_PREFIX_SEPARATOR

@krrish-berri-2 flagged a possible collision in the suffix fallback:
a local user function whose name happens to end in a bare canonical
substring (e.g. my_firecrawl_scrape vs canonical firecrawl_scrape)
would be spuriously selected.

Server-registered MCP tools are always emitted as
<server_name><MCP_TOOL_PREFIX_SEPARATOR><tool_name> via
add_server_prefix_to_name, so a canonical without the separator is
not a namespaced MCP tool and does not warrant suffix matching.
Added that guard to _name_matches_canonical with a regression test
(test_does_not_collide_with_local_function_on_unprefixed_canonical)
that reproduces the collision before the fix and is pinned after.

Pre-existing TestGetToolsByNames fixtures that relied on bare
canonicals (get_weather, search, read_file, write/delete/read) were
switched to realistic server-prefixed ones so they continue to
exercise the suffix-fallback path under the new guard. The opencode
scenario (client prefix on already-server-prefixed canonical) is
unchanged.

---------

Co-authored-by: sakenuGOD <sakenuGOD@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
2026-04-22 19:06:34 -07:00
ryan-crabbe-berri
c4c1861389
Merge pull request #26195 from BerriAI/litellm_team_member_total_spend
Track per-member total spend on team memberships
2026-04-22 18:20:16 -07:00
shin-berri
b6fdd46636
Merge pull request #26270 from BerriAI/litellm_/lucid-kowalevski-de832f
[Fix] Stabilize flaky spend accuracy tests + patch Redis buffer data-loss path
2026-04-22 15:02:24 -07:00
Yuneng Jiang
288d403529
[Fix] Preserve in-memory spend updates when Redis rpush fails
store_in_memory_spend_updates_in_redis drained the in-memory queues
into local variables before the rpush pipeline. If rpush raised (cloud
Redis hiccup, timeout, connection blip), those already-drained
transactions were garbage-collected with the scheduler job, silently
losing all spend aggregated during that tick.

Wrap the rpush in try/except. On failure, re-enqueue the aggregated
transactions into their respective in-memory queues so the next
scheduler tick retries.

Add a unit test that seeds real queues, simulates an rpush failure,
and asserts the transactions land back in-memory.
2026-04-22 14:10:40 -07:00
Milan
ec735074a2
fix(proxy): reapply Bedrock guardrail spend logging (#25854)
Restore guardrail spend/UI event_type wiring, request_data on streaming
OUTPUT paths, and centralized match redaction after the upstream revert.

Made-with: Cursor
2026-04-22 23:00:45 +03:00
Michael Riad Zaky
0bd49ecb8b Fix bug that bypasses per-team member budget limit
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (proxy-utils, tests/proxy_unit_tests/test_proxy_utils.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
2026-04-22 10:41:13 -07:00
Sameer Kankute
b9675109cb
fix(proxy): preserve anthropic_messages call type for /v1/messages logging
Keep anthropic_messages as the logged call_type for non-Anthropic /v1/messages adapter paths and add a regression test to prevent fallback to completion/acompletion.

Made-with: Cursor
2026-04-22 18:32:33 +05:30
Krrish Dholakia
c1954017cc feat(proxy): add /v1/memory CRUD endpoints with user/team scoping
New LiteLLM_MemoryTable stores user/team-scoped key/value entries with
optional JSON metadata. Value is a String (LLM-readable text) and metadata
is an optional Json? envelope, matching the Letta + mem0 hybrid model so
future structured fields can be added without a schema migration.

Endpoints:
  POST   /v1/memory         - create
  GET    /v1/memory         - list (caller-scoped; admins see all)
  GET    /v1/memory/{key}   - fetch one
  PUT    /v1/memory/{key}   - upsert
  DELETE /v1/memory/{key}   - delete

Non-admin callers cannot set a user_id/team_id other than their own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 20:42:12 -07:00
yuneng-jiang
eebb80fbef
Merge pull request #26208 from BerriAI/litellm_individual-team-member-budgets
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (proxy-utils, tests/proxy_unit_tests/test_proxy_utils.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
Litellm individual team member budgets
2026-04-21 18:38:00 -07:00
shivam
5770af0068
Merge branch 'litellm_internal_staging' into litellm_individual-team-member-budgets 2026-04-21 17:59:29 -07:00
shivam
27a105bcf9
fix: give each team member an independent budget instead of sharing the team default
Previously, members added to a team without an explicit per-member budget were
all linked to the same `litellm_budgettable` row referenced by the team's
`metadata.team_member_budget_id`. Updating one member's budget via
`/team/member_update` mutated the shared row and silently changed every other
member's budget too.

Now both write paths produce a private, per-member budget:

- `add_new_member` clones the team's default budget into a fresh row when a
  member is added without `max_budget_in_team`/`allowed_models`. If no team
  default exists, the membership is created with no budget.
- `_upsert_budget_and_membership` detects when an existing membership still
  points at the team's default budget id and clones-on-write, relinking the
  membership to the new private budget before applying the update.
- `team_member_update` reads `team_member_budget_id` from team metadata and
  passes it through so the helper can make this distinction.

Adds unit tests for clone-on-write, in-place update of a private budget, and
the no-default-no-budget add path.

Made-with: Cursor
2026-04-21 17:58:50 -07:00
yuneng-jiang
d9494b6990
Merge pull request #26144 from BerriAI/litellm_post_call_non_streaming
fix(bedrock_guardrails): use Bedrock OUTPUT source for apply_guardrail when scanning model responses
2026-04-21 17:50:16 -07:00
yuneng-jiang
5dc2926a1e
Merge pull request #26194 from BerriAI/litellm_fix_migration_thrashing
[Feature] Proxy: opt-in v2 migration resolver
2026-04-21 16:55:54 -07:00
Yuneng Jiang
88b1823f51
[Test] Fix setup_database call-signature assertions for v2 flag
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.

Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
2026-04-21 14:45:29 -07:00
Ryan Crabbe
e5f3e15969
Track per-member total spend on team memberships
Adds total_spend column to LiteLLM_TeamMembership that accumulates
continuously and is not zeroed by the budget cycle reset job. This
enables UI surfaces to distinguish current-cycle spend (the existing
spend column, which resets) from lifetime spend per team member.

Also exposes budget_reset_at on LiteLLM_BudgetTable so /team/info
callers can see when a member's budget window next resets. The field
was already stored in the DB but stripped by the response Pydantic
model.

Includes regression tests that:
- Guard the reset job against ever writing total_spend: 0
- Verify the spend writer increments both spend and total_spend in
  one UPDATE statement.
2026-04-21 13:56:44 -07:00
shivam
62c2c553d7
Merge remote-tracking branch 'upstream/litellm_internal_staging' into litellm_post_call_non_streaming 2026-04-21 11:45:26 -07:00