- test_prepare_key_update_data: replace bare MagicMock with
MagicMock(spec=LiteLLM_VerificationToken) and explicitly set
existing_key_row.metadata = {}, so reserved-field reads return real
values instead of MagicMock-returning-MagicMock. Fixes a regression
surfaced by the new reserved-metadata preservation logic.
- test_key_management_endpoints.py: black-format-only changes from
recent edits.
* feat(proxy): add /v1/memory CRUD endpoints with user/team scoping
New LiteLLM_MemoryTable stores user/team-scoped key/value entries with
optional JSON metadata. Value is a String (LLM-readable text) and metadata
is an optional Json? envelope, matching the Letta + mem0 hybrid model so
future structured fields can be added without a schema migration.
Endpoints:
POST /v1/memory - create
GET /v1/memory - list (caller-scoped; admins see all)
GET /v1/memory/{key} - fetch one
PUT /v1/memory/{key} - upsert
DELETE /v1/memory/{key} - delete
Non-admin callers cannot set a user_id/team_id other than their own.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(proxy/memory): omit metadata field when None on create
Prisma's Python client rejects `metadata=None` on a `Json?` field with
"A value is required but not set" — the field must be omitted from the
`data` dict entirely to store SQL NULL. Build the create payload
conditionally in both `create_memory` and the PUT-create branch of
`upsert_memory`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ui): add Memory page to view/manage /v1/memory entries
Adds a new "Memory" sidebar item under Tools so users can see what their
agents have stored. Lists all memories visible to the caller (scoped by
the backend), with a key-search filter, preview column, scope tags, and
view/edit/delete actions. Create modal accepts optional JSON metadata.
- networking.tsx: fetchMemoryList / createMemory / updateMemory / deleteMemory
wired to the /v1/memory CRUD endpoints.
- MemoryView + MemoryEditModal: new antd-based components (per CLAUDE.md:
use antd for new UI, not tremor).
- page.tsx + leftnav.tsx: wire the "memory" route + sidebar entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(memory): add key_prefix filter + promote Memory to AI GATEWAY nav
Backend:
- GET /v1/memory now accepts `key_prefix` for Redis-style namespace
scans (e.g. `?key_prefix=user:`). When both `key` and `key_prefix`
are passed, `key_prefix` wins.
- Prefix filter sits under the visibility filter in the Prisma where
clause, so it can never leak rows across user/team scopes.
- New tests: prefix match, and cross-scope isolation (another user's
`user:*` rows must not appear in the caller's results).
UI:
- Memory moved from a Tools submenu to a top-level AI GATEWAY item
(alongside Agents, MCP Servers, Skills) — it's an API primitive,
not a tool-management surface.
- Search box now drives prefix search, matching the Redis mental
model ("type the namespace, see everything under it").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): enforce unique key per scope by using NULLS NOT DISTINCT
The unique constraint `(key, user_id, team_id)` on LiteLLM_MemoryTable
silently allowed duplicates when user_id or team_id was NULL, because
Postgres treats every NULL as distinct by default (ANSI semantics). A
caller with no team_id could POST the same key three times and get
three rows.
Migration:
1. Dedupe existing rows, keeping the most recent per (key, user_id,
team_id), using `IS NOT DISTINCT FROM` so NULL == NULL.
2. Drop the old unique index.
3. Recreate it with `NULLS NOT DISTINCT` (Postgres 15+).
No code change: POST already returns 409 on unique-violation error
messages — it just wasn't firing before because the constraint didn't
catch the NULL-team case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): make key globally unique, 409 on any duplicate
Switches from the compound unique `(key, user_id, team_id)` to a simple
`key @unique`. The compound form silently allowed duplicates when
user_id or team_id was NULL (Postgres treats each NULL as distinct), so
callers could POST the same key repeatedly. Globally-unique key means
one row per key, period — any duplicate create → 409.
- schema.prisma (×3): `key String @unique`, drop `@@unique(...)`.
- initial add_memory_table migration: unique index on (key) only.
- Remove the now-unused follow-up NULLS NOT DISTINCT migration.
- Endpoint error message simplified ("already exists" — no "for this scope").
- Test fake's create() now enforces global key uniqueness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): full-width layout + user/teams-style columns
- Add `w-full` to the MemoryView outer div so the page fills the
flex-flex-1 container (was collapsing to intrinsic width).
- Replace the combined "Scope" column with separate User ID / Team ID
columns, matching the layout of the Users / Teams pages: ID, Name,
Preview, User ID, Team ID, Updated, Actions.
- IDs render with a truncated mono label + copy-to-clipboard button,
same pattern as view_users.
- Detail drawer now shows Memory ID / User ID / Team ID as separate
fields instead of stacked color tags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): use clean MCP-style ID pill, drop copy icons
The ID / User ID / Team ID columns showed a mono text blob with a
copy-to-clipboard icon next to each value — too busy compared to the
MCP Servers page. Swap the renderer for MCP's pill style:
- Truncated mono ID inside a blue Tailwind pill
(`font-mono text-blue-600 bg-blue-50 ... rounded-md border`).
- No copy icon. Full ID surfaces via tooltip.
- ID column is a button that opens the detail drawer on click;
user/team ID pills are static (not clickable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): address greptile review feedback
Addresses 5 greptile findings (3/5 → higher confidence target):
1. Identity-less orphan rows (P1): non-admin callers with no user_id AND
no team_id could create rows that the visibility filter would never
match again. Now rejected up front with 400 — caller must authenticate
with a scoped key or act as PROXY_ADMIN.
2. Upsert race returning 500 (P1): PUT's check-then-create isn't atomic;
a concurrent writer could slip a row in between the 404-check and the
create call. Now catch unique-violation on create, re-read, and fall
through to update — PUT stays idempotent. If the conflicting row
belongs to a different scope, surface a 409 instead of 500.
3. PUT-create scope inconsistency (P2): PUT's create branch always used
the caller's own user_id/team_id, so admins couldn't bootstrap rows
scoped elsewhere via PUT (only POST). Now PUT-create calls the shared
`_resolve_scope()` helper, matching POST semantics.
4. Stale schema comment (P2): schema said "Keyed by (key, user_id,
team_id)" but `key` is globally unique. Updated all three schema
copies to reflect the actual design.
5. UI silently truncated at 200 (P2): MemoryView fetched pageSize=200
with no load-more. Swapped to real server-side pagination driven by
`data.total`; page size is now 50 and the pager is a real AntD
control.
Also extracts a shared `_resolve_scope()` helper and `_is_unique_violation()`
from create_memory so POST and PUT don't drift on the scope/error logic.
Tests: +3 new (identity-less 400, PUT admin bootstrap, PUT race →
update), 18/18 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): typed Prisma error + explicit-null metadata on PUT
Two more greptile threads from the last review:
- Unique-violation detection was string-matching "Unique"/"UniqueViolation"
in the exception message, fragile across Prisma/driver versions. Now
check the typed error `code == "P2002"` first, with string fallback.
- PUT could not distinguish "metadata omitted" from "metadata: null" —
both parsed as `None`, so callers had no way to clear stored metadata.
Switch to Pydantic v2's `model_fields_set` to tell which fields the
caller actually sent; explicit null now clears the column.
New tests:
- explicit null clears metadata
- omitted metadata preserves existing value
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): send explicit null when user clears metadata
Addresses the remaining P1 from the last greptile review:
When the edit modal's metadata textarea was cleared and saved,
`metadataParsed` stayed `undefined`, `JSON.stringify` dropped the key
entirely, and the backend's `model_fields_set` guard therefore left
the stored metadata untouched — UI showed success but nothing changed.
Now: empty textarea on edit → send explicit `null` so the backend
sees `metadata` in `model_fields_set` and clears the column.
Empty textarea on create still maps to `undefined` (field omitted)
to avoid Prisma's `Json? = None` quirk on insert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): preserve slashes in key path encoding
The backend route `/v1/memory/{key:path}` supports keys with slashes,
but `encodeURIComponent` encoded `/` as `%2F`. Some proxies (nginx
default, CloudFlare, AWS ALB) reject or re-decode `%2F` mid-flight,
so UI update/delete calls on slash-containing keys could fail or
silently misroute.
New helper `encodeMemoryKeyForPath` splits by `/`, URL-encodes each
segment, then rejoins with literal `/`. Every other unsafe char
(spaces, `?`, `#`, `%`) stays encoded per-segment; slashes stay as
path delimiters, matching what the `:path` converter expects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ui/memory): drop misleading client-side column sorters
With server-side pagination, client sorters on `key` and `updated_at`
only reorder the current page while pretending to sort the full
dataset — users would see "sorted by name" but only the visible 50
rows would actually be sorted.
Remove the sorters. The backend already returns rows in
`updated_at DESC` order (sensible default for a memory view), and
users can narrow the result with the key-prefix filter.
Greptile also flagged missing `@@map` on the new model as a
"consistency" issue, but only 1 of 59 tables in this repo uses
`@@map` — the dominant pattern is to rely on Prisma's default
(model name == table name). Skipping that finding as a
false-positive on convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): compose visibility + key filters via explicit AND
Greptile P1 (filter-fragility): `where.update(vis)` was semantically
correct today, but dict-merging by key meant any future visibility
filter that grew a new top-level "OR" would silently clobber the
existing key filter.
Compose explicitly instead:
where = {"AND": [key_filter, vis]}
Applied to both `list_memory` and `_find_memory_for_caller`. When
either side is empty (admin has no visibility filter; list has no
key filter), skip the wrapper and use the non-empty side directly
to keep the generated SQL clean.
Test fake's `_matches` now understands top-level `AND` too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(ui/memory): wrap write helpers with react-query useMutation
Previously the Memory view read via `useQuery` but called the raw
create/update/delete fetch helpers directly in handlers, tracking
loading state with a local `submitting` flag and invalidating state
via `refetch()`. That mixes two concerns:
- it skips react-query's mutation state (isPending / isError / isSuccess)
- `refetch()` only retouches the currently-mounted query instance, not
other cached pages, so navigating back to an older page could show
stale rows
Switch the three write paths to `useMutation`:
- `createMutation`, `updateMutation`, `deleteMutation` — each owns
the mutation fn, success toast, and error toast.
- Success handlers invalidate the whole `["memoryList", ...]` prefix
via `queryClient.invalidateQueries`, so every cached page refetches
(pagination + filter-aware).
- Refresh button now invalidates instead of `refetch()`, keeping all
behavior consistent.
- handleSave/handleDelete become thin adapters that call `.mutateAsync`;
their errors are swallowed locally since the mutation's onError has
already surfaced the toast.
Also tightened the edit modal's key-field tooltip to reflect the
actual global-unique semantics (was "Unique per user/team scope").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): close cross-user write gap + sanitize 500 errors (Veria)
Addresses two Veria findings:
**High — cross-user memory tampering via team membership.** The
visibility filter uses an OR (`user_id == caller OR team_id == caller`)
so team members can SEE each other's team-scoped rows. That's
intentional for list/get. But because PUT/DELETE used the same filter
to find the target row, any team member could overwrite or delete a
teammate's *personal* row whenever both `user_id` and `team_id` were
stamped on it — broader visibility was being silently treated as
broader authority.
New `_assert_write_access(row, caller)` enforces ownership for
mutations. Non-admin rules:
- The row's `user_id` must match the caller (personal ownership), OR
- The row has no `user_id` and its `team_id` matches the caller's
team (a "pure team row" intended for shared writes).
Admins bypass the check. The same gate runs in PUT (both regular
and post-race-recovery branches) and DELETE.
**Medium — DB internals leaked through 500 detail.** Every `except`
block was raising `HTTPException(500, detail=str(e))`, which surfaces
Prisma error strings (table/column names, host:port, error class
names) to API callers. New `_internal_error()` helper logs the real
exception server-side and returns a generic, caller-safe `detail`.
Applied to create, list, upsert (general fallthrough), and delete.
Also tightened the race-recovery 409 message to drop the "in a
different scope" wording — the caller never needs to know whose
scope it lives in.
Tests (+5):
- teammate cannot overwrite personal row → 403
- teammate cannot delete personal row → 403
- teammate CAN modify pure team row (no user_id stamped) → 200
- admin bypasses write-auth → 200
- 500 response never echoes Prisma internals (table/host/class names)
25/25 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(memory): require team admin to modify pure team rows
Tightens the write-authorization rule for "pure team rows" (rows with
no user_id stamped, only team_id) to match the pattern used by
team-management endpoints (`_is_user_team_admin` + `_is_user_org_admin_for_team`):
- Plain team members can READ team rows via the OR visibility filter
(intentional, unchanged).
- Only PROXY_ADMIN, team admins of the row's team_id, or org admins
for the team's organization may MODIFY them. Plain members get 403.
`_assert_write_access` is now async and takes the prisma_client so it
can fetch the team and run the existing `_is_user_team_admin` /
`_is_user_org_admin_for_team` helpers from
`litellm.proxy.management_endpoints.common_utils`. The org-admin path
is best-effort: it calls `get_user_object`, which depends on the
proxy_server module being initialized, so any exception there is
treated as "not an org admin" rather than crashing the request.
Tests:
- team admin can modify pure team row → 200
- plain team member cannot modify pure team row → 403
- plain team member cannot delete pure team row → 403
Updates the test fake to add a tiny `litellm_teamtable.find_unique`
implementation and a `_make_team(team_id, admin_user_ids=[...])`
helper.
27/27 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: mypy + UI page-metadata sync for memory page
Two CI failures:
1. mypy: `_find_memory_for_caller` had `key_filter` inferred as
`dict[str, str]` (literal type) and the conditional `{"AND": [key_filter, vis]}`
returned `dict[str, list[...]]`, so the join site failed
`dict-item` typing. Annotate both intermediates as `dict` so mypy
widens the value type.
2. UI test (`page_utils.test.ts > should have descriptions for all
pages`): every leftnav entry must have a description in
`page_metadata.ts`, and `memory` was missing. Added a one-line
description, matching the style of neighboring entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449)
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro
Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:
- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
per 1M input/output/cached input
Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.
No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.
Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields
* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants
gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.
Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.
Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
* fix(schema): close LiteLLM_MemoryTable model brace dropped during merge
The rebase against `litellm_internal_staging` (which added
`LiteLLM_AdaptiveRouterState` / `LiteLLM_AdaptiveRouterSession`) left
the closing brace of `LiteLLM_MemoryTable` missing in all three
schema copies — the next model declaration ended up parsed as a field
of the memory table, surfacing as the CI prisma error:
error: This line is not a valid field or attribute definition.
--> schema.prisma:1250
|
1249 | // Per-(router, request_type, model) Beta posterior for the adaptive router.
1250 | model LiteLLM_AdaptiveRouterState {
Add the missing `}` (and the standard blank line) after the memory
table's `@@index([team_id])` in `schema.prisma`,
`litellm/proxy/schema.prisma`, and
`litellm-proxy-extras/litellm_proxy_extras/schema.prisma`.
`prisma generate --schema litellm/proxy/schema.prisma` now runs clean;
27/27 memory unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): infer team from DB when JWT has no team and user has one team
- When team_id is unset after JWT auth but the user row has exactly one
team, set team_id, team_object, and team_membership from DB.
- Skip when zero or multiple teams (ambiguous).
- Add parametrized unit tests in test_handle_jwt.py.
Made-with: Cursor
* fix(proxy): JWT single-team DB fallback: catch errors, tests match get_team_object
- Wrap get_team_object + get_team_membership in one try/except; log and skip on failure (stale/missing team id no longer fails auth).
- Parametrize tests: HTTP 404/500, membership error; use side_effect not return_value=None for missing team row.
Made-with: Cursor
* refactor(jwt): extract single-team fallback into _resolve_single_team_fallback helper
Made-with: Cursor
* feat(guardrails): add LLM_AS_A_JUDGE to SupportedGuardrailIntegrations
* feat(types): add EvalVerdict, StandardLoggingEvalInformation; wire eval_information into SpendLogsMetadata
* feat(guardrails): add self-contained llm_as_a_judge guardrail hook
* fix(a2a): filter agent-only litellm_params from acompletion kwargs; pass agent_id into body
* feat(ui): add LLMJudgeFields criteria builder component
* feat(ui): wire LLM-as-a-Judge into add guardrail form
* feat(ui): update EvalViewer — title 'LLM Judge Results', weighted score column, summary row
* fix(ui): wire EvalViewer into LogDetailContent to show LLM judge results on logs page
* fix(guardrails-ui): route llm_as_a_judge to criteria builder step; rename to LiteLLM LLM as a Judge; add litellm logo
* fix(guardrail-viewer): stack lifecycle + eval details vertically to avoid badge overflow in narrow drawer
* fix(guardrail-create): surface config validation errors on create instead of silently orphaning guardrail in DB
* fix(guardrail-registry): hardcode llm_as_a_judge in initializer registry so it loads regardless of package install path
* fix(llm-as-a-judge): fix P1 code quality issues - validate weights/on_failure, guard pre_call, handle multimodal, move imports to module level, fix spurious finally logging
* fix(guardrail_endpoints): use correct PK field in rollback delete and log rollback failure
* fix(llm_as_a_judge): support Pydantic object in _get_litellm_param fallback chain
* fix(LLMJudgeFields): replace @tremor/react Button with antd Button
* fix(llm_as_a_judge): remove dead registry dicts, fix KeyError in prompt builder, set correct status on judge failure
* test(llm_as_a_judge): add unit tests for guardrail hook
* fix(llm_as_a_judge): remove @log_guardrail_information decorator to fix duplicate guardrail_information entries
The decorator and the manual finally block both called add_standard_logging_guardrail_information_to_request_data, producing two entries per request. The decorator also misclassified HTTPException(422) blocks as guardrail_failed_to_respond (it checks for 400). The finally block correctly tracks status throughout, so removing the decorator is sufficient.
* fix(test_gcs_pub_sub): ignore metadata.eval_information in comparison
* fix(test_spend_management): ignore metadata.eval_information in payload comparison
* fix(types/guardrails): add input_type and messages to ApplyGuardrailRequest
* fix(guardrail_endpoints): pass input_type and messages through apply_guardrail endpoint
* fix(guardrail_endpoints): auto-detect post_call guardrails and use input_type=response
* fix(a2a_endpoints): merge agent litellm_params guardrails into data before post_call hooks
* fix(llm_as_a_judge): use float sum with tolerance for weight validation
* fix(guardrail_registry): split long import line for black formatting
* fix(llm_as_a_judge): guard guardrail_name Optional for mypy
* fix(llm_as_a_judge): set guardrail_status=guardrail_intervened when score fails, regardless of on_failure mode
* fix(a2a_endpoints): use try/finally so deferred spend log fires even when guardrail blocks with 422
* fix(litellm_logging): declare _defer_async_logging and _enqueue_deferred_logging on Logging class for mypy
* fix(logging_worker): restore queue.join() in flush() to wait for in-flight callbacks
* fix(vertex passthrough): log :embedContent and :batchEmbedContents responses
* test(vertex passthrough): add unit tests for :embedContent and :batchEmbedContents logging
* fix(vertex passthrough): extract input text from request body for embedContent token counting
* fix(vertex passthrough): add embedContent and batchEmbedContents to TRACKED_VERTEX_ROUTES
* fix(vertex passthrough): detect Google AI Studio URLs in embedContent handler
* test(vertex passthrough): add unit test for Google AI Studio URL embedContent provider detection
* style: black format vertex_passthrough_logging_handler
create_model does not inherit the base class docstring, so once an
extension registered a field the effective class had no description.
The UI renders schema.description as a header paragraph — losing it
broke the 'Configuration for UI-specific flags' text. Pass __doc__
through explicitly and add a regression test.
Extract the admin team-header attachment into a helper so
auth_builder stays under the 50-statement lint threshold; apply
black formatting to the two files flagged on the prior commit.
No behavior change.
Scope the header-driven team fetch to LLM API routes so admin
management routes keep the pre-existing bypass behavior (no
phantom teams, no 404s on mgmt calls). Team context is threaded
onto UserAPIKeyAuth so spend logs, rate limits, and team_models
attribution are correctly applied when admins act on behalf of
a team via x-litellm-team-id.
* fix(proxy): honor object_permission for managed vector store access
* perf(proxy): preload team object_permission on UserAPIKeyAuth
Populate team_object_permission during virtual-key and JWT auth when the
team is loaded, so can_user_access_vector_store uses it in memory first
and only falls back to get_object_permission by id when missing.
Made-with: Cursor
* fix(team_endpoints): auto-add SSO team members to org for proxy admins
* test: proxy_admin vs team_admin security boundary for team→org move
* screenshots: before/after for team-org SSO fix
* fix(team_endpoints): restore staging security features dropped in SSO commit
Co-Authored-By: Ishaan Jaff <ishaan@berri.ai>
* style: black formatting for team_endpoints
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
Temporary MCP OAuth sessions were kept in process-local memory, so on
multi-instance/LB proxy deployments a session created on instance A could
not be found when the follow-up /server/oauth/{server_id}/... request
landed on instance B.
Persist temporary session records to Redis (encrypted with the existing
proxy encryption helpers) as a best-effort L2 cache alongside the current
in-memory L1. Convert get_cached_temporary_mcp_server to async and await
it from the authorize/token/register OAuth endpoints.
Made-with: Cursor
Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.
common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.
Tests cover us/eu for the common_utils helper.
Made-with: Cursor
Sibling tests were mutating litellm.proxy.proxy_server.master_key and
prisma_client with raw setattr. Values leaked across tests in the same
xdist worker, flipping the auth short-circuit in user_api_key_auth and
causing unrelated tests (e.g. test_ui_view_session_spend_logs_pagination)
to return 401 instead of 200.
Replace raw setattr with monkeypatch in the two offending files and add
an autouse conftest fixture that snapshots/restores the known-leaky
module globals for every proxy test.
Two fixes to proxy-db CI:
1. test_realtime_webrtc_endpoints.py's `proxy_app` fixture mutated the
module-global `proxy_server.master_key` without restoring it, leaking
state into any test that shared the same xdist worker. Under
--dist=loadscope with 2 workers (GHA proxy-endpoints), this caused the
google_endpoints tests to fail with "No api key passed in." because
user_api_key_auth saw a set master_key and a missing API key on the
test request. The fixture now saves and restores the original value.
2. Address the Greptile note that the semantic shard design has no
catch-all, so a new test file added to tests/proxy_unit_tests/ without
a matrix entry would silently skip CI. Adds an assert-shard-coverage
job that enumerates test_*.py files and fails the workflow if any are
not referenced by a matrix entry, with a clear message telling the
author which semantic shard to place it in. All proxy-db shards now
depend on this guard.
The mocked async_increment_cache_pipeline is invoked from Router's
deployment_callback_on_success, registered as an async success callback.
Those callbacks are enqueued to GLOBAL_LOGGING_WORKER and run on a
background task, so the mock may not have been called yet when the test
asserts on it. Flush the worker before asserting.
Two independent deflakes:
1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
of 401/403 when earlier tests in the file left proxy-auth globals
(prisma_client, master_key, user_custom_auth, general_settings,
user_api_key_cache) in a state that let invalid tokens pass auth and
fall through to the endpoint's own start_date/end_date validation.
Add an autouse fixture that pins those globals to their import-time
defaults for every test in the file. Harden the assertion to include
response body so future flakes are diagnosable.
2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
on the Redis transaction buffer flushing spend to Postgres. The buffer
uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
and a single global buffer list key. Pointing the proxy at the shared
remote Redis means concurrent CI pipelines contend for the same lock
and can drain each other's buffer into the wrong database. Add a
start_redis reusable command that boots a per-job redis:7-alpine
container (digest-pinned), and switch proxy_spend_accuracy_tests to
REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
isolated per CI run.
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
* feat: add gpt-5.5 to model cost map
Add gpt-5.5 entry with pricing from OpenAI flagship page:
input $5/1M, cached input $0.50/1M, output $30/1M, 272K context.
* test: add gpt-5.5 coverage for model cost map and gpt-5 routing
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
The periodic budget-window reset job filtered keys/teams with
`where={"budget_limits": {"not": None}}`. The prisma-client-python
library does not support null-filtering on `Json?` columns (no
DbNull/JsonNull sentinel — upstream issue #714). The client drops the
`None` value during serialization and the engine rejects the query with
`MissingRequiredValueError: where.budget_limits.not: A value is
required but not set`, so neither the key nor team reset path runs.
Switch those two `find_many` calls to `query_raw` with
`WHERE budget_limits IS NOT NULL`, selecting only the PK and the
`budget_limits` column. Writes still go through the ORM. Add unit tests
covering the expired/unexpired paths for keys and teams, string-encoded
JSON payloads, empty payloads, error isolation between the two paths,
and a regression guard asserting the query still uses `IS NOT NULL`.
Align the Ruby, Node.js, and npm install path with the rest of the
config. Three separate upstream installers were being invoked via
\`curl ... | bash\` or unlocked \`npm install\`:
- RVM's \`get.rvm.io/stable\` installer (mutable upstream script).
Replace with a shallow git clone of the rvm/rvm repo at tag 1.29.12
and verify HEAD matches the published commit SHA before running the
local \`./install\` script. Same pattern already used for the
helm-unittest plugin in .github/workflows/helm_unit_test.yml.
- NodeSource's \`deb.nodesource.com/setup_18.x\` piped into sudo bash.
Replace with a direct download of the Node.js 18.20.8 linux-x64
tarball from nodejs.org, verified against the published
SHASUMS256.txt digest before extraction.
- \`npm install @google-cloud/vertexai @google/generative-ai\` and
\`--save-dev jest\` resolved fresh from the npm registry on every
run. Add \`tests/pass_through_tests/package.json\` with pinned
direct-dep versions and commit the generated package-lock.json, then
switch CI to \`npm ci\` (exact lockfile install, fails on drift).
Also scopes the Ruby+JS test runners to \`tests/pass_through_tests/\`
so they pick up the committed package.json rather than writing
node_modules at repo root.
