Currently the in memory cache uses a hash map which returns references to elements and not deep copies. Since we dont want to enforce this at the interface level, allowing copy creation at call sites necessary. This minimizes performane overhead.
- Updated `get_key_object` to return a deep copy of cached UserAPIKeyAuth objects.
- Modified `update_valid_token_with_end_user_params` to create a deep copy of the valid token before applying updates.
- Added unit tests to verify that original cached tokens remain unmodified after updates.
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
test_virtual_key_max_budget_alert_check_per_key_overrides_global asserted
override semantics but the implementation does additive merge. Renamed test
and updated assertion to match: per-key and global thresholds are unioned,
not replaced.
- Guard empty recipients in _handle_multi_threshold_max_budget_alert:
log warning and skip instead of falling through to old path error loop
- Widen max_budget_alert_emails type to Dict[str, Union[str, List[str]]]
to match _parse_email_list runtime behavior (accepts comma-separated strings)
- Pre-filter asyncio.create_task with min threshold check to avoid
unnecessary task allocation on every request when spend is below
all configured thresholds
- Add `default_key_max_budget_alert_emails` litellm_settings config as
global fallback for all virtual keys (per-key metadata takes priority)
- Fix crash when key has no user_id/user_email by passing recipient email
to _get_email_params (same pattern as team soft budget path)
- Use owner email for greeting, falling back to key_alias or token
- Rename setting from default_max_budget_alert_emails to
default_key_max_budget_alert_emails for clarity
Users can set metadata.max_budget_alert_emails as a JSON map of threshold
percentages to email recipients on virtual keys. When configured, the email
handler loops over each threshold, checks per-threshold dedup cache, and
sends to the configured recipients (auto-including the key owner's email).
When no map is set, the existing single 80% threshold behavior is preserved
unchanged. Teams support is out of scope for this v0.
Close three variant bypasses adjacent to VERIA-28 found during post-fix
variant audit:
1. _guardrail_modification_check had the same isinstance(dict) bypass
Veria-AI just flagged on the pre-call strip. A caller sending
`{"metadata": "{…}"}` as a JSON-encoded string (multipart/form-data
or extra_body) skipped the guard, got parsed to dict downstream, and
reached guardrail logic with bypass flags intact. Coerce strings via
safe_json_loads before evaluating.
2. The allow_client_tags strip only covered body metadata.tags and
litellm_metadata.tags — caller-supplied tags arriving via the
x-litellm-tags header or root-level data["tags"] bypassed it. Gate
add_request_tag_to_metadata's result on the same flag.
3. requester_metadata was deepcopied BEFORE the strip, so attacker
injections (user_api_key_metadata shadows, disallowed tags,
_pipeline_managed_guardrails) persisted in the snapshot. The PANW
guardrail (and any future consumer) trusting requester_metadata
would see forged values. Move the deepcopy to after the strip.
Regression tests added for each.
Per VERIA-28's secondary recommendation. The existing check only gated
metadata.guardrails. User-supplied values for disable_global_guardrails
(plural and the original singular typo variant) and opted_out_global_guardrails
are already silently ignored by _get_admin_metadata at read time, but the
silent-ignore makes diagnosis confusing and relies on one specific read
site catching them.
Reject at auth time with a 403 when any of:
- guardrails list (existing)
- disable_global_guardrails (new)
- disable_global_guardrail (new — historical singular-key variant)
- opted_out_global_guardrails (new)
are present in metadata, litellm_metadata, or at the request root, and the
caller's team lacks can_modify_guardrails. Defense in depth: the strip at
the pre-call layer still runs; this check fails loudly one layer earlier
so operators see an explicit 403 rather than a silent-ignore.
* fix: remove leading space from license public_key.pem
PEM must begin with -----BEGIN; a leading ASCII space breaks
cryptography.load_pem_public_key on older cryptography (e.g. 41.x),
causing OpenSSL no start line / deserialize errors.
Made-with: Cursor
* test: assert license public_key.pem loads as valid PEM
Regression guard for leading whitespace before -----BEGIN, which breaks
load_pem_public_key on older cryptography (e.g. 41.x).
Made-with: Cursor
Allow JWT tokens matching routing_overrides to use OAuth2 introspection without enabling global OAuth2 while keeping OAuth2 routing limited to LLM/info routes. Add regression coverage for management-route boundary and tighten opaque-token assertions; update docs to reflect selective-mode route scope.
Made-with: Cursor
* feat: multiple concurrent budget windows per API key and team (#24883)
* feat(proxy): add BudgetLimitEntry type and wire budget_limits into key/team models
* feat(schema): add budget_limits Json column to VerificationToken and TeamTable
* feat(migrations): add migration for budget_limits column on keys and teams
* feat(keys): initialize budget_limits windows with reset_at on key create/update
* feat(teams): initialize budget_limits windows with reset_at on team create/update
* feat(auth): add _virtual_key_multi_budget_check and _team_multi_budget_check
* feat(auth): call multi-budget checks from common_checks for keys and teams
* feat(proxy): increment per-window Redis spend counters after each request
* feat(budget): reset individual budget windows on schedule via reset_budget_job
* feat(ui): add hourly option to BudgetDurationDropdown
* feat(ui): add budget_limits field to KeyResponse type
* feat(ui): add Budget Windows editor to key edit view
* feat(ui): add Budget Windows editor to create key form
* fix(proxy): strip budget_limits=None before Prisma upsert to fix login 500
Prisma rejects nullable JSON fields (Json? without @default) when passed as
Python None — it needs the field omitted entirely so the DB stores NULL via
the column's nullable constraint. This was breaking /v2/login because the UI
session key creation path hit the upsert with budget_limits=None.
* ui(key-edit): use antd InputNumber+Button for budget windows, add reset hints
* ui(create-key): use antd InputNumber+Button for budget windows, add reset hints
* docs(users): add multiple budget windows section with API + dashboard walkthrough
* fix: BudgetExceededError returns HTTP 429 instead of 400
- Add status_code=429 to BudgetExceededError class
- auth_exception_handler hardcoded code=400 → code=429
* fix: no-op else branch in multi-budget auth checks causes KeyError
- BudgetLimitEntry objects must be coerced via model_dump() not left as-is
- Move _virtual_key_multi_budget_check into common_checks (was asymmetric
with _team_multi_budget_check which already lived there)
* fix: len() on JSON string returns char count not window count
Guard with isinstance check + json.loads() before iterating per-window
Redis counters in increment_spend_counters
* fix: silent except:pass hides Redis reset failures in reset_budget_windows
Log Redis counter reset failures as warnings so they are observable
* test: add unit tests for multi-budget window enforcement
5 tests covering: no budget_limits passes, under budget passes,
over hourly window raises 429, over monthly window raises 429,
BudgetLimitEntry objects coerced without KeyError
* fix: key per-window counters stable across reorders (duration key, not index)
* fix: team+key per-window spend increments use duration key, not index
* fix: budget window reset uses duration key; log failures instead of swallowing
* refactor: extract BudgetWindowsEditor to shared component
* refactor: key_edit_view imports BudgetWindowsEditor from shared component
* refactor: create_key_button imports BudgetWindowsEditor from shared component
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(reset_budget_job): extract _reset_expired_window helper to fix PLR0915 too many statements
* feat(skills): Skills Registry & Hub — register skills, browse in AI Hub, public skill hub (#25118)
* feat(skills): add domain and namespace fields to plugin types
* feat(skills): store and return domain/namespace inside manifest_json
* feat(skills): add /public/skill_hub endpoint for unauthenticated access
* feat(skills): whitelist /public/skill_hub from auth requirements
* feat(skills): add domain, namespace to Plugin and RegisterPluginRequest types
* feat(skills): smart URL parser — paste github URL, auto-detect source type and name
* feat(skills): replace enable toggle with Public badge, make rows clickable
* feat(skills): add skill detail view with Overview and How to Use tabs
* feat(skills): add MakeSkillPublicForm modal for publishing skills to the hub
* feat(skills): rename panel to Skills, wire in skill detail view on row click
* feat(skills): add skill hub table columns — name, description, domain, source, status
* feat(skills): add SkillHubDashboard with stats row, domain dropdown filter, and table
* feat(skills): add Skill Hub tab to AI Hub with Select Skills to Make Public button
* feat(skills): move Skills to top-level nav item directly under MCP Servers
* feat(skills): add skillHubPublicCall and NEXT_PUBLIC_BASE_URL support
* feat(skills): add Skill Hub tab to public AI Hub page
* feat(skills): add skills page routing in main app router
* feat(skills): add /skills page route
* chore: update package-lock after npm install
* docs(skills): add Skills Gateway doc page with mermaid architecture diagram
* docs(skills): add Skills Gateway to sidebar under Agent & MCP Gateway
* docs(skills): add loom walkthrough video to Skills Gateway doc
* chore: fixes
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* fix(proxy): enforce key-level model allowlist for custom auth
custom_auth_run_common_checks only runs common_checks (team/user/project model checks).
Custom auth now also enforces key-level model restrictions via can_key_call_model.
Move the custom-auth key-access regression tests to test_user_api_key_auth.py and keep test_custom_auth_end_user_budget.py focused on end-user budget behavior.
Made-with: Cursor
* fix(proxy): gate custom-auth key model checks behind opt-in
Keep key-level model allowlist enforcement in custom auth behind `custom_auth_run_common_checks` to preserve backwards compatibility, and update tests to verify default non-enforcement and opt-in enforcement behavior.
Made-with: Cursor
* test(proxy): isolate custom auth default check from shared settings state
Patch `proxy_server.general_settings` to an empty dict in the default custom-auth key-access test so it remains deterministic under shared module state.
Made-with: Cursor
* test(proxy): strengthen custom auth post-check assertions
Tighten custom auth regression tests by asserting exact can_key_call_model args and remove an unused common_checks mock from the default behavior path.
Made-with: Cursor
* fix(agentcore): parse A2A JSON-RPC responses in AgentCore provider
* fix(prompt-templates): ensure_alternating_roles handles tool-call chains
* feat(auth): add JWT claim routing overrides for OAuth2 validation
Made-with: Cursor
* docs(auth): document JWT-to-OAuth2 routing overrides
Add generic docs for running JWT and OAuth2 together, including routing_overrides YAML examples and list-based selector behavior for iss/client_id/aud.
Made-with: Cursor
---------
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
PR #24755 renamed `azure_api_key_header` to `AZURE_AI_API_KEY_header` in
the test file but did not update the actual function signatures of
`get_api_key()` and `_user_api_key_auth_builder()`, causing TypeError
on all affected test cases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL to both async_set_cache
calls in sync_user_role_and_teams for consistency with all other user cache
writes. Add 3 tests covering cache invalidation on role change, team change,
and no-op when nothing changes.
Add None-token test cases to both proxy_unit_tests and test_litellm
to cover the guard added in the previous commit. Also add -> bool
return type annotation to is_jwt().
Budget checks on API keys, teams, and team members were not enforced in
multi-pod deployments because user_api_key_cache is intentionally
in-memory-only. Each pod tracked spend independently, so with N pods
the effective budget was N × max_budget.
Introduces a separate spend_counter_cache (DualCache wired to
redis_usage_cache) with atomic increment/read helpers:
- increment_spend_counters(): awaited in cost callback (not create_task)
to update both in-memory and Redis before the next auth check
- get_current_spend(): reads Redis first (cross-pod authoritative),
falls back to in-memory, then to cached object .spend from DB
Budget check functions (_virtual_key_max_budget_check,
_team_max_budget_check, _check_team_member_budget) now read spend via
get_current_spend() instead of cached object .spend fields.
When Redis is not configured, falls back to in-memory-only counters
(same as current single-instance behavior).
Fixes#23714
* feat(redis): add circuit breaker to RedisCache to fast-fail when Redis is down (#24181)
* feat(redis): add circuit breaker env var constants
* feat(redis): add RedisCircuitBreaker and apply guard decorator to all async ops
* fix(dual_cache): fall back to L1 instead of re-raising on Redis increment failures
* test(caching): add circuit breaker unit tests
* fix(redis): fast-fail concurrent HALF_OPEN probes — only one probe at a time
* fix(dual_cache): return None fallback when in_memory_cache is absent and Redis fails
* test(caching): add regression tests for HALF_OPEN concurrency and None fallback
* Fix blocking sync next in __anext__ (#24177)
* Fix blocking sync next
* Update tests/test_litellm/litellm_core_utils/test_streaming_handler.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix PEP 479 regression in __anext__ sync iterator exhaustion
asyncio.to_thread re-raises thread exceptions inside a coroutine, where
PEP 479 converts StopIteration to RuntimeError before any except clause
can catch it. Add _next_sync_or_exhausted() module-level helper that
catches StopIteration in the thread and returns a sentinel instead, then
raise StopAsyncIteration in the coroutine.
Also rewrites the non-blocking test to use asyncio.gather() instead of
asyncio.create_task() (which returned None on Python 3.9 / pytest-asyncio
in CI), and adds an exhaustion regression test that drains the wrapper
fully and asserts no RuntimeError leaks out.
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: add git-subdir source type to claude-code/plugins API (#24223)
Support a third plugin source type `git-subdir` alongside the existing
`github` and `url` types, as documented in the official Claude Code
plugin marketplaces spec.
New format: {"source": "git-subdir", "url": "...", "path": "subdir/path"}
- Validates url and path fields are present and non-empty
- Rejects absolute paths, '..' segments, backslashes, and percent-encoded
traversal sequences (including double-encoded variants via regex check)
- Extracts path validation into _validate_git_subdir_path() helper
- Updates Pydantic field description to document all three source types
- Adds isValidUrl() check for url/git-subdir source types in the UI form
- Adds "Git Subdir" option to the UI form with a required Path field
- Adds unit tests covering success, update, missing/empty fields,
path traversal variants, and unknown source type
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* [FEAT] add extract_header and extract_footer to Mistral OCR supported params (#24213)
* docs: add git-subdir source type to claude-code plugin marketplace docs (#24289)
* fix(ui): swap J/K keyboard navigation in log details drawer (#24279) (#24286)
J should navigate down (next) and K should navigate up (previous),
matching vim/standard conventions.
* fix: use async_set_cache in user_api_key_auth hot path (#24302)
* fix: use async_set_cache in auth hot path to avoid blocking event loop
* test: assert no blocking set_cache call in _user_api_key_auth_builder
* test: broaden blocking call check to all sync DualCache methods
* test: fix regression test to actually catch blocking cache calls
* fix: ruff lint unused variable + UI build MessageManager error
- litellm/caching/redis_cache.py: remove unused variable 'e' in circuit
breaker exception handler (F841)
- add_plugin_form.tsx: use MessageManager.error() instead of undefined
message.error() for git URL validation
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add REDIS_CIRCUIT_BREAKER env vars to config_settings reference
Add REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD and
REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT to the environment variables
reference table so test_env_keys.py passes.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Vincenzo Barrea <manamana88@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Robert Kirscht <rkirscht242@gmail.com>
Co-authored-by: Imgyu Kim <kimimgo@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
- Change `if team_limit:` to `if team_limit is not None:` in both
get_key_model_rpm_limit and get_key_model_tpm_limit so that an
explicitly-empty team rate-limit map ({}) is returned as-is instead
of silently falling through to deployment defaults (P1 fix).
- Replace the bare `int()` list comprehension in _get_deployment_default_limit
with a loop that catches ValueError/TypeError so malformed config strings
do not raise an unhandled exception during request handling (P2 fix).
- Add corresponding unit tests for both edge cases.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Replace bare _get_deployment_default_tpm/rpm_limit calls in the
async_log_success_event condition with get_key_model_tpm/rpm_limit
(model_name=model_group). The higher-level getters short-circuit on
key/team metadata hits before ever reaching the router, so requests
that don't use deployment defaults incur no extra router lookup. Remove
the now-unused bare helper imports.
Also fix invalid `int = None` type hints in test helper signatures
to `Optional[int] = None`.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Use min() across all matching deployments instead of first-wins when
resolving default_api_key_tpm/rpm_limit for a model group, so
load-balanced setups with different per-deployment limits always apply
the most conservative value
- Replace the global SensitiveDataMasker non_sensitive_overrides change
with a targeted excluded_keys set at the remove_sensitive_info_from_deployment
call site, avoiding unintended suppression of other fields
- Update the v1 parallel request limiter to pass model_name to
get_key_model_tpm/rpm_limit so deployment defaults apply there too
- Add 4 tests covering multi-deployment min semantics
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Adds `default_api_key_tpm_limit` and `default_api_key_rpm_limit` to
`GenericLiteLLMParams` so operators can set per-deployment rate limit
defaults in config.yaml. When a key has no model-specific tpm/rpm limit
configured, the proxy falls back to these deployment defaults (Case 2 in
spec). Key-level limits always take priority (Case 1).
- Extends `get_key_model_tpm_limit` / `get_key_model_rpm_limit` with a
`model_name` param and a priority-4 deployment-default fallback
- Passes `model_name=requested_model` in the parallel request limiter so
the fallback is triggered at enforcement time
- Adds `"limit"` to `SensitiveDataMasker` non-sensitive overrides so
`*_limit` fields are not masked in `/model/info` responses
- Adds 17 unit tests covering both spec cases and the `/model/info` path
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Only remove wildcard path from openai_routes when the route entry has
type="subpath", avoiding accidental removal when two endpoints share
the same base path but differ in include_subpath
- Clean up _registered_pass_through_routes in the test finally block to
prevent stale entries from polluting subsequent tests on failure
- Add dedup guard for base path registration (prevents unbounded list
growth on config reload)
- Clean up base path and wildcard path from openai_routes when an
endpoint is removed via remove_endpoint_routes
- Rewrite test to exercise initialize_pass_through_endpoints directly,
covering registration, dedup on reload, and cleanup on removal
When a pass-through endpoint has both auth=true and include_subpath=true,
non-admin users got 401 errors on subpath requests because only the base
path was registered in openai_routes. Now the wildcard path is also
registered so the auth check recognizes subpath requests as LLM API routes.
Also fixes pre-existing pyright error where logging_obj was possibly
unbound in the except block.
Keep both sets of tests: upstream's OAuth2 token injection test and
our case-insensitive tool matching tests. Use upstream's version of
the bedrock output_config test (more comprehensive).
