* Litellm team model group name routing fix (#25148)
* fix(team-routing): use deterministic team model group names
Use a deterministic internal model_name for team-scoped deployments so sibling deployments with the same public model share a routing group. This makes team alias writes idempotent and preserves multi-deployment failover/load balancing behavior.
Made-with: Cursor
* fix(team-routing): keep team model routing on public names
Remove team model_alias rewrites and resolve team deployments by team_public_model_name with team_id so sibling deployments stay in the routing candidate pool, with explicit logs showing candidate selection before load balancing.
Made-with: Cursor
* chore(team-routing): remove temporary candidate pool logs
Remove temporary fire-emoji router logs used for local verification while keeping team sibling deployment routing behavior unchanged.
Made-with: Cursor
* fix(router): address Greptile review comments
- Add None guard for original_model_name in _add_team_model_to_db
- Remove stale old public name when renaming team model
- Add comment clarifying team deployment early-return priority
Made-with: Cursor
* fix(router): address remaining Greptile P0/P1 issues
- Update map_team_model test to expect public name return
- Only remove old public name if no sibling deployments use it
Made-with: Cursor
* fix(router): address Greptile P1/P2 performance issues
- Guard against llm_router=None to prevent silent deletion
- Add O(1) team_model index to avoid O(n) scan on every team request
Made-with: Cursor
* fix(router): prevent cross-team deployment leakage in fallback path
Guard should_include_deployment fallback to only return deployments
matching the requested team_id, preventing public-name collisions
from leaking deployments across teams
Made-with: Cursor
* fix(management): query DB directly for sibling deployments on rename
- Add clarifying comments to test assertions
- Query prisma DB instead of in-memory router to avoid stale state
- Prevents incorrect deletion of old public name when siblings exist
Made-with: Cursor
* fix(router): guard None model_info and deduplicate team index logic
- Guard against None model_info in sibling deployment check
- Extract _update_team_model_index helper to eliminate duplication
Made-with: Cursor
* fix(routing): prevent stale model_aliases from interfering with team routing
- Skip model_aliases rewrite if model resolves to team deployments
- Add test coverage for sibling-preservation branch
- Update MockPrismaClient to support sibling deployment scenarios
Made-with: Cursor
* perf(routing): optimize team model checks and improve test coverage
- Use O(1) team index lookup instead of map_team_model in alias guard
- Fix MockPrismaClient to validate where clause filters
- Add comment explaining DB query trade-off for team deployments
Made-with: Cursor
* fix(routing): address state consistency and type safety issues
- Check alias target pattern to detect stale team aliases
- Fix PrismaClient type annotation to Optional
- Eliminate in-place mutation in index update logic
Made-with: Cursor
* Fix greptile comments
* Fix greptile comments
* Fix greptile comments
* Fix greptile comments
* Fix greptile comments
* Fix code qa issues
* Fix greptile reviews and mock test
* Fix greptile reviews and mock test
* Fix greptile reviews and mock test
* fix(router): address Greptile P1/P2 review comments
- Add deduplication guard in _update_team_model_index to prevent duplicate indices
- Add wildcard comment in map_team_model for clarity
- Add monkeypatch to test_team_alias_stale_bypass_disabled_by_default for determinism
- Extract _get_team_deployments helper to centralize DB access pattern
- Add clarifying comments for team_public_model_name assignment ordering
Made-with: Cursor
* fix(router): address remaining Greptile review comments
- Cache LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS at module level to avoid hot-path secret lookups
- Add clarifying comments for should_include_deployment team isolation logic
- Add negative assertion for update_team.assert_not_called() in test
- Add docstring clarification for _get_team_deployments helper pattern
- Add explicit assertion message in test_get_model_list_alias_optimization
Made-with: Cursor
* fix(router): address final Greptile P1/P2 comments
- Reorder team_public_model_name assignment to happen before model_name mutation for clarity
- Add comment explaining no-rename fast-exit case in _update_existing_team_model_assignment
- Add comment explaining final patch_data.model_name = None applies to all code paths
Made-with: Cursor
* fix(tests): reset module-level cache in stale alias bypass tests
Reset _ENABLE_TEAM_STALE_ALIAS_BYPASS to None in both test functions
to ensure test isolation and prevent ordering-dependent failures
Made-with: Cursor
* feat(router): add order-based fallback so higher order deployments are tried on failure
When order=1 deployments fail, the router now automatically tries order=2,
then order=3, etc. before falling through to external fallbacks. This removes
the need for enable_pre_call_checks and makes order work as a true priority-based
fallback chain within a model group.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(router): address Greptile P0/P1 review comments on order fallback
- P0: Skip order-based fallback for ContextWindowExceededError and
ContentPolicyViolationError so their dedicated fallback handlers run
- P1: Read _target_order from kwargs to skip already-tried order levels,
preventing wasteful retries and exponential retry storms
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(router): add order-based fallback so higher order deployments are tried on failure
When a request to an order=1 deployment fails, the router now
automatically tries order=2, order=3, etc. before falling through to
external fallbacks. Works for all error types (429, 404, connection
errors). Requires enable_pre_call_checks=True.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(router): handle non-standard fallback formats with order-based fallback
When fallbacks use non-standard formats (e.g. ["claude-3-haiku"] or
[{"model": "...", "messages": [...]}]), detect them with
_check_non_standard_fallback_format and pass them through directly
instead of trying to parse with get_fallback_model_group which only
handles the standard dict-keyed format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: remove enable_pre_call_checks requirement from order docs
Order-based routing and fallback work without enable_pre_call_checks
in the current code. Remove the stale requirement from both doc files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix tests
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* Potential fix for code scanning alert no. 4373: Log Injection
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry
Missing unversioned entry causes cost tracking to return $0.00 for
all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI
Claude models have both versioned and unversioned entries.
* fix(router): skip misleading tags error when no candidates (e.g. cooldown)
Return early from get_deployments_for_tag when healthy_deployments is empty so
tag-based routing does not raise no_deployments_with_tag_routing after cooldown
filters all deployments. Adds regression test.
Made-with: Cursor
* feat(oci): add embedding support and update model catalog
- Add OCIEmbeddingConfig for OCI GenAI embedding models
- Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini)
- Add 8 embedding models (Cohere embed v3.0, v4.0)
- Update documentation with embedding examples
- Update pricing for all new models
* test(oci): add unit tests for OCI embedding support
- 17 unit tests covering OCIEmbeddingConfig
- Tests for URL generation, param mapping, request/response transform
- Tests for model pricing JSON completeness
* style(oci): format with black and ruff
* fix(oci): correct embedding request body format
OCI embedText API expects inputs, truncate, and inputType at the
top level of the request body, not nested under embedTextDetails.
Fixed transformation and updated tests accordingly.
Verified with real OCI API: 3/3 embedding models working.
* docs: clarify tag routing early return and test intent
Made-with: Cursor
* fix(oci): address code review findings from Greptile
- P1: Fix signing URL mismatch with custom api_base by accepting
api_base parameter in transform_embedding_request
- P2: Remove encoding_format from supported params (OCI does not
support it, was silently dropped)
- P2: Raise ValueError for token-array inputs instead of silently
converting to string representation
- Add test for token-list rejection
* fix(mcp): add STS AssumeRole support for MCP SigV4 authentication
MCPSigV4Auth only supported static AWS credentials or the boto3 default
credential chain. Production Kubernetes environments typically authenticate
via IAM role assumption (sts:AssumeRole), which was not possible.
Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth
stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole
to obtain temporary credentials before signing requests. Explicit keys,
if also provided, are used as the source identity for the STS call;
otherwise ambient credentials (pod role, instance profile) are used.
* fix: stop logging credential values and add missing redaction patterns
Replaces raw credential values in debug/error log messages with
boolean presence checks or type names. Adds PEM block, GCP token,
JWT, SAS token, and service-account blob patterns to the redaction
filter. Fixes private_key pattern to capture full PEM blocks instead
of stopping at the first whitespace.
Addresses: Vertex AI credential JSON (including RSA private key)
being logged to stderr on health check failures.
* fix: log only field names for UserAPIKeyAuth, not full object
* style: apply black formatting to experimental_mcp_client/client.py
* style: fix black/isort formatting and mypy error in proxy_server.py
- Fix black formatting in experimental_mcp_client/client.py (done in prev commit)
- Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py
- Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py
* fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue
When test_proxy_cli.py tests run before test_check_migration.py in the same
xdist worker, litellm.proxy.db.check_migration is already in sys.modules.
Patching litellm._logging.verbose_logger has no effect on the already-bound
reference. Patch the correct target (check_migration.verbose_logger) and
import the module before patching so the order doesn't matter.
* fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature
---------
Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com>
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Tests cover: matching deployments, wrong team_id filtering, string-encoded
model_info, empty results, invalid model_info, and mixed deployment filtering.
Also updates MockPrismaClient.find_many to support the new startswith query.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
prisma-client-py 0.11.0 does not support JSON path filtering (path/equals)
on Json fields. Replace with model_name prefix query + Python-side team_id
confirmation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Follow-up to the previous commit. The initial fix correctly prevented
injecting vector_store_ids: [] when the user never set any, but broke
the inverse case: a user who had ["vs_abc"] set and cleared the
selector would have their change silently ignored, because the handler
deleted the key from the PATCH payload and the backend's merge kept
the old value.
Distinguish "never touched" from "explicitly cleared" by initializing
the form field to undefined (not []) when the model has no stores, and
adding a middle branch in the submit handler that sends [] when the
form value is [] (user cleared) versus deleting the key when it's
undefined (user never touched).
Verifies that saving an edit to a model without vector stores does not
inject vector_store_ids into the PATCH payload. The test fails against
the pre-fix handler (payload contains vector_store_ids: []) and passes
with the length-guard fix.
The edit-model form initialized vector_store_ids to [] and then
unconditionally included it in the update payload. Editing any unrelated
field (e.g. temperature) on a model without vector stores would inject
vector_store_ids: [] into the model's litellm_params via the PATCH-merge
backend, which then propagated to inference requests and broke Anthropic
calls.
Mirror the cache_control_injection_points pattern directly below:
only include the field when length > 0, otherwise delete the key from
the payload so PATCH leaves the stored value untouched.
AntD <Select allowClear> returns undefined when the user clears the
selection. Coalescing to "" caused the team-update payload to carry
organization_id: "" instead of null, relying on the backend to coerce
it. Send null directly so the intent is explicit at the source.
- useKeyAliases, PaginatedKeyAliasSelect: add trailing `undefined` to
spy matchers for the new `team_id` param on `useInfiniteKeyAliases`
and `keyAliasesCall`.
- EntityUsage: mock new `TeamMultiSelect` child so QueryClientProvider
is not required for team-entity tests.
- ModelsCell: replace the overflow-accordion test with one that
verifies the new collapse-on-`all-proxy-models` behavior (no
accordion, single badge).
Extract `_apply_non_admin_alias_scope` from `key_aliases`,
`_resolve_team_access_group_resources` from `team_info`, and
`_enforce_list_team_v2_access` from `list_team_v2` to bring each
function under ruff's 50-statement limit. No behavior changes.
* bump litellm-proxy-extras version to 0.4.64
* bump litellm-proxy-extras==0.4.64 in requirements.txt
* bump litellm-proxy-extras==0.4.64 in pyproject.toml
Replace per-ID _resolve_access_group_resources loop with a single
find_many call that deduplicates IDs across all teams. Removes the
N+1 query pattern on cold cache for the team list endpoint.
The TeamData interface was missing access_group_models,
access_group_mcp_server_ids, and access_group_agent_ids fields,
causing a TypeScript build failure.
Three tests were patching the non-existent `get_access_object` instead
of `_get_access_object` (the lazy-import wrapper), causing AttributeError.
Also added missing `prisma_client` mock so tests get past the early-exit
guard and actually exercise the resolution logic.
The Key Alias dropdown on the Virtual Keys page was showing aliases from
all teams regardless of which team was selected. The team_id was never
passed through the frontend chain to the backend /key/aliases endpoint.
- Backend: add optional team_id query param to /key/aliases endpoint
- networking.tsx: add team_id param to keyAliasesCall
- useKeyAliases: accept and forward team_id to API call and query key
- filter.tsx: pass allFilters context to custom filter components
- PaginatedKeyAliasSelect: read Team ID from allFilters and pass to hook
* Litellm ishaan april1 (#25103)
* fix(proxy): enforce upperbound key params on key/update and add custom_key_update hook
The /key/update endpoint did not enforce upperbound_key_generate_params,
allowing users to bypass configured limits (tpm_limit, rpm_limit,
max_budget, duration, budget_duration) by updating an existing key
instead of generating a new one.
Extract the upperbound enforcement logic from _common_key_generation_helper()
into a standalone _enforce_upperbound_key_params() function and call it from
both the generate and update paths. For updates, None values are skipped
(not filled with defaults) since they mean "don't change this field".
Also adds a custom_key_update config option and user_custom_key_update global,
mirroring the existing custom_key_generate pattern, so custom key validation
logic can fire during key updates as well.
* fix(proxy): invoke custom_key_update hook in bulk update path
The user_custom_key_update hook was only called in update_key_fn
(single key update) but not in _process_single_key_update (bulk
update path), allowing custom validation to be bypassed via the
/key/update/bulk endpoint. Mirror the hook invocation in both paths.
* fix(proxy): pass UpdateKeyRequest to hook in bulk path, not BulkUpdateKeyRequestItem
Move the custom_key_update hook invocation to after UpdateKeyRequest
is constructed so the hook receives the same type in both single and
bulk update paths. Previously the bulk path passed
BulkUpdateKeyRequestItem (5 fields only), which would cause
AttributeError for hooks accessing fields like tpm_limit or models.
* fix(bedrock): promote cache usage to message_delta for Claude Code (#24850)
Ensure Bedrock/Anthropic-compatible streaming exposes cache usage where Claude Code reads it by promoting message_stop usage onto message_delta and preserving usage fields in fake-streamed message_delta events.
Made-with: Cursor
* fix(search): Support self-hosted Firecrawl response format in search transform (#24866)
The `transform_search_response` method only handled Firecrawl Cloud (v2)
response format where `data` is a dict with `web`/`news` keys. Self-hosted
Firecrawl (v1) returns `data` as a flat list of result objects, causing an
`AttributeError: 'list' object has no attribute 'get'`.
Detect the response format by checking if `data` is a list (self-hosted)
or dict (cloud) and handle both cases.
Cloud format: {"data": {"web": [...], "news": [...]}}
Self-hosted: {"success": true, "data": [{"url": "...", "title": "...", ...}]}
Co-authored-by: Synergy <synergyoclaw@gmail.com>
* feat: add environment and user tracking to prompt management (#24855)
* feat: add environment and user tracking to prompt management
- Add environment (development/staging/production) and created_by columns to LiteLLM_PromptTable
- Update unique constraint to [prompt_id, version, environment]
- All CRUD endpoints support environment filtering and user tracking
- Redesigned prompt detail page with environment tabs and version history
- UI: environment filter on list page, environment selector in editor
- 8 new tests for environment and user tracking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Black formatting and add environments to PromptInfoResponse TypeScript type
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address Greptile review findings
- P1: delete_prompt scopes in-memory cleanup to environment when provided
- P2: dotprompt_content parsed directly regardless of environment flag
- P2: use distinct for environments query
- P2: fix double-fetch on initial mount in prompt_info.tsx
- fix: remove unsupported select kwarg from find_many
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address remaining Greptile review comments
- Remove unused useCallback import (index.tsx)
- Remove unused ENV_COLORS variable (prompt_info.tsx)
- P1: in-memory fallback in get_prompt_versions now respects environment filter
- P1: reset selectedEnv when promptId changes to avoid stale state
- Cyclic imports are pre-existing pattern, not introduced by this PR
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: scope patch_prompt to environment using primary key
- Add environment query param to patch_prompt endpoint
- Look up target row by composite key (prompt_id + version + environment)
- Update by primary key (id) to target exactly one row
- Fixes Greptile finding: patch with multiple environments no longer ambiguous
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use actual start_time for failed request spend logs (#24906)
async_post_call_failure_hook set both start_time and end_time to
datetime.now(), making all failed requests show duration=0. Use the
actual start_time from litellm_logging_obj instead, so spend logs
reflect the real request duration on timeout and other failures.
Fixes#24888
* feat(bedrock): add nova canvas image edit support (#24869)
* feat(bedrock): add nova canvas image edit support
* fix(bedrock): support PathLike inputs for nova image edit
* chore: sync schema.prisma copies from root
* fix(mypy): correct type-ignore code for delta_usage arg-type
* fix(mypy): cast status_code to str, suppress intentional str yield
* fix(lint): extract _create_content_block_chunks to fix PLR0915
* fix(lint): extract helpers to fix PLR0915 in prompt endpoints
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(test): update model armor streaming test to handle string or int error code
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: redhelix <amin.lalji@gmail.com>
Co-authored-by: Synergy <synergyoclaw@gmail.com>
Co-authored-by: Talha Anwar <37379131+talhaanwarch@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: madhu19991 <madhu@thunkai.com>
Co-authored-by: Srikanth @adobe <devarakondasrikanth@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Replace the static team dropdown on the usage page with a new
TeamMultiSelect component that uses the paginated v2/team/list
endpoint with debounced server-side search and infinite scroll.
When Model Armor blocks a streaming response, it correctly raises
HTTPException(status_code=400) but create_response() catches it with a
bare except Exception and hardcodes a 500 response, discarding the
original status code.
Fix create_response() to preserve status_code from HTTPException instead
of hardcoding 500. Also update Model Armor's streaming hook to yield an
SSE error event instead of raising (matching the Prisma Airs pattern),
and fix make_model_armor_request() to return 400 for upstream API
failures instead of passing through the upstream status code.
