* fix(proxy): reconcile stale key spend counter after budget reset
* fix(proxy): invalidate stale key spend counter after budget reset or manual spend update
* fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass
* revert: undo unrelated formatting changes in enterprise directory
* test(proxy): add unit test for key spend update invalidating counter
* test(proxy): fix mocked update_data and hash token expectations in unit test
* feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses)
Bedrock Mantle serves the Responses API on two upstream paths:
- gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses
- every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses
BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in
utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses;
any model declared mode=responses (price-map entry or user model_info) -> /v1/responses;
everything else returns None and keeps the existing chat-completions emulation.
Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests.
* feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path
Addresses the Greptile review point that frontier detection should be a
price-map field rather than a hardcoded name match. The gate now routes a
model to /openai/v1/responses when its price-map entry declares
use_openai_responses_path, so a frontier model whose name does not follow the
openai.gpt- convention can be onboarded by JSON alone. The name-convention
check is kept as a fallback that needs no price-map entry, which preserves
zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4
get the flag in both price maps. Adds tests for the data-driven flag path and
for the flag presence on the gpt-5.x entries; both branches are mutation-tested.
* test(model_prices): allow use_openai_responses_path in price-map schema
The model_prices_and_context_window.json schema validator
(test_aaamodel_prices_and_context_window_json_is_valid) enforces
additionalProperties: false, so the new use_openai_responses_path flag on the
gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean,
alongside the other supports_* / capability flags.
* fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes
When output_parse_pii=true on the Anthropic native path (anthropic/claude-*),
response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was
yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with
the original values before reaching the caller.
Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta
/ text_delta events, and apply _unmask_pii_text before re-encoding. Wire it
into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist.
* fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask
Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks
don't bypass it and silently swallow a JSONDecodeError. Add
ensure_ascii=False to json.dumps so non-ASCII replacement values like
accented names are preserved as UTF-8 on the wire rather than being
\uXXXX-escaped. Add regression tests for both cases.
* fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update
Fixes#27734
Sending null for budget_duration, team_member_budget,
team_member_budget_duration, team_member_rpm_limit, or
team_member_tpm_limit via /key/update or /team/update returned 200 OK
but silently ignored the null value. The fields remained unchanged in
the database.
Root causes:
- /key/update: prepare_key_update_data() popped budget_duration from the
update dict but never re-added it (or budget_reset_at) when the value
was None.
- /team/update: _set_budget_reset_at() only acted when budget_duration
was non-None, leaving a stale budget_reset_at in the DB.
- /team/update: team_member_* null values bypassed the budget table
update entirely because should_create_budget() requires at least one
non-None field.
* test(proxy): cover no-budget-row path in clear_team_member_budget_fields
Replace "test_location" (contains underscore) with "us-central1" so tests
pass the regex validation added in get_vertex_base_url().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: propagate upstream status code in proxy API exception handler
When Google GenAI / Vertex returns a 404 for deprecated or missing
models via streamGenerateContent, the exception was falling through to
a generic handler that defaulted to 500. Now provider exceptions
carrying a valid HTTP status_code correctly propagate it through to
the ProxyException.
* fix: apply black formatting to common_request_processing.py
* fix: tighten status code range to 400-599 and deduplicate ProxyException raise
Context caching built the cachedContents URL as
https://{location}-aiplatform.googleapis.com, which is an invalid host for the
eu/us multi-region endpoints and returns 404. The inference path already
resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com)
via get_vertex_base_url(); reuse that helper in
_get_token_and_url_context_caching so caching uses the same host as inference.
Adds tests covering the eu/us multi-region cachedContents URLs (v1 and
v1beta1).
Fixes#29571
* fix(ui): show newest session logs first
* test(ui): keep session log pagination coverage
* fix(ui): show all session logs in the drawer, not just the first page
The session detail drawer fetched session logs via sessionSpendLogsCall
without page/page_size, so it only ever received the backend default of one
page (50 rows). Sessions with more than 50 calls had the rest unreachable in
the UI (#29153).
sessionSpendLogsCall now takes page/page_size, and the drawer fetches the
first page, reads total_pages, then fetches the remaining pages and
accumulates them before the existing client-side sort. This keeps the single
continuous list (and the selected-log lookup and keyboard navigation, which
all assume the full session) correct. Fetching is bounded by a page cap, and
the sidebar shows a "showing most recent N" note if a session exceeds it.
The rows are lightweight metadata (the endpoint excludes messages/response),
so the full set is small; request/response bodies are still loaded per log on
demand.
* fix(ui): default session drawer to most recent log, newest first
Open a session with its most recent log selected, and order the sidebar
newest-first to match the all-sessions logs overview. MCP calls stay
grouped last. The latest log by time is computed explicitly, since the
MCP grouping means it is not always the first row.
* Apply fetching pages in batches suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(ui): derive session total from accumulated rows when backend omits it
Compute the session total after all pages are fetched, falling back to the
accumulated row count rather than the first page's. Guards the truncation
note against a backend response that omits total but spans multiple pages.
---------
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix mcp scoped server name
* Update litellm/proxy/_experimental/mcp_server/mcp_context.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(mcp): cover scoped server name in the SSE initialize handler
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI
Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).
Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.
The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drop removed sampling params for Claude 4.7+ when drop_params is set
Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.
Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drive sampling param gating from the cost map and cover top_k
Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.
Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Declare supports_sampling_params in the cost map schema
The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary
Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)
After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)
Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"
This reverts commit 30d2e96f77.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(ui): remove dead dashboard files and unused dependencies
knip flagged seven orphaned source/config files with no importers and
five declared dependencies that nothing in the tree uses. Removing them
shrinks the dashboard bundle's source surface and keeps the manifest
honest; vite stays installed transitively via vitest, so test tooling is
unaffected.
* fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow
The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec
(tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml
workflow step still depend on it, so the redirect e2e job failed to load a
config that no longer existed.
* feat(ui): add admin flag to disable in-product UI nudges for everyone
Admins can now suppress the survey and Claude Code feedback popups for
all users via a single disable_ui_nudges UI setting, instead of relying
on each user dismissing them individually.
* fix(ui): suppress nudges while ui settings are loading
Gate nudgesDisabled on the ui-settings loading state so an admin with
disable_ui_nudges on doesn't see the survey prompt flash, and the
getInProductNudgesCall fetch doesn't fire, on a cold page load before
the flag resolves. Falls back to showing nudges if the fetch errors.
* test(ui): wrap CreateKeyPage test in QueryClientProvider
page.tsx now calls useUISettings (react-query), which needs a
QueryClient that layout.tsx supplies in production but the test did
not. Add the provider and mock getUiSettings so the query resolves.
With AI models capable of automated vulnerability discovery now publicly
available, we expect a large increase in report volume, much of it
unverified. Requiring a video of the exploit running against a live
instance raises the bar for submissions and keeps triage focused on
reproducible issues. Reports without a video will be closed and reopened
if one is added later.
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session
* fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss
models-and-endpoints, organizations, and virtual-keys each had a page.tsx
route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and
deep links never resolve to it and the route is unreachable. Each was a thin
wrapper that handed the shared view empty or no-op props (empty modelData with
a no-op setModelData, hardcoded empty organizations, no-op
setUserRole/setUserEmail), so reaching one would render a degraded page in any
case. The real wrapper belongs in the PR that flips each page into
MIGRATED_PAGES, written with eyes on it and a test
This continues the dead-scaffolding cleanup from #28891. The shared components
these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay,
since the legacy ?page= switch in app/page.tsx and src/components still import
them
* test(ui): add a data-driven App Router migration E2E smoke
Add a growing Playwright smoke for migrated pages: for each segment it deep-links
to the path route, asserts the URL and that the dashboard shell rendered, then
clicks off to a legacy page and asserts navigation still works. Driven by
e2e_tests/fixtures/migratedPages.ts, so adding a page is one line.
Runs in two situations against the same proxy: the default mount (npm run
e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root).
globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage
state is valid under a prefix. Seeded with api-reference; append the rest as their
migrations merge.
* test(ui): support headed slow-motion + watch pauses in the migration smoke
Honor SLOWMO in the server-root-path config (the default config already did),
and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state.
Both are no-ops by default, so CI behavior is unchanged.
* test(ui): make the migration smoke a sidebar-click user journey
Rework the smoke from deep-linking to a real navigation journey: start at the
landing page, click the migrated page in the sidebar (expanding submenus for
nested items), assert the path route rendered, reload it (the check a wrong
server_root_path breaks), bounce to a legacy page and back, and — once two pages
are migrated — navigate directly between two migrated pages. Verifies via URL +
shell render, driven by the same fixture list.
* test(ui): address review on the migration smoke
Escape ROOT and segment before interpolating them into RegExp URL matchers so a
future segment containing regex metacharacters can't silently widen the match.
Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead
of silently re-running the default mount and passing without exercising the prefix.
* test(ui): drop unused watch helper and fix stale smoke README
* test(ui): run the migration smoke under a server root path in CI
* test(ui): harden + instrument the server-root-path proxy reboot in CI
* test(ui): run the server-root-path migration smoke as its own CI job
Replace the in-place proxy reboot in e2e_ui_testing with a dedicated
e2e_ui_testing_server_root_path job that boots the proxy once with
SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the
config gets its own job rather than killing and relaunching the live proxy.
The reboot was failing deterministically: after pkill -9 and relaunch the
prefixed proxy never came back up on :4000 (connection refused), so the smoke
never ran. The readiness step that was supposed to surface the cause could
never reach its boot-log tail because CircleCI runs steps under bash -eo
pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's
exit 7. Booting the proxy as the job's own background step lets any boot crash
land in that step's log instead of being swallowed.
The default e2e_ui_testing job is unchanged aside from dropping the reboot,
prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at
the root mount there via the default Playwright config.
The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a
team's spend ceiling has nothing to do with the admin's own key budget. That
comparison was an unintended side effect of reusing _check_user_team_limits()
(which exists for the /team/new path) and broke the UI, which re-sends the
unchanged budget on every save.
New behavior on /team/update for standalone teams:
- A team admin (already authorized via _verify_team_access) may freely KEEP or
LOWER the team budget, and change models/tpm/rpm, without being gated by their
personal limits.
- GROWING a team's spend ceiling is a budget-authority action reserved for proxy
admins -> 403 for team admins. "Growing" covers both raising max_budget above
the team's current finite value and removing the cap entirely (max_budget=null,
detected via model_fields_set so an explicit null is distinguished from an
omitted field). For a team that currently has no cap, setting a finite value is
a restriction and is allowed.
- Org-scoped teams remain governed by _check_org_team_limits() (capped by the
org budget).
Also reverts the #29525 existing_team_max_budget workaround in
_check_user_team_limits() back to the create-only form; /team/new still enforces
the creator's personal caps.
docs(access_control): resolve the contradiction in the team-admin section —
team admins can keep/lower the budget and manage rate limits/models, but cannot
raise the team budget (proxy-admin only).
tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team
admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed,
keep/lower/resend allowed, and unchanged create-path guards.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path
* fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools
* fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return)
* fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize
* fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk
* fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect
* Fix MCP OAuth edit token handling
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(azure_ai): add MAI-Image-2.5 image generation support
Route azure_ai MAI models to /mai/v1/images/generations and map OpenAI size to width/height for the serverless API.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure_ai): address MAI image generation review feedback
Validate unsupported size values, default width/height independently, add MAI-Image-2.5 pricing, and expand test coverage.
@greptileai
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(azure_ai): add MAI image edit and expand model cost map
Add MAI image edit support with usage normalization for Azure response format,
and register MAI-Image-2.5-Flash and MAI-Image-2e pricing in the model map.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure_ai): validate MAI edit size by consuming map iterator
Greptile: lazy map() never evaluated int() so values like 1024xabc passed through.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure_ai): normalize MAI usage in generation response handler
Apply normalize_mai_image_usage before building ImageResponse so token-based
cost calculation works when Azure returns num_output_tokens fields.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure_ai): narrow MAI edit size param type for mypy
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Azure MAI image response handling
* Fix MAI image generation base model routing
* fix(azure_ai): preserve zero num_output_tokens in MAI usage normalization
* fix(azure_ai): wrap MAI generation response JSON parsing in error handling
* fix(azure_ai): build MAI image edit URL correctly for /mai/ root bases
* fix(azure_ai): build MAI image generation URL correctly for /mai/ root bases
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first
* feat(vantage): include organization metadata in FOCUS Tags export
Join LiteLLM_OrganizationTable when building Vantage/FOCUS export rows so
organization_id and organization_alias appear in Tags for org-level filtering.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(focus): include api_requests in organization Tags tests
FocusTransformer now requires api_requests after staging merge; add the
column to test fixtures so integrations CI can run the Tags assertions.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
A team's BYOK models (rows in LiteLLM_ProxyModelTable with model_info.team_id set)
were left orphaned when the team was deleted; they lingered in the database and kept
showing on the Models + Endpoints page. delete_team now removes them via a new
delete_team_models helper that deletes the rows in one transaction and syncs the
in-memory router only after that transaction commits, run before the team rows are
deleted so a mid-flight failure never leaves the team gone with its models orphaned
Raise the PyJWT floor in pyproject (>=2.13.0,<3.0) and re-resolve uv.lock so
the proxy installs 2.13.0 instead of 2.12.0. Bump the ws transitive-version
override in the dashboard from 8.19.0 to 8.20.1 and regenerate package-lock;
jsdom and openai both dedupe onto the single 8.20.1 copy.
Both are routine dependency maintenance bumps to keep pinned versions current.
* fix(vertex): propagate Vertex AI metadata in streaming success callbacks
Streaming calls assembled via stream_chunk_builder were missing
vertex_ai_grounding_metadata and vertex_ai_url_context_metadata in
standard_logging_object.response. Merge metadata from chunks into the
assembled response and mirror non-streaming hidden_params on Gemini chunks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(vertex): move streaming metadata merge into provider config hook
Address review feedback by delegating assembled-stream metadata propagation
to VertexGeminiConfig via BaseConfig.apply_assembled_streaming_response_metadata,
and only write chunk hidden_params when metadata is non-empty.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(redaction): scrub Vertex provider metadata when message logging is off
Clear vertex_ai_grounding_metadata and related fields from standard
logging responses and assembled streaming ModelResponse objects so
turn_off_message_logging cannot leak prompt-derived web search queries.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Use assembled model for streaming metadata hook
* Fix Vertex metadata redaction bypass in logging callbacks.
Scrub Vertex provider fields from litellm_params.metadata.hidden_params during perform_redaction so streaming success_handler merges do not leak prompt-derived metadata when message logging is disabled.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Vertex streaming metadata from hidden params
* fix(vertex): mirror vertex_ai_safety_results on assembled streaming responses
The non-streaming transform_response stores safety data under
vertex_ai_safety_results, but the streaming path only wrote
vertex_ai_safety_ratings. Assembled streaming responses therefore never
carried vertex_ai_safety_results, so any consumer reading that field saw
a silent difference between streaming and non-streaming calls.
Set vertex_ai_safety_results alongside vertex_ai_safety_ratings in the
shared stream metadata setter and add it to the assembled metadata field
list so it propagates through stream_chunk_builder.
* fix(streaming): log provider streaming metadata hook failures instead of swallowing them
* refactor(vertex): share single Vertex metadata field tuple across redaction and streaming
* refactor(vertex): move Vertex metadata redaction helpers into llms/vertex_ai
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
glm-5p1 supports native tools on Fireworks; explicit false flags caused
drop_params to strip tools and tool_choice before the provider request.
Co-authored-by: Cursor <cursoragent@cursor.com>
On /team/update for a standalone (no-org) team, _check_user_team_limits()
compared the request max_budget against the caller's personal max_budget
whenever max_budget was present in the payload. A team admin whose personal
budget is lower than the team's budget could not edit any field (tpm_limit,
team name, etc.) because the UI re-sends the unchanged max_budget on every
update, tripping the personal-budget check.
Pass the team's current max_budget into _check_user_team_limits() and skip the
personal-budget comparison when the incoming value is unchanged or lower than
the team's current budget. Only genuine increases above the team's current
budget are still validated against the caller's personal limit, so no
over-relaxation. Proxy admins and the org-scoped path are unaffected.
Adds two regression tests for the standalone update path (unchanged budget +
tpm_limit change, and lowering the budget), both for a caller whose personal
budget is below the team budget.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(jwt-auth): defer to single-team DB fallback on claim mismatch
Extends the single-team DB fallback introduced in #26418 to two more
cases where it previously could not run:
* `find_and_validate_specific_team_id`: when `team_id_jwt_field` is
configured and a claim value is present in the token but the team
does not exist in the LiteLLM DB (HTTPException 404 from
`get_team_object`), return `(None, None)` instead of raising — the
auth_builder fallback then attributes the request to the user's
single DB team. Only HTTPException is caught; other errors (e.g.
"No DB Connected") still propagate.
* `find_team_with_model_access`: when none of the `team_ids_jwt_field`
groups resolve to a real LiteLLM team, return `(None, None)` instead
of raising 403 so the same fallback path runs. If at least one group
DID resolve to a team but none granted the requested model, the
original 403 is preserved (legitimate access denial — not a claim
mismatch). Tracked via the new `any_claim_team_resolved` flag.
The strict `is_required_team_id` raise and `enforce_team_based_model_access`
raise remain unchanged. Unit tests cover both new soft-fail paths and
guard each preserved path (strict required, enforce_team_based, the
preserved 403, and the non-HTTPException propagation).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(jwt-auth): narrow HTTPException catch to 404 (greptile review)
Address Greptile review comments on #28913:
* `find_and_validate_specific_team_id`: re-raise HTTPException when
`status_code != 404`, pinning the catch to the "team doesn't exist
in db" path documented for `get_team_object`. A future change that
introduces a different status code (e.g. 403 for a blocked team)
will now propagate instead of silently falling through to the
single-team DB fallback.
* Add `test_find_and_validate_specific_team_id_non_404_http_exception_propagates`
parametrised over 400 / 403 / 500 to lock in the contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(jwt-auth): gate claim-mismatch fallback behind opt-in flag
The unresolved-team-claim fallback added in the previous commit
weakened the strict claim-based authorization contract by default —
an authenticated user whose JWT carries a stale or invalid team
claim could still consume their single DB team's models/quota via
the fallback.
Gate both soft-fail paths in `find_and_validate_specific_team_id`
and `find_team_with_model_access` behind a new opt-in flag
`team_claim_fallback` on `LiteLLM_JWTAuth` (default False).
Default-off preserves the pre-existing strict behavior. Operators
who intentionally treat IdP team claims as advisory (e.g. machine
tokens whose group claims live in a separate namespace from
LiteLLM team_ids) opt in via config.
Adds two regression tests guarding the default-off behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(model-management): allow deleting a BYOK model after its team is deleted
A team BYOK model (model_info.team_id set) became undeletable once its team
was deleted: POST /model/delete ran can_user_make_model_call, which looked the
team up and raised 400 "Team id=... does not exist in db" before the delete
could run, so the model lingered on the Models + Endpoints page with no way to
remove it.
Drop the team-existence prerequisite from the delete path. When the model's
team still exists the normal auth check runs unchanged; when it is gone a proxy
admin may delete the orphan and any other caller gets a 403. The check is
fail-closed, so a missing or errored team lookup can only block the delete or
require an admin, never grant a non-admin access. Add/update/health keep their
team-existence validation.
* refactor(model-management): drop redundant team lookup on model delete
Move the orphaned-team handling into can_user_make_model_call behind an
allow_missing_team flag instead of pre-checking team existence in delete_model.
The endpoint no longer issues its own litellm_teamtable lookup, so deleting a
model whose team still exists hits the team table once instead of twice. The
auth behavior is unchanged: a proxy admin can delete a model whose team was
deleted, any other caller gets a 403, and add/update/health keep the strict
"team must exist" validation.
* feat(galileo): add health check support for UI callback test
Register galileo in /health/services so the proxy UI callback connection test works.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(galileo): verify API key via /current_user health check
Call Galileo's current_user endpoint so the UI callback test validates credentials against the provider.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(ui): regenerate schema.d.ts for galileo health service
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): return IntegrationHealthCheckStatus from async_health_check
Fixes mypy assignment error in health_services_endpoint where response was
narrowed to IntegrationHealthCheckStatus from earlier branches.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Galileo logging to match Langfuse across all endpoint types.
Stop skipping ingest when output is empty and log embeddings with a placeholder so embedding, speech, and other non-text responses are recorded like Langfuse.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): remove unreachable health-check guard and None output sentinel
The use_v2_api flag is derived from bool(api_key), so the inner
GALILEO_API_KEY check inside the v2 branch could never run; collapse the
credential validation into the username/password path with a combined
message. _serialize_galileo_output now returns an empty string for None,
so _get_galileo_input_output_content always yields a str and the
post-call None coalescing guard is no longer needed.
* test(galileo): cover async_health_check failure paths and empty model response
Add regression tests for the Galileo health check unhealthy branches
(missing project id, missing base url, missing credentials, auth
failure, and request exception) and for logging a model response with
no choices, which now queues an empty output instead of being skipped.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes#29665) (#29788)
* feat(responses): add default no-op sign_request to BaseResponsesAPIConfig
* feat(responses): call sign_request after body is final, send signed bytes when signed
* feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path)
* test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency
* feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer
* docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route
* test(responses): cover fake-stream signing order and mantle bearer arg/env precedence
* fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance
* fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked
* fix(bedrock_mantle): sign the compact Responses route too, not just create
* fix(github-copilot): route per-model on /v1/responses based on model info (#29747)
* feat(focus): add GCS destination for FOCUS export (#29751)
* test: add failing tests for FocusGCSDestination
* feat: add FocusGCSDestination reusing GCSBucketBase auth
* feat: register FocusGCSDestination in factory; export from __init__
* fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config
* style: apply Black formatting to gcs_destination and tests
* style: apply Black formatting to factory.py
* fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565)
Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input
request when additionalModelRequestFields: {} or system: [] are present in the
payload. Both fields are optional in CommonRequestObject (total=False) and must
be omitted rather than sent as empty structures.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730)
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible
Azure OpenAI resources created via the newer "Azure AI Foundry" /
Cognitive Services pathway live on `*.cognitiveservices.azure.com`
subdomains, not the older `openai.azure.com`. Both are valid Azure
OpenAI surfaces in production today.
The OpenAI pass-through cost-tracking handler hard-codes only the older
hostname in five places (four `is_openai_*_route` methods on
OpenAIPassthroughLoggingHandler, plus is_openai_route on
PassThroughEndpointLogging). As a result, calls from newer Azure
deployments are silently classified as "not an OpenAI route", the
dispatch into the cost-tracking handler is skipped, and tokens/cost
never get extracted into LiteLLM_SpendLogs — the row gets written with
prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'.
Reproduced 2026-06-04 against a real Azure OpenAI deployment on
`*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0.
Fix: factor the hostname check into a single helper
`_is_openai_compatible_host` listing all three recognized surfaces
(api.openai.com, openai.azure.com, cognitiveservices.azure.com), and
have all five call sites delegate to it. Purely additive — never
weakens recognition for the originally-supported hostnames.
Adds a test
`test_is_openai_route_recognizes_cognitiveservices_azure_com` that
exercises all four `is_openai_*_route` static methods against
`*.cognitiveservices.azure.com` URLs (positive cases per route + a
small cross-route negative to confirm route-specific path matching
still works on the new hostname).
Out of scope for this PR (separate followup):
- `openai_passthrough_handler` calls chat/completions
`transform_response` on Responses API payloads (`output:` not
`choices:`), which throws inside the dispatch and drops the
SpendLogs row entirely. Recognized + tracked separately.
* ci: trigger fresh run
Empty commit to re-run checks. The previous auth-and-jwt failure was
a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads
in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to
this PR's scope (hostname recognition in pass-through cost tracking).
No code change.
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812)
The Responses API forces a specific function with a top-level name
({"type": "function", "name": "X"}), but _transform_tool_choice only handled the
nested Chat Completions shape and fell through to returning "required" for the flat
form, silently dropping the function name and degrading a forced function call to
force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the
"required" fallback when no name is present.
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584)
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic
PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for
every Anthropic target. That block is how the first-party Anthropic API recognizes
Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry
only that block, such as the auto-mode tool-safety classifier, fail with a
misleading 429 rate_limit_error; normal turns still work because they also carry
the "You are Claude Code" identity block.
Gate the strip behind should_strip_billing_metadata(), defaulting to False on the
first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and
overridden to True on the providers that reach these transforms and reject the
block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek
for the messages path). Behavior for those providers is unchanged.
* Strip billing header on Bedrock invoke and Vertex messages pass-through
Two more subclasses reach the gated strip but inherited keep-by-default.
AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request,
which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig
(Vertex messages pass-through) calls super().transform_anthropic_messages_request.
Override should_strip_billing_metadata() to True on both.
Add a parametrized test asserting the flag for every first-party base (False) and
provider subclass (True), covering all overrides, plus a translate_system_message
regression test for the Bedrock invoke path.
* fix(cache): log hashed cache keys (#29890)
* fix(ui): save routing groups as list (#29889)
* Revert "fix(ui): save routing groups as list (#29889)" (#29928)
This reverts commit 9b1f78ffa7.
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider (#29842)
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider
Registers parasail in the openai_like JSON provider loader with both
/v1/chat/completions and /v1/responses support. Parasail's Responses API
rejects store:true and any request that omits store, so the loader gains a
force_store_false special_handling flag; the parasail entry sets it and
the generated Responses config overrides store=false on every call. This
keeps callers from hitting "State storage not supported" and matches what
Parasail's docs require.
Adds the PARASAIL enum value, listing under openai_compatible_providers,
provider documentation at docs/my-website/docs/providers/parasail.md, and
a focused unit test file under tests/test_litellm/llms/parasail/ that
covers JSON registration, chat URL construction, Responses URL
construction with PARASAIL_API_BASE override, and the force_store_false
regression in both the caller-sent-store=true and caller-omitted cases.
* fix(parasail): register in provider_endpoints_support, drop in-repo docs
Greptile review feedback. The provider doc belongs in the litellm-docs
repo, not this one's docs/my-website tree; removing it here. Adds the
parasail entry to provider_endpoints_support.json so the
check_provider_folders_documented.py CI check passes (chat_completions
and responses true; others false).
* fix: normalize Anthropic passthrough server tool usage (#29827)
* test(anthropic): cover server_tool_use dict cost tracking
* fix: normalize Anthropic server tool usage
(cherry picked from commit 982f726bed)
* fix: keep server tool usage subscriptable
(cherry picked from commit 70280b9b27)
---------
Co-authored-by: Genmin <joey@joeyroth.com>
* fix(proxy): fix typo generic_role_mappoings -> generic_role_mappings in ui_sso.py (#29753)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): add disable_budget_reservation general setting (#27639) (#29493)
* feat(proxy): add disable_budget_reservation general setting (#27639)
* feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639)
* docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639)
* ci: re-trigger flaky docker build (prisma generate ECONNRESET)
* fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639)
* feat(gemini_tts): adding support to Gemini TTS languageCode parameters (#29623)
* Adding support to Gemini TTS Language Code parameters
* Mapping Gemini TTS languageCode param in Docstring
* Use snake_case for language_code input keyMapping Gemini TTS languageCode param in Docstring
* Restoring files modified under enterprise/litellm_enterprise due to lint/formatting checks
---------
Co-authored-by: João Garrido <joaogarrido@google.com>
* feat(guardrails): capture user and model metadata in CrowdStrike AIDR (#29517)
* fix(proxy): require OpenAI path segment for shared Azure Cognitive Services domains
Address Greptile review: the `*.cognitiveservices.azure.com` /
`*.openai.azure.com` domains are shared by every Azure Cognitive Service
(Speech, Vision, Language, ...), so a hostname-only substring match
misclassified non-OpenAI Azure traffic as OpenAI routes.
- Replace the substring host test with suffix matching (rejects look-alike
domains like cognitiveservices.azure.com.attacker.example).
- Add `_is_openai_compatible_url` that requires an OpenAI-style path marker
(`/openai/` or `/v1/`) on the shared Azure domains, and use it in
PassThroughEndpointLogging.is_openai_route (previously hostname-only).
- Add negative tests for Azure Speech/Vision paths and look-alike domains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: support Responses input in Redis semantic cache (#29581)
* fix: support responses input in redis semantic cache
* test: cover redis semantic prompt extraction
* test: handle blank redis semantic text fallbacks
* chore: remove async cache dead statement
* test: cover redis semantic cache miss paths
* fix: filter sensitive cache lookup kwargs
* chore: rerun ci after huggingface rate limit
* chore(ui): regenerate dashboard API types (npm run gen:api)
Sync src/lib/http/schema.d.ts with the proxy OpenAPI spec: adds the
disable_budget_reservation general-settings field and picks up the
RateLimitError docstring reindent. Fixes the gen:api CI drift check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(bedrock): assert empty additionalModelRequestFields is omitted
The Converse transformer now drops an empty additionalModelRequestFields
block instead of sending it as `{}`. Update test_bedrock_top_k_param so
models without top_k support (llama3) assert the key is absent rather than
equal to an empty dict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: codgician <15964984+codgician@users.noreply.github.com>
Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com>
Co-authored-by: Roi <roytev@gmail.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Scott <liam@uilliam.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
Co-authored-by: Ceder Dens <cederdens@gmail.com>
Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com>
Co-authored-by: Kai Huang <kaihuang724@gmail.com>
Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com>
Co-authored-by: Genmin <joey@joeyroth.com>
Co-authored-by: Arnav Bhilwariya <arnavbhilwariya0408@gmail.com>
Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com>
Co-authored-by: João Garrido <48538534+johngarrido@users.noreply.github.com>
Co-authored-by: João Garrido <joaogarrido@google.com>
Co-authored-by: Kenan Yildirim <kenan@kenany.me>
Co-authored-by: Dávid Balatoni <balcsida@gmail.com>
* fix(ui): let non-creator users OAuth into OBO-mode MCP servers from the Tools page
* fix(ui): clear OBO Tools-tab one-shot on navigate-back and gate on credential-status errors
* fix(ui): unify migrated-route URLs and cut the API Reference page over to path routing
Route all migrated-page navigation through one /ui-prefixed, serverRootPath-aware
builder in migratedPages.ts (migratedHref/legacyPageHref/legacyKeyForPathname),
replacing the three divergent base-URL constructions that lived in the dashboard
layout's withBase, leftnav, and the page.tsx redirect. The previous migratedHref
read NEXT_PUBLIC_BASE_URL, which no build sets, so it produced URLs without the
/ui prefix the app is served under; every other internal link hardcodes /ui and
this now matches that convention.
Remove the sidebar's own pushState navigation so the parent (legacy root page or
dashboard layout) is the single owner of navigation, fixing the double-navigate
that fired when moving between a path route and a legacy ?page= route.
Cut API Reference over to its path route: add api_ref -> api-reference to
MIGRATED_PAGES and delete its arm from the legacy switch. Visiting
/ui/?page=api_ref redirects to /ui/api-reference, the sidebar links to and
highlights it, and navigating away returns to the legacy switch.
* fix(ui): address review on migrated-page routing
Keep the legacy hyphenated ?page=api-reference form working by mapping it to the
api-reference route alongside api_ref; the old switch matched both, so a bookmark
using the hyphen would otherwise fall through to the Usage default.
Add legacyKeyForPathname coverage: a migrated path (with and without trailing
slash) resolves to the api_ref sidebar key rather than the alias, a non-migrated
path returns null, and a non-root serverRootPath prefix is stripped before matching.
* fix(ui): populate serverRootPath from getUiConfig so migrated nav links keep the root path
getUiConfig updated proxyBaseUrl but never called updateServerRootPath, so the
module-level serverRootPath stayed at its "/" default. Under a custom
server_root_path the unified migratedHref/legacyPageHref builders then dropped
the prefix and the sidebar produced /ui/api-reference (404) instead of
/<root>/ui/api-reference. Adds the missing updateServerRootPath call plus a
regression test asserting getUiConfig sets serverRootPath and that migratedHref
carries the prefix
The key edit page showed the default key type (no allowed_routes restriction)
as "Default", while the key creation form already labels the same value
"Full Access". Align the edit page to the create form so the two surfaces agree
on both the label and its description.
The create guardrail modal used antd's default maskClosable, so clicking
outside it dismissed the modal and reset every field the user had entered.
Setting maskClosable={false} keeps the modal open; it now closes only via
the explicit close button or Cancel, matching the other form modals in the
dashboard
The provider logo base path was relative ("../ui/assets/logos/"). With
trailingSlash enabled, the public model hub is served at /ui/model_hub_table/,
so the browser resolved the base to /ui/ui/assets/logos/ (a doubled /ui/), which
404s every icon. The authenticated hub renders inside the single-level /ui/ SPA
route where the relative path resolves correctly, so only the public hub broke.
Make the base root-absolute so it resolves at any route depth.
Consolidate the three hand-synced copies of the migrated-pages map
(LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in the dashboard layout,
and MIGRATED_PAGES in leftnav) into one shared module,
src/utils/migratedPages.ts, which also owns the migratedHref helper. Delete
the unused, incomplete Sidebar2 prototype.
No runtime behavior change: the map is still empty and Sidebar2 had no
importers, so this is pure deduplication ahead of the per-page App Router
migration. Follow-up work will unify the remaining base-URL builders
(layout's withBase and page.tsx's redirect) onto migratedHref.
* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec
Expose the v2 model info endpoint in /docs by removing include_in_schema=False
and documenting query parameters used by the admin UI and proxy CLI consumers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The dashboard calls UI-internal proxy routes that the public /openapi.json hides with include_in_schema=False, so they never reached schema.d.ts and could not be typed. The type generator now force-includes those routes when it dumps the spec for openapi-typescript; this mutates a throwaway interpreter only, so the spec the proxy actually serves is unchanged.
Regenerates schema.d.ts so 86 internal route families (for example /v2/model/info, /global/spend/*, /config/*, /v2/login, /sso/*) are now typed, with no public route removed. This unblocks migrating the dashboard's data fetching onto the typed $api client.
Branch CI note: schema.d.ts is generated; CI regenerates and diffs it via the same gen:api script.
