Global env var values are always stored encrypted, so a value that no longer
decrypts (typically a rotated LITELLM_SALT_KEY) was being forwarded into
upstream ${NAME} headers as ciphertext with only a debug log. Drop the value
and log a warning so the failure surfaces instead of silently sending ciphertext.
Per-user env var stores now merge over the existing values instead of replacing
them. Per-user credentials are write-only and never shown back, so requiring the
full set on every save forced users to re-enter credentials they could not see
just to change one field. Omitting (or sending empty) a field now keeps its
stored value; DELETE still clears everything. The modal no longer marks
already-set fields as required.
The (user_id, server_id) unique index cannot serve the
delete_many(where={server_id}) orphan cleanup in delete_mcp_server,
since server_id is not the leading column, so it falls back to a full
table scan. Add a server_id index to cover that delete path
The per-user env var cache is process-local, so in multi-worker
deployments a user who stored values on another worker could hit a stale
cached negative and receive a misleading 412 'missing credentials'
response until the entry expired. Before raising MCPMissingUserEnvVarsError
the resolver now re-reads straight from the DB with force_refresh, so a
process-local stale entry can never mask values stored elsewhere; the hot
success path still serves from cache.
Also drops the inaccurate claim that env vars are interpolated into the
server URL from the schema and type docs; only static_headers are
interpolated.
The health check and initialize-instructions prefetch copied
server.static_headers verbatim, so any auth header backed by a
${NAME} global env var was sent upstream as the literal placeholder.
Servers that migrated to the new ${NAME} convention authenticated fine
on real tool calls but flipped to 'unhealthy' in the dashboard. Both
probes now resolve static headers through
_resolve_static_headers_with_env_vars (no user context, best-effort)
so global values are substituted before the connection opens.
Global-scope MCP env vars hold admin-supplied secrets (API keys, passwords) that interpolate into static headers, but their raw value was serialized into the env_vars JSON column in plaintext, so anyone with read access to the database could recover those upstream credentials. Credentials and the per-user values_b64 column are already encrypted; global env var values now match that, encrypted on write in _prepare_mcp_server_data and decrypted when the server is built into the runtime registry and when records are read back for admin views. Per-user placeholder values are not secrets and stay verbatim.
A stored per-user value for a var that the admin later switched to global
scope was still able to override the admin's global value, because the
header merge applied the full user blob over the globals. Filter the user
blob to vars that are currently user-scoped and let admin globals win.
MCPServer.env_vars is stored as List[Dict[str, Any]] (deserialized from
the JSON column), but LiteLLM_MCPServerTable.env_vars is typed as
List[MCPEnvVar]. Passing the raw dicts relied on pydantic coercion at
runtime and tripped mypy's dataclass_transform __init__ check, failing
the lint job. Normalize the dicts into MCPEnvVar models at both
construction sites.
These values interpolate into Static Headers and Auth via ${NAME};
they are not exported into the MCP server's process environment, so
'Environment Variables' overpromised. The backend env_vars field is
unchanged.
The per-user value column rendered as a plain input, so it read like a static
value shared by every user, defeating the purpose of per-user variables. Add a
persistent "Hint" addon with an explanatory tooltip, lighten the typed text,
and align the row to the top so the addon group no longer sits lower than its
neighbours and the layout stays put when the name field shows a validation
error.
The GET /v1/mcp/server list and health endpoints build LiteLLM_MCPServerTable
from the in-memory registry via _build_mcp_server_table and health_check_server.
Both copied static_headers but dropped env_vars, so the list always returned
env_vars: null. The admin edit form is populated from that list data, so it
loaded an empty env-var list; saving any edit then persisted env_vars: [],
silently wiping the stored variables. With nothing left to interpolate, the
${VAR} static headers were forwarded upstream verbatim as literal text.
Carry env_vars through both conversions, mirroring static_headers. Add
regression tests asserting both paths round-trip name, scope, value, and
description.
add_mcp_server wrote the new row and then reloaded the entire registry from the
database inside one try block. A single pre-existing malformed row made the
reload raise, so the endpoint returned 500 even though the new server was already
persisted; callers assumed failure and retried, creating duplicate servers.
Split the flow so the database write is the commit point and still 500s on
failure, while the in-memory registry refresh is best-effort and only logged on
error. Add regression tests for both the refresh-fails-after-commit path and the
db-write-fails path
A stale admin secret left in form state after switching an env var's
scope from instance to per-user was forwarded to the backend as the
user-scope value, which is returned unredacted to authorized non-admin
users. Per-user entries carry no admin value, so drop it on submit.
The single-server and bulk per-user env var status endpoints echoed the
decrypted credential value back to any holder of the user's LiteLLM token,
so a leaked token could exfiltrate the raw upstream secret (e.g. a personal
access token) for use outside the proxy. Drop the value field from
MCPUserEnvVarSpec and the include_values plumbing so the status reports only
whether each credential is_set; users overwrite a field to rotate it. The
fill-in modal no longer pre-populates from the secret and flags already-set
fields instead.
getMCPUserEnvVars now throws on non-2xx so UserEnvVarsModal reports the
error instead of silently rendering the empty 'no per-user fields' state.
The per-user env-var endpoints now run the access check before the server
lookup so a non-admin cannot tell a missing server (404) apart from one
they lack access to (403), closing a server-id enumeration leak.
PROXY_ADMIN_VIEW_ONLY callers are treated as admin_view by GET /v1/mcp/server
and GET /v1/mcp/server/{id}, so they received unredacted scope=global env var
values, which can carry upstream API keys used in Authorization headers. Only
a full PROXY_ADMIN needs those values (to pre-fill the edit form); read-only
admins now get the same global-secret redaction already applied to non-admin
and restricted virtual-key views.
The single-server DB fetch returned the raw Prisma model, whose JSONB
env_vars deserialize to plain dicts. The non-admin and virtual-key
sanitizers read env_var.scope as an attribute, so GET /v1/mcp/server/{id}
for a server with env_vars raised AttributeError and returned a 500.
Wrap the result like every bulk fetch helper so env_vars become MCPEnvVar
objects before redaction.
* fix(vertex): strip output_config.effort for models that reject it
Haiku 4.5 on Vertex AI does not support output_config.effort and 400s with
"output_config.effort: Extra inputs are not permitted". PR #27074 emptied
VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS so effort would forward for Opus/Sonnet
4.6+, but that made the strip unconditional across every Vertex Anthropic
model, including ones that don't support it. Claude Code injects effort into
its default Messages payload, so `claude --model claude-haiku-4.5` started
failing.
Make the sanitizer model-aware: drop output_config.effort for models that
don't advertise output_config support (or any reasoning effort level) while
forwarding it for those that do. The fix covers both the chat-completion and
Messages pass-through transformation paths since they share the helper.
* chore(vertex): log at debug when dropping unsupported output_config.effort
Operators pointing an unregistered Vertex Claude alias that does support
effort would otherwise see it stripped with no signal. Debug level keeps it
out of normal logs since Claude Code sends effort on every request.
* ci: reproduce default-Windows wheel install to guard MAX_PATH
The existing using_litellm_on_windows job installs the project with
`uv sync`, an editable source install that never copies package files
into a deep site-packages path, so it cannot see the 260-char MAX_PATH
overflow that breaks `pip install litellm` on default Windows. The
content-filter benchmark fixtures have hit that limit three times
(#21941, #22039, #29536), each caught only after release.
This adds a guard to the same job that builds the wheel and installs it
the way an end user would: into a venv whose site-packages prefix is
padded to a realistic worst-case Windows length (~100 chars), then
asserts the install completes and litellm imports. Any packaged path
long enough to bust MAX_PATH at that prefix is reported up front, so the
check is deterministic regardless of the runner's long-path setting,
while the real install also covers failure modes a length heuristic
cannot (half-unpacked packages, reserved names, case collisions).
This commit is the guard only; on the current tree it correctly fails
because nine fixtures still exceed the limit. The rename that brings
them back under it follows on this branch.
* fix(packaging): shorten content-filter benchmark fixtures under MAX_PATH
The 10 content-filter benchmark result fixtures used the legacy
block_{topic}_-_contentfilter_({yaml}).json naming, up to 176 chars
inside the wheel, which busts the Windows 260-char MAX_PATH limit once
extracted under a realistic site-packages prefix and aborts
`pip install litellm` on default Windows.
Rename them to the short {topic}_cf.json scheme that
_save_confusion_results already emits today (it splits the label on the
em-dash and writes f"{topic}_cf"), matching the insults_cf.json and
investment_cf.json files fixed earlier. Re-running the eval suite now
regenerates these same short names rather than recreating the long ones.
This drops the longest packaged path from 176 to 128, so the guard added
in the previous commit goes from red to green with a 32-char margin.
* test(windows): tidy MAX_PATH guard per review
Close the wheel zip via a context manager rather than leaning on
refcount collection, and select the wheel under dist/ by newest mtime so
a stale artifact from an earlier build cannot be tested instead of the
one just produced. Also pin down the venv-depth formula with a short
note: the +2 is the separator joining the venv root to "Lib" plus the
trailing separator before the entry, which lands the simulated
site-packages prefix at exactly 100 chars.
* fix(anthropic/adapter): open thinking block for reasoning_content-only streaming chunks
The /v1/messages streaming content-block classifier (_translate_streaming_openai_chunk_to_anthropic_content_block) only recognized thinking_blocks. OpenAI-compatible reasoning backends (vLLM/SGLang reasoning parsers: DeepSeek-R1, Qwen3, gpt-oss, ...) populate reasoning_content with thinking_blocks=None, so the classifier fell through to a text block. The delta translator already emits thinking_delta for reasoning_content, so those deltas landed inside a text block and Anthropic streaming clients (Claude Code, SDK .stream()) silently dropped the chain-of-thought.
Mirror the reasoning_content fallback already present in the non-stream translator and the streaming delta translator so the classifier opens a thinking block. Adds a focused regression test.
* fix(anthropic/adapter): reach reasoning_content branch when thinking_blocks attr is absent
Delta deletes the thinking_blocks attribute when unset, so the prior nested check was unreachable for reasoning-only chunks (vLLM/SGLang). Make it a sibling elif so the content block is classified as thinking.
* test(proxy): stop component-allowlist test leaking DATABASE_URL into xdist peers
The component-allowlist test pins throwaway DATABASE_URL/LITELLM_MASTER_KEY
values at import time via os.environ so importing proxy_server doesn't need a
live database. Those values persisted for the whole pytest-xdist worker, so a
sibling test sharing the worker (test_key_rotation_e2e's DB-backed E2E case)
saw the leaked sqlite DATABASE_URL, treated it as an available database instead
of skipping, and the Prisma engine rejected the non-postgres URL (P1012 ->
httpx.ConnectError). Restore the prior environment after the import so the
throwaway values never escape the module.
---------
Co-authored-by: Tai An <antai12232931@outlook.com>
The bulk /user-env-vars/status feed only drives the dashboard "N fields
missing" badge, which needs is_set, so it no longer returns the stored
credential values; the single-server endpoint still returns them for the
fill-in modal to pre-populate.
Adds a description input to the admin env-var form for per-user scope so
admins can tell users what to enter; the per-user modal already surfaces
that description as a hint.
* Fix error code and context id injection bug
* Add support for all A2A methods
* Add logging
* address greptile review: relay upstream JSON-RPC errors, move _PASCAL_TO_WIRE to module level, add error path tests
* fix(a2a): run pre_call_hook for tasks/resubscribe SSE path to enforce guardrails
tasks/resubscribe was returning the raw SSE stream without calling proxy_logging_obj.pre_call_hook, silently bypassing any guardrails configured on the agent. This patch calls pre_call_hook before streaming begins and wires post_call_failure_hook into the SSE generator so errors are logged. Adds a regression test verifying the hook is called.
* fix(a2a): use get_async_httpx_client instead of creating httpx clients per request
Creating httpx.AsyncClient instances per-request adds ~500ms latency. Switch _forward_jsonrpc and _forward_jsonrpc_sse to use the shared client from get_async_httpx_client(httpxSpecialProvider.A2A).
* fix(a2a): forward caller identity headers on task ops; validate push notification URL
Two security fixes for task management methods:
1. All task operations (tasks/get, tasks/list, tasks/cancel, tasks/resubscribe, push notification config methods) now forward X-LiteLLM-User-Id and X-LiteLLM-Team-Id headers to the upstream agent, so the agent can scope task access to the authenticated caller.
2. tasks/pushNotificationConfig/set validates the callback URL before forwarding: requires HTTPS and rejects private/loopback/reserved IP ranges and localhost hostnames to prevent SSRF.
* Fix A2A task hook and push URL handling
* fix(a2a): fix mypy type errors for request_id and header_name dict key types
* Fix A2A request id and params forwarding
* Forward trace IDs for A2A task calls
* fix(a2a): strip client-forwarded X-LiteLLM-* headers before applying authenticated identity
A client could send x-a2a-<agent>-x-litellm-user-id in their request and have it forwarded to the upstream agent as an authenticated identity header. Fix: sanitize any X-LiteLLM-* headers from agent_extra_headers before merging, then apply the authenticated identity headers last so they always override client-supplied values.
* Fix A2A SSE fallback JSON-RPC error code
* Fix A2A SSE error id backfill
* fix(a2a): validate both push notification url fields to close SSRF bypass
* fix(a2a): widen request_id annotation to match JSON-RPC id call sites
* fix(a2a): run post-call streaming hook for tasks/resubscribe so agent guardrails apply
tasks/resubscribe returned the raw upstream SSE stream without routing events
through the post-call streaming hook, so output guardrails configured on the
agent were silently skipped for streaming task subscriptions while every other
task method and message/stream applied them. Parse upstream JSON-RPC SSE events
and feed them through async_streaming_data_generator, matching message/stream,
so guardrails inspect the streamed task content. Adds a regression test that
fails when the streamed events bypass the guardrail hook.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Make the per-user env-var delete idempotent with delete_many so a missing
row is a no-op, and remove the bare except on the clear endpoint that
swallowed real DB failures; a failed delete previously reported success to
the dashboard while the row remained in the database, so the next tool call
would 412 for credentials the user believed they had cleared.
Also use the module-level json instead of a redundant inline import, and
drop the no-op duplicate static_headers/env_vars assignment in the
create-server UI payload.
The per-user env-var endpoints (GET/POST/DELETE /server/{id}/user-env-vars)
fetched the server by id and returned its name, alias, and required-credential
metadata, or persisted/cleared stored values, without checking that the caller
can access that server. A non-admin could query or mutate env-var state for any
server id. Apply the same access gate fetch_mcp_server uses so non-admins are
limited to servers in their allowed set.
* Fix incorrect agent API request example payload structure (#29556)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs
On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span
is stored in litellm_params['litellm_metadata'] instead of
litellm_params['metadata']. When the request body contains a native
'metadata' field (e.g. Anthropic's {"user_id": "..."}),
litellm_params['metadata'] gets overwritten and the parent span is lost,
producing orphan root spans with a different trace_id.
Add fallback checks to litellm_metadata in:
- _get_span_context(): so child spans find the correct parent
- _end_proxy_span_from_kwargs(): so the proxy span gets closed
Fixes: https://github.com/BerriAI/litellm/issues/27934
* test(otel): tighten assertions per Greptile review
- test_span_context_metadata_takes_priority: assert litellm_metadata
span is never accessed, proving metadata takes priority
- test_span_context_no_parent_when_neither_has_span: assert both ctx
and detected_span are None
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: remove premature end-user budget check from get_end_user_object (#29420)
* fix(proxy): remove premature end-user budget check from get_end_user_object
Problem:
- `_check_end_user_budget()` was called inside `get_end_user_object()`
- This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated
- Zero-cost models (e.g., local vLLM) were incorrectly blocked when
end-users exceeded their budget, even though they should bypass budget checks
Solution:
- Remove `_check_end_user_budget()` calls from `get_end_user_object()`
- Budget enforcement now happens exclusively in `common_checks()` where
`skip_budget_checks` context is available
- `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation.
* refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object
- test_get_end_user_object() verifies data fetching
- test_check_end_user_budget() verifies enforcement
- test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget()
- test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object()
* Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534)
* Fix Gemini image config mapping
* Address Gemini image config review
* Format Gemini image generation transform
* Fix Gemini image token usage logging
* Share Gemini image request helpers
* Fix Gemini Imagen model routing
* Fixes as per self code review
* Fixes per internal code review
* Stop gating Imagen imageSize forwarding
* Document Gemini image size mapping source
* chore: retrigger lint
* Clarify Gemini candidate count precedence
* Add Inception provider (#29522)
* add inception as provider (chat, fim)
* linting
* seperate test suite for chat and fim
* fix test coverage
* fix: model hub custom pricing model info (#29293)
* Opik user auth key metadata extractors (#28397)
* fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic
* test: add unit tests for OPik metadata extraction logic
* fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy
* fix(ci): clarified comments and edited unit tests
* test: add unit tests for OPik metadata extraction with auth and requester overrides
* fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532)
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
* fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561)
`_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls`
so a following tool result can be matched back to its tool call. The assignment
was inside a branch guarded by
`assistant_msg.get("tool_calls", []) is not None`, which is also True for a
text-only assistant message (an empty list is not None). As a result, an
assistant message with no tool calls that appears between a tool call and its
tool result overwrote the reference, and conversion failed with:
Exception: Missing corresponding tool call for tool response message.
This shape is common: a model emits a short narration/assistant message after a
tool call before the tool result is appended.
Only update `last_message_with_tool_calls` when the assistant message actually
carries tool_calls (or a function_call). Adds a regression test.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models
The 1-hour prompt-cache write tier
(`cache_creation_input_token_cost_above_1hr`) was added to the
us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but
the eu./au./jp. cross-region inference profiles were left without it.
AWS Bedrock pricing applies the same +10% regional premium across all
geo profiles, so eu./au./jp. should carry the same 1-hour rates as
us. (1.6x the 5-minute regional rate).
Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL
prompt caching falls back to the 5-minute write rate and undercounts
spend by ~60% for European, Australian, and Japanese tenants.
Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where
AWS publishes one) to 14 regional Bedrock entries in both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- eu./au. Opus 4.6 ($11.00 / MTok)
- eu./au. Opus 4.7 ($11.00 / MTok)
- eu./au./jp. Sonnet 4.6 ($6.60 / MTok)
- eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC)
- eu./au./jp. Haiku 4.5 ($2.20 / MTok)
Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py`
with a `REGIONAL_EXPECTED` parametrized block covering all 13 new
entries plus the existing 1.6x ratio invariant.
Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the
wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06),
which would break the 1.6x ratio check. It is intentionally left out
of this PR so the scope stays "1-hour cache tier addition" — a
separate follow-up should correct the EU 5m rates for Opus 4.5.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models
GCP Vertex AI publishes a separate 1-hour cache write column for the
Claude family (1.6x the 5-minute write rate, matching the documented
Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the
5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}`
on Vertex AI Claude is undercounted in cost tracking by ~60%.
The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig`
extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and
`_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`.
Only the price registry was missing data.
Adds the field to 19 vertex_ai/claude-* entries across both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- Haiku 4.5 ($1.25 -> $2.00 / MTok)
- Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok)
- Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok)
- Opus 4 / 4.1 ($18.75 -> $30.00 / MTok)
Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py`
mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model
and asserts the 1.6x ratio across the family.
Fixes#27781.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Fix Gemini multimodal function responses (#29325)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* address greptile review: add _transform_image_usage method and model-map supports_image_size flag
- Add _transform_image_usage instance method to GoogleImageGenConfig that
delegates to transform_gemini_image_usage, fixing the regression test
- Replace hardcoded "2.5-flash" string check in supports_gemini_image_size
with a get_model_info lookup on supports_image_size (default true)
- Add supports_image_size: false to all gemini-2.5-flash model entries in
model_prices_and_context_window.json so capability is controlled via the
model map rather than embedded in code
* fix test failures: schema validation, mypy type, model info plumbing, pricing test
- Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it
- Pass supports_image_size through _get_model_info_helper constructor call
- Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True)
- Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid
- Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values
* Add Azure AI Kimi K2.6 metadata (#27052)
* Add Azure AI Kimi K2.6 metadata
* Scope Kimi metadata test cost map setup
* fall back to substring check for models not in model_prices_and_context_window.json
Models like gemini-2.5-flash-image-preview are not in the pricing JSON,
so get_model_info raises. Fall back to "2.5-flash" not in model when the
JSON has no explicit supports_image_size entry for the model.
* fix(inception): don't forward global litellm.api_key to Inception FIM
Match the Inception chat config: resolve only an Inception-specific key
(param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion
FIM path. The global litellm.api_key (often an OpenAI key) was both leaking
to api.inceptionlabs.ai and taking precedence over the configured Inception
key when set.
* fix(auth): enforce end-user budget on custom-auth path that skips common_checks
get_end_user_object() no longer raises BudgetExceededError, so custom-auth
deployments with custom_auth_run_common_checks unset (which skip the
centralized common_checks gate) stopped enforcing the end-user budget,
letting an over-budget end user keep making requests. Re-enforce the
budget in _run_post_custom_auth_checks on that path.
---------
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com>
Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com>
Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com>
Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk>
Co-authored-by: Lovro Seder <vrovro@gmail.com>
Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com>
Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A transient DB failure while loading per-user env vars on the tool-call
path was swallowed and turned into a misleading MCPMissingUserEnvVarsError
(412 'set up your credentials'). _load_user_env_vars now propagates DB
errors; the resolver keeps them non-blocking only on the listing path.
Global-scope env var values hold admin-supplied plaintext secrets. They
were returned verbatim inside LiteLLM_MCPServerTable.env_vars to non-admin
and virtual-key callers via the server list/detail endpoints. Both
sanitizers now blank global-scope values while leaving per-user
placeholders intact.
The MCP servers list moved from a DataTable to an MCPServerCard grid, so
the create-flow e2e test's `table tbody` lookup could never match and
e2e_ui_testing failed once the server landed in the list. Tag the grid
with a stable data-testid and assert against it, keeping the original
scoping that prevents the closing modal's server_name input from
satisfying the check early
Prisma hydrates the env_vars JSON column on LiteLLM_MCPServerTable into
MCPEnvVar models, but build_mcp_server_from_table fed them straight into
MCPServer.env_vars (typed List[Dict[str, Any]]). Pydantic rejected the
model instances with a dict_type error, reload_servers_from_database
swallowed the exception, and the server was dropped from the registry; any
request for it then returned "server_not_found". This silently broke every
dashboard/DB-created MCP server that declared env_vars (the whole per-user
env-vars flow), while config-defined servers kept working because their
env_vars arrive from YAML as plain dicts.
_deserialize_json_list now normalizes Pydantic-model entries to plain dicts
via model_dump(mode="json"), matching the config path and the
List[Dict[str, Any]] contract.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(pass-through): move Gemini pass-through tests to gemini-3.1-flash-lite
gemini-2.5-flash-lite is a generation behind and is slated for discontinuation on Vertex AI no earlier than October 16, 2026, so the pass-through suite was exercising an aging model. Every reference now points at gemini-3.1-flash-lite, which is GA and already priced in the cost map so the spend-logging assertions still compute a real cost
test_vertex.test.js also gains jest.retryTimes(3) to match the sibling spend tests. The CI failures were intermittent 429 RESOURCE_EXHAUSTED from Vertex quota pressure, and that file was the only one without a retry, so a single rate-limited request was failing the whole job
* test(pass-through): point Vertex tests at the global endpoint for gemini-3.1-flash-lite
gemini-3.1-flash-lite is not served on the Vertex us-central1 regional endpoint for the CI project, so the Vertex pass-through tests were returning a deterministic 404 "Publisher Model ... was not found or your project does not have access to it" while the Gemini API tests passed. Move the Vertex clients to the global location, which the pass-through router maps to aiplatform.googleapis.com, where the 3.1 family is served
* fix(key_generate): allow team members to create keys on org-scoped teams
When a virtual key is created for a team, enterprise logic inherits the
team's organization_id onto the key (add_team_organization_id). Since the
VERIA-55 org-IDOR fix, /key/generate then required the caller to be an
explicit LiteLLM_OrganizationMembership member of that org, returning
403 "Caller is not a member of organization_id=<uuid>". Admins normally
only add users to teams (not orgs), so self-serve key creation regressed
for any user on an org-scoped team (regression since v1.84.0-rc.1).
Skip the org-membership check when organization_id was inherited from the
key's team (organization_id == team_table.organization_id). Team-level
authorization already gates this path, so team membership is sufficient.
The membership check still runs when a caller assigns an organization_id
that did not come from the key's team, preserving the IDOR protection.
Adds regression tests covering both the team-inherited (allowed) and
foreign-org (still blocked) cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(key_generate): cover mismatched team org IDOR path on generate
Add test_generate_key_foreign_org_with_mismatched_team_still_enforces_membership
for the case where a team is present but request organization_id differs from
team_table.organization_id. Enterprise inheritance is no-op'd in the test so
the guard is exercised directly; membership validation must still run.
Addresses Greptile review on #29310.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Retarget the per-server MCP env-vars PR from patch/v1.87.0-rc.1 to
litellm_internal_staging for the next nightly.
Conflicts resolved:
- _experimental/out/: synced the prebuilt dashboard wholesale to
litellm_internal_staging (the branch only renamed build artifacts;
internal_staging already uses the X/index.html layout).
- _experimental/mcp_server/db.py: kept the env_vars serialization block
and adopted internal_staging's mcp_info handling (read from data_dict
to respect the exclude_unset partial-update filter).
* fix(auth): let internal users view search tools
Internal users could not see search tools in the UI even when an admin
created them, while vector stores were visible. The Search Tools page
rendered but its list calls 403'd because the read routes were not in
internal_user_routes. Grant internal users read-only access to the
listing and provider routes; create/update/delete stay admin-only.
Resolves LIT-3150
* fix(auth): scope /search_tools/list to caller-allowed tools
Adding the read routes to internal_user_routes let any internal user
call /search_tools/list, which returned every configured tool's id,
provider, api_base, and metadata regardless of the caller's
object_permission.search_tools allowlist. The api_key was masked, so
this was metadata disclosure rather than a credential leak, but it
ignored the key/team scoping that /search already enforces.
Filter the listing through the same can_key_call_search_tool /
can_team_call_search_tool checks (exposed as a boolean
can_user_view_search_tool), so non-admin callers only see tools they
may invoke; admins still see all. Mirrors how /vector_store/list
scopes results.
* fix(types): resolve mypy errors in list_search_tools
The config and DB build loops reused one loop variable, so mypy pinned
it to the config element type (SearchToolTypedDict); its .get() calls
returned object and the DB element (SearchTool) failed the reuse
assignment. Give each loop its own variable so each gets its real type,
and coerce the config tool's SearchToolInfoTypedDict to a plain dict to
match SearchToolInfoResponse.search_tool_info.
* test(proxy/utils): pin ProxyLogging behavior
Add behavior-pinning tests for the ProxyLogging cluster in
litellm/proxy/utils.py under tests/test_litellm/proxy/utils/proxy_logging/.
Covers InternalUsageCache, _CallbackCapabilities, top-of-file helpers
(print_verbose, _get_email_logger_class, _accepts_litellm_call_info,
_enrich_http_exception_with_guardrail_context), the full ProxyLogging
class (lifecycle, MCP-LLM bridging, capability probes, guardrail
pipeline, pre/during/post/streaming hooks, alerting), plus the
bottom-of-region helpers (on_backoff, jsonify_object, _lookup_deprecated_key).
Each pinned symbol has happy-path and error-path coverage; happy paths
use direct dict-equality with three or more keys (or HiddenParams /
Pydantic model_validate where the surface is a Pydantic shape). The
subdirectory carries a local _pin_check.py and _coverage_check.py that
enforce the gate without surfacing numeric thresholds in CI logs.
Wires tests/test_litellm/proxy/utils into the existing test-path block
in .github/workflows/test-unit-proxy-endpoints.yml.
* test(proxy/utils): drop unused mock_httpx_client fixture
Declared in conftest.py but never referenced by any test. Removing
the dead fixture per Greptile P2 feedback.
* test(proxy/utils): drop local-only gate scripts from PR
_pin_check.py and _coverage_check.py are local stopping signals (not
wired into CI, consume a gitignored .pin_list.txt). They served their
purpose telling the engineer when to stop writing tests; the pytest
suite is the artifact that belongs in the repo.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test(proxy/utils): pin PrismaClient and spend-update behavior
PR2 of the litellm/proxy/utils.py behavior-pinning plan
(https://www.notion.so/37343b8acdab81f68f39f66915f62bcf). Adds
tests/test_litellm/proxy/utils/prisma_and_spend/, with happy + error
pins for every symbol in the PR2 list: the config-param cache,
PrismaClient lifecycle/data ops/engine watcher/reconnect/health
clusters, the user-row cache and SMTP helper, password/token helpers,
ProxyUpdateSpend, and the module-level spend functions.
Tests run against fully-mocked Prisma stacks (patched ``Prisma`` /
``PrismaWrapper`` at fixture setup), with a fake SMTP transport and a
clock-driven asyncio.sleep for the monitor loop, so unit runs need no
DB or network. ``_pin_check.py`` enforces happy + error coverage for
every symbol; ``_coverage_check.py`` filters branch + line coverage to
the PR2 source range (lines 2,668-5,541) and prints PASS / FAIL with
no numbers. Workflow shard ``tests/test_litellm/proxy/utils`` is added
to the existing proxy-endpoints job.
* test(proxy/utils): commit pin list and drop dead exclusion line
Addresses Greptile review feedback on PR #29488:
- Check in ``.pin_list.txt`` (force-added, overriding the repo-wide
``.gitignore`` rule) so reviewers can reproduce the ``_pin_check.py``
PASS shown in the PR description without first regenerating the
file from Notion.
- Remove the unreachable ``_harness_smoke_test.py`` continue in
``_pin_check.py``: the surrounding ``test_*.py`` glob already
excludes underscore-prefixed files; rephrase the docstring instead.
* test(proxy/utils): shift PR2 coverage line range by +1 after merge
``litellm_internal_staging`` added one line in ``ProxyLogging`` at
``utils.py:645`` (PR1 territory, before the PR2 region). Bump the
``_PR2_LINE_START`` / ``_PR2_LINE_END`` constants accordingly so the
coverage gate keeps scoring the same source region after the merge.
* test(proxy/utils): drop committed pin-list and gate scripts
``_pin_check.py``, ``_coverage_check.py``, and ``.pin_list.txt`` are
local-only stopping signals: no workflow or pytest collection invokes
them, so committing them adds rot risk (line-range drift in the
coverage check, pin-list staleness) without any enforcement upside.
The pin-list contract lives in the Notion plan; the tests themselves
are the durable artifact.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test(proxy/utils): pin bottom-of-file helper behavior
Pin current behavior of the bottom-of-file pure-function helpers in
litellm/proxy/utils.py (projection, team config, time helpers,
guardrail merge, error helpers, URL/path helpers, premium gate, model
access, and misc DB/API-key helpers).
Adds tests/test_litellm/proxy/utils/helpers/ with one happy + one error
test per pinned symbol; folds the prior single-test
tests/test_litellm/proxy/test_utils.py into test_url_helpers.py and
deletes the old file. _pin_check.py and _coverage_check.py serve as
local stopping gates.
Adds tests/test_litellm/proxy/utils to the existing test-path block in
.github/workflows/test-unit-proxy-endpoints.yml.
Plan: https://www.notion.so/37343b8acdab81f68f39f66915f62bcf
Pin list: https://www.notion.so/37343b8acdab8150acdbf40e5756869f
* test(proxy/utils): apply greptile fixes to behavior-pinning gates
Address findings from the sibling PR1/PR2 greptile reviews that also
apply to this PR:
- Commit pin_list.txt alongside the gate script (was previously a
gitignored .pin_list.txt fetched from Notion). The gate is now
reproducible without out-of-band setup.
- Resolve the coverage region by locating the first pinned symbol's
def line in litellm/proxy/utils.py at runtime, instead of hardcoded
line numbers that drift when lines above shift.
- Word-boundary the pin reference check so pins like update_spend do
not falsely match update_spend_logs_job.
- Drop the dead _harness_smoke_test.py exclusion; the test_*.py glob
already filters underscore-prefixed files.
* test(proxy/utils): drop local-only stopping-signal scripts
Remove _pin_check.py, _coverage_check.py, and pin_list.txt. These were
dev-time tooling for knowing when test authoring was done; they are
not wired into CI and the test files themselves are the merge artifact.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Auth failures on management endpoints such as team/list and
organization/list (invalid or expired keys) were raised as
ProxyException, whose __str__ returned an empty string, so the OTEL
SERVER span recorded an error with no message. ProxyException now
stringifies to its message, get_error_information prefers the explicit
.message attribute, and the proxy exception handlers stamp a consistent
error.type, error.code and error.message on the span
Resolves LIT-3515
* fix(auth): preserve 401 status for expired JWTs in OTel traces
Expired JWT access tokens raised a generic Exception with no status
code attached. Because the codeless exception was logged to OTel via
post_call_failure_hook before auth_exception_handler re-wrapped it as
ProxyException(401), the OTel span never set http.response.status_code
and trace viewers displayed it as a generic 500. Clients still got a
401 back, so traces and actual responses diverged.
Raise ProxyException(code=401, type=expired_key) directly at the source
in both JWT decode paths so the 401 is consistent across the client
response and the OTel http.response.status_code attribute, matching how
virtual-key expirations are handled.
* fix(auth): preserve 401 for expired JWTs on issuer-scoped path
The issuer-scoped JWT path (_auth_jwt_with_issuer) still raised a generic
Exception on expiry, surfacing as a 500 in client responses and OTel traces.
Raise ProxyException with expired_key/401 there too, matching auth_jwt, and
add a regression test exercising the issuer path end-to-end
