mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-03 02:22:24 +00:00
39560 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
e33ec2ed2b |
fix(mcp/ui): rename env vars section to Variables
These values interpolate into Static Headers and Auth via ${NAME};
they are not exported into the MCP server's process environment, so
'Environment Variables' overpromised. The backend env_vars field is
unchanged.
|
||
|
mateo-berri
|
ba7cab2c34 |
fix(mcp/ui): clarify per-user env var field as a hint, not a value
The per-user value column rendered as a plain input, so it read like a static value shared by every user, defeating the purpose of per-user variables. Add a persistent "Hint" addon with an explanatory tooltip, lighten the typed text, and align the row to the top so the addon group no longer sits lower than its neighbours and the layout stays put when the name field shows a validation error. |
||
|
mateo-berri
|
8852c0df95 |
fix(mcp): preserve env_vars in registry to table conversion
The GET /v1/mcp/server list and health endpoints build LiteLLM_MCPServerTable
from the in-memory registry via _build_mcp_server_table and health_check_server.
Both copied static_headers but dropped env_vars, so the list always returned
env_vars: null. The admin edit form is populated from that list data, so it
loaded an empty env-var list; saving any edit then persisted env_vars: [],
silently wiping the stored variables. With nothing left to interpolate, the
${VAR} static headers were forwarded upstream verbatim as literal text.
Carry env_vars through both conversions, mirroring static_headers. Add
regression tests asserting both paths round-trip name, scope, value, and
description.
|
||
|
mateo-berri
|
ba20078c05 |
fix(mcp): keep a created MCP server when post-write registry refresh fails
add_mcp_server wrote the new row and then reloaded the entire registry from the database inside one try block. A single pre-existing malformed row made the reload raise, so the endpoint returned 500 even though the new server was already persisted; callers assumed failure and retried, creating duplicate servers. Split the flow so the database write is the commit point and still 500s on failure, while the in-memory registry refresh is best-effort and only logged on error. Add regression tests for both the refresh-fails-after-commit path and the db-write-fails path |
||
|
mateo-berri
|
0dcf316f59
|
fix(mcp/ui): clear value for user-scope env vars before submit
A stale admin secret left in form state after switching an env var's scope from instance to per-user was forwarded to the backend as the user-scope value, which is returned unredacted to authorized non-admin users. Per-user entries carry no admin value, so drop it on submit. |
||
|
mateo-berri
|
e9eeca1bb7
|
fix(mcp): make per-user env var status write-only
The single-server and bulk per-user env var status endpoints echoed the decrypted credential value back to any holder of the user's LiteLLM token, so a leaked token could exfiltrate the raw upstream secret (e.g. a personal access token) for use outside the proxy. Drop the value field from MCPUserEnvVarSpec and the include_values plumbing so the status reports only whether each credential is_set; users overwrite a field to rotate it. The fill-in modal no longer pre-populates from the secret and flags already-set fields instead. |
||
|
mateo-berri
|
1f1bf4c421
|
fix(mcp): surface env-var fetch errors and authorize before existence check
getMCPUserEnvVars now throws on non-2xx so UserEnvVarsModal reports the error instead of silently rendering the empty 'no per-user fields' state. The per-user env-var endpoints now run the access check before the server lookup so a non-admin cannot tell a missing server (404) apart from one they lack access to (403), closing a server-id enumeration leak. |
||
|
mateo-berri
|
102b73a261
|
fix(mcp): redact global env var secrets from read-only admin server views
PROXY_ADMIN_VIEW_ONLY callers are treated as admin_view by GET /v1/mcp/server
and GET /v1/mcp/server/{id}, so they received unredacted scope=global env var
values, which can carry upstream API keys used in Authorization headers. Only
a full PROXY_ADMIN needs those values (to pre-fill the edit form); read-only
admins now get the same global-secret redaction already applied to non-admin
and restricted virtual-key views.
|
||
|
mateo-berri
|
236407a57b
|
fix(mcp): wrap get_mcp_server result in LiteLLM_MCPServerTable
The single-server DB fetch returned the raw Prisma model, whose JSONB
env_vars deserialize to plain dicts. The non-admin and virtual-key
sanitizers read env_var.scope as an attribute, so GET /v1/mcp/server/{id}
for a server with env_vars raised AttributeError and returned a 500.
Wrap the result like every bulk fetch helper so env_vars become MCPEnvVar
objects before redaction.
|
||
|
mateo-berri
|
e030c24800
|
Merge branch 'litellm_internal_staging' into litellm_mcp_server_env_vars | ||
|
mateo-berri
|
3392cd0e98
|
fix(mcp): URL-encode server_id in env-var setup deep link | ||
|
Mateo Wang
|
cc55662e5f
|
fix(vertex): strip output_config.effort for Vertex Claude models that reject it (Haiku 4.5) (#29585)
* fix(vertex): strip output_config.effort for models that reject it Haiku 4.5 on Vertex AI does not support output_config.effort and 400s with "output_config.effort: Extra inputs are not permitted". PR #27074 emptied VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS so effort would forward for Opus/Sonnet 4.6+, but that made the strip unconditional across every Vertex Anthropic model, including ones that don't support it. Claude Code injects effort into its default Messages payload, so `claude --model claude-haiku-4.5` started failing. Make the sanitizer model-aware: drop output_config.effort for models that don't advertise output_config support (or any reasoning effort level) while forwarding it for those that do. The fix covers both the chat-completion and Messages pass-through transformation paths since they share the helper. * chore(vertex): log at debug when dropping unsupported output_config.effort Operators pointing an unregistered Vertex Claude alias that does support effort would otherwise see it stripped with no signal. Debug level keeps it out of normal logs since Claude Code sends effort on every request. |
||
|
yuneng-jiang
|
34293fa80a
|
ci: reproduce default-Windows wheel install to guard MAX_PATH (#29597)
* ci: reproduce default-Windows wheel install to guard MAX_PATH The existing using_litellm_on_windows job installs the project with `uv sync`, an editable source install that never copies package files into a deep site-packages path, so it cannot see the 260-char MAX_PATH overflow that breaks `pip install litellm` on default Windows. The content-filter benchmark fixtures have hit that limit three times (#21941, #22039, #29536), each caught only after release. This adds a guard to the same job that builds the wheel and installs it the way an end user would: into a venv whose site-packages prefix is padded to a realistic worst-case Windows length (~100 chars), then asserts the install completes and litellm imports. Any packaged path long enough to bust MAX_PATH at that prefix is reported up front, so the check is deterministic regardless of the runner's long-path setting, while the real install also covers failure modes a length heuristic cannot (half-unpacked packages, reserved names, case collisions). This commit is the guard only; on the current tree it correctly fails because nine fixtures still exceed the limit. The rename that brings them back under it follows on this branch. * fix(packaging): shorten content-filter benchmark fixtures under MAX_PATH The 10 content-filter benchmark result fixtures used the legacy block_{topic}_-_contentfilter_({yaml}).json naming, up to 176 chars inside the wheel, which busts the Windows 260-char MAX_PATH limit once extracted under a realistic site-packages prefix and aborts `pip install litellm` on default Windows. Rename them to the short {topic}_cf.json scheme that _save_confusion_results already emits today (it splits the label on the em-dash and writes f"{topic}_cf"), matching the insults_cf.json and investment_cf.json files fixed earlier. Re-running the eval suite now regenerates these same short names rather than recreating the long ones. This drops the longest packaged path from 176 to 128, so the guard added in the previous commit goes from red to green with a 32-char margin. * test(windows): tidy MAX_PATH guard per review Close the wheel zip via a context manager rather than leaning on refcount collection, and select the wheel under dist/ by newest mtime so a stale artifact from an earlier build cannot be tested instead of the one just produced. Also pin down the venv-depth formula with a short note: the +2 is the separator joining the venv root to "Lib" plus the trailing separator before the entry, which lands the simulated site-packages prefix at exactly 100 chars. |
||
|
Mateo Wang
|
53a206a179
|
fix(anthropic/adapter): emit thinking block for reasoning_content-only streaming chunks (#29600)
* fix(anthropic/adapter): open thinking block for reasoning_content-only streaming chunks The /v1/messages streaming content-block classifier (_translate_streaming_openai_chunk_to_anthropic_content_block) only recognized thinking_blocks. OpenAI-compatible reasoning backends (vLLM/SGLang reasoning parsers: DeepSeek-R1, Qwen3, gpt-oss, ...) populate reasoning_content with thinking_blocks=None, so the classifier fell through to a text block. The delta translator already emits thinking_delta for reasoning_content, so those deltas landed inside a text block and Anthropic streaming clients (Claude Code, SDK .stream()) silently dropped the chain-of-thought. Mirror the reasoning_content fallback already present in the non-stream translator and the streaming delta translator so the classifier opens a thinking block. Adds a focused regression test. * fix(anthropic/adapter): reach reasoning_content branch when thinking_blocks attr is absent Delta deletes the thinking_blocks attribute when unset, so the prior nested check was unreachable for reasoning-only chunks (vLLM/SGLang). Make it a sibling elif so the content block is classified as thinking. * test(proxy): stop component-allowlist test leaking DATABASE_URL into xdist peers The component-allowlist test pins throwaway DATABASE_URL/LITELLM_MASTER_KEY values at import time via os.environ so importing proxy_server doesn't need a live database. Those values persisted for the whole pytest-xdist worker, so a sibling test sharing the worker (test_key_rotation_e2e's DB-backed E2E case) saw the leaked sqlite DATABASE_URL, treated it as an available database instead of skipping, and the Prisma engine rejected the non-postgres URL (P1012 -> httpx.ConnectError). Restore the prior environment after the import so the throwaway values never escape the module. --------- Co-authored-by: Tai An <antai12232931@outlook.com> |
||
|
mateo-berri
|
dbc5193fc0
|
style(mcp/ui): hoist MCPUserEnvVarsStatus import to top of networking.tsx | ||
|
mateo-berri
|
26f15b420c
|
fix(mcp): stop echoing credential values in bulk env-var status and let admins describe per-user vars
The bulk /user-env-vars/status feed only drives the dashboard "N fields missing" badge, which needs is_set, so it no longer returns the stored credential values; the single-server endpoint still returns them for the fill-in modal to pre-populate. Adds a description input to the admin env-var form for per-user scope so admins can tell users what to enter; the per-user modal already surfaces that description as a hint. |
||
|
Sameer Kankute
|
48c9fabb26
|
Fix : a2a bugs 030626 (#29566)
* Fix error code and context id injection bug * Add support for all A2A methods * Add logging * address greptile review: relay upstream JSON-RPC errors, move _PASCAL_TO_WIRE to module level, add error path tests * fix(a2a): run pre_call_hook for tasks/resubscribe SSE path to enforce guardrails tasks/resubscribe was returning the raw SSE stream without calling proxy_logging_obj.pre_call_hook, silently bypassing any guardrails configured on the agent. This patch calls pre_call_hook before streaming begins and wires post_call_failure_hook into the SSE generator so errors are logged. Adds a regression test verifying the hook is called. * fix(a2a): use get_async_httpx_client instead of creating httpx clients per request Creating httpx.AsyncClient instances per-request adds ~500ms latency. Switch _forward_jsonrpc and _forward_jsonrpc_sse to use the shared client from get_async_httpx_client(httpxSpecialProvider.A2A). * fix(a2a): forward caller identity headers on task ops; validate push notification URL Two security fixes for task management methods: 1. All task operations (tasks/get, tasks/list, tasks/cancel, tasks/resubscribe, push notification config methods) now forward X-LiteLLM-User-Id and X-LiteLLM-Team-Id headers to the upstream agent, so the agent can scope task access to the authenticated caller. 2. tasks/pushNotificationConfig/set validates the callback URL before forwarding: requires HTTPS and rejects private/loopback/reserved IP ranges and localhost hostnames to prevent SSRF. * Fix A2A task hook and push URL handling * fix(a2a): fix mypy type errors for request_id and header_name dict key types * Fix A2A request id and params forwarding * Forward trace IDs for A2A task calls * fix(a2a): strip client-forwarded X-LiteLLM-* headers before applying authenticated identity A client could send x-a2a-<agent>-x-litellm-user-id in their request and have it forwarded to the upstream agent as an authenticated identity header. Fix: sanitize any X-LiteLLM-* headers from agent_extra_headers before merging, then apply the authenticated identity headers last so they always override client-supplied values. * Fix A2A SSE fallback JSON-RPC error code * Fix A2A SSE error id backfill * fix(a2a): validate both push notification url fields to close SSRF bypass * fix(a2a): widen request_id annotation to match JSON-RPC id call sites * fix(a2a): run post-call streaming hook for tasks/resubscribe so agent guardrails apply tasks/resubscribe returned the raw upstream SSE stream without routing events through the post-call streaming hook, so output guardrails configured on the agent were silently skipped for streaming task subscriptions while every other task method and message/stream applied them. Parse upstream JSON-RPC SSE events and feed them through async_streaming_data_generator, matching message/stream, so guardrails inspect the streamed task content. Adds a regression test that fails when the streamed events bypass the guardrail hook. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
b38320d29c
|
fix(mcp): propagate delete DB errors; drop redundant import and duplicate payload assign
Make the per-user env-var delete idempotent with delete_many so a missing row is a no-op, and remove the bare except on the clear endpoint that swallowed real DB failures; a failed delete previously reported success to the dashboard while the row remained in the database, so the next tool call would 412 for credentials the user believed they had cleared. Also use the module-level json instead of a redundant inline import, and drop the no-op duplicate static_headers/env_vars assignment in the create-server UI payload. |
||
|
mateo-berri
|
9764f54f4d
|
fix(mcp): gate per-user env-var endpoints by server access
The per-user env-var endpoints (GET/POST/DELETE /server/{id}/user-env-vars)
fetched the server by id and returned its name, alias, and required-credential
metadata, or persisted/cleared stored values, without checking that the caller
can access that server. A non-admin could query or mutate env-var state for any
server id. Apply the same access gate fetch_mcp_server uses so non-admins are
limited to servers in their allowed set.
|
||
|
Sameer Kankute
|
c7ab9adde5
|
Litellm oss staging 030626 (#29578)
* Fix incorrect agent API request example payload structure (#29556) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span is stored in litellm_params['litellm_metadata'] instead of litellm_params['metadata']. When the request body contains a native 'metadata' field (e.g. Anthropic's {"user_id": "..."}), litellm_params['metadata'] gets overwritten and the parent span is lost, producing orphan root spans with a different trace_id. Add fallback checks to litellm_metadata in: - _get_span_context(): so child spans find the correct parent - _end_proxy_span_from_kwargs(): so the proxy span gets closed Fixes: https://github.com/BerriAI/litellm/issues/27934 * test(otel): tighten assertions per Greptile review - test_span_context_metadata_takes_priority: assert litellm_metadata span is never accessed, proving metadata takes priority - test_span_context_no_parent_when_neither_has_span: assert both ctx and detected_span are None --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: remove premature end-user budget check from get_end_user_object (#29420) * fix(proxy): remove premature end-user budget check from get_end_user_object Problem: - `_check_end_user_budget()` was called inside `get_end_user_object()` - This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated - Zero-cost models (e.g., local vLLM) were incorrectly blocked when end-users exceeded their budget, even though they should bypass budget checks Solution: - Remove `_check_end_user_budget()` calls from `get_end_user_object()` - Budget enforcement now happens exclusively in `common_checks()` where `skip_budget_checks` context is available - `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation. * refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object - test_get_end_user_object() verifies data fetching - test_check_end_user_budget() verifies enforcement - test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget() - test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object() * Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534) * Fix Gemini image config mapping * Address Gemini image config review * Format Gemini image generation transform * Fix Gemini image token usage logging * Share Gemini image request helpers * Fix Gemini Imagen model routing * Fixes as per self code review * Fixes per internal code review * Stop gating Imagen imageSize forwarding * Document Gemini image size mapping source * chore: retrigger lint * Clarify Gemini candidate count precedence * Add Inception provider (#29522) * add inception as provider (chat, fim) * linting * seperate test suite for chat and fim * fix test coverage * fix: model hub custom pricing model info (#29293) * Opik user auth key metadata extractors (#28397) * fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic * test: add unit tests for OPik metadata extraction logic * fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy * fix(ci): clarified comments and edited unit tests * test: add unit tests for OPik metadata extraction with auth and requester overrides * fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532) Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> * fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561) `_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls` so a following tool result can be matched back to its tool call. The assignment was inside a branch guarded by `assistant_msg.get("tool_calls", []) is not None`, which is also True for a text-only assistant message (an empty list is not None). As a result, an assistant message with no tool calls that appears between a tool call and its tool result overwrote the reference, and conversion failed with: Exception: Missing corresponding tool call for tool response message. This shape is common: a model emits a short narration/assistant message after a tool call before the tool result is appended. Only update `last_message_with_tool_calls` when the assistant message actually carries tool_calls (or a function_call). Adds a regression test. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models The 1-hour prompt-cache write tier (`cache_creation_input_token_cost_above_1hr`) was added to the us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but the eu./au./jp. cross-region inference profiles were left without it. AWS Bedrock pricing applies the same +10% regional premium across all geo profiles, so eu./au./jp. should carry the same 1-hour rates as us. (1.6x the 5-minute regional rate). Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL prompt caching falls back to the 5-minute write rate and undercounts spend by ~60% for European, Australian, and Japanese tenants. Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where AWS publishes one) to 14 regional Bedrock entries in both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - eu./au. Opus 4.6 ($11.00 / MTok) - eu./au. Opus 4.7 ($11.00 / MTok) - eu./au./jp. Sonnet 4.6 ($6.60 / MTok) - eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC) - eu./au./jp. Haiku 4.5 ($2.20 / MTok) Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py` with a `REGIONAL_EXPECTED` parametrized block covering all 13 new entries plus the existing 1.6x ratio invariant. Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06), which would break the 1.6x ratio check. It is intentionally left out of this PR so the scope stays "1-hour cache tier addition" — a separate follow-up should correct the EU 5m rates for Opus 4.5. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing tier for Vertex AI Anthropic models GCP Vertex AI publishes a separate 1-hour cache write column for the Claude family (1.6x the 5-minute write rate, matching the documented Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the 5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}` on Vertex AI Claude is undercounted in cost tracking by ~60%. The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig` extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and `_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`. Only the price registry was missing data. Adds the field to 19 vertex_ai/claude-* entries across both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - Haiku 4.5 ($1.25 -> $2.00 / MTok) - Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok) - Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok) - Opus 4 / 4.1 ($18.75 -> $30.00 / MTok) Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py` mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model and asserts the 1.6x ratio across the family. Fixes #27781. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Fix Gemini multimodal function responses (#29325) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * address greptile review: add _transform_image_usage method and model-map supports_image_size flag - Add _transform_image_usage instance method to GoogleImageGenConfig that delegates to transform_gemini_image_usage, fixing the regression test - Replace hardcoded "2.5-flash" string check in supports_gemini_image_size with a get_model_info lookup on supports_image_size (default true) - Add supports_image_size: false to all gemini-2.5-flash model entries in model_prices_and_context_window.json so capability is controlled via the model map rather than embedded in code * fix test failures: schema validation, mypy type, model info plumbing, pricing test - Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it - Pass supports_image_size through _get_model_info_helper constructor call - Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True) - Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid - Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values * Add Azure AI Kimi K2.6 metadata (#27052) * Add Azure AI Kimi K2.6 metadata * Scope Kimi metadata test cost map setup * fall back to substring check for models not in model_prices_and_context_window.json Models like gemini-2.5-flash-image-preview are not in the pricing JSON, so get_model_info raises. Fall back to "2.5-flash" not in model when the JSON has no explicit supports_image_size entry for the model. * fix(inception): don't forward global litellm.api_key to Inception FIM Match the Inception chat config: resolve only an Inception-specific key (param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion FIM path. The global litellm.api_key (often an OpenAI key) was both leaking to api.inceptionlabs.ai and taking precedence over the configured Inception key when set. * fix(auth): enforce end-user budget on custom-auth path that skips common_checks get_end_user_object() no longer raises BudgetExceededError, so custom-auth deployments with custom_auth_run_common_checks unset (which skip the centralized common_checks gate) stopped enforcing the end-user budget, letting an over-budget end user keep making requests. Re-enforce the budget in _run_post_custom_auth_checks on that path. --------- Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com> Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com> Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk> Co-authored-by: Lovro Seder <vrovro@gmail.com> Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com> Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
d781b62112
|
fix(mcp): surface DB errors on tool-call path and redact global env var secrets
A transient DB failure while loading per-user env vars on the tool-call path was swallowed and turned into a misleading MCPMissingUserEnvVarsError (412 'set up your credentials'). _load_user_env_vars now propagates DB errors; the resolver keeps them non-blocking only on the listing path. Global-scope env var values hold admin-supplied plaintext secrets. They were returned verbatim inside LiteLLM_MCPServerTable.env_vars to non-admin and virtual-key callers via the server list/detail endpoints. Both sanitizers now blank global-scope values while leaving per-user placeholders intact. |
||
|
Cursor Agent
|
7d11dd7def
|
style(mcp): black-format the env-vars deserialization regression test
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
ef341b82ed
|
fix(mcp/ui): scope create-server e2e assertion to the new card grid
The MCP servers list moved from a DataTable to an MCPServerCard grid, so the create-flow e2e test's `table tbody` lookup could never match and e2e_ui_testing failed once the server landed in the list. Tag the grid with a stable data-testid and assert against it, keeping the original scoping that prevents the closing modal's server_name input from satisfying the check early |
||
|
Cursor Agent
|
48e8f40150
|
fix(mcp): load DB-stored servers whose env_vars hydrate as MCPEnvVar models
Prisma hydrates the env_vars JSON column on LiteLLM_MCPServerTable into MCPEnvVar models, but build_mcp_server_from_table fed them straight into MCPServer.env_vars (typed List[Dict[str, Any]]). Pydantic rejected the model instances with a dict_type error, reload_servers_from_database swallowed the exception, and the server was dropped from the registry; any request for it then returned "server_not_found". This silently broke every dashboard/DB-created MCP server that declared env_vars (the whole per-user env-vars flow), while config-defined servers kept working because their env_vars arrive from YAML as plain dicts. _deserialize_json_list now normalizes Pydantic-model entries to plain dicts via model_dump(mode="json"), matching the config path and the List[Dict[str, Any]] contract. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
f3e2167730
|
test(pass-through): move Gemini pass-through tests to gemini-3.1-flash-lite (#29595)
* test(pass-through): move Gemini pass-through tests to gemini-3.1-flash-lite gemini-2.5-flash-lite is a generation behind and is slated for discontinuation on Vertex AI no earlier than October 16, 2026, so the pass-through suite was exercising an aging model. Every reference now points at gemini-3.1-flash-lite, which is GA and already priced in the cost map so the spend-logging assertions still compute a real cost test_vertex.test.js also gains jest.retryTimes(3) to match the sibling spend tests. The CI failures were intermittent 429 RESOURCE_EXHAUSTED from Vertex quota pressure, and that file was the only one without a retry, so a single rate-limited request was failing the whole job * test(pass-through): point Vertex tests at the global endpoint for gemini-3.1-flash-lite gemini-3.1-flash-lite is not served on the Vertex us-central1 regional endpoint for the CI project, so the Vertex pass-through tests were returning a deterministic 404 "Publisher Model ... was not found or your project does not have access to it" while the Gemini API tests passed. Move the Vertex clients to the global location, which the pass-through router maps to aiplatform.googleapis.com, where the 3.1 family is served |
||
|
milan-berri
|
b11833c737
|
fix(key_generate): allow team members to create keys on org-scoped teams (#29310)
* fix(key_generate): allow team members to create keys on org-scoped teams When a virtual key is created for a team, enterprise logic inherits the team's organization_id onto the key (add_team_organization_id). Since the VERIA-55 org-IDOR fix, /key/generate then required the caller to be an explicit LiteLLM_OrganizationMembership member of that org, returning 403 "Caller is not a member of organization_id=<uuid>". Admins normally only add users to teams (not orgs), so self-serve key creation regressed for any user on an org-scoped team (regression since v1.84.0-rc.1). Skip the org-membership check when organization_id was inherited from the key's team (organization_id == team_table.organization_id). Team-level authorization already gates this path, so team membership is sufficient. The membership check still runs when a caller assigns an organization_id that did not come from the key's team, preserving the IDOR protection. Adds regression tests covering both the team-inherited (allowed) and foreign-org (still blocked) cases. Co-authored-by: Cursor <cursoragent@cursor.com> * test(key_generate): cover mismatched team org IDOR path on generate Add test_generate_key_foreign_org_with_mismatched_team_still_enforces_membership for the case where a team is present but request organization_id differs from team_table.organization_id. Enterprise inheritance is no-op'd in the test so the guard is exercised directly; membership validation must still run. Addresses Greptile review on #29310. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
mateo-berri
|
af63258e40 |
Merge branch 'litellm_internal_staging' into litellm_mcp_server_env_vars
Retarget the per-server MCP env-vars PR from patch/v1.87.0-rc.1 to litellm_internal_staging for the next nightly. Conflicts resolved: - _experimental/out/: synced the prebuilt dashboard wholesale to litellm_internal_staging (the branch only renamed build artifacts; internal_staging already uses the X/index.html layout). - _experimental/mcp_server/db.py: kept the env_vars serialization block and adopted internal_staging's mcp_info handling (read from data_dict to respect the exclude_unset partial-update filter). |
||
|
Shivam Rawat
|
d45e9e4d56
|
fix(proxy): resolve managed video model ids for auth (#29545)
Some checks failed
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Unit Tests: Proxy DB Operations / schema-migration (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
* fix(proxy): resolve managed video model ids for auth Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): cover character_id router model resolution Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Yassin Kortam
|
8fbdfc7f0d
|
fix: missing mcp otel attributes (#29554) | ||
|
ryan-crabbe-berri
|
0a767ed14f
|
fix(auth): let internal users view search tools (#29542)
* fix(auth): let internal users view search tools Internal users could not see search tools in the UI even when an admin created them, while vector stores were visible. The Search Tools page rendered but its list calls 403'd because the read routes were not in internal_user_routes. Grant internal users read-only access to the listing and provider routes; create/update/delete stay admin-only. Resolves LIT-3150 * fix(auth): scope /search_tools/list to caller-allowed tools Adding the read routes to internal_user_routes let any internal user call /search_tools/list, which returned every configured tool's id, provider, api_base, and metadata regardless of the caller's object_permission.search_tools allowlist. The api_key was masked, so this was metadata disclosure rather than a credential leak, but it ignored the key/team scoping that /search already enforces. Filter the listing through the same can_key_call_search_tool / can_team_call_search_tool checks (exposed as a boolean can_user_view_search_tool), so non-admin callers only see tools they may invoke; admins still see all. Mirrors how /vector_store/list scopes results. * fix(types): resolve mypy errors in list_search_tools The config and DB build loops reused one loop variable, so mypy pinned it to the config element type (SearchToolTypedDict); its .get() calls returned object and the DB element (SearchTool) failed the reuse assignment. Give each loop its own variable so each gets its real type, and coerce the config tool's SearchToolInfoTypedDict to a plain dict to match SearchToolInfoResponse.search_tool_info. |
||
|
Yassin Kortam
|
08223e1ec3
|
fix: missing span for guardrail passthrough (#29552) | ||
|
yuneng-jiang
|
b175990b4a
|
test(proxy/utils): pin ProxyLogging behavior (#29485)
* test(proxy/utils): pin ProxyLogging behavior Add behavior-pinning tests for the ProxyLogging cluster in litellm/proxy/utils.py under tests/test_litellm/proxy/utils/proxy_logging/. Covers InternalUsageCache, _CallbackCapabilities, top-of-file helpers (print_verbose, _get_email_logger_class, _accepts_litellm_call_info, _enrich_http_exception_with_guardrail_context), the full ProxyLogging class (lifecycle, MCP-LLM bridging, capability probes, guardrail pipeline, pre/during/post/streaming hooks, alerting), plus the bottom-of-region helpers (on_backoff, jsonify_object, _lookup_deprecated_key). Each pinned symbol has happy-path and error-path coverage; happy paths use direct dict-equality with three or more keys (or HiddenParams / Pydantic model_validate where the surface is a Pydantic shape). The subdirectory carries a local _pin_check.py and _coverage_check.py that enforce the gate without surfacing numeric thresholds in CI logs. Wires tests/test_litellm/proxy/utils into the existing test-path block in .github/workflows/test-unit-proxy-endpoints.yml. * test(proxy/utils): drop unused mock_httpx_client fixture Declared in conftest.py but never referenced by any test. Removing the dead fixture per Greptile P2 feedback. * test(proxy/utils): drop local-only gate scripts from PR _pin_check.py and _coverage_check.py are local stopping signals (not wired into CI, consume a gitignored .pin_list.txt). They served their purpose telling the engineer when to stop writing tests; the pytest suite is the artifact that belongs in the repo. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
yuneng-jiang
|
457f65eff9
|
test(proxy/utils): pin PrismaClient and spend-update behavior (#29488)
* test(proxy/utils): pin PrismaClient and spend-update behavior PR2 of the litellm/proxy/utils.py behavior-pinning plan (https://www.notion.so/37343b8acdab81f68f39f66915f62bcf). Adds tests/test_litellm/proxy/utils/prisma_and_spend/, with happy + error pins for every symbol in the PR2 list: the config-param cache, PrismaClient lifecycle/data ops/engine watcher/reconnect/health clusters, the user-row cache and SMTP helper, password/token helpers, ProxyUpdateSpend, and the module-level spend functions. Tests run against fully-mocked Prisma stacks (patched ``Prisma`` / ``PrismaWrapper`` at fixture setup), with a fake SMTP transport and a clock-driven asyncio.sleep for the monitor loop, so unit runs need no DB or network. ``_pin_check.py`` enforces happy + error coverage for every symbol; ``_coverage_check.py`` filters branch + line coverage to the PR2 source range (lines 2,668-5,541) and prints PASS / FAIL with no numbers. Workflow shard ``tests/test_litellm/proxy/utils`` is added to the existing proxy-endpoints job. * test(proxy/utils): commit pin list and drop dead exclusion line Addresses Greptile review feedback on PR #29488: - Check in ``.pin_list.txt`` (force-added, overriding the repo-wide ``.gitignore`` rule) so reviewers can reproduce the ``_pin_check.py`` PASS shown in the PR description without first regenerating the file from Notion. - Remove the unreachable ``_harness_smoke_test.py`` continue in ``_pin_check.py``: the surrounding ``test_*.py`` glob already excludes underscore-prefixed files; rephrase the docstring instead. * test(proxy/utils): shift PR2 coverage line range by +1 after merge ``litellm_internal_staging`` added one line in ``ProxyLogging`` at ``utils.py:645`` (PR1 territory, before the PR2 region). Bump the ``_PR2_LINE_START`` / ``_PR2_LINE_END`` constants accordingly so the coverage gate keeps scoring the same source region after the merge. * test(proxy/utils): drop committed pin-list and gate scripts ``_pin_check.py``, ``_coverage_check.py``, and ``.pin_list.txt`` are local-only stopping signals: no workflow or pytest collection invokes them, so committing them adds rot risk (line-range drift in the coverage check, pin-list staleness) without any enforcement upside. The pin-list contract lives in the Notion plan; the tests themselves are the durable artifact. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
yuneng-jiang
|
1aed5e1bbd
|
test(proxy/utils): pin bottom-of-file helper behavior (#29509)
* test(proxy/utils): pin bottom-of-file helper behavior Pin current behavior of the bottom-of-file pure-function helpers in litellm/proxy/utils.py (projection, team config, time helpers, guardrail merge, error helpers, URL/path helpers, premium gate, model access, and misc DB/API-key helpers). Adds tests/test_litellm/proxy/utils/helpers/ with one happy + one error test per pinned symbol; folds the prior single-test tests/test_litellm/proxy/test_utils.py into test_url_helpers.py and deletes the old file. _pin_check.py and _coverage_check.py serve as local stopping gates. Adds tests/test_litellm/proxy/utils to the existing test-path block in .github/workflows/test-unit-proxy-endpoints.yml. Plan: https://www.notion.so/37343b8acdab81f68f39f66915f62bcf Pin list: https://www.notion.so/37343b8acdab8150acdbf40e5756869f * test(proxy/utils): apply greptile fixes to behavior-pinning gates Address findings from the sibling PR1/PR2 greptile reviews that also apply to this PR: - Commit pin_list.txt alongside the gate script (was previously a gitignored .pin_list.txt fetched from Notion). The gate is now reproducible without out-of-band setup. - Resolve the coverage region by locating the first pinned symbol's def line in litellm/proxy/utils.py at runtime, instead of hardcoded line numbers that drift when lines above shift. - Word-boundary the pin reference check so pins like update_spend do not falsely match update_spend_logs_job. - Drop the dead _harness_smoke_test.py exclusion; the test_*.py glob already filters underscore-prefixed files. * test(proxy/utils): drop local-only stopping-signal scripts Remove _pin_check.py, _coverage_check.py, and pin_list.txt. These were dev-time tooling for knowing when test authoring was done; they are not wired into CI and the test files themselves are the merge artifact. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
ryan-crabbe-berri
|
f047b1571e
|
fix(otel): capture 401 error details in management endpoint spans (#29535)
Auth failures on management endpoints such as team/list and organization/list (invalid or expired keys) were raised as ProxyException, whose __str__ returned an empty string, so the OTEL SERVER span recorded an error with no message. ProxyException now stringifies to its message, get_error_information prefers the explicit .message attribute, and the proxy exception handlers stamp a consistent error.type, error.code and error.message on the span Resolves LIT-3515 |
||
|
ryan-crabbe-berri
|
9d9558e78f
|
fix(auth): preserve 401 status for expired JWTs in OTel traces (#29510)
* fix(auth): preserve 401 status for expired JWTs in OTel traces Expired JWT access tokens raised a generic Exception with no status code attached. Because the codeless exception was logged to OTel via post_call_failure_hook before auth_exception_handler re-wrapped it as ProxyException(401), the OTel span never set http.response.status_code and trace viewers displayed it as a generic 500. Clients still got a 401 back, so traces and actual responses diverged. Raise ProxyException(code=401, type=expired_key) directly at the source in both JWT decode paths so the 401 is consistent across the client response and the OTel http.response.status_code attribute, matching how virtual-key expirations are handled. * fix(auth): preserve 401 for expired JWTs on issuer-scoped path The issuer-scoped JWT path (_auth_jwt_with_issuer) still raised a generic Exception on expiry, surfacing as a 500 in client responses and OTel traces. Raise ProxyException with expired_key/401 there too, matching auth_jwt, and add a regression test exercising the issuer path end-to-end |
||
|
Yassin Kortam
|
3a1c6bba97
|
feat(proxy): native /health/drain preStop hook for graceful shutdown (#29439) | ||
|
Mateo Wang
|
a5ccd96152
|
[internal copy of #29003] fix(vertex_ai): use user-supplied api_base as is for Model Garden OpenAI-compat path (#29530)
* fix(vertex_ai): use user-supplied api_base as is for Model Garden OpenAI-compat path
* chore(tests): url assertions and outputs
* fix(tests): fixing reference to unused test
* fix(aiohttp): drop octet-stream content-type on bodyless requests
The aiohttp transport forwarded httpx's empty request body straight
to aiohttp, which attaches a default Content-Type: application/octet-stream
for any bytes payload. Bodyless requests such as DELETE /responses/{id} then
hit OpenAI with that header and were rejected with unsupported_content_type,
breaking the e2e_openai_endpoints test_basic_response check. Coercing an
empty body to None makes aiohttp behave like the httpx transport and send no
content-type for bodyless requests.
---------
Co-authored-by: Steven Kessler <9701252+stvnksslr@users.noreply.github.com>
|
||
|
Mateo Wang
|
6a9f542f81
|
test: stabilize batch VCR coverage and stop live upload/network leaks (#29477)
* test: stabilize batch VCR coverage * test: replay bedrock batch s3 uploads * test: stop batch tests leaking live uploads * test: keep bedrock batch workflow off live s3 * test: mock bedrock batch workflow network * test: accept realtime guardrail refusal wording * test: update gemini thought signature model * test: quiet logging worker atexit flush * test: address Greptile review on batch VCR fixes Handle content= bodies in the bedrock batch post stub so payload extraction does not raise a TypeError when a request omits json and data. Restore litellm list state faithfully by preserving None instead of coercing it to an empty list, so callbacks that start as None are not turned into [] after a test. Set logging.raiseExceptions inside the try block in the atexit flush so the finally always restores the previous value. * test: scope atexit logging suppression to the drain loop Wrap only the queue drain loop in LoggingWorker._flush_on_exit with the logging.raiseExceptions toggle so the process-wide global is suppressed for the smallest possible window, keeping other threads' logging error reporting intact outside the loop. * test: cover atexit flush error-swallow branch in LoggingWorker The _flush_on_exit drain loop was wrapped in a try/finally to scope the logging.raiseExceptions toggle, which reindented the existing edge-case branches into the diff and dropped patch coverage below target. Add a regression test that enqueues a coroutine which raises during the atexit flush and asserts the failure is swallowed while later queued events are still drained, exercising the silent-failure path directly. |
||
|
Mateo Wang
|
3f33efdd57
|
fix(tests): drop import-time completion call in test_register_model (#29521)
* fix(tests): drop import-time completion call in test_register_model test_update_model_cost_via_completion() was invoked at module scope, so it ran during pytest collection and fired a live OpenAI completion. The local test jobs glob the whole tests/local_testing folder and let pytest import every file, narrowing what runs only afterward with -k, so this call executed in every one of those jobs regardless of their filter. When the request failed (for instance a 429 once the OpenAI account hit its quota), collection of the file errored and aborted the entire session, which is why langfuse, assistants, router and local_testing_part2 all reported "ERROR collecting tests/local_testing/test_register_model.py" and never ran their own tests. Remove the stray call and add a regression that parses the module and fails if any locally defined function is invoked at module scope again * test: also guard async def from module-scope invocation ast.AsyncFunctionDef is a distinct node from ast.FunctionDef, so an async test invoked at module scope would have slipped past the guard. Collect both kinds of definitions * fix(responses): send Content-Type application/json on OpenAI responses requests OpenAI's responses API now rejects body-less requests (GET/DELETE) that arrive without a content type, returning 500 "Unsupported content type: 'application/octet-stream'. This API method only accepts 'application/json' requests". litellm's create path got the header for free because httpx sets it when a json body is present, but the delete/get handlers send no body and so sent no content type. The official OpenAI SDK declares Content-Type: application/json on every request; mirror that in validate_environment so all OpenAI responses calls carry it. This is what made tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_basic_response fail on the responses.delete() call. |
||
|
Mateo Wang
|
f81d8ae077
|
[internal copy of #29232] feat: route future Claude models to Anthropic provider via pattern matching (#29239)
* feat: route future Claude models to Anthropic provider via pattern matching
Add pattern-based matching for Claude model names so that future models
(e.g., claude-opus-4-9, claude-sonnet-5-0) are automatically routed to
the Anthropic provider without requiring model_prices_and_context_window.json
updates.
The pattern matches: claude-{opus|sonnet|haiku}-{major}-{minor}[-YYYYMMDD]
https://claude.ai/code/session_017asCVDN5jBFMBcZRjiQR6C
* fix: don't hard-code the tier names
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* style: move import re to module level (PEP 8)
Move `import re` from inside the module body to the top-level imports
section, following PEP 8 style guidelines that all imports should
appear at the top of the file.
https://claude.ai/code/session_01Dt8fzn81eYMfxu1MoBa5hN
* test: fix claude-mini-4-5 assertion to match generic-tier pattern
The pattern intentionally accepts any [a-z]+ tier (see
|
||
|
ryan-crabbe-berri
|
d991c47018
|
fix(ui/agents): make A2A skill tags enterable and validated (#29512)
* fix(ui/agents): make A2A skill tags enterable and validated
Skill tags were marked required but rendered as a comma-split text input
that couldn't surface validation and let empty values save. Switch tags
and examples to Select tag inputs, drop the misleading "Required" skills
label (the API allows zero skills), and validate the full configure step
so an added skill must be complete before advancing.
Resolves LIT-3153
* fix(ui/agents): allow Enter to create skill tags/examples
Drop open={false} from the tags and examples Select inputs. With the
dropdown forced closed, AntD suppresses the "create from input" option,
so pressing Enter (as the placeholder instructs) did nothing. Matches the
existing extra_headers Select.
|
||
|
Sameer Kankute
|
ae7ac72331
|
feat(agents): add LangFlow agent provider with A2A session bridging (#28963)
* feat(agents): add LangFlow agent provider with A2A session bridging
Register LangFlow as a completion provider and agent type (UI + /api/v1/run),
and map A2A contextId to LangFlow session_id for multi-turn conversations.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(providers): document langflow in provider_endpoints_support.json
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(agents): address Greptile review for LangFlow integration
Move A2A contextId→session_id mapping into LangFlow A2A provider config,
add langflow.svg logo, remove live integration test, use model for token count.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(langflow): prevent flow_id override via request optional_params
Derive flow_id only from the authorized model name and reject flow_id
kwargs so callers cannot invoke a different LangFlow run endpoint.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(langflow): remove redundant flow_id branch in _get_flow_id
* fix(langflow): surface an error when the run response has no extractable message
Previously the response parser returned the raw JSON blob as the assistant
message when it could not find message text, silently presenting an
unparseable payload as a valid answer. It now returns None and the caller
raises a LangFlowError so the failure is visible to the client.
* fix(langflow): URL-encode flow_id path segment to prevent path injection
flow_id is taken from the model suffix and interpolated into
/api/v1/run/{flow_id}. Without path-segment encoding a model such as
langflow/../../x (or one containing ?) could move the request off the run
endpoint to another path on the configured LangFlow server using the
operator x-api-key. Encode the segment with quote(safe="") so it always
stays a single path segment.
* fix(langflow): reject empty flow_id from model name
* fix(langflow): return stripped flow_id so validation matches URL path
* fix(langflow): reject caller-supplied tweaks to prevent flow component override
* fix(langflow): reject caller-supplied tweaks injected via extra_body
The transform_request guard only inspected optional_params, but extra_body
is popped before transform_request runs and merged into the request body
afterward, letting a caller reintroduce tweaks and override the
operator-configured LangFlow flow components. Validate the final request
body in sign_request so tweaks cannot reach LangFlow through extra_body.
* test(langflow): move provider tests into mirrored coverage path
The langflow tests lived under tests/llm_translation/, whose CircleCI job
runs without --cov and uploads nothing to Codecov, so none of the new
langflow code counted toward patch coverage (codecov/patch reported 9.78%
of the diff hit against a 70.83% target).
Relocate them to tests/test_litellm/llms/langflow/, which the GitHub
Actions provider job runs with --cov=./litellm and uploads, and add
regression tests for the previously untested happy paths (transform_response
building the ModelResponse with usage, non-JSON body handling, last-user
message extraction, outputs-dict response shape, sign_request pass-through,
error class and stream flags). Patch coverage on the diff is now ~88%.
* fix(langflow): require litellm_params in A2A config instead of silent empty fallback
* fix(langflow): scope A2A session_id to the authenticated key
The LangFlow A2A bridge used the LangFlow session_id verbatim from the
client-controlled A2A contextId, so two distinct virtual keys authorized for
the same agent could read or append to each other's LangFlow conversation
memory by reusing a contextId.
Hand the authenticated key hash to the completion bridge through litellm_params
and namespace the forwarded session_id with it. The same key keeps a stable
session across turns, while different keys can no longer collide on a shared
contextId. The principal is hashed before it is embedded in the session_id, so
the stored token is never sent to the LangFlow backend; the original contextId
is preserved as a suffix for operator-side correlation.
* fix(langflow): wire authenticated key hash through A2A bridge and tests
Define A2A_USER_API_KEY_HASH_PARAM in the completion bridge handler, strip it
before litellm.acompletion, inject the authenticated key hash at the proxy A2A
endpoint, and add regression tests for per-key LangFlow session scoping.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
c1602587c1
|
fix(tests): drop module-level test calls that break local_testing collection (#29520)
* fix(tests): drop module-level test calls that break local_testing collection Several files in tests/local_testing invoked their test functions at module scope (e.g. test_register_model.py ran test_update_model_cost_via_completion() at the bottom of the file). Those calls execute during pytest collection, so they fire real network requests at import time. test_register_model.py's call hit an OpenAI 429 and raised, turning into a collection error. A collection error aborts the whole session for every job that globs tests/local_testing/**/test_*.py, which is why unrelated jobs like langfuse_logging_unit_tests (-k langfuse) and litellm_assistants_api_testing (-k assistants) both failed even though neither touches register_model; the -k filter only applies after collection. pytest discovers and runs these test_* functions on its own, so the top-level calls were dead and harmful. Removes them from test_register_model.py, test_wandb.py, test_lunary.py, and test_multiple_deployments.py, and adds a regression test that scans the directory for module-level test invocations. * test(local_testing): skip unparseable files in module-scope invocation guardrail A syntax error in any tests/local_testing file would make ast.parse raise an unhandled SyntaxError, so the guardrail itself would crash with a confusing traceback instead of its assertion message. Such a file already fails pytest collection on its own, which is the clearer signal, so the guardrail now skips files it cannot parse and stays focused on detecting module-scope test calls. Reads files as utf-8 for deterministic behavior across platforms. |
||
|
Sameer Kankute
|
4a81ec4982
|
feat(proxy): add per-MCP-server RPM rate limiting for keys and teams (#29482)
* feat(proxy): add per-MCP-server RPM rate limiting for keys and teams
Adds mcp_rpm_limit, a dict keyed by MCP server name (alias if set, else the
configured name) that caps requests per minute per server for a key or team.
The v3 rate limiter builds a per-server descriptor only when a limit is
configured for the server being called, so other servers stay uncapped and no
TPM reservation is engaged. Server identity is surfaced into the request data
via mcp_rate_limit_server_name so the limiter can resolve it.
* fix(proxy): gate MCP rpm descriptors on call_mcp_tool; document mcp_rpm_limit param
Only honor mcp_server_name when the call is an actual MCP tool call. Without
this, a normal LLM request could inject mcp_server_name in its body to consume
a target server's MCP quota and 429 legitimate tool calls. Also adds the
mcp_rpm_limit parameter docstring to update_key, new_user, and user_update so
the API docs validator passes.
* Fix MCP rate limit quota handling
* Delete scripts/test_mcp_rpm_limit.sh
* docs(proxy): clarify mcp_rpm_limit is enforced for keys and teams, not per user
* fix(proxy): accept mcp_rpm_limit in generate_key_helper_fn
NewUserRequest and GenerateKeyRequest inherit mcp_rpm_limit from
GenerateRequestBase, so /user/new and /key/generate forwarded the field
to generate_key_helper_fn, which did not accept it and returned a 500
("unexpected keyword argument 'mcp_rpm_limit'"). Accept the param and
store it in metadata, matching model_rpm_limit/model_tpm_limit, so the
limit is persisted where get_key_mcp_rpm_limit reads it.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
ryan-crabbe-berri
|
ebbc5cc787
|
feat(vector-stores): forward per-request params to Vertex AI Search (#29459)
* feat(vector-stores): forward per-request params to Vertex AI Search The vertex_ai/search_api search transform hardcoded the request body to query plus pageSize 10, dropping max_num_results and extra_body. Map max_num_results to pageSize and merge extra_body through with precedence, so callers can send native Discovery Engine fields such as dataStoreSpecs. Resolves LIT-3506 * fix(vector-stores): log effective query when extra_body overrides it When a caller passes a query inside extra_body, the outbound Vertex Search request used that value but model_call_details recorded the original, so the echoed search_query was stale. Log the effective query from the request body. * fix(vector-stores): allowlist Vertex AI Search extra_body fields Raw-merging extra_body let callers set dataStoreSpecs/branch to search a different Discovery Engine data store with the proxy's Vertex credentials, bypassing the vector_store_id path authorization. Reject target-selecting fields and forward only allowlisted per-request tuning fields. Resolves LIT-3506 * refactor(vector-stores): split Vertex AI Search extra_body allowlists by mode Data-store and engine/app serving configs accept different SearchRequest fields, so derive two TypedDicts (VertexSearchDataStoreExtraBody and VertexSearchEngineExtraBody) in types/vector_stores.py and make _filter_extra_body mode-aware via vertex_engine_id. dataStoreSpecs and numResultsPerDataStore now pass through in engine/app mode (where an app fans out across stores) and are rejected in data-store mode. branch/servingConfig/entity remain rejected in both modes. * fix(vector-stores): raise BadRequestError (400) for invalid Vertex Search extra_body Rejecting unsupported or target-selecting extra_body fields previously raised a bare ValueError, which the vector store error path mapped to a generic APIConnectionError (HTTP 500). Raise litellm.BadRequestError so invalid per-request input surfaces as HTTP 400 with a clear message. |
||
|
Mateo Wang
|
6d6eda8101
|
[internal copy of #28008] Support MCP OAuth passthrough and issuer-scoped JWT auth (#28356)
* fix(proxy): point /metrics 401 at the opt-out flag Operators upgrading past |
||
|
milan-berri
|
efaafbbd02
|
fix(proxy): strip NUL bytes from spend log payloads to prevent PostgreSQL 22P05 (#29515)
A raw NUL byte (\x00) in request/response content is serialized by json.dumps
into the \u0000 JSON escape. When update_spend_logs writes this to the
LiteLLM_SpendLogs jsonb columns, Postgres rejects the whole batch with
error 22P05 ("unsupported Unicode escape sequence ... cannot be converted to
text"), crashing the periodic update_spend job and dropping the spend-log batch.
Centralize stripping in safe_dumps (covers metadata/response paths and any
future caller) and route the messages, proxy_server_request, request_tags, and
response (string branch) payloads through it instead of json.dumps. Dict keys
are stripped too.
Adds regression tests for safe_dumps and the spend-log message, response, and
request_tags payload builders.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Yassin Kortam
|
ce7b1fd29d
|
fix(passthrough): emit otel guardrail span when a guardrail blocks (#29470)
* fix(passthrough): emit otel guardrail span when a guardrail blocks The otel_v2 logger emits guardrail spans from its post-call hooks by reading standard_logging_guardrail_information off the top-level metadata of the dict handed to those hooks. On passthrough, post-call guardrails run against a throwaway hook_data dict (metadata was already stripped off _parsed_body by _init_kwargs_for_pass_through_endpoint), so a deny that raises a non ModifyResponseException records its logging info on hook_data and then the generic failure handler forwards _parsed_body, which no longer carries it. The span was therefore present on allow but missing on block; the unified path keeps metadata on the same dict it passes to the failure hook, so its span always shows. Carry the guardrail logging entries recorded on hook_data over to the request_data forwarded to post_call_failure_hook so the failure path matches the unified path. Resolves LIT-3510 * test(passthrough): cover guardrail-logging carry helper; simplify helper Address review feedback on the guardrail-block span fix. Simplify _carry_guardrail_logging_info: the realistic failure path always builds fresh metadata on request_data, so the merge-into-existing-list branch was dead code. Use setdefault with a shallow-copied list so the carried entries never share the source hook_data list reference. Drop the module-level sys.modules proxy_server mock from the otel span test; pass_through_endpoints imports proxy_server lazily, so it is unnecessary and avoided the test-isolation risk of registering a mock under that key. Add pure unit tests for _carry_guardrail_logging_info (no otel dependency) that pin its contract: carries entries, copies the list, populates existing metadata without clobbering prior guardrail entries, and no-ops when there is nothing to carry. * test(passthrough): cover deny-path guardrail logging forwarding without otel The otel span regression test skips in coverage jobs that lack the optional opentelemetry package, leaving the failure-handler wiring (capturing hook_data and carrying its guardrail logging info) uncovered. Add an otel-independent regression that drives the real pass_through_request through a post-call deny and asserts post_call_failure_hook receives request_data carrying the standard_logging_guardrail_information. Fails on the pre-fix code. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yassin Kortam
|
b98a656254
|
Add MCP semantic conventions to otelv2 (#29468)
* Add MCP semantic conventions to otelv2
Emit OpenTelemetry GenAI MCP tool-call spans from the v2 logger. A closed
call_mcp_tool request now produces a CLIENT span named "tools/call {tool}"
carrying mcp.method.name, gen_ai.operation.name=execute_tool, gen_ai.tool.name,
the upstream server name, and (opt-in, content-gated) tool arguments/result.
Adds the MCP and JSON-RPC attribute vocabulary to the semconv module, an
MCPToolCallSpanData payload built from StandardLoggingMCPToolCall, an
MCP_TOOL_CALL span role, and mapper support.
* Complete the MCP span-attribute vocabulary in otelv2 semconv
Add the remaining OTel GenAI MCP semconv attribute keys: gen_ai.prompt.name,
the network.* transport keys with their well-known NetworkTransport values, and
the client.* peer keys for MCP server spans. A test pins the full vocabulary so
a dropped or renamed key fails loudly.
* Populate mcp.session.id on MCP tool-call spans
Capture the mcp-session-id header (case-insensitively) at the tool-call entry
point and thread it through StandardLoggingMCPToolCall into the span, so spans
for stateful MCP sessions carry mcp.session.id. Stateless calls have no such
header and the attribute is simply absent.
* Test that stateless MCP calls omit mcp.session.id
---------
Co-authored-by: Claude <noreply@anthropic.com>
|