A persistent Rubrik webhook outage previously let authenticated traffic
accumulate prompt/response payloads in the in-memory retry queue
without bound. The PR-introduced retry-on-failure behavior in
flush_queue() never trims the queue, so under sustained outage and
high request volume the proxy can run out of memory.
Cap the queue at RUBRIK_MAX_QUEUE_SIZE events (default 10_000) and
drop the oldest events when the cap is exceeded. Emit a throttled
verbose_logger warning so operators can detect a stuck webhook.
- Centralize SSE 'data:' chunk parsing in litellm.responses.sse_output_recovery
so the ChatGPT Responses transformer and the Responses->Chat-Completions bridge
share a single implementation.
- Log a warning when get_supported_openai_params drops 'tools' for a
fireworks_ai model whose JSON entry sets supports_function_calling=false,
so users notice the behavioral change instead of silently losing tools.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The TypeError raised in _normalize_tool_calls is caught by apply_guardrail's
broad except, which logs the message plus exc_info. Including repr(tc) in
the message could expose function arguments (potentially sensitive user
data) in the proxy log stream. Type name alone is enough for debugging.
The Bedrock Invoke chat and messages transformations strip output_config
when neither supports_output_config nor any supports_*_reasoning_effort
flag is set in the model JSON. This was silent; emit a verbose_logger
warning when the strip actually removes a present output_config so newly
released models (where the JSON entry hasn't caught up yet) surface a
clear log line instead of dropping the effort parameter without notice.
* feat: add Xiaomi MiMo-V2.5-Pro and MiMo-V2.5 OpenRouter model entries (#27700)
Squash-merged by litellm-agent from TorvaldUtne's PR.
* fix(ui): trim whitespace from MCP inspector tool call inputs (#28203)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* gemini-3.1-flash-lite pricing (#27933)
* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers
* fix pricing
* add service tier
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* fix: incorrect /v1/agents request example (#28131)
* fix(anthropic): accept dict-shape reasoning_effort from Responses bridge (#28201)
* fix(anthropic): accept dict-shape reasoning_effort from Responses bridge
Issue #28196 — the Responses->Chat parser (transformation.py:184-200) keeps the full dict as reasoning_effort when summary is set; that branch was added in #25359. But the Anthropic transformation here still guarded on isinstance(value, str), silently dropping the param. Result: callers using the standard Reasoning(effort, summary) OpenAI-shaped object on Anthropic lose thinking entirely (0 reasoning_tokens, no thinking_blocks).
Coerce dict -> string before mapping. Same shape tolerance that gpt_5_transformation._normalize_reasoning_effort_for_chat_completion already implements. summary is irrelevant for Anthropic's thinking_blocks.
Adds two regression tests: one parametrized over string + dict shapes (with and without summary), one covering unparseable dict inputs (drops silently, no crash).
* test(anthropic): add non-adaptive model coverage for dict-shape reasoning_effort
Per Greptile feedback on PR #28198: the original regression test only exercised the adaptive (4.6+) path. Add a parametrized test for the non-adaptive branch (claude-sonnet-4-5) verifying that dict-shape reasoning_effort still maps to thinking.type='enabled' + budget_tokens, and that output_config is NOT set on pre-4.6 models.
* test(anthropic): convert unparseable-dict test to @pytest.mark.parametrize
Per @greptile-apps inline review on PR #28201 — matches the parametrize style of the two adjacent dict-shape tests and produces clearer failure messages (test ID per case instead of one collapsing for-loop).
* feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite (#28280)
Squash-merged by litellm-agent from ro31337's PR.
* fix(router): wrap aresponses streaming iterator for mid-stream fallbacks (#28215)
Squash-merged by litellm-agent from cwang-otto's PR.
* fix(router): unblock staging — mypy + coverage for aresponses streaming fallback (#28318)
Squash-merged by litellm-agent from cwang-otto's PR.
* fix(responses): forward timeout on completion transformation path (Anthropic, Bedrock, Vertex) (#28133)
Squash-merged by litellm-agent from cwang-otto's PR.
* feat(ui): add pause/resume Switch to the models table (#28151)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(responses): merge sync completion kwargs to avoid duplicate keys
Double-splatting litellm_completion_request and kwargs raised TypeError
when metadata or service_tier were set. Match the async merge pattern.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Use proxy base URL for CLI SSO form action (#28271)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which was missing from the
cost map. This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
litellm.completion_cost lookup.
- Add mistral/ministral-8b-2512 entry to both the in-tree
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json (mirrors the
existing openrouter/mistralai/ministral-8b-2512 pricing).
- litellm.model_cost is loaded at import time from the URL pinned to
main, so the new backup entry isn't visible at test runtime until
it also lands on main. Backfill any entries missing from the
remote-fetched map into litellm.model_cost in the local_testing
conftest so cost-calculator lookups succeed on this branch.
* fix(tests): drop unnecessary del of conftest backfill loop vars
* fix(router): harden streaming fallback wrapper for bridge iterators
- FallbackResponsesStreamWrapper now uses getattr fallbacks when copying
attributes from the source iterator. The bridge path
(LiteLLMCompletionStreamingIterator used by Anthropic/Bedrock/Vertex)
does not call super().__init__ and is missing response, logging_obj
(it uses litellm_logging_obj), responses_api_provider_config,
start_time, request_data, call_type, and _hidden_params. Previously,
wrapper construction raised AttributeError for any streaming fallback
on the bridge path.
- _aresponses_with_streaming_fallbacks now deep-copies the
litellm_metadata (and metadata) dicts into fallback_kwargs. The
primary attempt mutates this dict in place via
_update_kwargs_with_deployment, so a shallow copy of kwargs was
leaking primary-deployment fields (deployment, model_info, api_base)
into the mid-stream fallback request.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(router): use safe_deep_copy for fallback metadata snapshot
The ban_copy_deepcopy_kwargs CI check rejects copy.deepcopy() on any
variable whose name contains 'kwargs' (incl. fallback_kwargs). Swap
the two copy.deepcopy(fallback_kwargs[...]) calls for safe_deep_copy,
which handles non-picklable values (OTEL spans, etc.) by per-key
deepcopy with fallback to the original reference.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(ci): skip chronically flaky build_and_test integration tests
Both tests have been failing on every recent run of build_and_test
against this PR's HEAD (1686967, 1688402, 1689993, 1690877), and the
same two tests also fail intermittently on unrelated commits and other
branches, independent of any code change in this PR (which only touches
router fallback wrappers, the Anthropic Responses bridge, and unrelated
UI/cost-map files).
- tests.test_spend_logs.test_spend_logs: /spend/logs?request_id=...
returns 500 even after a 20s wait for the spend log to be written.
Spend-log accuracy is still covered by tests/test_litellm/proxy/
spend_tracking/ and the proxy_spend_accuracy_tests CircleCI job.
- tests.test_team_members.test_add_multiple_members: /team/info?team_id=
... intermittently returns 404/400 mid-loop after add_team_member
calls in the same fixture-created team. Single-member coverage in
test_add_single_member already exercises the same endpoints, and
team-member CRUD has dedicated unit coverage under
tests/test_litellm/proxy/management_endpoints/.
Skipping unblocks the build_and_test job until the underlying race in
the dockerized integration setup is root-caused.
* fix: preserve explicit timeout=0 in responses API handler
Use 'timeout if timeout is not None else request_timeout' instead of
'timeout or request_timeout' so an explicit timeout=0/0.0 isn't silently
replaced by the default request_timeout.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui): guard model_info access in pause Switch with optional chaining
* fix(ui): guard model_info access in pause Switch onChange handler
Mirror the optional-chaining guard already applied to the isPausing
check so a config-model row with a missing model_info cannot throw
when the toggle's onChange fires.
---------
Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params
Operators have reported large numbers of idle Prisma connections that
never get closed. The proxy already forwards `connection_limit` and
`pool_timeout` to the DATABASE_URL, but had no knob for capping idle
or slow connections. Add three new `general_settings` keys that thread
through to the DATABASE_URL / DIRECT_URL query string:
- `database_connect_timeout` -> Prisma `connect_timeout`
- `database_socket_timeout` -> Prisma `socket_timeout` (the main
knob for closing idle connections from the LiteLLM side)
- `database_extra_connection_params` -> untyped passthrough dict for
any other Prisma URL param (`pgbouncer`, `statement_cache_size`,
`sslmode`, ...); keys here override LiteLLM defaults.
Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a
single `_build_db_connection_url_params` helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/proxy_cli.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
The aclose() method closed both HTTP clients but did not cancel the
periodic flush task. After close, the task would wake up every
flush_interval seconds and try to POST via the now-closed
async_httpx_client, generating recurring errors.
Cancel the task and await its termination before closing the clients.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The test creates a team via /team/new, adds a member via /team/member_add,
then queries /team/info — and intermittently gets a 404 for a team that
was just successfully created and mutated. The basic happy path is
already covered by test_add_single_member; we only lose the 10-iteration
stress loop.
* fix(interactions): never drop streamed text deltas; always emit terminal completion
The interactions streaming bridge had two bugs flagged by Greptile on PR #28153:
1. The first OutputTextDeltaEvent (and the second, when no ResponseCreatedEvent
precedes the deltas) was consumed to emit a synthetic interaction.created /
step.start event, but the chunk's text payload was never forwarded as a
step.delta. The text only reappeared in the terminal step.stop, which
defeats the purpose of incremental streaming.
2. When the upstream Responses API stream ended via StopIteration without a
ResponseCompletedEvent, the iterator emitted step.stop but never the
terminal interaction.completed event carrying the full collected text.
This refactors the iterator to translate each upstream chunk into a list of
events (instead of a single event) and buffers them in a deque. A text delta
now expands into [interaction.created, step.start, step.delta] on the first
chunk so no token is dropped, and the StopIteration / StopAsyncIteration
fallback always flushes a terminal interaction.completed event when one
hasn't already been sent.
Both behaviors are covered by new unit tests:
- test_no_text_token_is_dropped_during_streaming
- test_response_created_then_text_delta_emits_step_start_and_delta
- test_stop_iteration_fallback_emits_completion_event
- test_response_completed_emits_stop_then_completion (no double-emit)
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(interactions): correlate EOF terminal events with stream's interaction id
The StopIteration fallback path previously built the terminal step.stop /
interaction.completed events with id=None (legacy content.stop) and a
memory-address fallback string (interaction.completed), neither of which
matched the item_id used by the earlier interaction.created / step.start /
step.delta events in the same stream. Downstream consumers correlating
events by id would see a mismatch.
Persist the interaction id derived from the first upstream chunk (item_id
on an OutputTextDeltaEvent, or response.id on a ResponseCreatedEvent) and
reuse it when flushing the terminal events on EOF.
Author: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci(windows): raise UV_HTTP_TIMEOUT to 300s for uv sync
The using_litellm_on_windows job has been hitting flaky PyPI download
timeouts during 'uv sync --frozen --group dev' — different packages on
each rerun (six, pydantic-core), all surfacing the same uv error:
Failed to download distribution due to network timeout.
Try increasing UV_HTTP_TIMEOUT (current value: 30s).
uv's default 30s per-request timeout is too tight for the Windows runner
on this project (50+ deps, several multi-MB wheels), so bump it to 300s
to let slow individual downloads complete instead of failing the build.
* fix(interactions): correlate ResponseCompletedEvent terminal events with stream's interaction id
When a stream starts directly with OutputTextDeltaEvent (no preceding
ResponseCreatedEvent), interaction.created carries item_id while
interaction.completed previously carried response.id from
ResponseCompletedEvent. The two ids can differ, leaving consumers that
correlate events by id unable to match the start and completion events.
Fall back to self._interaction_id (set on the first chunk that derives
an id) before response.id, mirroring the EOF terminal path.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Reusing resolved_project (typed str from the fast path's tuple unpack)
for an Optional[str] assignment tripped mypy. Use project_id directly
after the None check.
- chatgpt responses: don't overwrite a meaningful error_message with None
when a later RESPONSE_FAILED/ERROR event lacks an error object.
- vertex_ai: serve STALE tokens from the lock-free fast path and only
schedule a deduplicated background refresh, eliminating per-key lock
contention near token expiry.
- rubrik: aclose() now closes both async_httpx_client and
tool_blocking_client to avoid leaking connections from the dedicated
client when the logger shuts down.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Regression tests for the case where the upstream caller (initialize_guardrail)
passes event_hook=None and the logger should still fall back to post_call,
and the sanity case where an explicitly-set non-None event_hook is preserved.
initialize_guardrail always passes event_hook=litellm_params.mode, so
setdefault never applied its default. When mode is omitted from the
guardrail config, event_hook ended up as None instead of post_call.
Use 'or' to fall back to the intended default when the value is None.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The non-streaming transform_response folds xAI's reasoning_tokens into
completion_tokens before calling _normalize_openai_compatible_usage_totals,
preserving the OpenAI invariant total = prompt + completion. The streaming
chunk_parser only ran the normalization, so when xAI streamed usage with
reasoning tokens (total = prompt + completion + reasoning), the normalize
check (total < prompt + completion) was a no-op and the invariant remained
violated.
Refactor _fold_reasoning_tokens_into_completion to also accept a raw usage
dict (in addition to ModelResponse / Usage) and call it from the streaming
chunk_parser before normalization, so streaming and non-streaming paths
report usage consistently for reasoning models.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
After reauthentication or fresh load, the resolved (cache_credentials, project_id)
cache key may point to stale credentials from a prior load. Skipping the write
when the key existed forced the next request to go through a redundant
refresh/reauth cycle. Always overwrite so callers using the resolved project_id
hit the fresh credentials object.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(mcp): allow native MCP OAuth redirect URIs (cursor://)
Discoverable OAuth /authorize rejected cursor:// callbacks because
validate_trusted_redirect_uri only accepted http/https. Add an
allowlisted native path with a built-in Cursor default and optional
MCP_TRUSTED_NATIVE_REDIRECT_URIS env for other clients.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): address Greptile native redirect URI review
Lowercase paths in normalizer so env allowlist entries match case-
insensitively. Tighten wildcard prefix matching to reject sibling
paths (e.g. callback-2) unless the prefix ends with /.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): reject query params on native OAuth redirect URIs
Greptile: normalization stripped query strings before allowlist compare,
so cursor://.../callback?injected=... could pass validation. Reject any
native redirect_uri with a query component (same as fragments).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(model_cost_map): add mistral/ministral-8b-2512 entry
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which is not in the cost map.
This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
completion_cost lookup. Add the entry mirroring the existing
openrouter/mistralai/ministral-8b-2512 pricing.
* fix(mcp): lowercase default native redirect URIs
Make _parse_trusted_native_redirect_uris apply the same lowercasing
to built-in defaults as it does to env-var entries.
* fix(tests): backfill local model_cost into remote-fetched map
litellm.model_cost is loaded at import time from the URL pinned to main,
so pricing entries that exist only in this branch (e.g.
mistral/ministral-8b-2512, freshly added because Mistral now returns this
id from mistral-tiny) are absent at test time and completion_cost lookups
raise. Backfill the in-tree backup so cassette-driven cost calculations
resolve against the entries that ship with the branch under test.
Fixes the local_testing_part1 failures on test_completion_mistral_api and
test_completion_mistral_api_modified_input.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
- Reject reducto:// file IDs sent through the proxy /v1/ocr JSON API.
The IDs are not bound to a LiteLLM key, so an authenticated user
could submit another user's file ID and receive OCR text via the
proxy's shared Reducto credentials. Force fresh uploads (multipart
form or inline base64 data URI) so every OCR call is server-mediated
and implicitly bound to the originating request.
- Add ocr_cost_per_credit=0.015 to reducto/parse-v3 and
reducto/parse-legacy in both pricing JSONs so successful Reducto OCR
calls debit key/team spend instead of recording zero.
- vertex_llm_base: guard background refresh's cache write with an
identity check so a stale write cannot overwrite a credentials
reference replaced by a concurrent reauthentication path.
- router: make shared backend mode preservation directional - only
preserve when an existing 'responses' mode would be downgraded to
'chat', or when the deployment mode is None (which would otherwise
clear the existing mode). Legitimate upgrades now apply.
- rubrik: remove unused preserve_events_added_during_flush attribute;
RubrikLogger overrides flush_queue, so the base-class flag never
applied. Drop the test that exercised the parent path on a Rubrik
instance since it does not reflect real flush behavior.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vertex_ai): omit function_call id on Vertex Gemini 3.5+ tool turns
Vertex AI rejects `id` on function_call/function_response parts; only Google AI Studio accepts it for Gemini 3.5+ strict tool matching.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(vertex_ai): forward custom_llm_provider in context caching
Pass custom_llm_provider through to _gemini_convert_messages_with_history
in the context caching path so Gemini 3.5+ tool-call `id` forwarding
behaves consistently between cached and non-cached completions on Google
AI Studio.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <claude@anthropic.com>
Previously the inner guard 'if _deployment_mode is not None' prevented
_shared_model_info['mode'] from being set back to the existing shared
mode when the deployment mode was None, which then overwrote the shared
backend's mode with None via register_model.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Both ChatGPTResponsesAPIConfig and LiteLLMResponsesTransformationHandler
duplicated the same OUTPUT_ITEM_DONE / OUTPUT_TEXT_DONE recovery
algorithm. Move that logic into litellm.responses.sse_output_recovery
and have both call sites use the shared helpers, so future fixes apply
in one place.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
refresh_auth is invoked from three call sites that can run on different
threads (sync get_access_token, async slow path via asyncify, and the
background proactive refresh task). Only the sync path was protected
by _sync_refresh_lock, so a concurrent sync + async/background call
could invoke google-auth's Credentials.refresh() on the same object
from two threads simultaneously, mutating internal credential state.
Move the lock acquisition into refresh_auth itself; the lock is an
RLock so reentrant acquisition from the sync path remains safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Replace the asyncio.Lock._waiters inspection in
_maybe_prune_async_refresh_lock with an explicit refcount so the entry
is pruned exactly when no coroutine is holding or waiting on the lock,
without depending on any private asyncio internals.
* test(ui-e2e): add admin key creation with a specific proxy model
Adds Playwright coverage for creating a key (no team) scoped to a single
proxy model, complementing the existing All-Proxy-Models test. Uses a
DOM-dispatched click on the antd dropdown option since the popup
animation can render the option outside the viewport.
* test(ui-e2e): verify scoped key works against mock /chat/completions
Extend the "Create a key with a specific proxy model" test to extract
the new key from the success modal and POST to /chat/completions for
the scoped model, asserting 200 and the mock response body. Without
this the test could pass even if the model selection failed to register.
- Build a memoized index of fireworks_ai/* entries from litellm.model_cost,
invalidated by (id, len) of the model_cost dict. Avoids re-scanning the
full ~30k-entry model_cost dictionary on every get_provider_info call.
- Replace plain substring containment with hyphen-aligned boundary matching
so a known short model name (e.g. 'some-model') cannot falsely match an
unrelated longer query (e.g. 'awesome-model').
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(interactions): migrate to Google Interactions API steps schema (May 2026)
Default to Api-Revision: 2026-05-20 (new `steps` schema). Add
`litellm.use_legacy_interactions_schema` global flag that sends
Api-Revision: 2026-05-07 for operators who need the legacy `outputs`
schema until June 8, 2026.
- Inject Api-Revision header in GoogleAIStudioInteractionsConfig.validate_environment()
- Auto-coalesce response_mime_type → response_format and image_config migration on new schema
- Add steps field to InteractionsAPIResponse and InteractionsAPIStreamingResponse
- Add StepStart/StepDelta/StepStop/InteractionCreated/etc. SSE event types
- Update streaming completion detection to handle interaction.completed event
- Bridge transformer populates both outputs and steps fields
- Bridge streaming iterator emits new-schema events by default
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): address greptile review feedback
- Avoid mutating caller's generation_config dict by shallow-copying
before popping image_config, preventing silent failures on retries
- Skip schema key in response_format when response_format is None to
avoid sending schema: null to the Google Interactions API
- Remove delta field from step.stop events (new schema only); the
StepStop model has no delta field and sending it duplicates already-
streamed text and breaks spec-conformant clients
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): parse use_legacy_interactions_schema string values safely
bool("false") returns True in Python, so quoted YAML values like
"false" or "False" silently activated the legacy Interactions API
schema. Match the env-var parsing pattern in litellm/__init__.py by
treating string inputs as true only when they equal "true" (case
insensitive).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(interactions): only set object/id/delta on step.stop for legacy schema
StepStop (new schema) has no object, id, or delta fields. Setting them
unconditionally caused spec-breaking extra fields on new-schema step.stop
events in all four construction sites (sync/async × main-loop/StopIteration).
Legacy content.stop still receives id, object, and delta unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): stabilize streaming bridge schema, dict aliasing, and lost first delta
- Capture use_legacy_interactions_schema once at iterator construction so
all events emitted by a single stream use a consistent schema, even if
the global flag is mutated mid-stream.
- Check for the buffered interaction.complete/completed event before the
finished check in __next__/__anext__ so the final completion event
(which carries the full collected text in steps) is not dropped after
self.finished is set.
- Copy text content entries before appending to both outputs and the
steps content list to avoid shared mutable dict aliasing between the
two response fields.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix tests
* fix greptile review
* fix(interactions): address Greptile P1 review on schema coalescing and legacy deltas
Skip response_mime_type merge when response_format is already a list, avoid
in-place list mutation on image_config append, and restore delta.type on
legacy content.delta events.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(interactions): black-format gemini transformation.py
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): JWT on tools/list, REST server_id resolution, tool_server_mismatch
Sign outbound MCP JWTs for list_mcp_tools and inject headers on the tools/list
path. Resolve server_id on /mcp-rest/tools/call and return 403 tool_server_mismatch
when the tool does not belong to the requested server. Default missing arguments to {}.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): restrict list JWTs to mcp:tools/list and default REST arguments to {}
- List-only JWTs (call_type=list_mcp_tools) no longer carry the broad
mcp:tools/call scope. _build_scope() now emits only mcp:tools/list
when no tool name is provided, mirroring the existing least-privilege
rule that tool-call JWTs omit mcp:tools/list.
- REST /tools/call now defaults a missing 'arguments' field to {} so
execute_mcp_tool() and downstream **arguments / .keys() calls don't
receive None and crash with TypeError/AttributeError.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): validate tool/server in call_tool; skip JWT signer when not configured or static auth present
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): align tests and mypy with user_api_key_auth on tools/list
Update mocks for the new _get_tools_from_server parameter, mock server
registry in REST access-denied test, and narrow static_headers for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): accept user_api_key_auth in get_tools_from_mcp_servers mock
The side_effect for the all-servers case did not accept the new kwarg,
so tools/list returned an empty list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): fail fast for unknown tools when server mapping exists
Server-name fallback in call_tool must not open an upstream session when
the tool is absent from a populated mapping. Update the HTTP transport test
to register a known tool before asserting not-found behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix mypy
* Fix mypy
* fix(mcp): preserve tools/call scope on missing tool name; pass user_api_key_auth in list_tools
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): match alias/server_name in _resolve_mcp_server_for_tool_call
The registry lookup in _resolve_mcp_server_for_tool_call previously only
compared candidate.name against the provided server_name, but tool name
prefixes can be derived from a server's alias or server_name (see
get_server_prefix). When the tool→server mapping is empty/stale (cold
start, dynamic tools), the lookup would fail for alias-configured
servers even though get_mcp_server_by_name (used by the REST path)
matches alias, server_name, and name.
Match the same priority of identifiers in both the registry pass and
the unprefixed fallback so the MCP protocol call_tool path is
consistent with the REST path.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): reuse proxy_logging DualCache in inject_mcp_jwt_headers_for_upstream
Instead of allocating a fresh DualCache() on every tools/list invocation,
prefer the shared proxy_logging_obj.internal_usage_cache.dual_cache when
available. The cache argument is currently unused by MCPJWTSigner, but
sharing the proxy's cache avoids per-call allocation overhead and matches
the cache identity used elsewhere in the proxy hook plumbing — so any
future per-request state stored in cache will survive across list calls.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): return 403 ip_filtering for IP-restricted servers in tools/call name lookup
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(test): accept user_api_key_auth kwarg in list_tools mocks
The proxy-infra job was failing on four TestMCPServerManager tests because
the mock_get_tools_from_server stubs did not accept the new
user_api_key_auth keyword argument that list_tools now forwards to
_get_tools_from_server. Add the kwarg to each stub so list_tools can call
through cleanly.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): skip JWT injection when per-user mcp_auth_header is set
MCPClient._get_auth_headers() applies extra_headers AFTER writing
Authorization from auth_value, so an injected JWT silently overwrites
the user's per-server OAuth token. Guard the JWT signer with
'not mcp_auth_header' so per-user OAuth (and any dict-form per-user
auth) takes precedence, mirroring the existing static_headers guard.
Adds a regression test that the signer's inject helper is not called
when mcp_auth_header is supplied.
* fix(mcp): skip JWT injection when extra_headers already has Authorization
When a server uses per-user OAuth tokens, the resolved token is passed
into _get_tools_from_server via extra_headers. The JWT injection guard
only checked mcp_auth_header and the server's static headers, so the
signer would silently overwrite the user's OAuth Authorization header.
Add a check for an existing Authorization entry in extra_headers so
caller-supplied per-user OAuth tokens take precedence over JWT signing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(mcp): cover JWT signer + tool-call resolution branches
Adds unit tests for the new MCPServerManager helpers (_resolve_mcp_server_for_tool_call,
_resolve_oauth2_headers_for_tool_call) and the new MCPJWTSigner paths
(_build_scope call_type branches and inject_mcp_jwt_headers_for_upstream).
Brings patch coverage above the auto target without changing behavior.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): retry tool-server lookup with prefixed name in REST mismatch check
When the REST /mcp-rest/tools/call path sends a raw tool name plus
requested_server_id, _get_mcp_server_from_tool_name(name) can return
None if the mapping only stores the prefixed form. That bypassed the
tool_server_mismatch 403 guard and let the call fall through to
trusting requested_server.
Retry the lookup with every known prefix of the requested server so
the mismatch check fires whenever the tool is actually registered.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): always reject unknown tools in server-name fallback
Defense-in-depth: _resolve_mcp_server_for_tool_call previously skipped
the unknown-tool check whenever the per-server mapping had no entries
yet (cold start, OAuth2 lazy listing, or upstream listing failure),
allowing arbitrary tool names to reach upstream servers.
Tighten the check so the server-name fallback always rejects tool
names not present in the mapping. Callers must call list_tools first
(standard MCP flow) before tools/call can resolve. Removes the
now-unused _mapping_has_tools_for_server helper and adds an
explicit empty-mapping rejection test alongside the existing
populated-mapping rejection test.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude (greptile subagent) <claude-greptile-bot@anthropic.com>
Raise a clear ValueError when Reducto /upload returns 200 without a
file_id key (or with a non-JSON body), instead of letting downstream
callers see a confusing KeyError.
Address the two new findings from greptile's 19:45 review of the
vertex+router surfaces.
- vertex_llm_base: when the slow path sees TokenState.INVALID, await any
in-flight background refresh task before invoking refresh_auth
ourselves. google-auth's Credentials.refresh() is not safe to call
concurrently on the same credentials object, and the background task
runs outside the per-key lock. After the wait, re-check the cached
token so we can short-circuit if the background refresh already
restored it. Extracted the helper into
_await_in_flight_background_refresh so get_access_token_async stays
under ruff's PLR0915 statement budget.
- router.py: when alias registration would overwrite the deployment's
declared `mode` to keep the shared backend mode stable, emit a
verbose_router_logger.warning so the override is visible to operators
instead of silently winning. The existing fix (preventing alias
registration from downgrading a shared `mode: responses` to chat) is
preserved; the warning just surfaces it.
- Fireworks: _get_model_cost_capability fallback now picks the longest
substring match in model_cost so more specific entries win over less
specific ones (instead of returning the first match by insertion order).
- Reducto OCR: drop per-request _api_key/_api_base instance attributes on
_BaseReductoOCRConfig and instead thread api_key/api_base through
transform_ocr_request/async_transform_ocr_request kwargs from the
shared OCR HTTP handler. Makes the config safe to share/cache across
concurrent requests with different credentials.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Swap the # 5a / # 5b step labels so they appear in numerical order
within the file. The new output_config-strip block was added with
label # 5b above the pre-existing # 5a 'remove custom field from
tools' block; rename the new block to # 5a and the pre-existing
block to # 5b so the labels match the order of the steps in the
file.
No behavior change.
Co-authored-by: Greptile Reviewer <greptile-apps@users.noreply.github.com>
Address greptile's P2 thread (#3144753707) about ocr_cost silently
under-reporting billing when response.usage_info.pages_processed is
missing. The credit-priced and unpriced fallback still has to return
0.0 (we don't know how to bill without usage), but emit a warning so
the missing-data case is visible in logs instead of disappearing.
The per-page-priced branch still raises, preserving the original
ValueError signal callers may catch.
* feat(ui): team allowed_passthrough_routes create parity + edit load fix
Add the Allowed Pass Through Routes selector to the create-team modal
(previously only on the edit form), and fix the edit form silently
dropping the field: it lives under team metadata, so initialValues must
read info.metadata.allowed_passthrough_routes — otherwise the selector
renders empty and saving wipes admin-set routes. Both selectors are
gated to premium proxy admins, mirroring the server-side gate.
Resolves LIT-3019
* fix(ui): persist team allowed_passthrough_routes edits on save
The edit form loaded the selector but the save path never wrote it back:
allowed_passthrough_routes stayed in the raw metadata JSON textarea and
parsedMetadata (from that textarea) always won, so selector edits were
silently discarded. Strip it from the textarea initialValues and overlay
values.allowed_passthrough_routes into updateData.metadata, mirroring how
guardrails is handled.
Resolves LIT-3019
* fix(ui): preserve team passthrough routes for non-proxy-admins on save
Only proxy admins may set allowed_passthrough_routes (server-side gate).
For non-proxy-admins, write the team's stored value back into metadata
instead of the form value, so saving an unrelated setting can't silently
wipe routes; omit the key entirely when the team never had any.
Resolves LIT-3019
* chore(e2e): migrate runner to uv, add All Proxy Models key test
Switches the local e2e runner (run_e2e.sh) from poetry to uv to match
the rest of the repo and CI. Adds a Playwright test for creating an
admin key with no team selected (all-proxy-models flow), a SLOWMO env
hook for headed debugging, and a MIGRATION_TRACKING.md doc that maps
the manual UI QA checklist to e2e tests so future migration work has
a single source of truth.
* chore(e2e): address greptile feedback
- Remove MIGRATION_TRACKING.md (docs belong in litellm-docs repo)
- playwright.config.ts: fall back to 0 when SLOWMO is non-numeric
(parseInt returns NaN, which Playwright accepts silently)
- run_e2e.sh: add --frozen to uv sync for CI determinism
When a new filter is applied to spend logs, React Query's keepPreviousData
left stale rows on screen for 10–15s with no indication that a fetch was
in progress. The previous custom isFilteringResults flag was removed in
the #25847 toolbar refactor and only partially restored on the Fetch
button. Use React Query's isPlaceholderData to discriminate a real
filter change (queryKey changed, data not yet arrived) from a same-key
live-tail refetch, and feed it into the existing isLoading prop on the
toolbar pagination text and the table body. Live-tail polls still keep
previous rows without flicker.
Co-authored-by: Ryan <ryan@Ryans-MBP.localdomain>
* fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints
Azure code-interpreter containers return provider-native IDs (cntr_ + hex)
that carry no LiteLLM routing payload, so _decode_container_id returns
model_id=None. The router was falling through to call the handler directly,
bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for
Azure deployments. Fall back to the model_id forwarded from the proxy
ownership check so deployment credentials are always applied.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url
When a deployment's api_base is the responses endpoint URL
(e.g. .../openai/responses?api-version=...), AzureContainerConfig was
appending /openai/containers on top of it, producing the broken path
.../openai/responses/openai/containers. Azure returns 404 for that URL
while the correct path is .../openai/containers.
Strip any /openai/responses suffix from api_base before constructing
the containers URL so the resource root is always used as the starting point.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): prefer api-version from api_base URL over deployment's api_version
The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses
API and is too old for the containers API, which requires 2025-04-01-preview.
The responses endpoint api_base already carries the correct api-version in its
query string. Extract it and use it for the containers URL, overriding the
stale deployment-level version.
Fixes DELETE and file-upload operations returning 404 due to wrong api-version.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(containers): pass params=None instead of params={} to httpx to preserve api-version
httpx erases a URL's query-string when params={} (empty dict) is passed,
silently stripping ?api-version=2025-04-01-preview from every container
POST/DELETE request. Azure's GET endpoints tolerate a missing api-version;
POST (upload) and DELETE are strict, so those returned 404.
Fix: use `params or None` in container_handler._async_handle and
llm_http_handler.async_container_delete_handler (and all sibling container
handlers) so that an empty params dict falls back to None, leaving httpx to
preserve the URL's existing query string intact.
Adds a regression test that directly documents the httpx behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): remove elif model_id branch from _init_containers_api_endpoints
Two reviewer findings addressed:
1. Truncated comment on the model_id fallback line — now complete.
2. Security: the elif branch that fired when container_id was absent allowed
any authenticated caller to supply model_id in a POST /v1/containers body
and route the request through an arbitrary deployment UUID, bypassing the
model-level access checks that only validate `model`. Removed the elif
branch; operations without container_id (create, list) route by the
caller-supplied `model` field as before. model_id forwarding is kept only
inside the container_id block, where the proxy ownership check has already
validated the container before forwarding the deployment ID.
Adds a regression test pinning the security boundary: no-container-id path
calls original_function directly even when model_id is in kwargs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(containers): validate proxy-to-router model_id forwarding for managed IDs
Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id
to verify that get_container_forwarding_params (the proxy-side half of the Azure
routing fix) correctly extracts and forwards model_id from a LiteLLM-managed
encoded container ID.
This closes the gap identified by Greptile P1: the previous regression test
only injected model_id as a direct kwarg, validating the router in isolation.
The new test exercises the actual proxy-to-router data flow through
ownership.get_container_forwarding_params, confirming that kwargs["model_id"]
is populated before _init_containers_api_endpoints is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): tighten endpoint-path strip to endswith match
Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so
the suffix strip only fires when api_base actually ends with one of the
endpoint-specific path suffixes. This is the more precise check greptile
flagged on the original find()-based implementation.
* Fix sync container handler to preserve URL query string
Mirror the async path fix: pass None instead of an empty params dict so
httpx does not strip the URL's existing query string (e.g.
?api-version=...), which is required for Azure container routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(azure-containers): strip trailing slash before endpoint suffix match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(containers): recover model_id from stored encoded id for native Azure container IDs
get_container_forwarding_params previously only set model_id when the
user-supplied container_id was a LiteLLM-managed encoded id. For native
upstream IDs (e.g. Azure 'cntr_<hex>') the decode fails and model_id was
never forwarded — making the router-side fallback in
_init_containers_api_endpoints unreachable in production.
Fall back to the stored 'unified_object_id' on the ownership row, which
is the encoded form captured at create time when the router selected a
specific deployment. Decoding that yields the deployment model_id and
restores router-based credential application (api_base, api_key) for
retrieve/delete and container-file operations on native IDs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Bedrock Invoke transformation files (chat and messages) called
_supports_factory(custom_llm_provider=None, ...) which relies on
auto-detection. For short Bedrock model names (e.g. 'anthropic.claude-opus-4-6'
without the version suffix) auto-detection fails and the lookup falls back
through the exception path. Passing the known 'bedrock' provider explicitly
makes the lookup deterministic for all Bedrock model variants, including
cross-region inference profile IDs.
Co-authored-by: Claude <noreply@anthropic.com>
Address greptile's open thread on _async_refresh_locks growing
unboundedly in high-cardinality deployments.
- Add _maybe_prune_async_refresh_lock: drops the per-key Lock from
the registry once no coroutine holds it and no coroutine is queued
in lock._waiters. The check-then-pop sequence is safe under
asyncio's cooperative scheduler — a waiter that arrives after the
pop simply creates a fresh lock under the same key, which is fine
because the previous batch is already done.
- Wrap the slow-path async with lock in a try/finally so the prune
runs on every exit (return, exception, reauth retry).
- Extract the existing background-refresh task scheduling into
_schedule_background_refresh so get_access_token_async stays under
ruff's PLR0915 ("Too many statements") limit. No behaviour change.
- Regression tests cover both pruning after release (the dict
shrinks back to zero after each call) and the safeguard that
keeps the lock alive while a waiter is still queued.
The `allowed_tools` field on `BlockedToolsResult` was computed in
`_extract_blocked_tools` but never read by the only caller — when any
tool was blocked the integration unconditionally raised
`ModifyResponseException` to reject the full response, never doing
partial filtering. Drop the dataclass and return the blocking
explanation directly as `Optional[str]` so there's no misleading shape
hinting at unused partial-filter capability.
Co-authored-by: Greptile <greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): normalize batch file IDs before ManagedObjectTable write
Run post_call_success_hook before update_batch_in_database on retrieve/cancel,
and ensure_batch_response_managed_file_ids so file_object never stores raw
provider output_file_id or error_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): address Greptile review on batch file ID normalization
Remove redundant resolve_* calls after update_batch_in_database and rename
loop variable to avoid shadowing hidden_params unified_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which was missing from the
cost map. This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
litellm.completion_cost lookup.
- Add mistral/ministral-8b-2512 entry to both the in-tree
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json (mirrors the
existing openrouter/mistralai/ministral-8b-2512 pricing).
- litellm.model_cost is loaded at import time from the URL pinned to
main, so the new backup entry isn't visible at test runtime until
it also lands on main. Backfill any entries missing from the
remote-fetched map into litellm.model_cost in the local_testing
conftest so cost-calculator lookups succeed on this branch.
* fix(tests): drop unnecessary del of conftest backfill loop vars
* fix: resolve batch response file IDs even when status unchanged
The status-unchanged early return in update_batch_in_database was
skipping ensure_batch_response_managed_file_ids, leaving raw provider
input_file_id (and other raw IDs) in the user-facing response when
polling an in-progress batch. Move the in-place file ID normalization
above the early return so the response always carries unified managed
IDs while still skipping the DB write when nothing changed.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(batches): cover ensure_batch_response_managed_file_ids branches
Add tests for the previously-uncovered paths in
ensure_batch_response_managed_file_ids: error_file_id normalization,
swallowed conversion errors, UserAPIKeyAuth fallback from
db_batch_object, model_name resolution from unified_file_id, and early
returns when managed_files_obj, model_id, or auth context are missing.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
- Fireworks _get_model_cost_capability: after exact-key lookups, fall back
to substring matching against fireworks_ai/* entries in model_cost so
model name variants (e.g. fine-tuned suffixes) continue to inherit
capability flags like supports_reasoning.
- Vertex vertex_llm_base: replace non-reentrant threading.Lock with RLock
on the sync refresh path so the reauthentication retry, which recurses
into get_access_token while still holding the lock, does not deadlock
when reloaded credentials are also expired.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Address bugbot review concerns:
- Sanitize proxy_server_request before forwarding to the Rubrik webhook.
The previous code passed the entire inbound HTTP context (Authorization,
Cookie, x-api-key, and the raw request body) through to a third-party
endpoint, which exfiltrates proxy credentials and upstream secrets. The
new _sanitize_proxy_server_request allowlists only url and method.
(Cursor Bugbot HIGH severity #3192354895)
- Treat a null choices[0].message.tool_calls as 'all blocked' rather than
letting iteration raise and silently fall through the outer except in
apply_guardrail (which would fail open). Iterate over a defensive
fallback list instead of relying on the dict default.
(Cursor Bugbot MEDIUM severity #3192349538)
Co-authored-by: Cursor Bugbot <bugbot@cursor.com>
