39448 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
mateo-berri	02aa7b2803	fix(gemini realtime): propagate usageMetadata on tool-call response.done ... Gemini Live emits usageMetadata as a sibling top-level key alongside the toolCall frame; the tool-call branch was unconditionally building response.done from get_empty_usage(), so tokens consumed by tool-call turns were recorded as zero spend and bypassed LiteLLM budget accounting. Mirror the non-tool-call RESPONSE_DONE path: when the same frame carries usageMetadata, run VertexGeminiConfig._calculate_usage and forward the real token counts.	2026-05-23 03:47:21 +00:00
mateo-berri	f54874f707	Merge branch 'litellm_internal_staging' into litellm_live_api_tool_calling_support	2026-05-23 03:26:21 +00:00
Cursor Agent	73b6da9e73	fix(realtime): check function_call_output before user role to prevent guardrail bypass ... Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-23 02:13:10 +00:00
mateo-berri	039e86f010	fix(gemini realtime): redact realtime payloads from debug logs ... The transform_realtime_response debug logs were dumping the raw inbound Gemini frame and each outbound OpenAI event payload (up to 500 chars). Realtime frames carry transcripts, model output, and tool-call arguments, so those strings ended up in application logs whenever DEBUG was enabled. Replace the inbound dump with just the top-level frame keys and the outbound dump with just the event type.	2026-05-23 02:01:32 +00:00
mateo-berri	0e13cfabef	fix(gemini realtime): preserve sibling keys on empty toolCall no-op ... Replace the early return on `functionCalls` empty/absent with a `continue` plus a `tool_call_handled` flag that mirrors the existing `server_content_handled` pattern. The post-loop guard already distinguishes intentionally-consumed known keys from genuinely-unknown messages, so adding `toolCall` to that exclusion list lets the loop continue iterating over any sibling top-level keys in the same Gemini frame instead of short-circuiting on the first empty toolCall. In practice Gemini's protobuf places `toolCall`/`serverContent`/ `setupComplete` in a `oneof` so the only realistic sibling is `usageMetadata` (already filtered as unknown-top-level), but the uniform handling avoids silently discarding any future sibling key should the wire format grow.	2026-05-23 01:58:30 +00:00
mateo-berri	70e1169989	fix(realtime): forward sanitized function_call_output on guardrail block ... Providers that pair every toolCall with a toolResponse (e.g. Gemini and Vertex Live) stay in the awaiting-tool-call state until a toolResponse arrives. Dropping a blocked function_call_output outright left those providers stalled — the subsequent guardrail clientContent and response.create were ignored because the prior toolCall had no matching toolResponse. When the client-supplied tool output fails the realtime guardrail check, forward a sanitized placeholder function_call_output (same call_id, generic policy marker as output) instead of dropping the message entirely. The placeholder carries no blocked content, so the model never sees it, while still completing the provider's tool-call cycle so the session can recover and the violation message reaches the user.	2026-05-23 01:55:32 +00:00
mateo-berri	d3490859a4	fix(ci): restore guardrail injection on duplicate session.created and cast realtime delta event ... - Re-enable the one-time guardrail turn_detection update on duplicate session.created. `_maybe_send_guardrail_turn_detection_update` is already idempotent via `_guardrail_turn_detection_update_sent`, so the previous guard was unnecessary and broke the deferred-setup path where the synthetic session.created is emitted by llm_http_handler outside this loop (no prior chance to inject). - Cast the response.function_call_arguments.delta dict appended to `returned_message: List[OpenAIRealtimeEvents]` so mypy is satisfied.	2026-05-23 01:52:45 +00:00
Cursor Agent	a27e6bcae9	fix(realtime): avoid stale session.created flag triggering guardrail re-injection ... Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-23 01:38:17 +00:00
mateo-berri	4ca26e88a5	fix(gemini realtime): emit function_call_arguments.delta before .done ... Gemini delivers the full function-call arguments in a single toolCall frame. The OpenAI Realtime spec orders the streaming events as output_item.added -> function_call_arguments.delta(+) -> function_call_arguments.done -> output_item.done. Emit a single delta carrying the complete arguments string before the matching .done so spec-compliant SDK clients that accumulate deltas and gate finalisation on at least one delta arriving do not stall on Gemini tool calls.	2026-05-23 01:29:30 +00:00
mateo-berri	16e44cbee8	fix(realtime): run guardrails on function_call_output content ... Tool result outputs are client-controlled and fed to the model, so they must pass the same content checks as user text messages. Otherwise an attacker can smuggle blocked content into a function_call_output and have the model process it.	2026-05-23 01:11:17 +00:00
Cursor Agent	295a9e6e13	fix(gemini realtime): promote nested turn_detection when flat value is not a dict ... When the session payload had `turn_detection: None` (or any non-dict value), the normalizer skipped promoting the GA nested `audio.input.turn_detection` because it only checked key presence. The stale None then flowed into `map_automatic_turn_detection` and raised TypeError on `'create_response' in value`. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-23 00:45:57 +00:00
Cursor Agent	ef98c98a12	refactor(gemini realtime): drop unused json_message arg from map_openai_event ... Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-23 00:31:39 +00:00
Cursor Agent	f1b76d99e8	fix(gemini realtime): preserve sibling toolCall when serverContent has only transcription ... Previously, when a Gemini frame contained both a transcription-only serverContent and a sibling toolCall, the transcription handler would early-return and silently drop the toolCall. Instead, mark serverContent as handled and fall through so the main loop still processes siblings like toolCall, while preserving the prior no-op behavior for empty/ transcription-only frames. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-23 00:15:54 +00:00
mateo-berri	84791fc3f5	fix(gemini realtime): tolerate sibling-only frames (e.g. standalone usageMetadata) ... A Gemini Live frame that contains only metadata keys outside _KNOWN_GEMINI_TOP_LEVEL_KEYS (e.g. a bare {"usageMetadata": {...}} emitted between turns) leaves returned_message empty after the transform loop and was tripping the 'Unknown message type' guard, which raised ValueError and terminated the WebSocket session. Treat such frames as no-ops and return the unchanged state instead.	2026-05-22 23:55:00 +00:00
Cursor Agent	1e4f86e19a	fix(realtime): inject guardrail turn_detection on subsequent session.update without one ... Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 23:52:35 +00:00
mateo-berri	1a838a1517	fix(gemini realtime): cast maxOutputTokens to int for typeddict assignment	2026-05-22 23:34:44 +00:00
Cursor Agent	3a1a5ae392	fix(gemini realtime): use camelCase maxOutputTokens in response.done ... Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 23:25:08 +00:00
mateo-berri	6299633138	fix(gemini realtime): cast maxOutputTokens to int for typeddict assignment	2026-05-22 23:12:13 +00:00
Cursor Agent	61928b9704	fix(gemini realtime): scope dotted-key event lookup and propagate session metadata to tool-call response.done ... - map_openai_event: only check the current key/value pair when resolving dotted map entries (e.g. serverContent.turnComplete) so a sibling key in the same frame can't misclassify the event being processed (e.g. toolCall returning RESPONSE_DONE). - tool-call path: extract generationConfig once and include modalities, temperature, and max_output_tokens on response.done so its shape matches response.created and the non-tool-call response.done. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 23:02:48 +00:00
milan-berri	7270f723de	fix(mcp): forward upstream initialize instructions on cold gateway init (#28231) ... Prefetch upstream InitializeResult.instructions before merging gateway initialize options when YAML/DB do not set instructions, so clients receive upstream server text on the first MCP initialize without list_tools. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-22 16:00:47 -07:00
Sameer Kankute	f35e7eb2f6	feat(guardrails): add Microsoft Purview DLP guardrail (#24966) ... * feat(guardrails): add Microsoft Purview DLP guardrail * fix(guardrails/purview): raise_for_status on HTTP errors, cap scope cache, reuse executor * fix(guardrails/purview): propagate litellm_call_id as correlation_id to Purview * chore: fixes * refactor(guardrails): delegate get_user_prompt to get_last_user_message PurviewGuardrailBase duplicated AzureGuardrailBase (and OpenAIGuardrailBase) user-prompt extraction. The same logic already lived in common_utils.get_last_user_message; wire guardrail bases to that helper, fix the helper docstring, and drop its redundant self-import of convert_content_list_to_str. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): make protection scope cache true LRU on hits OrderedDict.get() does not update insertion order; call move_to_end on TTL-valid cache hits so popitem(last=False) evicts least-recently-used users instead of FIFO by first insert. Add a regression test with a small max cache size. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * Fix mypy * fix(guardrails/purview): harden user-id resolution and broaden DLP text Prefer API key and proxy-injected metadata over client metadata for Entra identity. Scan full message transcript pre-call and all completion choices post-call. Align logging-only hook with the same user-id rules. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails/purview): scan /v1/completions prompt and TextChoices Normalize text-completion prompts (string or list of strings); skip token-id-only prompts. Run post-call DLP on TextCompletionResponse choices. Extend logging_only hook for text_completion. Add tests and completion_prompt_to_str helper. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview-dlp): return data after DLP pass; per-call executor; dedupe text extraction async_pre_call_hook now returns the request dict after a successful check so callers match skip-path behavior. logging_hook uses a fresh ThreadPoolExecutor per invocation like Presidio to avoid single-worker starvation. Response text extraction is centralized in _completion_response_text_parts. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): fix LRU cache refresh position and add Responses API scanning Two fixes to the Microsoft Purview DLP guardrail: 1. LRU cache bug (base.py): When a stale scope cache entry was re-fetched, the assignment updated the value but Python's OrderedDict.__setitem__ preserves the original insertion order for existing keys. This left the refreshed entry near the front of the dict, making it the first candidate for LRU eviction via popitem(last=False). Fix: call move_to_end(user_id) after every write to an existing key. 2. Responses API coverage gap (purview_dlp.py): Requests to /v1/responses use an 'input' field instead of 'messages' or 'prompt', so the pre-call hook returned without scanning the content. Similarly, post-call hook did not handle ResponsesAPIResponse.output. Fix: add _responses_api_input_to_str() helper and handle 'responses'/'aresponses' call types in async_pre_call_hook, async_post_call_success_hook (via _completion_response_text_parts), and async_logging_hook. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): message separator, non-blocking logging_hook, TextChoices type error Three bugs fixed in the Microsoft Purview DLP guardrail: 1. get_prompt_text_for_dlp message separator (base.py) - Previously called get_str_from_messages() which concatenated all message texts with NO separator, so 'end of msg1' + 'start of msg2' became 'end of msg1start of msg2'. - Now joins per-message text with '\n\n' via convert_content_list_to_str(), preserving DLP pattern detection accuracy across message boundaries. 2. logging_hook blocking the event loop thread (purview_dlp.py) - Previously called future.result() which blocked the calling thread (often the event loop thread) for the entire round-trip of two sequential Microsoft Graph API calls (_compute_protection_scopes + _process_content). - Now fires and forgets: when called inside a running loop, schedules the coroutine with loop.create_task(); otherwise spawns a daemon thread. Returns (kwargs, result) immediately in both cases. - Removes unused concurrent.futures.ThreadPoolExecutor import; adds threading. 3. Incompatible assignment type error (purview_dlp.py:180) - mypy inferred 'choice' as TextChoices from the first loop body, then flagged the assignment in the second loop as incompatible with Choices. - Fixed by using distinct loop variable names: text_choice (TextChoices) and chat_choice (Choices). Tests: 7 new tests added covering the separator fix (TestGetPromptTextForDlp) and the non-blocking logging_hook (TestLoggingHookNonBlocking). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): suppress API errors in logging-only mode and scan tool-call arguments Three issues fixed: 1. _check_content except block re-raised unconditionally even when block_on_violation=False. The docstring promised 'log only - do not raise' but network/API errors always propagated. Fixed by checking block_on_violation before re-raising; when False, log a warning and continue. 2. async_logging_hook used a single try/except wrapping both the prompt and response audit calls. When the first _check_content (uploadText) raised due to an API error the second call (downloadText) was silently skipped. Fixed by giving each audit call its own try/except so both always run independently. 3. convert_content_list_to_str() only reads message.content, so tool_calls[].function.arguments and function_call.arguments were invisible to the Purview pre-call and post-call scans. An authenticated caller could embed sensitive text in tool-call arguments and bypass DLP. Fixed by: - Adding PurviewGuardrailBase._extract_tool_call_args_from_message() which handles both dict and object-style messages, covering both tool_calls[] arrays and the legacy function_call field. - Updating get_prompt_text_for_dlp() to include those arguments alongside message content (request/prompt path). - Changing _completion_response_text_parts() from @staticmethod to an instance method and adding tool-call argument extraction for ModelResponse choices (response path). Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * chore(ui): restructure pre-built Next.js output to directory-based routing Flat page files (e.g. guardrails.html) replaced by directory-based index.html equivalents (e.g. guardrails/index.html) matching the Next.js App Router output format. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(purview): comprehensive security hardening — identity spoofing, streaming bypass, token-id gap Four security issues addressed: 1. end_user_id kwargs fallback missing in _resolve_user_id_from_logging_kwargs user_id already fell back to kwargs.get("user_api_key_user_id") when absent from metadata, but end_user_id only checked md.get("user_api_key_end_user_id") with no kwargs-level fallback. Added or kwargs.get("user_api_key_end_user_id"). 2. Streaming responses bypassed post_call blocking async_post_call_success_hook only runs on assembled non-streaming responses. For streaming requests the proxy already delivered all content before the hook ran, so raising HTTPException there had no effect. Added async_post_call_streaming_iterator_hook which buffers the entire stream, assembles it via stream_chunk_builder, runs the Purview DLP check, and only then re-yields chunks via MockResponseIterator. If a violation is detected the exception is raised before any bytes reach the client. The proxy automatically skips async_post_call_success_hook for guardrails that define this method, preventing duplicate scans. 3. Caller-controlled Purview user identity in blocking modes When a LiteLLM API key has no bound user_id the guardrail fell back to metadata[user_id_field], which is supplied by the caller. A caller could set this to any Entra object ID whose Purview policies are more permissive and bypass DLP. Added _resolve_trusted_user_id() that only returns identities from the proxy auth system (user_api_key_dict.user_id, end_user_id, or proxy-injected metadata["user_api_key_user_id"]). Added _resolve_user_id_for_blocking() used by all blocking-mode hooks: tries trusted sources first; if only caller-supplied is available, logs a SECURITY WARNING and still proceeds (backward compat); if nothing resolves, skips with a warning. 4. Token-id prompt DLP bypass When /v1/completions received a pure token-id array prompt, completion_prompt_to_str() returned None and the pre_call hook silently skipped the Purview scan. An authenticated caller could tokenize blocked text and send it without DLP evaluation. The hook now detects this case (raw_prompt present but prompt_text None) and logs a WARNING while letting the request pass through — token-id payloads are opaque at the text layer and cannot be scanned. This makes the gap explicit rather than silent. Tests: 94 total, all passing. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * Revert "chore(ui): restructure pre-built Next.js output to directory-based routing" This reverts commit c70c4303b7. * fix(purview): fail closed on identity spoofing, token prompts, and path encoding Encode Entra user IDs in Graph paths, guard caches with asyncio.Lock, scan Responses API instructions with string input, reject caller-only metadata and token-id completion prompts in blocking mode, and revert unrelated UI HTML restructure from the PR branch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview): use threading.Lock and getattr for LitellmParams - Replace asyncio.Lock with threading.Lock in PurviewGuardrailBase. The cache lock is acquired both from the proxy's main event loop and from short-lived event loops created by the logging_hook thread fallback. In Python 3.10+ an asyncio.Lock is bound to the first event loop that acquires it, so the second loop would silently break audit logging with RuntimeError. All critical sections are in-memory dict ops with no awaits, so a synchronous lock is safe. - Use getattr() on LitellmParams in initialize_guardrail() instead of .get(), which does not exist on Pydantic BaseModel instances and would raise AttributeError at runtime. Tests updated to construct Mock objects with spec= so they reflect the real interface. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(purview): dedupe trust-level user resolution and drop dead code - _resolve_user_id now delegates levels 1-3 to _resolve_trusted_user_id so blocking and non-blocking paths share a single source of truth. - Drop redundant event_hook override in MicrosoftPurviewDLPGuardrail.__init__ (initialize_guardrail already forwards event_hook=litellm_params.mode). - Drop unused self._logging_only attribute; blocking is controlled by the block_on_violation argument passed to _check_content. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): fail-closed on responses API transform error; avoid duplicate audit calls Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): fail-closed blocking DLP; revert directory-based UI HTML Blocking hooks now require UserAPIKeyAuth user_id/end_user_id only (no spoofable metadata), re-raise Responses API transform errors, scan streamed text completions, and reject requests with no bound identity. Reverts the accidental directory-based Next.js output from cc47081 (c70c4303b7). Co-authored-by: Cursor <cursoragent@cursor.com> * Remove dead code in purview_dlp: _resolve_user_id_for_blocking never returns falsy The method either returns a non-empty trusted user id or raises HTTPException, so the 'if not user_id' guards in async_pre_call_hook and async_post_call_success_hook were unreachable. Tighten the return type to str and drop the dead checks to make the fail-closed behavior explicit. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview): exclude caller-controlled end_user_id from blocking DLP Blocking Purview checks now use only API-key/JWT-bound user_id, not end_user_id populated from request user/metadata/safety_identifier. Co-authored-by: Cursor <cursoragent@cursor.com> * style(purview): apply Black formatting to base.py Co-authored-by: Cursor <cursoragent@cursor.com> * fix(purview): use post-await timestamp for cache TTL Capture the timestamp after the network call completes when storing it as the cache freshness marker, so the effective TTL reflects when the response was actually received rather than when the request started. Under high network latency the previous behavior shortened the effective cache lifetime. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): fail closed when stream_chunk_builder returns None stream_chunk_builder can return None (e.g., when ChunkProcessor filters all chunks), causing both isinstance checks to fail and the buffered chunks to be released without DLP scanning. Explicitly fail closed in that case by raising an HTTPException so the streaming DLP guardrail does not bypass policy enforcement. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): resolve user_id before buffering stream Co-authored-by: Yassin Kortam <yassin@berri.ai> * merge main (#28629) * test(vcr): classify cache verdicts, detect live calls, surface cost leaks Convert the per-test VCR verdict line from a single 'NOOP / HIT / MISS / PARTIAL' tag into a classified outcome that distinguishes the cases that silently bill the live API on every CI run from the ones that don't: HIT pure replay PARTIAL mixed replay + new recordings MISS:RECORDED new cassette saved to Redis (cached next run) MISS:OVERFLOW cassette > MAX_EPISODES_PER_CASSETTE; persister refused to save; re-bills every run MISS:NOT_PERSISTED test failed; save_cassette skipped; re-bills NOOP VCR-marked but no HTTP traffic (mocked elsewhere) UNMARKED:LIVE_CALL test bypassed VCR AND opened a TCP connection to a known LLM provider host -> wasted spend UNMARKED:NO_TRAFFIC test bypassed VCR but didn't call out The UNMARKED:LIVE_CALL signal is what converts 'this test probably hits live' into 'this test connected to api.openai.com'. We install a socket.connect / socket.create_connection wrapper for the duration of each non-VCR-marked test and record any outbound TCP to a known LLM provider hostname. The probe sits below the httpx layer so vcrpy and respx (which both patch above the socket) are unaffected. Replace the file-level _RESPX_CONFLICTING_FILES blacklists in the llm_translation and local_testing conftests with per-item respx detection in apply_vcr_auto_marker_to_items. A test now skips VCR when it actually carries @pytest.mark.respx or has respx_mock in its fixture chain - not just because some other test in the same file imports MockRouter. Items skipped by skip_files are split into respx_conflict (real conflict, the module wires up respx) vs file_opt_out (dead skip- list entry whose module never touches respx) so the session summary makes pruning obvious. Stabilize the AWS SigV4 fingerprint: the Authorization header on Bedrock requests rotates its Credential date and Signature on every call, which previously pushed every Bedrock test past the 50-episode overflow threshold. Extract the access-key id only ('aws-sigv4:AKIA...') so two requests with the same identity match. Always emit verdict logging when VCR is active (set LITELLM_VCR_VERBOSE=0 to opt back into the legacy quiet mode). Add a session-end classification summary that lists overflow tests, unmarked live-call tests, and the skip-reason breakdown. Wire the live-call probe + summary hook into every test directory that already uses the Redis-backed VCR cache (audio_tests, guardrails_tests, image_gen_tests, litellm_utils_tests, llm_responses_api_testing, llm_translation, local_testing, logging_callback_tests, ocr_tests, pass_through_unit_tests, router_unit_tests, search_tests, unified_google_tests). Add tests/llm_translation/test_vcr_classification.py covering the verdict classifier, skip-reason tagging, AWS SigV4 fingerprint stability, live-host classification, and session summary rendering. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): drop dead 'from respx import MockRouter' imports These seven test files were on _RESPX_CONFLICTING_FILES, which made the auto-marker skip them entirely. Inspecting the source shows the only respx artifact is a top-level 'from respx import MockRouter' that no test ever uses - no @pytest.mark.respx, no respx_mock fixture, no respx.mock context manager. The import is dead code left over from a previous mocking pattern. Now that apply_vcr_auto_marker_to_items detects respx per-item via the marker / fixture chain (b637d9f64a), the file-level skip is no longer needed for these files - they were the reason the OpenAI tests (test_o3_reasoning_effort, test_streaming_response[o1/o3-mini], TestOpenAIO1::test_streaming, TestOpenAIChatCompletion::test_web_search, TestOpenAIO3::test_web_search, etc.) ran live every CI build despite the cassette cache being healthy. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(image_edits): regenerate fixtures per call instead of holding open module-level file handles Module-level TEST_IMAGES = [ open(os.path.join(pwd, 'ishaan_github.png'), 'rb'), open(os.path.join(pwd, 'litellm_site.png'), 'rb'), ] SINGLE_TEST_IMAGE = open(...) opens the file once at import. After the first multipart upload, the file pointer is at EOF, so every subsequent test in the same xdist worker sends an empty multipart body. That non-determinism (a) blows the recorded cassette past MAX_EPISODES_PER_CASSETTE (50) so _RedisPersister.save_cassette refuses to save it, and (b) re-bills the live image edit endpoint on every CI run. Recent CI runs confirm the leak: tests/image_gen_tests/test_image_edits.py shows six tests parking at 51-52 cassette entries (TestOpenAIImageEditGPTImage1::test_openai_image_edit_litellm_sdk[False], TestOpenAIImageEditDallE2::..., test_openai_image_edit_with_bytesio, test_openai_image_edit_litellm_router, test_multiple_vs_single_image_edit[False], test_multiple_image_edit_with_different_formats). Replace the module-level file handles with _make_test_images() / _make_single_test_image() factories that return fresh _RewindableImage (BytesIO subclass) objects whose pointer always starts at 0. The image bytes are read once at import into module-level constants (_ISHAAN_GITHUB_BYTES, _LITELLM_SITE_BYTES), so disk I/O cost is unchanged. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(vcr): match real Bedrock hostnames in live-call probe The suffix '.bedrock-runtime.amazonaws.com' never matched real Bedrock endpoints, which use the format 'bedrock-runtime[-fips].{region}.amazonaws.com' (region between 'bedrock-runtime' and 'amazonaws.com'). Add an explicit host check for that pattern so Bedrock live calls are visible to the probe, and update the unit test accordingly. Also drop the unused '_LIVE_CALL_PROBE_INSTALLED' module variable. * fix(vcr): cover full RFC1918 172.16.0.0/12 range in local prefixes * fix(image_edits): drop _RewindableImage to prevent infinite multipart upload The _RewindableImage(BytesIO) wrapper auto-rewound on every read after EOF, which made the OpenAI SDK's multipart upload writer read the same bytes forever instead of seeing EOF. Workers OOM'd / SIGKILL'd: [gw0] node down: Not properly terminated replacing crashed worker gw0 ... worker 'gw1' crashed while running 'tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditGPTImage1::test_openai_image_edit_litellm_sdk[False]' The auto-rewind was added defensively for parametrized + flaky-retried tests, but BaseLLMImageEditTest::test_openai_image_edit_litellm_sdk already calls get_base_image_edit_call_args() once per invocation and that helper now constructs fresh streams via _make_test_images(), so rewinding inside the stream is unnecessary. Replace with plain BytesIO seeded with the cached image bytes. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): mark Bedrock prompt-caching cross-call tests VCR-incompatible The pass_through prompt-caching tests (test_prompt_caching_returns_cache_read_tokens_on_second_call, test_prompt_caching_streaming_second_call_returns_cache_read) make a warm-up call and then assert the *second* call sees a non-zero cache_read_input_tokens count from the upstream's prompt-cache. VCR replay can't model cross-call provider state — both calls match the same cassette episode, so the second call returns the first call's pre-warmup response and the assertion fails: AssertionError: Expected cache_read_input_tokens > 0 on second call, but got 0. Full usage: {'input_tokens': 4986, 'cache_creation_input_tokens': 4974, 'cache_read_input_tokens': 0} This started biting after the AWS SigV4 fingerprint stabilization (b637d9f64a): Bedrock requests now produce a stable per-access-key fingerprint instead of a per-request signature, so cassettes successfully replay where they previously always missed and re-recorded live. Opt these tests out via skip_nodeid_suffixes so they run live and match the existing pattern in tests/llm_translation/conftest.py (::test_prompt_caching). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(vcr): tighten OVERFLOW classification and switch respx detection to AST Address two greptile P2 review concerns on PR #27795: 1. MISS:OVERFLOW was firing whenever total > MAX_EPISODES_PER_CASSETTE regardless of cassette state. A cassette that grew past the cap historically but this run only *replayed* (dirty=False) is healthy — the persister never tries to save, so the cache state is stable and the next run will replay too. Only flag OVERFLOW when dirty=True (new episodes were recorded that the persister would refuse to save). Add a regression test covering the dirty=False + large-total case. 2. _module_uses_respx did substring matching on the module source, which false-positives on comments / docstrings / string literals. A comment like # Previously tried respx.mock but switched to vcrpy would keep a file pinned on the opt-out list, defeating the dead-import pruning goal of this PR. Replace the substring scan with an ast.NodeVisitor (_RespxUsageVisitor) that only counts: - @pytest.mark.respx / @respx.mock decorators - with respx.mock(): ... (sync + async) context managers - respx.mock(...) calls outside a with/decorator - function parameters / fixture names equal to respx_mock Add tests for the comment / docstring / string-literal cases plus each real-usage pattern. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(vcr): aggregate worker stats on the controller so the session summary actually renders under xdist `_session_stats` is a module-level dict mutated inside `_vcr_outcome_gate` — which runs in each xdist worker process. The controller's `pytest_terminal_summary` then reads its own empty `_session_stats` and bails on `if not counts: return`, so the OVERFLOW / LIVE_CALL sections the rest of this PR adds never make it into CI logs in the dist mode CI actually uses. Ship a structured `vcr_outcome` payload via `user_properties` (which xdist round-trips) and add `aggregate_report_outcome` on the controller to fold worker outcomes into `_session_stats`. The recording process tags `vcr_recorded_by` with `PYTEST_XDIST_WORKER` so the controller can tell "single-process — already counted locally" apart from "produced by a worker — needs aggregation here", and not double-count when there's no xdist. Covered by 9 new unit tests in test_vcr_classification.py including the end-to-end summary render path. * fix(guardrails): improve CrowdStrike AIDR input handling (#26658) * feat(lasso): add tool-calling support to LassoGuardrail (#27648) * feat(lasso): extend LassoGuardrail to support tool calling (RND-5748) * fix(lasso): PR review followups for tool-calling guardrail (RND-5748) * fix(lasso): handle object-style tool_calls in _update_tool_calls_from_masked (RND-5748) * fix(lasso): use model role for tool_use blocks (RND-5748) * test(lasso): add round-trip tests for message transformation (RND-5748) * fix(lasso): remove unused imports, handle Responses-API input masking, flatten multimodal content (RND-5748) * fix(lasso): inspect Responses-API input field (RND-5748) * fix(lasso): guard text-cursor remap against Lasso count mismatch (RND-5748) * fix(lasso): flatten list content in tool_result.content (RND-5748) * fix(lasso): remap multimodal list content during masking (RND-5748) Bug: _map_masked_messages_back counted list-content messages in original_text_count but the remap loop only handled isinstance(str). The positional text_cursor never advanced for list messages, causing all subsequent masked texts to be written onto the wrong messages. Fix: added elif isinstance(content, list) branch that replaces the list with the masked text string and advances the cursor — mirrors the existing string-content branch. Also handles the assistant + tool_calls combo for list-content messages. Test: test_map_masked_messages_back_list_content verifies a user message with [text + image_url] followed by an assistant message gets correct masked content on both (cursor stays aligned). * refactor(lasso): extract _get_field and _extract_tool_call_fields helpers (RND-5748) The dict-vs-object access pattern (x.get('y') if isinstance(x, dict) else getattr(x, 'y', None)) was duplicated 14 times across 5 methods. _get_field(obj, field) — single-point dict/Pydantic field access. _extract_tool_call_fields(call) — returns (call_id, name, parsed_input) with JSON argument parsing, replacing ~30 duplicate lines in both async_post_call_success_hook and _expand_messages_for_classification. Also simplified _update_tool_calls_from_masked, _prepare_payload tool mapping, and _apply_masking_to_model_response call_id extraction. Net ~60 lines removed. No behavior change — all 32 tests pass. * fix(lasso): add count guard to _apply_masking_to_model_response (RND-5748) _apply_masking_to_model_response used a bare text_cursor without verifying 1:1 correspondence between text-bearing choices and masked text entries. If Lasso returned a different number of text messages than choices with content, masked text would be applied to the wrong choice or silently skip choices. Added the same count-mismatch guard pattern already used in _map_masked_messages_back: count original text-bearing choices, compare to masked_text length, skip text remap on mismatch with a warning log. Tool_call masking via id-based lookup is unaffected. Tests: - test_apply_masking_to_model_response_multiple_choices: verifies correct per-choice masked text with 2 choices - test_apply_masking_to_model_response_count_mismatch: verifies content is left unchanged when counts disagree * fix(lasso): close two guardrail-bypass paths flagged in review (RND-5748) * tool-call args: when function.arguments is malformed JSON or parses to a non-object, preserve the raw string as {"arguments": <raw>} so Lasso still inspects it instead of receiving input=None. Covers both pre-call and post-call extraction (shared helper). Also resolves the CodeQL empty-except warning since the except body now assigns parsed=None. * Responses-API input: when a request carries both "messages" and "input", inspect both. Previously a benign messages array let the guardrail skip data["input"] entirely. The masking write-back is split via a count boundary so masked messages flow back to data["messages"] and masked input flows back to data["input"] without cross-contamination. Tests: malformed/non-object args round-trip, dual-field classification, dual-field masking write-back split. * chore(lasso): black formatting + comment on expand skip branch (RND-5748) * black: wrap two long expressions in lasso.py and reformat dict literals in test_lasso.py to satisfy CI lint. * add a short comment in _expand_messages_for_classification explaining why empty string and None content are intentionally skipped (None is the OpenAI shape for a pure tool-call turn). * fix(lasso): satisfy mypy in _handle_masking, _update_tool_calls_from_masked, _apply_masking_to_model_response (RND-5748) * Narrow `response.get("messages")` into a local before slicing so mypy doesn't see `Optional[List[Dict[str, str]]]` as non-indexable. * Rename the two write-side `func` bindings in `_update_tool_calls_from_masked` to `func_dict` / `func_obj` so mypy doesn't unify the dict and Any|None branches. * Rename the inner loop variable in `_apply_masking_to_model_response` from `msg` to `masked_msg` to avoid clashing with the `msg = choice.message` rebinding below. No behavior change; resolves the 7 mypy errors from the CI lint job. * perf: eliminate per-request callback scanning on proxy hot path (#27858) - Introduce `_CallbackCapabilities` dataclass and `ProxyLogging._callback_capabilities()` static method that inspects `litellm.callbacks` once and caches capability flags keyed on (list length, member ids); invalidates automatically when the callback list mutates without per-request iteration overhead - Replace O(n) `litellm.callbacks` walks in `async_pre_call_hook`, `during_call_hook`, `async_post_call_streaming_iterator_hook`, `async_post_call_streaming_hook`, and `post_call_response_headers_hook` with fast-path exits when no relevant callbacks are registered - Add `needs_iterator_wrap()` and `needs_per_chunk_streaming_hook()` instance methods to decouple iterator-level wrapping from per-chunk hook execution; avoids `get_response_string` materialization per chunk when no guardrail or chunk-hook callback is active - Introduce `_fast_serialize_simple_model_response_stream()` using `orjson` for common single-choice text streaming chunks, bypassing the full Pydantic serializer; falls back to `model_dump_json` for tool calls, logprobs, usage, and provider-specific fields - Add early-return in `_restamp_streaming_chunk_model` when downstream model already matches the requested model, avoiding unnecessary string comparisons on every chunk - Fix stale zero-cost cache bug in `_is_model_cost_zero`: move the per-router `_zero_cost_cache` dict onto the `Router` instance and clear it in `_invalidate_model_group_info_cache` so in-place pricing updates via `upsert_deployment` immediately resume budget enforcement - Add `scripts/benchmark_chat_completions_perf.py`: standalone async benchmarking tool with a mock OpenAI provider, LiteLLM proxy process management, non-streaming RPS, streaming TTFT, and full-stream latency measurements with repeat/median run support - Add comprehensive unit tests covering capability detection, cache invalidation, fast-path correctness, zero-cost cache regression, and the no-callback streaming fast path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> * ci(mutmut): enable mutate_only_covered_lines to fit in CI budget (#27910) The mutation-test workflow timed out at the 350-minute job cap when running whole-folder mutation against litellm/proxy/management_endpoints/ (~30 files, ~1.5 MB of source). Every mutant was running the full test suite, and mutants were generated for lines no test covers — which would survive regardless, just wasting compute. mutmut 3.x's mutate_only_covered_lines setting runs the suite once up front to compute coverage, then skips mutating uncovered lines. This cuts the mutant count dramatically and is the right semantic for the score (no test → no kill possible → uncountable). Per-mutant test filtering by function name is already automatic in mutmut 3.x; no external coverage step is needed. * fix(rate-limit): stop v3 limiter from leaking internal stash to provider body (#27913) * fix(rate-limit): stop v3 limiter from leaking internal stash to provider body PR #27001 (atomic TPM rate limit) introduced a reservation flow that writes four LiteLLM-internal keys onto the request data dict: _litellm_rate_limit_descriptors _litellm_tpm_reserved_tokens _litellm_tpm_reserved_model _litellm_tpm_reserved_scopes _litellm_tpm_reservation_released These keys are forwarded as request body params to the upstream provider, which rejects them as unknown fields: OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors' (mapped by litellm to RateLimitError / 429, hiding the bug behind a misleading 'throttling_error' code) Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are not permitted' Net effect: every chat completion against any real provider fails the moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check itself still runs (raises 429 on over-limit), but the success path poisons the upstream body. Reproduced on litellm_internal_staging HEAD (410ce761dc) against gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request fails with the provider's unknown-field error. Fix: the stash is metadata only. - Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS registry so we have a single source of truth for stash keys. - New helper _stash_value_in_metadata_channels writes to data['metadata'] / data['litellm_metadata'] without touching the top level. - _stash_reservation_in_data and the descriptor stash now route through that helper. _mark_reservation_released stops writing top-level. - _lookup_stashed_value also checks kwargs['metadata'] / kwargs['litellm_metadata'] (raw request_data shape) in addition to kwargs['litellm_params']['metadata'] (completion kwargs shape). - async_post_call_failure_hook now reads descriptors via the unified metadata lookup instead of request_data.get(top-level). - Defense in depth: async_pre_call_hook strips any stash key that somehow surfaced at the top level (stale cache, future refactor, test fixture) before returning. Tests: - New regression test asserts no _litellm_* stash key is present at the top level of data after async_pre_call_hook, and that the metadata channel still carries the reservation + descriptors so success / failure reconciliation works. - Existing test_tpm_concurrent.py tests that asserted top-level presence are updated to read from data['metadata'] — the location is an implementation detail; the spec is that post-call callbacks can resolve the stash. Verified end-to-end against OpenAI gpt-4o-mini and Anthropic claude-haiku-4-5 via /v1/chat/completions on a low-rpm key: - With limits not exceeded: HTTP 200, valid completion response, no leaked fields in body. - With RPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: requests'). - With TPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: tokens'). Full v3 hook test suite passes (171 tests). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments Address greptile P2: test fixture now uses the imported constant. Drop comments that re-explain what well-named identifiers already convey. * fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at the start of async_pre_call_hook. Without this, an authenticated caller can inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in body metadata, trigger a proxy-side rejection, and cause async_post_call_failure_hook to refund TPM counters against attacker-named scopes (e.g. another tenant's api_key). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: allow for allowlisted redirect URIs (#27761) * fix: allow for allowlisted redirect URIs * github comment addressing * Update litellm/proxy/_experimental/mcp_server/oauth_utils.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * harden oauth wildcard further * test: cover wildcard entry with dot-leading suffix rejection --------- Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * Emit native web_search_tool_result blocks for Anthropic clients (Claude Desktop / Cowork citations) (#27886) * feat(custom_logger): add async_post_agentic_loop_response_hook Lets a CustomLogger shape the response returned by the agentic-loop follow-up call without bypassing the loop's safety / observability machinery (depth tracking, fingerprinting, etc.). Default returns the response unchanged. Used by websearch_interception to inject Anthropic-native web_search_tool_result blocks when the originating client requested a native web_search_* tool. * feat(llm_http_handler): call post-agentic-loop hook on the originating callback In _execute_anthropic_agentic_plan, after anthropic_messages.acreate returns, call the originating callback's async_post_agentic_loop_response_hook so it can mutate the final response (e.g. inject native tool_result blocks). Pass the callback through from _call_agentic_completion_hooks. Exceptions in the post-hook are caught and logged so a buggy callback can't kill the request. * feat(websearch_interception): add is_anthropic_native_web_search_tool Identifies tools the Anthropic-native clients (Claude Desktop, the Anthropic SDK, the Anthropic Console) use to request native search: type starts with "web_search_" (e.g. web_search_20250305). Rejects the LiteLLM standard tool, the OpenAI-function variant, the bare "WebSearch" legacy name, and the bare "web_search" Claude Code shape. This lets us decide per-request whether the client expects web_search_tool_result content blocks in the response, without renaming any existing constants or touching native-provider skip logic. * feat(websearch_interception): add build_web_search_tool_result_block Produces the Anthropic-native web_search_tool_result content block from a structured SearchResponse. Anthropic-native clients use this block to populate citations / source links — the existing text-blob flatten path only feeds readable evidence to the model and discards the structure, so this builder gives us the missing piece. Shape matches https://docs.anthropic.com/en/api/web-search-tool — web_search_result items carry url, title, page_age, encrypted_content (empty string when the search provider doesn't supply one). * feat(websearch_interception): emit native web_search_tool_result blocks When the originating client request carried a native Anthropic web_search_* tool, the final response now also carries web_search_tool_result content blocks alongside the model's text answer — so Claude Desktop / Anthropic SDK clients can populate the citations panel and replay conversation history with structured search evidence. Wiring: - Pre-request hooks (both deployment + Anthropic path) set a flag on kwargs when they see a native web_search_* tool, so the signal survives the conversion-to-litellm_web_search step regardless of which hook fires first. - _execute_search now returns (text, SearchResponse) so the structured results aren't lost when the text is flattened for the follow-up model call. - _build_anthropic_request_patch returns the parallel list of SearchResponse objects. - async_build_agentic_loop_plan pre-builds the web_search_tool_result blocks (one per tool_use_id) and stashes them on plan.metadata when the flag is set. - async_post_agentic_loop_response_hook reads the metadata and prepends the blocks to response.content. - _execute_agentic_loop mirrors the injection for the legacy path so both paths behave identically. Clients that send the LiteLLM standard tool keep the existing text-only behavior — no regression. * test(websearch_interception): cover native web_search_tool_result emission 18 tests across: - detector branches (native vs litellm-standard, OpenAI-function shape, Claude Desktop builtin WebSearch, bare web_search, missing type) - block-builder shape (results, none, empty) - pre-request hook flag-setting (native sets, standard does not) - async_build_agentic_loop_plan attaches blocks to plan.metadata when the flag is present, leaves metadata untouched when absent - post-hook injection into dict and object responses - legacy _execute_agentic_loop mirrors the injection so both paths return the same shape * test(websearch_short_circuit): keep _execute_search mocks in sync with new tuple return * test(websearch_thinking_constraint): keep _execute_search mocks in sync with new tuple return * feat(websearch_interception): emit native blocks from try_short_circuit_search The agentic-loop post-hook only fires when the model returns a tool_use block. Cowork / Claude Desktop on Bedrock actually make TWO requests per user turn: the main /v1/messages with their builtin tool, and a separate standalone /v1/messages whose only tool is web_search_20250305. That second request hits try_short_circuit_search — no agentic loop, no post-hook — and was returning text-only, leaving the citations panel empty. When the short-circuit input carries a native web_search_* tool, build a synthetic server_tool_use + web_search_tool_result pair (using the structured SearchResponse already returned by _execute_search) so the client gets the native shape it expects. The legacy text block is preserved so non-native short-circuit callers (Claude Code, github_copilot, etc.) see the same payload as before. Failure path still emits the native block pair (with empty results) plus the text-error block, so the client gets a well-formed response rather than a malformed half-shape. * test(websearch_native_blocks): cover short-circuit native-block emission Three new cases on top of the existing 18: - native web_search_20250305 short-circuit → [server_tool_use, web_search_tool_result, text], ids paired, urls/titles carried. - litellm_web_search short-circuit → text-only (no regression). - native short-circuit on search failure → still emits the native block pair (empty results) plus the text-error block, so the client never sees a malformed half-shape. * test(websearch_short_circuit): index assertions by block type, not by position Native short-circuit responses now have [server_tool_use, web_search_tool_result, text] when the input carries web_search_20250305 — find the text block by type rather than relying on content[0]. * fix(websearch_interception): gate legacy WebSearch name on schema absence Clients like Cowork / Claude Desktop ship a client-side tool named "WebSearch" with a full input_schema — they handle it themselves and expect to make a separate native web_search_20250305 sub-request for the actual search. Today is_web_search_tool matches the bare name regardless of other fields, which hijacks the client's tool server-side. The agentic loop fires on the main request, the model never gets to emit the client-side tool_use, and the separate native sub-request (where citation data flows) is never made. Net: citations panel empty. Real Anthropic client tools always carry input_schema (the API rejects them otherwise), so a bare {name: "WebSearch"} with no schema is the only thing that could be a legacy interception marker. Gate the match on schema absence: legacy callers (if any) keep working, real client-side WebSearch tools pass through untouched. * fix(websearch_interception): drop "WebSearch" from response-detection lists Post-conversion the model always sees ``litellm_web_search``, so the "WebSearch" entry in the response-side tool_use detection lists was dead at best. If a model ever did return ``tool_use(name="WebSearch")`` it would now (incorrectly) hijack the client's own ``WebSearch`` tool again — same Cowork problem we just fixed on the input side. Drop it. * test(websearch_native_blocks): cover the WebSearch legacy-name schema gate Three new cases: - {name: "WebSearch"} (bare interception marker) → still matched - {name: "WebSearch", input_schema: {...}} (Cowork client tool) → passes through untouched - {name: "WebSearch", description: "..."} (no schema) → still matched on the assumption it's a legacy marker rather than a malformed real client tool. --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * ci(codecov): restore litellm/ prefix on uploaded coverage paths pytest-cov runs with --cov=litellm, which makes coverage.xml store paths relative to the package root (e.g. `proxy/proxy_server.py` instead of `litellm/proxy/proxy_server.py`). Codecov auto-resolves these only when the basename is unique in the repo. Files like proxy_server.py, router.py, utils.py, main.py, and constants.py — which have duplicates under enterprise/ or other subpackages — get silently dropped during ingest. The `fixes: ["::litellm/"]` rule prepends `litellm/` to every uploaded path so they resolve unambiguously. Confirmed against multiple recent coverage.xml artifacts that no uploader currently emits paths already prefixed with `litellm/`, so the rule is safe to apply universally. This restores Codecov visibility for the highest-fix-rate hotspots: proxy_server.py, router.py, proxy/utils.py, litellm_logging.py, constants.py, key_management_endpoints.py, utils.py, main.py, user_api_key_auth.py, team_endpoints.py, and litellm_pre_call_utils.py. * chore(ci): remove unused GitHub Actions workflows and orphan files Audit of .github/workflows/ via gh run history shows the following have either never run or have been dormant for 10+ weeks. CI coverage that still matters is preserved on CircleCI (e.g. llm_translation_testing). Removed workflows: - test-litellm.yml — workflow_dispatch only, last run 2026-02-12 (cancelled); CCI local_testing_part1/2 covers the same tests - llm-translation-testing.yml — last run 2025-07-10; replaced by CCI llm_translation_testing job (run_llm_translation_tests.py kept for the make test-llm-translation target) - run_observatory_tests.yml — last run 2026-03-03 (cancelled) - scan_duplicate_issues.yml — last run 2026-03-02 (failure) - publish_to_pypi.yml — never run - read_pyproject_version.yml — fires on every push to main but its echoed version output is not consumed by any downstream step Removed orphan files (no callers in workflows, CCI, or Makefile): - .github/workflows/README.md — documented only publish_to_pypi.yml - .github/workflows/update_release.py + results_stats.csv - .github/actions/helm-oci-chart-releaser/ * Revert "ci(codecov): restore litellm/ prefix on uploaded coverage paths" This reverts commit e25a988a3f. The `fixes: ["::litellm/"]` rule turned out to be applied *after* Codecov's auto-resolution, not before. Files with unique basenames (which were auto-resolving correctly to `litellm/<path>`) got an extra `litellm/` prepended, producing `litellm/litellm/<path>` storage. Files with ambiguous basenames (the actual target of the fix) continued to be dropped because the auto-resolution still failed for them. Net result on the verification run: 1375 files now stored under unresolvable `litellm/litellm/...` paths, and the 11 originally-missing hotspots are still missing. Reverting before piling on further changes. * test(ui): preserve global Button/Tooltip mocks in per-file @tremor/react vi.mock Per-file `vi.mock("@tremor/react", ...)` factories fully replace the setup-level mock from `tests/setupTests.ts`, so the global Button/Tooltip overrides are lost in any file that re-mocks `@tremor/react`. Without them, the real Tremor `<Button>` leaks through and its internal `useTooltip(300)` schedules a native 300ms `setTimeout` on pointer events. When the test environment is torn down before the timer fires, the trailing `setState` calls `getCurrentEventPriority`, which reads `window.event` against a destroyed jsdom -> "window is not defined" flake observed on CI. Patches the 7 leaky test files to re-supply `Button` (bare `<button>`) and `Tooltip` (Fragment) overrides matching `setupTests.ts`. Also drops a dead `afterEach` workaround in `user_edit_view.test.tsx` (the fake-timer dance it ran could not drain a real timer scheduled before the swap) and corrects a misleading comment in `MakeMCPPublicForm.test.tsx`. * ci: use --cov=./litellm so coverage paths resolve unambiguously in Codecov pytest-cov treats --cov=<module-name> as a Python package and emits XML paths relative to the package root, stripping the litellm/ prefix (`proxy/proxy_server.py` instead of `litellm/proxy/proxy_server.py`). Codecov's auto-prefix heuristic then drops every file whose basename is ambiguous in the repo — `proxy_server.py` (3 copies under enterprise/), `router.py` (2 copies), `utils.py` (20+), `main.py` (20+), `constants.py` (2). The 11 highest-fix-rate hotspots have never appeared in Codecov. Switching to --cov=./litellm treats the argument as a path, which makes coverage.xml emit repo-relative paths (`litellm/proxy/proxy_server.py`). Each path is unambiguous, so Codecov resolves all files correctly. Verified locally: rerunning a single proxy_unit_tests test with --cov=./litellm produced `filename="litellm/proxy/proxy_server.py"`, `filename="litellm/router.py"`, and `filename="litellm/types/router.py"` as distinct entries — exactly the disambiguation Codecov needs. Touches every workflow that uploads coverage: the two reusable GHA workflows (_test-unit-base.yml, _test-unit-services-base.yml), test-mcp.yml, and all 14 invocations in .circleci/config.yml. * fix(mcp): allow delegate PKCE bypass for internal MCP servers Remove available_on_public_internet gating from delegate-auth-to-upstream paths so oauth2 + delegate_auth_to_upstream interactive servers behave the same when marked internal. Keeps M2M exclusion. Updates tests. * chore(mcp): warn on internal + upstream PKCE delegate Log verbose_logger.warning when loading oauth2 interactive servers with available_on_public_internet=false and delegate_auth_to_upstream=true (config + DB). Dashboard Alert for the same combo. CLAUDE note for operators. Tests for log and M2M skip. * fix(mcp): dedupe load_servers_from_config alias block Removes accidental duplicate alias/mcp_aliases and get_server_prefix logic (fixes PLR0915 and avoids resetting alias after mapping). * fix(mcp): expose delegate_auth_to_upstream in MCP server list rows (#27936) _build_mcp_server_table omitted delegate_auth_to_upstream, so GET /v1/mcp/server always returned the default false while the registry kept the DB value. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(proxy): fix vector store retrieve/list/update/delete without model (#27929) * feat(proxy): fix vector store retrieve/list/update/delete routing without model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): remove unchecked query-param injection in vector store management endpoints Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): use subset assertion for vector store route test to allow extra kwargs like shared_session Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller (#27984) * fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller CheckBatchCost bypasses async_post_call_success_hook, causing raw provider output_file_ids to be persisted in LiteLLM_ManagedObjectTable. This fix converts output_file_id and error_file_id to managed base64 IDs before the DB write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(check_batch_cost): persist managed file before mutating response and propagate team_id - Move setattr after store_unified_file_id so the response only receives the managed ID once the DB record is successfully written. Avoids serializing an orphaned managed ID into file_object when the store call fails. - Populate team_id on the minimal UserAPIKeyAuth from job.team_id so the managed file record is created with the correct team ownership, allowing other team members to access the batch output file via /files/{id}/content. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(managed_batches): extend test to cover error_file_id conversion Co-authored-by: Cursor <cursoragent@cursor.com> * fix managed file test --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs (#27912) * fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs Vertex batch jobs recorded 0 spend and 0 tokens after PR #25627 added automatic transformation of GCS predictions.jsonl to OpenAI format. Two bugs fixed: 1. batch_utils.py: the Vertex-specific cost/usage reader (calculate_vertex_ai_batch_cost_and_usage) was always invoked and reads raw usageMetadata fields that no longer exist in the OpenAI-shaped output. Now the reader is only used when disable_vertex_batch_output_transformation=True; otherwise the generic path handles the already-transformed OpenAI-shaped content. 2. cost_calculator.py: batch_cost_calculator skipped the global litellm.get_model_info() lookup when a model_info dict was passed in, even when that dict had no pricing fields (e.g. deployment metadata with only id/db_model). It now falls back to the global pricing table when the provided model_info has no pricing data. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm/cost_calculator.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(cost-calculator): use not-any guard for pricing fallback in batch_cost_calculator Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cost-calculator): treat explicit zero batch pricing as set in model_info The fallback to litellm.get_model_info() used truthy checks on pricing fields, so 0.0 was treated as missing and replaced by global rates. Use `is not None` like elsewhere in cost calculation. Add regression test. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * feat: add weighted-routing failover (#27980) * Feat: Add Weighted-Routing Failover * test(router): cover weighted failover helper functions Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): align weighted failover deployment list type with mypy Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): address greptile review on weighted failover - Narrow exception swallowing in `_maybe_run_weighted_failover` to `openai.APIError` so model failures defer to the regular fallback while programming bugs (AttributeError/KeyError/TypeError) surface. - Note async-only limitation of `enable_weighted_failover` in the Router constructor docstring. - Make the weighted distribution test less flaky (1000 iterations, looser bound) and make the non-simple-shuffle test deterministic by failing both deployments instead of relying on the latency strategy's first pick. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): ensure weighted failover metadata persists in kwargs The previous `kwargs.setdefault(metadata_variable_name, {}) or {}` returned a brand-new dict whenever the existing metadata was falsy (empty dict or None), so writes to `_failover_excluded_ids` never made it back into `kwargs`. Multi-hop weighted failover then re-selected previously failed deployments and exhausted `max_fallbacks` prematurely. Explicitly assign a fresh dict into kwargs when metadata is missing so mutations are visible to subsequent failover hops. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(router): regression for weighted failover metadata persistence Asserts kwargs["metadata"]["_failover_excluded_ids"] is populated after _maybe_run_weighted_failover, proving the metadata dict written by the helper is the same object that lives in kwargs (no disconnected copy). Pairs with the prior fix that replaced `setdefault(..., {}) or {}` with an explicit get/assign so writes survive across hops. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): harden weighted failover error/state handling - Catch RouterRateLimitError (ValueError) alongside openai.APIError in _maybe_run_weighted_failover so an exhausted intra-group retry falls through to the regular cross-group fallback path instead of bubbling out and bypassing configured fallbacks. - Stop mutating the shared input_kwargs dict; build a local copy with the weighted-failover keys so the entry (with _excluded_deployment_ids) cannot leak into later fallback paths reading the same dict. - _get_excluded_filtered_deployments now returns an empty list when the exclusion filter removes every healthy deployment, instead of falling back to the original list. The original-list behavior risked re-picking the just-failed deployment; callers already handle the empty case by raising their no-deployments error, which weighted failover now catches and converts into a normal cross-group fallback. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): fall through to rpm/tpm when total weight is zero When the weight metric's total is zero (e.g. after weighted-failover exclusion leaves only zero-weight backups), continue to the next metric (rpm/tpm) instead of returning a uniform random pick immediately. This lets rpm/tpm still drive routing when present, and only falls back to the uniform random pick at the end if no metric provides a positive total weight. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): skip weighted failover when remaining deployments are all in cooldown _maybe_run_weighted_failover was computing 'remaining' from all_deployments (every deployment in the model group, including those in cooldown). This meant that when all non-excluded deployments were in cooldown the method still invoked run_async_fallback unnecessarily, which propagated into async_get_healthy_deployments, found no eligible deployments, and raised RouterRateLimitError — only safely caught thanks to the earlier exception-broadening fix. The fix: before computing 'remaining', fetch the current cooldown set via _async_get_cooldown_deployments and subtract it from all_ids. This allows _maybe_run_weighted_failover to return None immediately (skipping the run_async_fallback call entirely) when every non-failed deployment is in cooldown, letting the caller fall through to the correct cross-group fallback path without the wasteful extra round-trip. Tests added: - unit: _maybe_run_weighted_failover returns None without calling run_async_fallback when all remaining deployments are in cooldown - unit: _maybe_run_weighted_failover still calls run_async_fallback when at least one healthy (non-cooldown) deployment is available - integration: end-to-end fallthrough to cross-group fallback when remaining deployments are in cooldown Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpo… (#27976) * fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943) * docs: add one-line docstring to _disable_debugging (#27894) Squash-merged by litellm-agent from oss-agent-shin's PR. * Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831) Squash-merged by litellm-agent from Cyberfilo's PR. * Sanitize empty text content blocks on /v1/messages (#27832) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not Found. Both AmazonMantleConfig (chat/completions caller route) and AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded the wrong path, so every Mantle request 404'd before reaching the model. Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages API at /anthropic/v1/messages with SSE streaming." https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock Confirmed independently against the live endpoint: /v1/chat/completions -> 200 OK /v1/messages -> 404 Not Found (what litellm used) /anthropic/v1/messages -> 200 OK (Claude only) Adds a regression test asserting both Mantle configs build the /anthropic/v1/messages path, and updates the existing assertions that encoded the wrong path. --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> * fix: sanitize empty text blocks in sync anthropic_messages_handler path Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(utils): import get_secret at runtime (#28014) * fix(proxy): make /config/update env-var encryption idempotent A single decrypt-then-encrypt chokepoint (_encrypt_env_variables_for_db) now backs both update_config and save_config. Re-submitting a value the Admin UI read back from /get/config/callbacks as ciphertext no longer stacks a second encryption layer, which previously decrypted to garbage and silently broke the callback. The chokepoint decrypts with the pure _decrypt_db_variables (no os.environ mutation on the write path) and encrypts exactly once; update_config merges only the sent keys so untouched env vars keep their stored ciphertext byte-for-byte. * test(proxy): add endpoint-level regression for /config/update double-encryption Adds test_update_config_env_var_round_trip_not_double_encrypted, which drives the real /config/update handler: first write plaintext, then re-POST the stored ciphertext (the Admin UI round-trip) and assert the value is not stacked with a second encryption layer and untouched keys stay byte-identical. Verified to fail against the pre-fix handler and pass after. Also tightens the unit test to exactly three ciphertext re-feeds. * chore(ci): modernize model references in tests and configs (#27856) * test: modernize models used in CircleCI e2e test suites Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo, claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current equivalents across the e2e_openai_endpoints and proxy_e2e_anthropic_messages_tests CircleCI jobs. - gpt-4o -> gpt-5.5 (responses API e2e tests) - gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config) - gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning, still actively fine-tunable) - gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 / gpt-5-mini - bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001 (also aligning oai_misc_config model_name with what test_bedrock_batches_api.py actually requests) - bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15) -> claude-sonnet-4-5-20250929 * test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5 Greptile/Cursor flagged that after the previous commit, the bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5 (both pointed to claude-sonnet-4-5-20250929). Rename to bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID (us.anthropic.claude-sonnet-4-6, already in the litellm model registry) so the alias name matches the underlying model version. * test: modernize models across remaining CI-mounted configs & tests Expands the modernization sweep to all CircleCI-mounted proxy configs and to test directories where the model literal is a fixture/route key (not the test's subject). Config changes: - proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 / gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump text-embedding-ada-002 underlying to text-embedding-3-small. User- facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.) preserved for backward compatibility with tests. - simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml: bump gpt-3.5-turbo underlying to gpt-5-mini. - pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet / claude-3-haiku entries replaced with claude-sonnet-4-5 / claude- haiku-4-5 / claude-opus-4-7. - oai_misc_config.yaml: align alias name with the gpt-5-mini rename. Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4- 20250514 retire 2026-06-15): - tests/llm_translation/test_anthropic_completion.py: bump 3 references + paired Vertex AI ID to claude-sonnet-4-5. - tests/llm_translation/test_optional_params.py: bump 2 references. - tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py and test_bedrock_anthropic_messages_test.py: bump router fixtures using the deprecated model IDs. - tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py: modernize docstring examples. - tests/test_end_users.py: update references to renamed alias. * test: modernize placeholder model literals in router_unit_tests Mass replace_all on fixture/placeholder model literals across the router_unit_tests/ suite (model name is a routing key / label, not the test subject). Sub-agent sweep so far — additional commits will follow for logging_callback_tests/, enterprise/, top-level tests/test_*.py, and other CI-mounted dirs. Mappings applied: - gpt-3.5-turbo -> gpt-5-mini - gpt-4 (bare) -> gpt-5.5 - gpt-4o (bare) -> gpt-5 - text-embedding-ada-002 -> text-embedding-3-small - claude-3-sonnet-20240229 / claude-3-opus-20240229 / claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 -> claude-sonnet-4-5-20250929 / claude-opus-4-7 / claude-haiku-4-5-20251001 as appropriate Explicitly preserved: - gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current - gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals) - JSONL batch body literals - Mock LLM response model fields (must match upstream) - Fake/mock identifiers * test: modernize placeholder model literals across remaining CI suites Sub-agent sweep across logging_callback_tests/, guardrails_tests/, enterprise/, pass_through_unit_tests/, otel_tests/, llm_responses_api_testing/, batches_tests/, spend_tracking_tests/, litellm_utils_tests/, unified_google_tests/, and a few top-level tests/test_*.py files where the model literal is a fixture or placeholder (router model_list, mock standard logging payload, mock callback data) rather than the test's subject. Mappings applied (see scope notes below): - gpt-3.5-turbo -> gpt-5-mini - gpt-4 (bare) -> gpt-5.5 - gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5 is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex / gpt-5-mini exist) - gpt-4o-mini (bare) -> gpt-5-mini - text-embedding-ada-002 -> text-embedding-3-small - claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929 - claude-3-opus-20240229 -> claude-opus-4-7 - claude-3-haiku-20240307 -> claude-haiku-4-5-20251001 - claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929 - claude-3-7-sonnet-20250219 -> claude-sonnet-4-6 - gemini-1.5-flash -> gemini-2.5-flash - gemini-1.5-pro -> gemini-2.5-pro Explicitly preserved (not modernized): - llm_translation/ tests where model is the SUBJECT (provider-specific translation/transformation logic). Only the deprecated 20250514 references were already bumped in a prior commit. - Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges documented by the sub-agent). - Bedrock model IDs in test_health_check.py path-stripping tests. - JSONL batch request bodies and mock LLM response bodies (must match upstream literal). - Langfuse expected-request-body JSON fixtures (cost values are exact- match-asserted; changing the model would shift response_cost). - gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI equivalent). - Top-level tests calling the proxy through user-facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases in proxy_server_config.yaml stay; only the underlying model was bumped. - tests/test_gpt5_azure_temperature_support.py (the test's whole point is model-name handling). - Fake / mock / openai/fake identifiers. Notable side fixes: - test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini), resolving a latent inconsistency. - proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5` (bare gpt-5 is not a valid OpenAI alias). - test_batches_logging_unit_tests.py: explicit_models list entries kept distinct (gpt-5-mini + gpt-5.5) after bulk rename. * test: fix CI failures from model modernization sweep CI surfaced 4 categories of regression from the bulk modernization: 1. Azure deployment names are customer-specific. Reverted: - tests/litellm_utils_tests/test_health_check.py: azure/text- embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure account does not have a text-embedding-3-small deployment). - tests/logging_callback_tests/test_custom_callback_router.py: same revert for two router fixtures driving aembedding. 2. gpt-5 family does not accept temperature != 1. Tests that pass a custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern non-reasoning OpenAI mini that still accepts temperature/logprobs): - tests/logging_callback_tests/test_datadog.py - tests/logging_callback_tests/test_langsmith_unit_test.py - tests/logging_callback_tests/test_otel_logging.py 3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to gpt-5.5 (a reasoning model that rejects logprobs). The proxy test tests/test_openai_endpoints.py::test_chat_completion_streaming exercises logprobs/top_logprobs through that alias. Bumped the underlying model to gpt-4.1 (non-reasoning, still modern). 4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with hardcoded model="gpt-4o" and a model-specific spend value. Reverted the litellm.acompletion calls in the test to model="gpt-4o" so the fixture's exact-match assertions still hold. 5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py: anthropic.messages.create routing to openai/gpt-5-mini returned an empty content[0] with max_tokens=100 (reasoning-token consumption). Swapped to openai/gpt-4.1-mini. * test: fix Assistants API model + 2 cursor[bot] review nits 1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5 isn't accepted by the /v1/assistants endpoint ("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants- API-supported, non-reasoning). 2. example_config_yaml/pass_through_config.yaml: the previous sweep bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the Sonnet tier intact. (Cursor bugbot review.) 3. example_config_yaml/simple_config.yaml: model_name was left as gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which muddles the "simple" example. Make both sides gpt-5-mini so the most basic example is a straight 1:1 mapping again. (Cursor bugbot review.) * fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models tests/test_openai_endpoints.py::test_completion calls the proxy alias "gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with custom temperature / logprobs / the legacy /v1/completions endpoint. The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini, which are reasoning models that reject temperature != 1 and don't expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini (modern non-reasoning OpenAI models) instead — keeps user-facing aliases preserved while picking a current underlying that still supports the parameters/endpoints the tests exercise. * test(proxy): isolate run_server CLI tests from prisma DB-setup path test_keepalive_timeout_flag and test_timeout_worker_healthcheck_flag were the only run_server tests in test_proxy_cli.py that neither stripped DATABASE_URL/DIRECT_URL nor mocked the prisma DB path. When a DATABASE_URL is present (CI/env leak), run_server --local enters the DB block and blocks in the un-timeout'd subprocess.run(["prisma"]) at proxy_cli.py:987 plus the ProxyExtrasDBManager migrate-deploy retry loops, ~370s per test on the CI runner. --dist=loadscope pins both to one xdist worker, so the proxy-infra job appears stuck at 99% and hits the 20-min timeout. Apply the same isolation every other run_server test in this file already uses: mock PrismaManager.setup_database + should_update_prisma_schema and strip DATABASE_URL/DIRECT_URL. Full module drops from 31.7s to 2.9s locally; both tests fall off the slow list. * feat: add OTEL GenAI latest-experimental semantic convention support (#27418) - Introduce `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` opt-in that switches OTEL traces to conform with the OpenTelemetry GenAI semantic conventions specification - Extract all semconv behavior into a new `OTELGenAISemconvMixin` class in `gen_ai_semconv.py`, mixed into `OpenTelemetry` to keep concerns separated - In semconv mode, span name follows `{operation} {model}` pattern (e.g. `chat gpt-4`) and span kind is set to `CLIENT` instead of legacy `litellm_request` - Replace `gen_ai.system` with `gen_ai.provider.name` and drop `llm.is_streaming` in semconv mode; add `gen_ai.request.{frequency_penalty,presence_penalty,top_k,seed,stop_sequences,stream,choice.count}` and `gen_ai.usage.cache_{creation,read}.input_tokens` attributes - Replace per-message `gen_ai.content.prompt` / per-choice `gen_ai.content.completion` log events with a single consolidated `gen_ai.client.inference.operation.details` event; omit `gen_ai.input/output.messages` when content capture is disabled - Suppress the non-standard `raw_gen_ai_request` child span entirely in semconv mode - Support both programmatic (`OpenTelemetryConfig.semconv_stability_opt_in` field) and environment variable activation; the two sources are unioned so either or both can enable the opt-in - Extract OTEL SDK `LogRecord` / `SeverityNumber` version-compatibility shim into a reusable `_otel_log_types()` static method to deduplicate the `< 1.39.0` / `>= 1.39.0` import branching - Add 30+ unit tests covering opt-in gating, span naming, attribute emission/omission rules, stop sequence normalization, cache token attributes, and the consolidated event lifecycle Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> * chore: retrigger CI * test(ci): add reasoning_effort grid v4 e2e regression suite Encode the 231-cell QA sweep (21 provider x model combos x 11 effort values) from #27039 / #27074 as an automated CircleCI-gated regression suite. Each cell hits the real provider endpoint, captures the outgoing wire body via a pre-call CustomLogger, and asserts: - thinking.type, output_config.effort, thinking.budget_tokens, max_tokens in the captured request body (regression signal for silent drops/strips in any provider transformation) - HTTP status (200 vs BadRequestError -> 400) returned by litellm (regression signal for clean-error vs leaked-500 mappings) The matrix is encoded as a small rule set keyed by (model_mode, effort) plus per-model xhigh/max capability overrides, then expanded across the five chat-completion routes (Anthropic direct, Azure AI Foundry, Vertex AI, Bedrock Converse, Bedrock Invoke /chat) and the Bedrock Invoke /v1/messages route. Cells skip at runtime when the route's provider env vars are absent, so PR builds without credentials no-op gracefully. Wired into CircleCI as the reasoning_effort_grid_v4_e2e job behind the existing main / litellm_* branch filter. * fix(reasoning_effort_grid_v4): cleanup unused fixture, parse converse body, guard budget tokens - Remove unused vertex_credentials_path fixture (and now-unused os import) from conftest.py. - Parse Bedrock Converse complete_input_dict (logged as a JSON string by converse_handler.py) before passing to _assert_cell, so dict accessors work uniformly across routes. - Extend _BUDGET_TOKENS with xhigh and max entries so the budget-mode branch in expected() cannot KeyError if a future budget model gains the matching cap. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(reasoning_effort_grid_v4): grant sonnet-4-6 entries the max-effort cap The runtime _validate_effort_for_model allows effort='max' for any Claude 4.6 model (opus or sonnet), and model_prices_and_context_window sets supports_max_reasoning_effort: true for claude-sonnet-4-6. The grid spec previously gave sonnet-4-6 entries _CAPS_NONE, so expected() returned status=400 for effort='max', which mismatched the runtime's status=200 and caused 6 cells (one per route) to fail. Rename _CAPS_OPUS_4_6 to _CAPS_4_6 (since the cap set is shared by opus and sonnet 4.6) and assign it to all sonnet-4-6 entries. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(tests): move reasoning_effort grid suite under llm_translation, drop v4 naming - Drop the "v4" suffix throughout: it referred to the QA sweep iteration, not this test suite. There's only one regression suite, so just call it reasoning_effort_grid. - Move tests/test… * Revert "merge main (#28629)" This reverts commit e4870f7bb6. * refactor(purview): remove unused logging_only constructor param The logging_only kwarg was passed in but only echoed in a log message; behavioral mode is driven by event_hook. Drop the dead parameter and the no-op derivation in the initializer. * fix(purview): log post-call success hook via @log_guardrail_information * fix(purview): raise HTTPException 400 on Responses API transform failure When the Responses API input cannot be transformed in blocking mode, raise an HTTPException with a clear 400 detail instead of re-raising the raw transformation exception. The latter would surface as a 500 and be indistinguishable from a backend provider failure; the former matches the docstring and the existing pre-call HTTPException patterns. * Fix Purview audit prompt extraction for Responses API logging hook The async_logging_hook receives kwargs == litellm_logging_obj.model_call_details, where function_setup mirrors the raw responses input under the 'messages' key (string or list of input items, not chat-format messages). The previous logic took the generic 'messages' branch first, calling get_prompt_text_for_dlp on data that is not in the chat-message format, and the responses-specific extractor was never reached. As a result, the prompt half of the Purview audit was silently skipped (or produced garbage text) for Responses API calls in logging_only mode. Check call_type first and route Responses API calls to _responses_api_input_to_str, which reads the original 'input' and 'instructions' keys that pre_call and update_environment_variables persist on model_call_details. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): check call_type for responses before messages in pre-call hook Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): don't reject empty-string prompts as token-id prompts completion_prompt_to_str returns None for both token-id lists *and* empty/whitespace-only strings (stripped). The previous check 'raw_prompt is not None and prompt_text is None' conflated these cases, raising the misleading 'Token-id completion prompts cannot be scanned' error for harmless empty-string prompts like {"prompt": ""}. Tighten the check to only reject true token-id prompts (non-empty list of ints). Empty/whitespace string prompts now fall through to the 'no prompt text → skip scan' path. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): scan ResponsesAPI streaming completed event for DLP The streaming iterator hook previously routed all assembled streams through stream_chunk_builder, which only knows chat/text-completion deltas. Responses API streams emit typed events (response.created, response.completed, ...) whose final event carries the full ResponsesAPIResponse, so stream_chunk_builder would raise APIError or pass the assembled response through unchanged. Detect Responses API streaming chunks before the chat/text fallthrough and extract the assembled ResponsesAPIResponse from the latest response.completed (or response.failed / response.incomplete) event, then scan its output_text via the same _completion_response_text_parts path used by non-streaming. * fix(purview_dlp): fail closed on incomplete Responses API streams Previously, _assemble_responses_api_from_chunks returned None both when the stream was not a Responses API stream and when it was a Responses API stream but no final ResponsesAPIResponse-bearing event was received. The caller treated both cases identically and fell through to stream_chunk_builder, which does not understand Responses API events. Return a (is_responses_api_stream, assembled) tuple so the caller can fail closed with an accurate error when Responses API events were seen but no final response event arrived, instead of misrouting events to the chat chunk builder. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): wrap upstream DLP errors as HTTPException(400) in blocking mode Previously a bare `raise` in `_check_content` re-propagated raw network / HTTP errors (e.g. httpx.HTTPStatusError, ConnectionError) to the client, which would surface as a 500. Now blocking-mode failures from the Graph `processContent` call (and OAuth token / protection-scopes calls) are converted to HTTPException(400) with a structured detail payload, while HTTPException instances raised by upstream layers continue to propagate unchanged. Logging-only mode is unaffected. * test(purview_dlp): cover HTTP error paths for token + Graph POST * fix(purview): include Responses API function_call arguments in DLP scan ResponsesAPIResponse.output_text only aggregates output_text content blocks and ignores function_call items, so sensitive data in model-generated tool-call arguments would bypass the DLP scan. Mirror the ModelResponse path by extracting function_call arguments explicitly from the output list. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(purview_dlp): scan ResponsesAPIResponse in stream_chunk_builder fallback If a Responses API stream slips past _assemble_responses_api_from_chunks (no chunks with type starting with 'response.') and stream_chunk_builder somehow returns a ResponsesAPIResponse, route it through _completion_response_text_parts instead of the 'not a ModelResponse' pass-through that would leak content unscanned. * fix(purview_dlp): preserve upstream Graph status on `_check_content` errors httpx.HTTPStatusError from Graph API (429, 503, etc.) was always wrapped as HTTPException(400), making rate-limits and infrastructure errors indistinguishable from a DLP policy block and stripping retry-after info. Now: - 429 and 5xx pass through with their original status code; the upstream Retry-After header is forwarded. - 401/403 (proxy-side credential/consent issue, not actionable by the client) map to 502 Bad Gateway. - A debug log makes the logging_hook -> async_logging_hook deferral observable so audit failures don't silently disappear if the framework stops dispatching async_logging_hook for some code path. * fix(purview_dlp): reject nested token-id completion prompts OpenAI /v1/completions accepts prompt: [[token, ids]] (multi-prompt token-id batches). The previous blocking-mode check only fired on a flat list[int], so nested or mixed token-id prompts skipped the Purview scan while the model still received the data. Extract the token-id detection into PurviewGuardrailBase.is_token_id_prompt and use it from the pre-call hook so every list shape Purview cannot decode fails closed. * fix(purview): drop caller-influenceable identity fallbacks in audit resolver Logging-only hook now resolves the Purview user from only the proxy-injected user_api_key_user_id (which mirrors UserAPIKeyAuth.user_id after the proxy strips caller-supplied user_api_key_* keys). Skipping the audit when no trusted identity is available prevents a caller from submitting metadata.user_id pointing at a victim's Entra object id and having their prompt/response sent to Purview under that user's identity. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: claude <claude@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Kenan Yildirim <kenan@kenany.me> Co-authored-by: vladpolevoi <vladp@lasso.security> Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Dennis Henry <dennis.henry@okta.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Vincent <yimao1231@gmail.com> Co-authored-by: Kris Xia <xiajiayi0506@gmail.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com> Co-authored-by: Tom Denham <tom@tomdee.co.uk> Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com> Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com> Co-authored-by: robin-fiddler <robin@fiddler.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com> Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com> Co-authored-by: harish-berri <harish@berri.ai> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Ryan <ryan@Ryans-MBP.localdomain> Co-authored-by: Claude (greptile subagent) <claude-greptile-bot@anthropic.com> Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com> Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com> Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com> Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: Kevin Zhao <zkm8093@gmail.com> Co-authored-by: Matthew Lapointe <lapointe683@gmail.com> Co-authored-by: Elon Azoulay <elon.azoulay@gmail.com> Co-authored-by: afoninsky <andrey.afoninsky@gmail.com> Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com> Co-authored-by: Maruti Agarwal <88403147+marutilai@users.noreply.github.com> Co-authored-by: Cursor Bugbot <bugbot@cursor.com> Co-authored-by: Greptile <greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Greptile Reviewer <greptile-apps@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com> Co-authored-by: withomasmicrosoft <withomas@microsoft.com> Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com>	2026-05-22 15:59:04 -07:00
yuneng-jiang	574ee7526d	test(streaming): tolerate Vertex 429 wrapped in MidStreamFallbackError (#28669) ... Streaming 429s are wrapped in MidStreamFallbackError so the Router can fall back; the existing 'except litellm.RateLimitError: pass' in test_vertex_ai_stream no longer matches, causing the generic pytest.fail branch to fire when upstream Vertex returns 429. Add a sibling except for MidStreamFallbackError that only swallows it when e.original_exception is a RateLimitError, so unrelated streaming failures still fail the test.	2026-05-22 15:57:29 -07:00
Cursor Agent	2c41400bdf	fix(gemini realtime): skip unknown sibling keys in transform loop ... Gemini realtime messages can include sibling metadata keys like usageMetadata alongside primary payload keys (toolCall, serverContent). Previously, the transform loop called map_openai_event for every top-level key, raising ValueError for unknown ones and terminating the WebSocket session. Skip top-level keys not present in MAP_GEMINI_FIELD_TO_OPENAI_EVENT to keep the session alive when Gemini emits usage metadata with a toolCall response. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 22:49:23 +00:00
mateo-berri	42169c8578	test(gemini realtime): wrap test_gemini_tool_call_resets_ids fixture in setup envelope ... The cached session_configuration_request the proxy stores is always serialized as {"setup": ...}; this test passed a bare config dict, so transform_session_created_event's .get('setup', {}) returned an empty dict and the responseModalities lookup ran against the default rather than the fixture. Wrap the fixture in the same shape the production cache uses.	2026-05-22 22:34:53 +00:00
mateo-berri	d1a9da3514	fix(gemini realtime): cast merged realtimeInputConfig for typeddict assignment ... mypy flagged the assignment of the merged dict into BidiGenerateContentSetup.realtimeInputConfig with [typeddict-item]: the intermediate variable widens to dict[Any, Any], losing the TypedDict narrowing the previous dict-literal form had.	2026-05-22 22:29:15 +00:00
mateo-berri	bafc187248	test(gemini realtime): wrap remaining cached session configs in setup envelope ... The session_configuration_request the proxy caches is always serialized as {"setup": ...}; three modality-related tests dumped a bare config dict instead, so transform_session_created_event's `.get('setup', {})` quietly returned an empty dict and the responseModalities lookup ran against the default rather than the fixture. Wrap the remaining tests in the same shape the production cache uses so any regression in modality forwarding actually trips.	2026-05-22 22:27:14 +00:00
mateo-berri	e5ffd021f7	fix(gemini realtime): bound _tool_call_id_to_name with an LRU; exercise modality forwarding test ... Two minor follow-ups from review: * Switch _tool_call_id_to_name to a 256-entry LRU OrderedDict so a long session with many tool calls doesn't grow the dict without bound, while retried function_call_output lookups still hit for recently-seen call_ids. * Fix test_gemini_realtime_transformation_session_created to wrap the cached session config in {"setup": ...} so the modality lookup in transform_session_created_event actually exercises responseModalities forwarding (the prior payload was silently treated as empty).	2026-05-22 22:23:37 +00:00
mateo-berri	20764dd342	fix(realtime): record synthetic session.created in deferred-setup mode ... The deferred-setup path emits a synthetic session.created directly to the client websocket but did not run it through RealTimeStreaming's store_message, so the event was missing from the session log used by success_handler / async_success_handler. Call store_message before forwarding so the synthetic event lands in the same log stream as provider-driven events.	2026-05-22 22:14:15 +00:00
mateo-berri	459c1973b4	fix(gemini realtime): deep-merge automaticActivityDetection on follow-up session.update ... The follow-up setup merge already deep-merged generationConfig and realtimeInputConfig, but realtimeInputConfig.automaticActivityDetection itself is a nested dict. A partial VAD update (e.g. the guardrail-injected disabled=True from create_response=False) silently dropped unrelated knobs such as silenceDurationMs and prefixPaddingMs from the original setup. Deep-merge that block too so partial overrides only touch the fields they specify.	2026-05-22 22:10:57 +00:00
mateo-berri	d56d875ea6	fix(vertex realtime): warn when dropping guardrail turn-detection update ... In non-deferred mode the auto-setup is sent on connect, so the audio-transcription guardrail's subsequent session.update carrying turn_detection.create_response=False cannot be forwarded as a second setup (Vertex Live closes the WebSocket with 1007). Surface a warning when this specific drop happens so operators know the model will auto-respond before the guardrail can gate it, instead of failing silently at debug level.	2026-05-22 22:04:19 +00:00
mateo-berri	0780e5f69f	fix(gemini realtime): empty toolCall must not terminate the WebSocket ... If Gemini sends a toolCall whose functionCalls list is empty (or absent), the previous `continue` left returned_message empty and the "Unknown message type" guard fired, killing the WebSocket session. Return a normal (empty) result instead so the session keeps going.	2026-05-22 21:54:16 +00:00
mateo-berri	c59260fc70	fix(gemini realtime): keep call_id→name mapping across function_call_output retries ... A client SDK that retries function_call_output (or sends the same result twice) would previously hit a missing-name lookup on the second send because _handle_function_call_output popped the call_id → name entry. Without name, Gemini may silently reject the response. Use dict.get so the mapping persists for the lifetime of the session.	2026-05-22 21:50:22 +00:00
mateo-berri	b60cc950f3	fix(gemini realtime): mirror modalities/temperature/max_output_tokens on tool-call response.created ... The audio/text response.created preamble includes modalities, temperature, and max_output_tokens on the response object so spec-compliant clients can initialise per-response state. The tool-call response.created was missing these fields, leaving clients without consistent response metadata when a response starts with a tool call instead of content. Read them from the cached session_configuration_request the same way the audio/text path does.	2026-05-22 21:44:43 +00:00
milan-berri	1b141bc588	fix(bedrock): decouple STS region from Bedrock aws_region_name (#28245) ... * fix(bedrock): decouple STS region from Bedrock aws_region_name STS AssumeRole now resolves signing region from aws_sts_endpoint (parsed host) or AWS_REGION/AWS_DEFAULT_REGION instead of aws_region_name, fixing air-gapped cross-region Bedrock setups and endpoint/signature mismatches. Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): add regression coverage for _build_sts_client_kwargs Parametrize _resolve_sts_region and _build_sts_client_kwargs matrix cases, and assert IRSA/web-identity paths use aligned STS endpoint and region_name. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): tighten STS region helpers and drop redundant web-identity endpoint synthesis Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): cover FIPS, GovCloud, and China STS endpoints Addresses greptile P2: regex sts(?:-fips)? supported sts-fips hosts but was not exercised by the parametrized parse test. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-23 00:39:24 +03:00
mateo-berri	615a7da9ba	fix(realtime): deep-merge generationConfig and refresh cache on follow-up setup ... A subsequent Gemini session.update that touches any generationConfig sub-field (e.g. just temperature) was clobbering the original generationConfig — silently dropping responseModalities and switching the session to text-only. Deep-merge generationConfig so existing keys (responseModalities, maxOutputTokens, ...) are preserved when the client updates only a subset. Also drop the early-return in _cache_session_configuration_request so the cached payload tracks the latest setup sent to the backend. Without this, downstream readers (transform_session_created_event, modality lookup in return_new_content_delta_events) keep reading stale modalities/system instruction after a follow-up setup.	2026-05-22 21:37:04 +00:00
mateo-berri	3efd803cd1	fix(gemini realtime): default conversation_id before tool-call response.done ... mypy flagged that response.done's conversation_id (str on the TypedDict) could be None when current_response_id was already set on entry. Ensure the fallback runs unconditionally before the response is constructed.	2026-05-22 21:34:34 +00:00
Cursor Agent	a0043494a0	fix(gemini realtime): deep-merge nested config in follow-up session update ... Previously, the follow-up setup performed a shallow merge between the original setup and new overrides. If a session.update touched any field inside generationConfig (e.g. modalities), the entire generationConfig would be replaced, silently dropping unrelated sub-keys like temperature or maxOutputTokens. Apply the same deep-merge to realtimeInputConfig so partial automatic-activity-detection updates don't drop other realtime input config fields either. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 21:33:07 +00:00
Cursor Agent	76a82447e8	fix(gemini realtime): include usage on tool-call response.done; coerce non-dict tool output to struct ... - Tool-call response.done now includes an empty usage object, matching the non-tool-call path so OpenAI-compatible clients always see usage. - _handle_function_call_output wraps non-dict JSON parses under a 'result' key so Gemini's functionResponses[].response (a Struct) always receives a mapping. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 21:20:42 +00:00
mateo-berri	ba75de7848	fix(realtime): preserve client session.update fields on follow-up Gemini setup ... In non-deferred mode the auto-setup pre-populates session_configuration_request, so a later client session.update carrying tools or instructions used to fall into the subsequent path and only forward turn_detection. Rebuild a merged follow-up setup that overlays the new client fields on top of the original setup so tools/instructions/etc. are no longer silently dropped.	2026-05-22 21:19:13 +00:00
Cursor Agent	9efcc02776	fix(realtime): correct conversation_id, VAD disable, modality state, empty toolCall ... - Gemini tool-call response.done now includes conversation_id so clients can match it against the preceding response.created. - Vertex AI setup no longer overrides an explicit guardrail-injected create_response: False back to disabled: False; the guardrail's intent to disable VAD auto-response is now respected. - Modality handler is now passed the locally-updated response/item IDs rather than the original input snapshot, preventing stale IDs after a prior tool-call/response.done in the same JSON message resets them. - Skip emitting orphaned response.created/response.done events when Gemini sends an empty functionCalls array. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 21:06:06 +00:00
mateo-berri	24b8e17a4b	test(gemini realtime): exercise toolCall → function_call_output name round-trip ... Update test_gemini_realtime_function_call_output_transformation to pre-load the call_id → name mapping by transforming a Gemini toolCall first, then assert that the resulting Gemini toolResponse functionResponses entry carries the function name. This pins the production round-trip rather than the degenerate 'name missing' branch.	2026-05-22 20:48:11 +00:00
Cursor Agent	36f6d84c22	fix(realtime): avoid double-serialization and normalize non-dict turn_detection in guardrail override ... - Skip the force-override block when the injection block already ran for the same session.update to avoid redundant JSON re-serialization. - Normalize non-dict client-provided turn_detection values (flat and nested audio.input.turn_detection) to a dict before enforcing create_response=False, matching the injection block's behavior and preventing potential bypass on backends that accept non-dict values. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 20:44:58 +00:00
Cursor Agent	50ef9c8c21	fix(gemini realtime): preserve original setup config on follow-up session.update ... Gemini Live treats a second BidiGenerateContentSetup as a full session replacement, not a partial merge. The guardrail-driven turn_detection-only session.update was emitting a setup containing only model + realtimeInputConfig, which would silently drop tools, generationConfig, inputAudioTranscription, and systemInstruction from the original setup. Carry forward the cached original setup and only override realtimeInputConfig. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 20:31:15 +00:00
mateo-berri	00cde9da7c	fix(lint): silence PLR0915 on client_ack_messages ... The function exceeded the 50-statement limit (64 > 50) after recent realtime guardrail additions. Matches the existing project pattern for inherently complex event/message-mapping methods (see _process_event, translate_messages_to_responses_input, transform_realtime_response, _arealtime, etc.).	2026-05-22 20:20:11 +00:00
mateo-berri	f2d8a2a8fe	fix(realtime): force create_response=False in all client session.update turn_detection when audio guardrails active ... Prevents a client from re-enabling Gemini/GA VAD auto-response (and thereby bypassing the audio transcription guardrail) by sending a later session.update with turn_detection.create_response: true.	2026-05-22 20:11:59 +00:00
mateo-berri	fb935426eb	fix(vertex_ai/realtime): normalize all GA-remapped session fields before mapping ... Previously _build_vertex_ai_setup_config only lifted nested turn_detection back to the top level. GA clients' output_modalities and audio.input.transcription were silently dropped because map_openai_params only recognises the flat OpenAI-beta keys. Use the parent's _normalize_session_payload_for_mapping so modalities, transcription, and turn_detection are all surfaced before mapping.	2026-05-22 19:43:11 +00:00
Cursor Agent	2325041913	fix(gemini realtime): normalize GA-remapped session fields before mapping ... map_openai_params only recognises the flat OpenAI-beta keys (modalities, input_audio_transcription, turn_detection). For GA clients the upstream shim renames these into the nested GA schema (output_modalities, audio.input.transcription, audio.input.turn_detection), causing them to be silently dropped in _handle_session_update. Add a normalization helper that surfaces the GA-remapped values back at the top level so the existing mapping logic picks them up. Without this, a GA client explicitly requesting modalities=['text'] would still default to audio output. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 19:29:36 +00:00
Krrish Dholakia	a3c953ed4e	style: apply black formatting to fix lint CI (LIT-3274) (#28639) (#28641) ... * fix(bedrock): strip bedrock/ prefix and URL-encode ARNs in get_bedrock_model_id for invoke path The invoke path (used by /v1/messages → Anthropic SDK / Claude Code) called get_bedrock_model_id() which, when falling back to the raw model string, did not strip the 'bedrock/' routing prefix and did not URL-encode ARNs. For a model like: bedrock/arn:aws:bedrock:us-east-1:<ACCOUNT>:inference-profile/global.anthropic... the URL built was: /model/bedrock/arn:aws:bedrock:…/invoke-with-response-stream ❌ Bedrock returned a JSON error body. LiteLLM's AWSEventStreamDecoder passed those bytes into botocore's EventStreamBuffer which expects binary event-stream framing. Checksum validation failed on the JSON prelude (0x223a7b22 == ':{"') producing a misleading botocore.eventstream.ChecksumMismatch instead of the actual Bedrock error. Fix: strip 'bedrock/' (and 'invoke/') routing prefix from model string, then URL-encode if the result is an ARN — matching what the converse path already does in converse_handler.py. Fixes: LIT-3274 * fix(bedrock): use strip_bedrock_routing_prefix to handle compound prefixes Address greptile review: the original fix used a loop with break, so bedrock/invoke/arn:... only stripped bedrock/ leaving invoke/arn:... which is not an ARN → fell through to .replace('invoke/','',1) → bare unencoded ARN → same malformed-URL bug. strip_bedrock_routing_prefix() iterates without break, correctly stripping bedrock/ then invoke/ in sequence. Also adds test case for the compound-prefix scenario. * style: apply black formatting to fix lint CI (LIT-3274) --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: LiteLLM Bot <bot@berri.ai>	2026-05-22 12:10:37 -07:00
Cursor Agent	8cf3f4eda1	fix(vertex_ai realtime): keep VAD enabled when guardrails inject create_response: False ... map_automatic_turn_detection sets disabled=True whenever create_response is absent OR False. Transcription guardrails inject create_response: False to suppress auto-responses while expecting VAD to stay active, but the previous override in _build_vertex_ai_setup_config only fired when create_response was absent, leaving disabled=True and silently breaking speech detection and transcription events. Vertex Live has no 'VAD on, no auto-response' mode, so always keep VAD active in the setup config. Co-authored-by: Yassin Kortam <yassin@berri.ai>	2026-05-22 19:06:38 +00:00
milan-berri	9600fda2cc	fix(sagemaker): send native Cohere embed payload to Cohere SageMaker endpoints (#28613) ... * fix(sagemaker): use Cohere embed payload for Marketplace endpoints SageMaker embedding only special-cased Voyage; every other endpoint received HuggingFace TGI `{"inputs": [...]}`. AWS Marketplace Cohere containers expect the native Cohere embed payload (`texts`, `input_type`) and reject the HF shape with `422 EmbedReqV2.inputs is of type string but should be of type Object`. Add `SagemakerCohereEmbeddingConfig` that reuses Bedrock/Cohere request and response transforms, and route SageMaker endpoint names containing `cohere` or a Cohere embed model fragment (`embed-multilingual`, `embed-english`, `embed-v3`, `embed-v4`) to it. Supports `input_type`, `dimensions`, and `encoding_format`. Voyage and HuggingFace SageMaker endpoints are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(sagemaker): simplify cohere detection and align with file conventions - Detect Cohere SageMaker endpoints with a single `"cohere" in model.lower()` check, mirroring the existing Voyage branch instead of a separate helper function and marker constant. - Drop instance caches of sub-configs; instantiate `BedrockCohereEmbeddingConfig` / `CohereEmbeddingConfig` per call to match the existing pattern in `BedrockCohereEmbeddingConfig._transform_request`. - Match `SagemakerEmbeddingConfig`'s signatures, defaults, and `Any` typing for `logging_obj`; collapse the input-normalization helper inline. - Inline `transform_embedding_response` input lookup; no behavior change. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sagemaker): restore provider-supported embedding params after map Cohere input_type is advertised in get_supported_openai_params but was filtered out of non_default_params by OPENAI_EMBEDDING_PARAMS before map_openai_params ran. Merge supported params from passed_params after map (same path Greptile flagged). Handle input_type explicitly in SagemakerCohereEmbeddingConfig.map_openai_params and add an integration test through get_optional_params_embeddings. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(embeddings): only restore non-OpenAI supported params after map The post-map restore loop must skip OPENAI_EMBEDDING_PARAMS so mapped fields (e.g. dimensions -> output_dimension) are not duplicated under their OpenAI names. Align SageMaker embedding import order with sibling files and add a regression test for dimensions mapping. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(sagemaker): avoid double post_call on Cohere embedding response Greptile review on #28613 caught that `CohereEmbeddingConfig._transform_response` calls `logging_obj.post_call` internally. The SageMaker embedding handler already calls `post_call` once before invoking the transform, so the Cohere SageMaker path fired callbacks, cost calculators, and log handlers twice per request. Extract the parsing body of `_transform_response` into `_populate_embedding_response` (pure extract-method, no behavior change for existing Cohere direct or Bedrock Cohere paths, which keep calling `_transform_response`). Have `SagemakerCohereEmbeddingConfig` call the new helper directly so it parses the response without re-logging. Add a regression test asserting `logging_obj.post_call` is not invoked by the SageMaker Cohere transform. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-22 12:00:42 -07:00