mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
39460 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cursor Agent
|
a8b592b634
|
fix(gemini realtime): event_id, item copy, and dict guard for tool-call events
- Emit event_id on response.output_item.added for tool calls so spec-compliant OpenAI Realtime SDK clients can index/deduplicate the event like every other server-sent event in the sequence. - Pass a shallow copy of function_call_item to response.output_item.done and conversation.item.created so downstream handlers (e.g. the beta-protocol translator) that mutate the item dict don't corrupt sibling events sharing the same reference. - Guard map_openai_event against non-dict values (e.g. Gemini's 'setupComplete: true' boolean payload) so the WebSocket session doesn't die with an AttributeError on the unguarded .get() call. Add NotRequired event_id field on OpenAIRealtimeStreamResponseOutputItemAdded to keep existing call-sites that don't set event_id compatible. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
ec79cec7ae
|
test(model_prices): allow audio_transcription_config in schema | ||
|
mateo-berri
|
e0e65bc7f5
|
Merge branch 'litellm_internal_staging' into litellm_live_api_tool_calling_support | ||
|
yuneng-jiang
|
8513d7fc0c
|
chore: update Next.js build artifacts (2026-05-23 19:21 UTC, node v20.20.2) (#28707) | ||
|
ryan-crabbe-berri
|
886e91b85e
|
fix(otel): stamp http.response.status_code on all error responses (#28405)
* fix(otel): stamp http.response.status_code on all error responses
httpx.HTTPStatusError exposes status under .response.status_code, not as a
top-level attr, so unified-endpoint 5xx failures left the SERVER span without
a status. The admin hooks only wrote a child span and never stamped or ended
the parent at all, so admin 4xx/5xx (and success) responses were invisible
to dashboards. Adds a fallback to .response.status_code in get_error_information,
and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook
with the same _record_exception_on_span helper the unified path uses.
Resolves LIT-3193
* test(otel): exercise httpx.HTTPStatusError through admin path
Pins the contract that get_error_information's response.status_code fallback
is reachable from any entry point — without this, a future refactor that
bypasses _record_exception_on_span in the admin hooks could regress for
httpx-wrapped exceptions while the unified suite still passes.
* chore(otel): trim verbose comments in LIT-3193 changes
Tighten docstrings and remove redundant section dividers/inline narration.
Behavior is unchanged.
* fix(otel): set span.status on management hook parent SERVER span
Mirror the unified failure path: stamp StatusCode.ERROR on the parent
SERVER span before recording the exception, and StatusCode.OK before
ending it on success. Without this, OTEL backends filtering on span
status (the idiomatic primitive) miss admin-endpoint failures even
though the http.response.status_code attribute is correct.
Extend assert_server_span_attrs to assert span.status.status_code
matches the expected outcome so the gap can't regress.
* fix(otel): close SERVER span on body-validation and unhandled errors
Stash the SERVER span on request.state in auth so FastAPI exception
handlers can finish it for failures that occur after auth but before
the route handler (e.g. /model/new TypeError, /key/generate
RequestValidationError). Without this, those requests left dangling
spans missing http.response.status_code.
Resolves LIT-3193
* fix(otel): generic 500 body, log exception details server-side
Don't leak str(exc) and type(exc).__name__ to clients on uncaught
exceptions. The full traceback is logged via verbose_proxy_logger and
the SERVER span still gets http.response.status_code=500.
Resolves LIT-3193
* fix(otel): stamp http.response.status_code on every SERVER span path
Closes three remaining gaps where the proxy SERVER span ended without
the http.response.status_code attribute:
1. ProxyException raised from _read_request_body (e.g. invalid JSON
body) bubbled out of user_api_key_auth before the SERVER span was
created, so the FastAPI handler had nothing to close and the trace
never reached the backend. Hoist the span creation to a new
idempotent _ensure_parent_otel_span_on_request_state helper called
at the top of user_api_key_auth; wire openai_exception_handler to
close the dangling span. Covers /v1/chat/completions, /v1/messages,
/v1/responses (shared handler).
2. /v1/responses success — _handle_success ends the proxy span before
async_post_call_success_hook fires on this path, so the hook's
set_response_status_code_attribute(200) silently no-op'd against an
ended span. Stamp 200 + set OK status at the close site in
_handle_success / _end_proxy_span_from_kwargs via a shared
_close_proxy_span_ok helper, so the attribute lands regardless of
which success hook runs first.
3. Failure path for exceptions without code/status_code (e.g. a bare
TypeError surfacing through _handle_llm_api_exception) — empty
error_information.error_code → _record_exception_on_span skips the
stamp → the hook ends the span. Default to 500 in
async_post_call_failure_hook so the attribute is always set.
Resolves LIT-3193
|
||
|
ishaan-berri
|
14c0a2b3e2
|
feat(prometheus): emit per-token-type detail metrics (LIT-3220) (#28372) (#28378)
* feat(prometheus): emit per-token-type detail metrics (LIT-3220) (#28372) Adds five sparse counter metrics that break out the token detail fields providers already report in `usage.prompt_tokens_details` and `usage.completion_tokens_details`: - litellm_input_cached_tokens_metric (provider prompt-cache reads) - litellm_input_cache_creation_tokens_metric (Anthropic prompt-cache writes) - litellm_input_audio_tokens_metric (audio input tokens) - litellm_output_reasoning_tokens_metric (reasoning tokens) - litellm_output_audio_tokens_metric (audio output tokens) These are additive — existing input/output/total counters are unchanged, so no dashboards break. Each new counter is only incremented when the underlying detail is populated and > 0, keeping scrape output sparse for providers that don't report a given field. Data is read from the canonical Usage dict that `get_standard_logging_object_payload` already attaches at `standard_logging_payload["metadata"]["usage_object"]`, so no new plumbing through the logging pipeline is required. Tests: 10 new unit tests covering registration, label-set parity, all-types increment, zero/None/negative skip behaviour, and the no-metadata/no-usage_object no-op paths. Closes LIT-3220 Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> * chore: remove proof folder image --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
yuneng-jiang
|
5e16f20962
|
test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints (#28681)
* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents ( |
||
|
ishaan-berri
|
203b529c9d
|
feat(azure): add speech transcription config support (#27482)
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com> |
||
|
Yassin Kortam
|
2eab9ee2c0
|
perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths (#28289)
* perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths
- Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path
- Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config
- Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories
- Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config)
- Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default)
- Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each)
- Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk
- Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch
- Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk
- Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params)
- Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels
- Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches
- Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution
* perf: address greptile review for anthropic streaming hot path
- Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta
events from different block indexes are observed without an intervening
flush. Anthropic sends blocks strictly sequentially, but defensive bail
prevents silent text-merging if the protocol ever interleaves.
- Replace leaf-class `__dict__` check for `async_post_call_streaming_hook`
in `_callback_capabilities` with a function-identity comparison that
walks the MRO. A vendor base class can carry the override and the
registered class can add nothing else; before this PR the hook was
unconditionally invoked, so an inherited-override miss would silently
drop the hook on the streaming path.
- Add unit tests for both behaviors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mypy): narrow model_name to str in cost-injection branch
The hoisted cost_injection_active flag in chunk_processor encodes the
`bool(model_name)` requirement but mypy can't track that invariant
through the local, so the per-chunk `_process_chunk_with_cost_injection(
chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed
non-None local inside the cost-injection branch so mypy narrows
correctly without changing runtime behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Michael-RZ-Berri
|
3b2ce201d8
|
encrypt callback_vars in key/team metadata at rest (#27141)
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
Mateo Wang
|
492891cad8
|
CI: copy of #25177 (OCI GenAI: embeddings, streaming/reasoning fixes, model catalog) (#28223)
* fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455)
Squash-merged by litellm-agent from Anai-Guo's PR.
* feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508)
Squash-merged by litellm-agent from yimao's PR.
* fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503)
Squash-merged by litellm-agent from krisxia0506's PR.
* Fix Gemini MIME detection for extensionless GCS URIs (#27278)
Squash-merged by litellm-agent from krisxia0506's PR.
* fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107)
Squash-merged by litellm-agent from voidborne-d's PR.
* feat(chart): add support for autoscaling behavior in HPA (#27990)
Squash-merged by litellm-agent from FabrizioCafolla's PR.
* feat(proxy): add blocked flag to models for pause/resume from the UI (#27927)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix: pass socket timeouts to Redis cluster clients (#27920)
Squash-merged by litellm-agent from tomdee's PR.
* Fix/cache token (#28009)
Squash-merged by litellm-agent from escon1004's PR.
* fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080)
Squash-merged by litellm-agent from Divyansh8321's PR.
* fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617)
* fix: reset org and tag budgets (#27326)
* reset org budgets
* reset tag budgets
---------
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
* fix(ui): omit allowed_routes from key edit save when unchanged (#27553)
* fix(ui): omit allowed_routes from key edit save when unchanged
When a team admin opens Edit Settings on a key with key_type=AI APIs and
saves without changing anything, the UI re-sends the existing allowed_routes
value, which the backend's _check_allowed_routes_caller_permission gate
rejects for non-proxy-admins (LIT-2681).
Strip allowed_routes from the patch in handleSubmit when it deep-equals the
original keyData.allowed_routes. The backend treats absence as "leave alone,"
so no-op saves now succeed for non-admins. Admins explicitly editing the
field still send the new value.
* fix(ui): order-insensitive allowed_routes diff + cover null-original case
Address Greptile review:
- Switch the "is allowed_routes unchanged" check to a Set-based comparison so
a server-side reorder of the array doesn't register as a user edit and
re-trigger LIT-2681.
- Add two regression tests: (1) keyData.allowed_routes is null and the form
is untouched — patch should strip the field; (2) server returned routes in
a different order than the user originally entered — patch should still
recognize the value as unchanged.
* chore(ui): strip ticket refs and tighten comments in key edit fix
- Remove internal-tracker references from in-code comments
- Tighten the WHY comment in handleSubmit to two lines
- Drop redundant test-block comments — test names already describe the case
* fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc
* fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests
GuardrailRaisedException and BlockedPiiEntityError both lacked a
status_code attribute. When these exceptions reached the proxy
exception handler (getattr(e, 'status_code', 500)), the fallback
defaulted to HTTP 500 — making intentional guardrail blocks
indistinguishable from server errors and causing unnecessary client
retries.
Changes:
- Add status_code=400 (keyword-only) to GuardrailRaisedException
- Add status_code=400 (keyword-only) to BlockedPiiEntityError
- Update _is_guardrail_intervention() to recognize both exceptions
so downstream loggers record 'guardrail_intervened' instead of
'guardrail_failed_to_respond'
- Add 6 unit tests for default/custom status codes and getattr pattern
- Strengthen existing blocked-action test with status_code assertion
Fixes #24348
---------
Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* fix(router/proxy): address Greptile P1+P2 review comments on PR #28161
- router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429)
when a specifically-addressed deployment is administratively blocked; 429 misleads
retry-enabled clients into spinning forever against a paused model
- proxy_server: compute get_fully_blocked_model_names() once before both branches in
model_list() instead of duplicating the call in each branch
- deepseek: upgrade silent debug log to warning when injecting placeholder
reasoning_content so callers are clearly notified of degraded multi-turn quality
- tests: update two blocked-deployment assertions to expect ServiceUnavailableError
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address bug detection findings (cache token order, mutable defaults)
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: address bugs in async pass-through, anthropic cache token detection, rerank tests
- async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments
- cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0
- dashscope rerank tests: pass request to httpx.Response constructions for consistency
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix code qa
* fix(vertex_ai/gemini): strip MIME parameters from GCS contentType
GCS object metadata's contentType field can include parameters such as
'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases
so downstream get_file_extension_from_mime_type sees a bare MIME type.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vertex_ai/gemini): clarify mime-type error message string concatenation
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(oci): add embeddings, fix streaming/reasoning, expand model catalog
- Add OCIEmbedConfig with full Cohere embed support (7 models, batch up to 96)
- Fix sync streaming: split SSE events on \n\n before JSON parsing
- Fix reasoning models (Gemini 2.5, xAI Grok): make completionTokens and message
optional in OCIResponseChoice to handle max_tokens exhausted on reasoning
- Fix compartment_id resolution in chat transform to use resolve_oci_credentials
- Fix tool call id: make OCIToolCall.id optional, generate UUID fallback for
providers (Google via OCI) that omit it
- Add OCI_KEY env var support for inline PEM keys
- Fix datetime.utcnow() deprecation in request signing
- Expand model catalog: 29 OCI models including Llama 4, Gemini 2.5, xAI Grok,
Cohere Command A, and all Cohere embed variants
- Add 37 live integration tests: sync/async completions for Meta/Google/xAI/Cohere,
sync/async embeddings, tool use across all vendors, streaming, env var auth
- Add 23 embed unit tests covering all transform and validation paths
* fix(oci): remove dead OCI elif branch in utils.py, align async split_chunks with sync version
* test(oci): add unit tests for split_chunks fix and no-duplicate-OCI-branch guard
* fix(oci): address remaining bugs from issue #25082 — streaming signed body, Cohere stop sequences, hardcoded defaults
- Bug 1: sync and async streaming paths now use signed_json_body when provided
instead of re-serializing data with json.dumps() — the OCI RSA-SHA256 signature
covers the exact request body bytes, so re-serializing produces an invalid sig
- Bug 3: Cohere stop sequences now map to 'stopSequences' (was incorrectly 'stop')
- Bug 4: removed hardcoded Cohere defaults (maxTokens=600, temperature=1, topK=0,
topP=0.75, frequencyPenalty=0) that silently overrode user intent on every call
- Added 6 unit tests covering all three fixes
* fix(oci): comprehensive code quality pass — bugs, tests, schema accuracy
- Fix Cohere tool call IDs (was always call_0; now UUID per call)
- Fix TOOL_CALL finish reason mapping in both sync and streaming paths
- Fix Cohere stop parameter mapping (stop → stopSequences)
- Remove hardcoded Cohere defaults (maxTokens/topK/topP/frequencyPenalty)
- Fix content[0] safety guard against empty content arrays
- Fix streaming signed body used consistently (not re-serialized)
- Raise OCIError (not bare Exception/ValueError) throughout
- Centralize OCI_API_VERSION constant; import uuid at module level
- Fix embed get_complete_url to strip trailing slashes from api_base
- Fix OCIEmbedResponse schema: add inputTextTokenCounts (actual OCI field)
- Fix embed usage computed from inputTextTokenCounts (sum of per-input counts)
- Fix Cohere toolCallId included in tool result messages
- Add OCIToolCall.id as Optional (absent in Google/xAI streaming chunks)
- Update tests to reflect correct behavior (no hardcoded defaults, UUID ids,
deferred credential validation, OCIError vs ValueError, real response schema)
* test(oci): move integration tests to tests/llm_translation/
Addresses greptile P1: tests/test_litellm/ is for mock-only unit tests
(make test-unit target). Real-network OCI tests now live in the correct
location alongside other provider integration tests.
* fix(oci): align types and transformation with official OCI SDK
- Remove OCIVendors.GEMINI — apiFormat="GEMINI" is invalid; all non-Cohere
models use apiFormat="GENERIC"
- Add toolChoice, logitBias, logProbs to OCIChatRequestPayload so params
present in the mapping are no longer silently dropped by Pydantic
- Exclude n→numGenerations from Cohere param map (not a Cohere API field)
- Fix CohereToolResult: change callId/result to call/outputs matching
the OCI SDK's CohereToolResult structure
- Fix CohereToolMessage: replace non-existent toolCallId with toolResults
list; update adapt_messages_to_cohere_standard to build proper tool-result
history entries by resolving tool call name+params from preceding assistant
messages
- Map generic-model stream finish reasons to OpenAI convention
(COMPLETE→stop, MAX_TOKENS→length, TOOL_CALLS→tool_calls), consistent
with the existing Cohere streaming path
- Add optional id field to OCIEmbedResponse so valid API responses
carrying an id are not rejected by the Pydantic model
* fix(oci): use 'output' key in Cohere tool result outputs (matches reference impl)
* fix(oci): port schema/type utilities from langchain-oracle reference impl
- Add resolve_oci_schema_refs: inline $ref/$defs — OCI rejects JSON Schema refs
- Add resolve_oci_schema_anyof: flatten Optional[T] anyOf (Pydantic v2 emits these)
- Add sanitize_oci_schema: strip title, normalise null types, ensure array items
- Add OCI_JSON_TO_PYTHON_TYPES: Cohere expects Python type names (str/int/float),
not JSON Schema names (string/integer/number)
- Add enrich_cohere_param_description: embed enum/format/range/pattern constraints
into description since CohereParameterDefinition has no dedicated fields
- Apply all of the above in adapt_tool_definitions_to_cohere_standard and
adapt_tool_definition_to_oci_standard
- Fix toolChoice conversion: map OpenAI string ('auto','none','required') to OCI
dict form ({"type":"AUTO"} etc.) — the API rejects plain strings
- Update unit test expectations to match correct Python type names and enriched
descriptions
* refactor(oci): split transformation.py into cohere.py and generic.py
transformation.py was 1 243 lines doing too many jobs. Split along the
same boundaries as the langchain-oracle reference (providers/cohere.py,
providers/generic.py):
chat/cohere.py — Cohere message/tool building, response + stream parsing
chat/generic.py — Generic message/tool building, response + stream parsing
transformation.py — thin OCIChatConfig orchestrator + OCIStreamWrapper
Public symbols (OCIChatConfig, OCIStreamWrapper, adapt_messages_to_*,
OCIRequestWrapper, version, …) remain importable from transformation.py
for backward compatibility. OCIStreamWrapper gains delegating shims for
_handle_cohere_stream_chunk and _handle_generic_stream_chunk so existing
test call sites keep working unchanged.
transformation.py: 1 243 → 620 lines
* refactor(oci): principal-level code quality pass
- Remove _extract_text_content duplication — single definition in cohere.py,
imported where needed; instance method on OCIChatConfig eliminated
- Move cryptography imports to module level with _CRYPTOGRAPHY_AVAILABLE flag
and _require_cryptography() guard; no more re-import on every signing call
- Move litellm version import to module level via litellm._version; remove
inline import inside validate_oci_environment
- sign_with_manual_credentials now returns Tuple[dict, bytes] matching
sign_with_oci_signer — asymmetry eliminated, Optional[bytes] guards removed
throughout stream wrappers (signed_json_body: bytes = b"")
- Rename _openai_to_oci_cohere_param_map → openai_to_oci_cohere_param_map
for consistency with openai_to_oci_generic_param_map
- Remove double-key bug in map_openai_params where responseFormat was stored
under both OCI and OpenAI key names simultaneously
- Remove delegating shims (adapt_messages_to_cohere_standard,
adapt_tool_definitions_to_cohere_standard, _handle_generic_stream_chunk)
from OCIChatConfig/OCIStreamWrapper; tests now import directly from
cohere.py and generic.py where symbols live
- Trim __all__ to 7 genuine public symbols; remove the 13-symbol list that
existed only to support test imports
- Collapse per-model integration test classes into pytest.mark.parametrize;
CHAT_MODELS list is the single source of truth for model-specific config
- Black + Ruff clean across all OCI files
* fix(oci): address PR review findings
- types/llms/oci.py: add "TOOL_CALL" to CohereChatResponse.finishReason
Literal so Pydantic does not raise ValidationError on non-streaming
Cohere tool-use calls (Greptile P1)
- test_oci_cohere_tool_calls.py: add test covering TOOL_CALL finish reason
- model_prices_and_context_window.json: remove 6 duplicate oci/cohere.embed-*
keys that were silently overridden by the more complete entries already
present in the file (Greptile P1)
- common_utils.py: move OCI_API_VERSION here from chat/transformation.py
so embed/transformation.py does not need to import chat/transformation;
change Protocol stub body from ... to pass (CodeQL "statement no effect");
add comment to sha256_base64 clarifying it implements OCI HTTP signing
spec, not password hashing (CodeQL false positive)
- chat/transformation.py: import CustomStreamWrapper from
litellm_core_utils.streaming_handler instead of litellm.utils to reduce
import cycle depth (CodeQL cyclic import)
- chat/cohere.py, chat/generic.py: import Usage and
ChatCompletionMessageToolCall from litellm.types.utils instead of
litellm.utils for the same reason
- embed/transformation.py: import OCI_API_VERSION from common_utils
instead of chat/transformation (removes the embed→chat import edge)
* test(oci): add unit tests to improve patch coverage
- test_oci_common_utils.py (new): covers sha256_base64, build_signature_string,
OCIRequestWrapper.path_url, resolve_oci_credentials, get_oci_base_url,
validate_oci_environment, sign_with_oci_signer error paths, sign_oci_request
routing, load_private_key_from_file error paths, resolve_oci_schema_refs
(including circular ref and external $ref), resolve_oci_schema_anyof,
sanitize_oci_schema (all branches), enrich_cohere_param_description
- test_oci_generic_chat.py (new): covers content-message error paths (non-dict
item, unsupported type, non-string text, invalid image_url), tool-call
validation error paths, adapt_messages_to_generic_oci_standard error paths,
handle_generic_response (None message, text content, tool calls),
handle_generic_stream_chunk (finish reasons, streaming tool calls),
OCIStreamWrapper non-string chunk error
- test_oci_chat_transformation.py: add error paths for validate_environment
(empty messages), transform_request (missing compartment_id, Cohere without
user messages), transform_response (error key), map_openai_params
(unsupported param with and without drop_params), tool_choice string mapping
- test_oci_cohere_tool_calls.py: add edge cases for stream chunk finish
reasons (TOOL_CALL, MAX_TOKENS, unknown), _extract_text_content with
non-dict list items and non-string input,
adapt_messages_to_cohere_standard with malformed JSON tool arguments
* fix(oci): rename supports_streaming to supports_native_streaming in model prices
The JSON schema for model_prices_and_context_window.json uses
`supports_native_streaming` (not `supports_streaming`) and has
`additionalProperties: false`. Rename the field across all OCI
entries to pass the schema validation test.
* test(oci): add 67 tests targeting uncovered happy paths for coverage
Boost patch coverage on the four lowest-coverage OCI files:
- common_utils.py: sign_with_manual_credentials (oci_key / oci_key_file
paths), sign_oci_request routing, _require_cryptography
- generic.py: adapt_messages_to_generic_oci_standard (all roles),
adapt_tool_definition_to_oci_standard, adapt_tools_to_openai_standard,
handle_generic_stream_chunk text/finish-reason paths
- cohere.py: _extract_text_content, adapt_messages_to_cohere_standard
(all roles including tool results), handle_cohere_response /
handle_cohere_stream_chunk all finish-reason branches
- transformation.py: get_vendor_from_model, OCIChatConfig._get_optional_params
(toolChoice string→dict, responseFormat, tools for both vendors),
transform_request for GENERIC model, get_sync/async_custom_stream_wrapper
with mocked HTTP, OCIStreamWrapper.chunk_creator happy paths
* fix(oci): suppress CodeQL false positive on sha256_base64 (OCI HTTP signing, not password hashing)
* fix(oci): remove 6 duplicate model price entries and reconcile conflicting values
Six OCI chat model keys appeared twice in model_prices_and_context_window.json
with conflicting pricing/context data (JSON parsers silently discard the first).
Remove the first-occurrence entries and update the surviving entries:
- meta.llama-4-maverick / llama-4-scout: keep updated entries (free preview
pricing, larger context windows, vision support)
- meta.llama-3.1-70b: keep original pricing, restore supports_native_streaming
- google.gemini-2.5-{flash,pro,flash-lite}: keep OCI pricing page values,
restore supports_native_streaming
* fix(oci): route GPT-5 family to maxCompletionTokens
GPT-5 / GPT-5-mini / GPT-5-nano / GPT-5.5 on OCI reject "maxTokens"
with HTTP 400:
Invalid 'maxTokens': Unsupported parameter: 'maxTokens' is not
supported with this model. Use 'maxCompletionTokens' instead.
(Same convention as OpenAI's reasoning-API contract.)
Add a model-aware rename in OCIChatConfig._get_optional_params so the
request payload uses maxCompletionTokens when the model id starts with
openai.gpt-5. Regular Llama / Cohere / Gemini / GPT-4.x continue to use
maxTokens unchanged.
Also widen OCIChatRequestPayload to carry the new optional field so it
survives Pydantic serialization.
Verified live against OCI us-chicago-1:
- openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.5 all return 200
- Full feature sweep on gpt-5.5 (basic, system, multi-turn, streaming,
tools, usage) all green
- meta.llama-3.3-70b-instruct still uses maxTokens (no regression)
4 new unit tests cover the helper, the routing in both pre- and
post-translation states, and Pydantic serialization.
* ci(oci): fix CI failures — black formatting + recursive_detector ignore
- Run black on litellm/llms/oci/common_utils.py + 3 OCI test files
that drifted out of black-compliance during the rebase.
- Add the three bounded recursive functions in oci/common_utils.py
(`_resolve`, `resolve_oci_schema_anyof`, `sanitize_oci_schema`) to
the recursive_detector IGNORE_FUNCTIONS list. All three are bounded:
`_resolve` uses a `resolving_stack` cycle guard; the other two are
bounded by JSON-schema tree depth (no cycles in well-formed input),
matching the pattern of the existing OCI/Vertex schema walkers
already on the list.
* fix(oci): silence MyPy errors in cohere.py — typed-dict access
Two errors flagged by `lint` CI:
llms/oci/chat/cohere.py:73: "object" has no attribute "__iter__"
llms/oci/chat/cohere.py:119: No overload variant of "get" of "dict"
matches argument types "object", "CohereToolCall"
Both stem from `msg.get("tool_calls")` / `msg.get("tool_call_id")`
returning `object` per the AllMessageValues TypedDict union. Bind to
`Any` locally for the iteration and coerce the lookup key with `str()`,
removing the now-unused `# type: ignore` on those lines.
No behaviour change — pure type-narrowing for the type checker.
* fix(oci): silence CodeQL py/weak-sensitive-data-hashing on sha256_base64
CodeQL's taint analysis traces request bodies back to environment-loaded
secrets and flags `hashlib.sha256(body).digest()` as
`py/weak-sensitive-data-hashing` — even though SHA-256 is the algorithm
mandated by the OCI HTTP request signing spec for the
`x-content-sha256` header (not a password/secret hash).
The previous suppression used legacy `# lgtm[...]` syntax which the
modern CodeQL action ignores. Switch to Python's standard
`hashlib.sha256(..., usedforsecurity=False)` (Python 3.9+) which CodeQL
honours as a non-security declaration. Behaviour unchanged.
* feat(oci): add reasoning_effort passthrough — only true missing primitive
OCI's GenericChatRequest exposes a reasoningEffort field
(NONE/MINIMAL/LOW/MEDIUM/HIGH) that's the single biggest cost knob for
reasoning-capable models on the service:
- GPT-5 family
- Gemini 2.5
- Grok reasoning variants (3-mini, 4-fast, 4.20)
- Cohere Command-A-Reasoning
Setting reasoning_effort=LOW typically cuts reasoning-token spend 5-10×
vs the default. Without exposing this, litellm users had no way to tune
cost-vs-quality on these models.
The other GenericChatRequest fields (verbosity, parallel_tool_calls,
logit_bias, n, metadata, web_search_options, prediction) are not
exposed because they are not missing primitives — they either duplicate
prompt-engineering, framework-level controls, or are too niche to
justify the maintenance surface. We only ship what users genuinely
can't accomplish another way.
Excluded from the Cohere v1 param map: CohereChatRequest has no
reasoningEffort field, and Cohere reasoning models
(cohere.command-a-reasoning) use COHEREV2 which is a separate request
type not covered by this PR.
Verified live: GPT-5.5 + reasoning_effort="HIGH" sends
{"reasoningEffort": "HIGH"} on the wire and OCI accepts the request.
* feat(oci): reasoning_effort + reasoning_tokens for OCI GenAI
Three small additions for OCI reasoning models, requested by users
testing the PR in production fork builds:
1. **reasoning_effort param mapping (GENERIC vendors).** OCI expects
uppercase levels ("LOW"/"MEDIUM"/"HIGH"/"NONE") on `reasoningEffort`,
but OpenAI-compatible clients send lowercase. Mapped + uppercased in
`_get_optional_params`. Marked unsupported on Cohere V1/V2 since OCI
Cohere has no reasoning models (avoids Pydantic validation failure
on CohereChatRequest).
2. **"disable" → "NONE" mapping.** OpenAI uses "disable" to turn off
reasoning; OCI uses "NONE". Without this, callers get a 400.
3. **reasoning_tokens propagated to Usage.** OCI returns
`completionTokensDetails.reasoningTokens` but it wasn't being passed
to LiteLLM's Usage object. Now flows through to
`Usage.completion_tokens_details.reasoning_tokens` so callers can
track reasoning token consumption for cost/observability.
Tests: 7 new unit tests in TestOCIReasoningEffort covering upper/lower
case, "disable"→"NONE", Cohere drop/raise paths, and reasoning_tokens
extraction (with and without completionTokensDetails). 5 new live
integration tests against xai.grok-3-mini in us-chicago-1 verifying the
full request/response loop end-to-end. Existing
test_transform_response_simple_text assertion that
completion_tokens_details was None has been updated to assert
reasoning_tokens flows through.
Verified live on xai.grok-3-mini: reasoning_effort=low → OCI accepts
"LOW", returns reasoningTokens=316 in usage. reasoning_effort=disable
→ OCI accepts "NONE". Full suite: 370/370 unit + 51/51 integration.
* fix(codeql): re-scope py/weak-sensitive-data-hashing exclusion to OCI signing file
CodeQL's taint analysis re-fires the `py/weak-sensitive-data-hashing`
alert at `litellm/llms/oci/common_utils.py:103` whenever upstream code
paths into the OCI signing module change (touching `transformation.py`
opens new flow paths that CodeQL re-evaluates from scratch). The
`hashlib.sha256(..., usedforsecurity=False)` declaration silences the
direct-call form of the query but not the taint-flow form.
SHA-256 here is mandated by the OCI HTTP signing specification for the
x-content-sha256 content-integrity header — not for password storage:
https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm
CodeQL has no per-query path filter and GitHub Code Scanning ignores
inline lgtm/codeql comments, so path-ignoring this single ~560-line
signing utility file is the narrowest available suppression. All other
files retain full coverage of py/weak-sensitive-data-hashing — including
litellm/proxy/utils.py where the rule legitimately applies.
This restores the NEUTRAL CodeQL state the PR had on prior commits
(see `2111c98af7` for the same approach on the previous branch
evolution that the cherry-pick was rebased onto a different baseline).
* fix(oci): drop duplicate text on Cohere streaming terminal chunk
OCI Cohere's terminal SSE event re-sends the full assembled response in
`text` alongside a populated `chatHistory`. Emitting that text as another
delta concatenates the entire response onto the already-streamed output
(e.g. "How can I help?How can I help?").
Use `chatHistory is not None` as the discriminator for the consolidated
terminal event — `finishReason` is a weaker signal that could in principle
appear on a non-consolidated chunk. The two coincide today; this preserves
correctness if OCI ever ships finishReason on an incremental chunk.
Adds a live-OCI integration regression test that compares streamed vs
non-streamed length and asserts the response prefix appears only once.
Verified to fail under the previous code with the exact reported
reproduction: 'Hello! How can I help you today?Hello! How can I help you today?'.
Reported by @gotsysdba on PR #25177.
* fix(oci): buffer SSE stream across HTTP read boundaries
The old split_chunks helper split each individual HTTP read on "\n\n",
which assumed SSE event boundaries always aligned with read boundaries.
In practice the OCI streaming endpoint delivers events that may:
- straddle two reads (chunk_creator gets a truncated JSON and crashes)
- arrive separated by a single "\n" instead of "\n\n"
- share a read with multiple complete events
Replace the inline split with module-level helpers _iter_sse_events
(sync) / _aiter_sse_events (async) that maintain a buffer across reads,
split on any newline, and yield only complete "data:" lines.
Add 25 regression tests covering event-split-across-reads, tiny-chunk
reads, single-newline separators, keepalive/comment lines, trailing
partial events flushed at EOF, "\r\n" line endings, and an end-to-end
smoke test that feeds an awkwardly-chopped payload through the splitter
into OCIStreamWrapper.chunk_creator.
Reported by John Lathouwers.
* test(oci): repoint TestOCIKeyNormalization to sign_with_manual_credentials
The signing helper moved from OCIChatConfig._sign_with_manual_credentials
to a module-level sign_with_manual_credentials in common_utils.py. Four
tests in TestOCIKeyNormalization still called the old method:
- 2 failed outright with AttributeError
- 2 passed by accident because they used pytest.raises(Exception),
which happily caught the AttributeError instead of exercising the
intended OCIError path
Repoint all four to the new module-level function so they exercise the
actual oci_key type-validation branch.
* fix(oci): validate oci_region before URL interpolation to prevent SSRF
Anchor oci_region to ^[a-z][a-z0-9-]{0,30}[a-z0-9]$ inside get_oci_base_url
so user-supplied regions that would redirect the signed request to an
attacker-controlled host (e.g. 'evil.com/#') fail with HTTP 400 before
the URL or signature is built. Empty string still falls back to the
us-ashburn-1 default, so existing callers are unaffected.
* test(audio): skip when gpt-4o-audio-preview is unavailable upstream
OpenAI retired `gpt-4o-audio-preview` (404 model_not_found in CI as of
2026-05-19), and the existing try/except in these tests only re-raised
on 'openai-internal' errors. Other exceptions were silently swallowed,
so the next line ran with an unbound `response`/`completion` and
failed with an unrelated UnboundLocalError that masked the real cause.
Extend the skip condition to also cover model_not_found / 'does not exist'
so the suite reports the upstream outage cleanly, matching the pattern
used in
|
||
|
Cursor Agent
|
35cc424923
|
fix(realtime): send sanitized toolResponse before guardrail clientContent
Some checks are pending
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
Two related fixes for the function_call_output blocked-by-guardrail path: 1. Ordering: Gemini Live requires a matching toolResponse immediately after a toolCall before any other client message. Previously we ran the guardrail first (which sends clientContent/cancel) and only then forwarded the sanitized function_call_output. Add an optional pre_block_backend_message arg to run_realtime_guardrails so the sanitized toolResponse is emitted before the guardrail's own backend messages. 2. Stale pending flag: stop setting _pending_guardrail_message in the tool-output block. That flag exists to swallow the reflexive response.create an OpenAI client sends right after a user text message. In tool-calling flows the client may never send a response.create (e.g. Gemini SDKs auto-respond), so leaving the flag set would consume an unrelated response.create from a later turn. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
02aa7b2803
|
fix(gemini realtime): propagate usageMetadata on tool-call response.done
Gemini Live emits usageMetadata as a sibling top-level key alongside the toolCall frame; the tool-call branch was unconditionally building response.done from get_empty_usage(), so tokens consumed by tool-call turns were recorded as zero spend and bypassed LiteLLM budget accounting. Mirror the non-tool-call RESPONSE_DONE path: when the same frame carries usageMetadata, run VertexGeminiConfig._calculate_usage and forward the real token counts. |
||
|
mateo-berri
|
f54874f707
|
Merge branch 'litellm_internal_staging' into litellm_live_api_tool_calling_support | ||
|
Cursor Agent
|
73b6da9e73
|
fix(realtime): check function_call_output before user role to prevent guardrail bypass
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
039e86f010
|
fix(gemini realtime): redact realtime payloads from debug logs
The transform_realtime_response debug logs were dumping the raw inbound Gemini frame and each outbound OpenAI event payload (up to 500 chars). Realtime frames carry transcripts, model output, and tool-call arguments, so those strings ended up in application logs whenever DEBUG was enabled. Replace the inbound dump with just the top-level frame keys and the outbound dump with just the event type. |
||
|
mateo-berri
|
0e13cfabef
|
fix(gemini realtime): preserve sibling keys on empty toolCall no-op
Replace the early return on `functionCalls` empty/absent with a `continue` plus a `tool_call_handled` flag that mirrors the existing `server_content_handled` pattern. The post-loop guard already distinguishes intentionally-consumed known keys from genuinely-unknown messages, so adding `toolCall` to that exclusion list lets the loop continue iterating over any sibling top-level keys in the same Gemini frame instead of short-circuiting on the first empty toolCall. In practice Gemini's protobuf places `toolCall`/`serverContent`/ `setupComplete` in a `oneof` so the only realistic sibling is `usageMetadata` (already filtered as unknown-top-level), but the uniform handling avoids silently discarding any future sibling key should the wire format grow. |
||
|
mateo-berri
|
70e1169989
|
fix(realtime): forward sanitized function_call_output on guardrail block
Providers that pair every toolCall with a toolResponse (e.g. Gemini and Vertex Live) stay in the awaiting-tool-call state until a toolResponse arrives. Dropping a blocked function_call_output outright left those providers stalled — the subsequent guardrail clientContent and response.create were ignored because the prior toolCall had no matching toolResponse. When the client-supplied tool output fails the realtime guardrail check, forward a sanitized placeholder function_call_output (same call_id, generic policy marker as output) instead of dropping the message entirely. The placeholder carries no blocked content, so the model never sees it, while still completing the provider's tool-call cycle so the session can recover and the violation message reaches the user. |
||
|
mateo-berri
|
d3490859a4
|
fix(ci): restore guardrail injection on duplicate session.created and cast realtime delta event
- Re-enable the one-time guardrail turn_detection update on duplicate session.created. `_maybe_send_guardrail_turn_detection_update` is already idempotent via `_guardrail_turn_detection_update_sent`, so the previous guard was unnecessary and broke the deferred-setup path where the synthetic session.created is emitted by llm_http_handler outside this loop (no prior chance to inject). - Cast the response.function_call_arguments.delta dict appended to `returned_message: List[OpenAIRealtimeEvents]` so mypy is satisfied. |
||
|
Cursor Agent
|
a27e6bcae9
|
fix(realtime): avoid stale session.created flag triggering guardrail re-injection
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
4ca26e88a5
|
fix(gemini realtime): emit function_call_arguments.delta before .done
Gemini delivers the full function-call arguments in a single toolCall frame. The OpenAI Realtime spec orders the streaming events as output_item.added -> function_call_arguments.delta(+) -> function_call_arguments.done -> output_item.done. Emit a single delta carrying the complete arguments string before the matching .done so spec-compliant SDK clients that accumulate deltas and gate finalisation on at least one delta arriving do not stall on Gemini tool calls. |
||
|
mateo-berri
|
16e44cbee8
|
fix(realtime): run guardrails on function_call_output content
Tool result outputs are client-controlled and fed to the model, so they must pass the same content checks as user text messages. Otherwise an attacker can smuggle blocked content into a function_call_output and have the model process it. |
||
|
Cursor Agent
|
295a9e6e13
|
fix(gemini realtime): promote nested turn_detection when flat value is not a dict
When the session payload had `turn_detection: None` (or any non-dict value), the normalizer skipped promoting the GA nested `audio.input.turn_detection` because it only checked key presence. The stale None then flowed into `map_automatic_turn_detection` and raised TypeError on `'create_response' in value`. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor Agent
|
ef98c98a12
|
refactor(gemini realtime): drop unused json_message arg from map_openai_event
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor Agent
|
f1b76d99e8
|
fix(gemini realtime): preserve sibling toolCall when serverContent has only transcription
Previously, when a Gemini frame contained both a transcription-only serverContent and a sibling toolCall, the transcription handler would early-return and silently drop the toolCall. Instead, mark serverContent as handled and fall through so the main loop still processes siblings like toolCall, while preserving the prior no-op behavior for empty/ transcription-only frames. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
84791fc3f5
|
fix(gemini realtime): tolerate sibling-only frames (e.g. standalone usageMetadata)
A Gemini Live frame that contains only metadata keys outside
_KNOWN_GEMINI_TOP_LEVEL_KEYS (e.g. a bare {"usageMetadata": {...}}
emitted between turns) leaves returned_message empty after the
transform loop and was tripping the 'Unknown message type' guard,
which raised ValueError and terminated the WebSocket session.
Treat such frames as no-ops and return the unchanged state instead.
|
||
|
Cursor Agent
|
1e4f86e19a
|
fix(realtime): inject guardrail turn_detection on subsequent session.update without one
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
1a838a1517
|
fix(gemini realtime): cast maxOutputTokens to int for typeddict assignment | ||
|
Cursor Agent
|
3a1a5ae392
|
fix(gemini realtime): use camelCase maxOutputTokens in response.done
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
6299633138
|
fix(gemini realtime): cast maxOutputTokens to int for typeddict assignment | ||
|
Cursor Agent
|
61928b9704
|
fix(gemini realtime): scope dotted-key event lookup and propagate session metadata to tool-call response.done
- map_openai_event: only check the current key/value pair when resolving dotted map entries (e.g. serverContent.turnComplete) so a sibling key in the same frame can't misclassify the event being processed (e.g. toolCall returning RESPONSE_DONE). - tool-call path: extract generationConfig once and include modalities, temperature, and max_output_tokens on response.done so its shape matches response.created and the non-tool-call response.done. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
milan-berri
|
7270f723de
|
fix(mcp): forward upstream initialize instructions on cold gateway init (#28231)
Some checks are pending
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
Prefetch upstream InitializeResult.instructions before merging gateway initialize options when YAML/DB do not set instructions, so clients receive upstream server text on the first MCP initialize without list_tools. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
f35e7eb2f6
|
feat(guardrails): add Microsoft Purview DLP guardrail (#24966)
* feat(guardrails): add Microsoft Purview DLP guardrail
* fix(guardrails/purview): raise_for_status on HTTP errors, cap scope cache, reuse executor
* fix(guardrails/purview): propagate litellm_call_id as correlation_id to Purview
* chore: fixes
* refactor(guardrails): delegate get_user_prompt to get_last_user_message
PurviewGuardrailBase duplicated AzureGuardrailBase (and OpenAIGuardrailBase)
user-prompt extraction. The same logic already lived in
common_utils.get_last_user_message; wire guardrail bases to that helper,
fix the helper docstring, and drop its redundant self-import of
convert_content_list_to_str.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): make protection scope cache true LRU on hits
OrderedDict.get() does not update insertion order; call move_to_end on
TTL-valid cache hits so popitem(last=False) evicts least-recently-used
users instead of FIFO by first insert.
Add a regression test with a small max cache size.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* Fix mypy
* fix(guardrails/purview): harden user-id resolution and broaden DLP text
Prefer API key and proxy-injected metadata over client metadata for Entra
identity. Scan full message transcript pre-call and all completion choices
post-call. Align logging-only hook with the same user-id rules.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails/purview): scan /v1/completions prompt and TextChoices
Normalize text-completion prompts (string or list of strings); skip token-id-only
prompts. Run post-call DLP on TextCompletionResponse choices. Extend logging_only
hook for text_completion. Add tests and completion_prompt_to_str helper.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(purview-dlp): return data after DLP pass; per-call executor; dedupe text extraction
async_pre_call_hook now returns the request dict after a successful check so
callers match skip-path behavior. logging_hook uses a fresh ThreadPoolExecutor
per invocation like Presidio to avoid single-worker starvation. Response text
extraction is centralized in _completion_response_text_parts.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): fix LRU cache refresh position and add Responses API scanning
Two fixes to the Microsoft Purview DLP guardrail:
1. LRU cache bug (base.py): When a stale scope cache entry was re-fetched,
the assignment updated the value but
Python's OrderedDict.__setitem__ preserves the original insertion order for
existing keys. This left the refreshed entry near the front of the dict,
making it the first candidate for LRU eviction via popitem(last=False).
Fix: call move_to_end(user_id) after every write to an existing key.
2. Responses API coverage gap (purview_dlp.py): Requests to /v1/responses use
an 'input' field instead of 'messages' or 'prompt', so the pre-call hook
returned without scanning the content. Similarly, post-call hook did not
handle ResponsesAPIResponse.output. Fix: add _responses_api_input_to_str()
helper and handle 'responses'/'aresponses' call types in async_pre_call_hook,
async_post_call_success_hook (via _completion_response_text_parts), and
async_logging_hook.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): message separator, non-blocking logging_hook, TextChoices type error
Three bugs fixed in the Microsoft Purview DLP guardrail:
1. get_prompt_text_for_dlp message separator (base.py)
- Previously called get_str_from_messages() which concatenated all message
texts with NO separator, so 'end of msg1' + 'start of msg2' became
'end of msg1start of msg2'.
- Now joins per-message text with '\n\n' via convert_content_list_to_str(),
preserving DLP pattern detection accuracy across message boundaries.
2. logging_hook blocking the event loop thread (purview_dlp.py)
- Previously called future.result() which blocked the calling thread
(often the event loop thread) for the entire round-trip of two sequential
Microsoft Graph API calls (_compute_protection_scopes + _process_content).
- Now fires and forgets: when called inside a running loop, schedules the
coroutine with loop.create_task(); otherwise spawns a daemon thread.
Returns (kwargs, result) immediately in both cases.
- Removes unused concurrent.futures.ThreadPoolExecutor import; adds threading.
3. Incompatible assignment type error (purview_dlp.py:180)
- mypy inferred 'choice' as TextChoices from the first loop body, then
flagged the assignment in the second loop as incompatible with Choices.
- Fixed by using distinct loop variable names: text_choice (TextChoices) and
chat_choice (Choices).
Tests: 7 new tests added covering the separator fix (TestGetPromptTextForDlp)
and the non-blocking logging_hook (TestLoggingHookNonBlocking).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): suppress API errors in logging-only mode and scan tool-call arguments
Three issues fixed:
1. _check_content except block re-raised unconditionally even when
block_on_violation=False. The docstring promised 'log only - do not
raise' but network/API errors always propagated. Fixed by checking
block_on_violation before re-raising; when False, log a warning and
continue.
2. async_logging_hook used a single try/except wrapping both the prompt
and response audit calls. When the first _check_content (uploadText)
raised due to an API error the second call (downloadText) was silently
skipped. Fixed by giving each audit call its own try/except so both
always run independently.
3. convert_content_list_to_str() only reads message.content, so
tool_calls[].function.arguments and function_call.arguments were
invisible to the Purview pre-call and post-call scans. An authenticated
caller could embed sensitive text in tool-call arguments and bypass DLP.
Fixed by:
- Adding PurviewGuardrailBase._extract_tool_call_args_from_message()
which handles both dict and object-style messages, covering both
tool_calls[] arrays and the legacy function_call field.
- Updating get_prompt_text_for_dlp() to include those arguments
alongside message content (request/prompt path).
- Changing _completion_response_text_parts() from @staticmethod to an
instance method and adding tool-call argument extraction for
ModelResponse choices (response path).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* chore(ui): restructure pre-built Next.js output to directory-based routing
Flat page files (e.g. guardrails.html) replaced by directory-based
index.html equivalents (e.g. guardrails/index.html) matching the
Next.js App Router output format.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): comprehensive security hardening — identity spoofing, streaming bypass, token-id gap
Four security issues addressed:
1. end_user_id kwargs fallback missing in _resolve_user_id_from_logging_kwargs
user_id already fell back to kwargs.get("user_api_key_user_id") when absent
from metadata, but end_user_id only checked md.get("user_api_key_end_user_id")
with no kwargs-level fallback. Added or kwargs.get("user_api_key_end_user_id").
2. Streaming responses bypassed post_call blocking
async_post_call_success_hook only runs on assembled non-streaming responses.
For streaming requests the proxy already delivered all content before the
hook ran, so raising HTTPException there had no effect. Added
async_post_call_streaming_iterator_hook which buffers the entire stream,
assembles it via stream_chunk_builder, runs the Purview DLP check, and only
then re-yields chunks via MockResponseIterator. If a violation is detected the
exception is raised before any bytes reach the client. The proxy automatically
skips async_post_call_success_hook for guardrails that define this method,
preventing duplicate scans.
3. Caller-controlled Purview user identity in blocking modes
When a LiteLLM API key has no bound user_id the guardrail fell back to
metadata[user_id_field], which is supplied by the caller. A caller could set
this to any Entra object ID whose Purview policies are more permissive and
bypass DLP. Added _resolve_trusted_user_id() that only returns identities
from the proxy auth system (user_api_key_dict.user_id, end_user_id, or
proxy-injected metadata["user_api_key_user_id"]). Added
_resolve_user_id_for_blocking() used by all blocking-mode hooks: tries
trusted sources first; if only caller-supplied is available, logs a
SECURITY WARNING and still proceeds (backward compat); if nothing resolves,
skips with a warning.
4. Token-id prompt DLP bypass
When /v1/completions received a pure token-id array prompt,
completion_prompt_to_str() returned None and the pre_call hook silently
skipped the Purview scan. An authenticated caller could tokenize blocked
text and send it without DLP evaluation. The hook now detects this case
(raw_prompt present but prompt_text None) and logs a WARNING while letting
the request pass through — token-id payloads are opaque at the text layer
and cannot be scanned. This makes the gap explicit rather than silent.
Tests: 94 total, all passing.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* Revert "chore(ui): restructure pre-built Next.js output to directory-based routing"
This reverts commit
|
||
|
yuneng-jiang
|
574ee7526d
|
test(streaming): tolerate Vertex 429 wrapped in MidStreamFallbackError (#28669)
Streaming 429s are wrapped in MidStreamFallbackError so the Router can fall back; the existing 'except litellm.RateLimitError: pass' in test_vertex_ai_stream no longer matches, causing the generic pytest.fail branch to fire when upstream Vertex returns 429. Add a sibling except for MidStreamFallbackError that only swallows it when e.original_exception is a RateLimitError, so unrelated streaming failures still fail the test. |
||
|
Cursor Agent
|
2c41400bdf
|
fix(gemini realtime): skip unknown sibling keys in transform loop
Gemini realtime messages can include sibling metadata keys like usageMetadata alongside primary payload keys (toolCall, serverContent). Previously, the transform loop called map_openai_event for every top-level key, raising ValueError for unknown ones and terminating the WebSocket session. Skip top-level keys not present in MAP_GEMINI_FIELD_TO_OPENAI_EVENT to keep the session alive when Gemini emits usage metadata with a toolCall response. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
42169c8578
|
test(gemini realtime): wrap test_gemini_tool_call_resets_ids fixture in setup envelope
The cached session_configuration_request the proxy stores is always
serialized as {"setup": ...}; this test passed a bare config dict, so
transform_session_created_event's .get('setup', {}) returned an empty
dict and the responseModalities lookup ran against the default rather
than the fixture. Wrap the fixture in the same shape the production
cache uses.
|
||
|
mateo-berri
|
d1a9da3514
|
fix(gemini realtime): cast merged realtimeInputConfig for typeddict assignment
mypy flagged the assignment of the merged dict into BidiGenerateContentSetup.realtimeInputConfig with [typeddict-item]: the intermediate variable widens to dict[Any, Any], losing the TypedDict narrowing the previous dict-literal form had. |
||
|
mateo-berri
|
bafc187248
|
test(gemini realtime): wrap remaining cached session configs in setup envelope
The session_configuration_request the proxy caches is always serialized
as {"setup": ...}; three modality-related tests dumped a bare config
dict instead, so transform_session_created_event's
`.get('setup', {})` quietly returned an empty dict and the
responseModalities lookup ran against the default rather than the
fixture. Wrap the remaining tests in the same shape the production
cache uses so any regression in modality forwarding actually trips.
|
||
|
mateo-berri
|
e5ffd021f7
|
fix(gemini realtime): bound _tool_call_id_to_name with an LRU; exercise modality forwarding test
Two minor follow-ups from review:
* Switch _tool_call_id_to_name to a 256-entry LRU OrderedDict so a long
session with many tool calls doesn't grow the dict without bound,
while retried function_call_output lookups still hit for recently-seen
call_ids.
* Fix test_gemini_realtime_transformation_session_created to wrap the
cached session config in {"setup": ...} so the modality lookup in
transform_session_created_event actually exercises responseModalities
forwarding (the prior payload was silently treated as empty).
|
||
|
mateo-berri
|
20764dd342
|
fix(realtime): record synthetic session.created in deferred-setup mode
The deferred-setup path emits a synthetic session.created directly to the client websocket but did not run it through RealTimeStreaming's store_message, so the event was missing from the session log used by success_handler / async_success_handler. Call store_message before forwarding so the synthetic event lands in the same log stream as provider-driven events. |
||
|
mateo-berri
|
459c1973b4
|
fix(gemini realtime): deep-merge automaticActivityDetection on follow-up session.update
The follow-up setup merge already deep-merged generationConfig and realtimeInputConfig, but realtimeInputConfig.automaticActivityDetection itself is a nested dict. A partial VAD update (e.g. the guardrail-injected disabled=True from create_response=False) silently dropped unrelated knobs such as silenceDurationMs and prefixPaddingMs from the original setup. Deep-merge that block too so partial overrides only touch the fields they specify. |
||
|
mateo-berri
|
d56d875ea6
|
fix(vertex realtime): warn when dropping guardrail turn-detection update
In non-deferred mode the auto-setup is sent on connect, so the audio-transcription guardrail's subsequent session.update carrying turn_detection.create_response=False cannot be forwarded as a second setup (Vertex Live closes the WebSocket with 1007). Surface a warning when this specific drop happens so operators know the model will auto-respond before the guardrail can gate it, instead of failing silently at debug level. |
||
|
mateo-berri
|
0780e5f69f
|
fix(gemini realtime): empty toolCall must not terminate the WebSocket
If Gemini sends a toolCall whose functionCalls list is empty (or absent), the previous `continue` left returned_message empty and the "Unknown message type" guard fired, killing the WebSocket session. Return a normal (empty) result instead so the session keeps going. |
||
|
mateo-berri
|
c59260fc70
|
fix(gemini realtime): keep call_id→name mapping across function_call_output retries
A client SDK that retries function_call_output (or sends the same result twice) would previously hit a missing-name lookup on the second send because _handle_function_call_output popped the call_id → name entry. Without name, Gemini may silently reject the response. Use dict.get so the mapping persists for the lifetime of the session. |
||
|
mateo-berri
|
b60cc950f3
|
fix(gemini realtime): mirror modalities/temperature/max_output_tokens on tool-call response.created
The audio/text response.created preamble includes modalities, temperature, and max_output_tokens on the response object so spec-compliant clients can initialise per-response state. The tool-call response.created was missing these fields, leaving clients without consistent response metadata when a response starts with a tool call instead of content. Read them from the cached session_configuration_request the same way the audio/text path does. |
||
|
milan-berri
|
1b141bc588
|
fix(bedrock): decouple STS region from Bedrock aws_region_name (#28245)
* fix(bedrock): decouple STS region from Bedrock aws_region_name STS AssumeRole now resolves signing region from aws_sts_endpoint (parsed host) or AWS_REGION/AWS_DEFAULT_REGION instead of aws_region_name, fixing air-gapped cross-region Bedrock setups and endpoint/signature mismatches. Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): add regression coverage for _build_sts_client_kwargs Parametrize _resolve_sts_region and _build_sts_client_kwargs matrix cases, and assert IRSA/web-identity paths use aligned STS endpoint and region_name. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): tighten STS region helpers and drop redundant web-identity endpoint synthesis Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): cover FIPS, GovCloud, and China STS endpoints Addresses greptile P2: regex sts(?:-fips)? supported sts-fips hosts but was not exercised by the parametrized parse test. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
mateo-berri
|
615a7da9ba
|
fix(realtime): deep-merge generationConfig and refresh cache on follow-up setup
A subsequent Gemini session.update that touches any generationConfig sub-field (e.g. just temperature) was clobbering the original generationConfig — silently dropping responseModalities and switching the session to text-only. Deep-merge generationConfig so existing keys (responseModalities, maxOutputTokens, ...) are preserved when the client updates only a subset. Also drop the early-return in _cache_session_configuration_request so the cached payload tracks the latest setup sent to the backend. Without this, downstream readers (transform_session_created_event, modality lookup in return_new_content_delta_events) keep reading stale modalities/system instruction after a follow-up setup. |
||
|
mateo-berri
|
3efd803cd1
|
fix(gemini realtime): default conversation_id before tool-call response.done
mypy flagged that response.done's conversation_id (str on the TypedDict) could be None when current_response_id was already set on entry. Ensure the fallback runs unconditionally before the response is constructed. |
||
|
Cursor Agent
|
a0043494a0
|
fix(gemini realtime): deep-merge nested config in follow-up session update
Previously, the follow-up setup performed a shallow merge between the original setup and new overrides. If a session.update touched any field inside generationConfig (e.g. modalities), the entire generationConfig would be replaced, silently dropping unrelated sub-keys like temperature or maxOutputTokens. Apply the same deep-merge to realtimeInputConfig so partial automatic-activity-detection updates don't drop other realtime input config fields either. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor Agent
|
76a82447e8
|
fix(gemini realtime): include usage on tool-call response.done; coerce non-dict tool output to struct
- Tool-call response.done now includes an empty usage object, matching the non-tool-call path so OpenAI-compatible clients always see usage. - _handle_function_call_output wraps non-dict JSON parses under a 'result' key so Gemini's functionResponses[].response (a Struct) always receives a mapping. Co-authored-by: Yassin Kortam <yassin@berri.ai> |