mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
443 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tin-berri
|
a7d01cb1ac
|
Merge pull request #33573 from BerriAI/litellm_lit4478_anthropic_auto_cache
feat(anthropic): add enable_anthropic_prompt_caching for automatic cache_control injection |
||
|
Tin Chi Lo
|
53c285a94a |
fix(anthropic): stand down when the client caches its tool definitions
_request_has_cache_control only looked at messages and system, so a client that marks cache_control on tools alone did not suppress auto-injection. Tool breakpoints count toward the provider's four-block limit, so three of them plus the two injected here is five, which Anthropic rejects. Thread tools through both entry points and treat a client-marked tool as the stand-down signal it already is for messages and system. |
||
|
Yassin Kortam
|
2162da5015
|
fix(langfuse_otel): build per-request OTLP exporter from key and team dynamic Langfuse credentials (#32437)
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials Key-scoped langfuse_otel callbacks only injected Authorization headers into the init-time exporter, so a proxy without global LANGFUSE_* env vars kept its fallback exporter and never exported traces to Langfuse. Dynamic params now build a full per-request OTLP config (endpoint from the key's langfuse_host, otlp_http, basic auth from the key's credentials). Resolves LIT-3976 * fix(otel): log dynamic config endpoint in span processor debug output * fix(otel): redact authorization headers in exporter debug logs |
||
|
Tin Chi Lo
|
f7a3e22b22 |
feat(anthropic): allow enabling prompt caching via environment variables
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on without a config file. An unsupported ttl falls back to the provider default rather than reaching the provider verbatim |
||
|
Tin Chi Lo
|
04afc962b1 |
feat(anthropic): add enable_anthropic_prompt_caching for automatic cache_control injection
Anthropic only caches a prompt when the request carries explicit cache_control breakpoints, unlike OpenAI where prompt caching is automatic and needs no configuration. Today litellm can inject those breakpoints server-side, but only when an admin hand-writes cache_control_injection_points into a model's litellm_params (or router_settings.default_litellm_params). Clients such as Claude Code and Claude Desktop never set cache_control themselves, and the admin recipe is easy to miss, so Anthropic traffic through the proxy silently pays full price on every repeated prefix. This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When it is on and the request has no injection points configured and no client-supplied cache_control, litellm synthesizes a default pair of breakpoints (the system prompt and the trailing turn) so the stable prefix is cached while the breakpoint advances with the conversation. It is wired into both surfaces: /chat/completions seeds the points before the existing prompt-management gate, and /v1/messages resolves them in maybe_inject_cache_control, so the existing AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and its refusal to overwrite client breakpoints. The default is off, so no existing deployment changes behavior. Injection is gated to providers that actually consume cache_control markers (anthropic and bedrock) and to models the cost map flags as supporting prompt caching; note that supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and Gemini models report it as well but never take cache_control markers. The default ttl is Anthropic's 5 minute ephemeral cache, with an optional anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to ChatCompletionCachedContent, which the bedrock and anthropic transforms already read at runtime but the type never declared Resolves LIT-4478 |
||
|
Devin AI
|
75ccb4f416 | Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_websearch_responses_interception | ||
|
Krrish Dholakia
|
7f598c6a9b |
fix(websearch): address Responses review findings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
32af83d63a
|
fix(s3): sanitize slashes in response-id-derived object key file name (#33271) | ||
|
yucheng-berri
|
939117bb8d
|
fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook (#33136)
* fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook * fix(guardrails): keep request-body dispatch predicate unchanged * fix(guardrails): fail closed when proxy extras are missing at deployment hook |
||
|
yucheng-berri
|
07ea4b3e14
|
feat(prometheus): expose video duration and image count consumption metrics (#33138) | ||
|
Krrish Dholakia
|
aa7b480f4c |
fix(websearch): add Responses API surface to websearch interception
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
011e8e7f52
|
fix(prometheus): read v3 rate limiter remaining values for per-key model gauges (#33119) | ||
|
yucheng-berri
|
60d557c930
|
fix(datadog): split log batches proactively under intake payload limits (#32860)
* fix(datadog): split log batches proactively under intake payload limits * fix(datadog): size intake chunks with exact wire serialization |
||
|
Mateo Wang
|
1ab1515d9e
|
fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834)
adds a top-level guard in _increment_remaining_budget_metrics that returns early when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config), and per-entity guards in each _set_*_budget_metrics_after_api_request helper for partial disabling. eliminates four async DB/cache round-trips per successful LLM request when budget metrics are disabled. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
Yassin Kortam
|
99b4c5ed3e
|
feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655)
* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. |
||
|
yucheng-berri
|
74623b12b1
|
fix(guardrails): mask credentials embedded in guardrail_response before persist (LIT-4314) (#32687)
Team-level callback_vars (e.g. langsmith_api_key) get spread into data["metadata"] as four aliases (user_api_key_metadata, user_api_key_team_metadata, user_api_key_auth_metadata, user_api_key_auth). When a guardrail hook echoes that metadata into its guardrail_response, the plaintext credential landed five times inside LiteLLM_SpendLogs.metadata.standard_logging_guardrail_information[i].guardrail_response and every downstream sink that reads it (OTel via emit_guardrail_span, Langfuse, custom loggers). Add a purpose-built payload walker (mask_credentials_in_payload) that only masks strings under sensitive-named keys and preserves every other value (None, ints, floats, bools, tuples, typed objects) verbatim. The walker reuses SensitiveDataMasker.is_sensitive_key so the pattern list stays in one place, and unwraps Pydantic models via model_dump() so nested UserAPIKeyAuth values reached by the walk get scanned as plain dicts (they are JSON-serialized downstream anyway). Apply the walker at add_standard_logging_guardrail_information_to_request_data after the existing secret_fields pop and match/regex redaction, so every downstream sink sees masked values from a single seam. |
||
|
Yassin Kortam
|
1d87084212
|
refactor(otel): move litellm error detail keys under the litellm.* namespace (#32591)
The v2 OTel integration stamped litellm-specific error details as error.code, error.stack_trace, and error.llm_provider, squatting on the semconv-owned error.* namespace. They now live at litellm.provider.error.code, litellm.provider.error.stack_trace, and litellm.provider.error.llm_provider alongside the other vendor-extension keys. error.type and error.message stay on the semconv keys. |
||
|
yucheng-berri
|
85d1fe6e2a
|
fix(otel): restore error.* span attributes on v2 error spans (LIT-4179) (#32524)
The v2 emitter has never stamped error.message / error.code / error.stack_trace / error.llm_provider as span attributes; only error.type reached the wire. Backends that flatten span attributes into label indexes (Elastic APM labels.error_*, Datadog span tags) lost these four fields when v2 became the active integration on v1.90+ for otel_v2-flagged deployments. The pre-existing exception span event carrying the full message (LIT-3758) is unchanged; the message now rides both places at once, matching v1s shape. SpanError grows three optional detail fields; _parse_error threads them from StandardLoggingPayloadErrorInformation; the emitters error branch stamps them via a new module-level helper, guarded per field so guardrail-shape errors are not polluted with empty attributes. New semconv constants mirror open_inference.ErrorAttributes byte-for-byte, so v1 and v2 consumers read the same keys. Regression tests extend the mapped test files under tests/test_litellm/integrations/otel/. pytest reports 243 passed. |
||
|
Yassin Kortam
|
6f6bd45681
|
perf(auth): negative-cache missing user/key lookups on the request hot path (#32368) | ||
|
Yassin Kortam
|
3116ed211b
|
feat(otel): stamp gen_ai.response.time_to_first_chunk on streaming LLM spans (#32236) | ||
|
Krrish Dholakia
|
2967bc9bef
|
fix: merge websearch tool params (#32162)
* fix: pass websearch tool params * fix: load db websearch tool params * fix: merge search tools in proxy * fix: satisfy websearch lint budget * fix: enforce websearch tool auth * fix: preserve search tools on empty sync * chore: rerun circleci |
||
|
yucheng-berri
|
7a6a070370
|
feat(prometheus): add api_provider label to token, latency, request and cache metrics (#32126)
* feat(prometheus): add api_provider label to token, latency, request and cache metrics The token (input/output/total), latency (llm_api, time_to_first_token, request_total, request_queue_time), proxy request (total/failed) and cache metrics were emitted from the same call sites as litellm_spend_metric and litellm_requests_metric, which already carry api_provider, yet these were missing it. That left no way to break tokens, latency, request counts or cache hits down by upstream provider even though the provider is already on the payload as custom_llm_provider. Add api_provider to each metric's label allow-list. The success path already populates enum_values.api_provider from standard_logging_payload, so those metrics emit it with no further plumbing. The cache label is added to the shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the provider prompt-cache read/creation token metrics; the label-presence test asserts all of them. For the client-side failure path, where a deployment may not have been resolved, derive it best-effort from litellm_params.custom_llm_provider, a partial standard_logging_object, or inference from the requested model name via litellm.get_llm_provider, falling back to empty rather than guessing. Resolves LIT-4178 * fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions - Suppress the strict-rule BLE001 budget breach with a justified noqa; the broad except in the failure-path provider extraction is intentional defense-in-depth (covered by test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors), not dead code to delete - Update tests/enterprise assertions for litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric, the three latency metrics, and the proxy request counters to expect the new api_provider label, matching what litellm_mapped_enterprise_tests caught in CI --------- Co-authored-by: Shivi Jain <mobile.350017@gmail.com> |
||
|
yucheng-berri
|
93cdcca1c5
|
fix(azure_sentinel): resolve audit stream from AZURE_SENTINEL_AUDIT_STREAM_NAME (#32010)
When AzureSentinelLogger is resolved from the string callback name "azure_sentinel", it is constructed with no arguments, so audit_stream_name is always None and resolved_audit_stream_name fell back to the standard resolved_stream_name. Audit logs then ingested into the access-log DCR stream whose schema is built from StandardLoggingPayload, so Azure Monitor Logs Ingestion silently dropped the audit-specific columns and audit rows arrived effectively empty. Add an AZURE_SENTINEL_AUDIT_STREAM_NAME env var fallback in __init__, mirroring the AZURE_SENTINEL_STREAM_NAME idiom already used for the standard stream, so audit logs can target a separate DCR stream without a custom callbacks file. |
||
|
Sameer Kankute
|
321345d4c8
|
feat: litellm oss staging (#31935)
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632) * fix(prometheus): bound per-request budget metric emission with a timeout Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising * fix(prometheus): reject non-finite and non-positive budget-metrics timeout env float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default * fix: report the blocked LLM response's real token usage (#31217) When a guardrail blocks a post-call response, the synthetic violation response reported hard-coded zero usage, discarding the token usage the upstream call had already consumed. Fix the root cause rather than re-counting tokens: - Add an optional `original_response` field to ModifyResponseException. - The unified guardrail's post-call success hook attaches the blocked LLM response to the exception. - The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions) block handlers report `original_response.usage` directly. Pre-call blocks never invoked the LLM, so usage is zero. Mock-based tests cover the helper (returns original usage / zero), the success hook attaching original_response, and the endpoint reporting it end-to-end. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389) Streaming moderation improvements for the unified guardrail post-call streaming iterator hook: - streaming_buffer_until_moderated: withhold all chunks until end-of-stream moderation passes, then release the original response (clean) or only the block message (blocked) -- the original content is never delivered on a block. Snapshot chunks with a shallow list() copy (end-of-stream builds a separate assembled response; chunks aren't mutated in place). - Clean Anthropic SSE on block: synthesize a well-formed termination sequence instead of a bare data: {"error": ...} blob that truncates the stream. Provider-specific synthesis lives in AnthropicMessagesHandler via build_block_sse_chunks (format-agnostic routing stays in the hook). - Mid-stream blocks continue the in-progress message (close open content block, append block message, terminate) rather than emitting a second message_start, which clients reject. Standalone envelope only when no chunks were sent (buffered path). - ModifyResponseException imported under TYPE_CHECKING + locally at runtime to avoid a module-level cyclic import. Adds regression tests for buffering (content withheld on block) and mid-stream continuation (single message_start). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails - _standalone_block_chunks and _block_continuation_chunks now read real token usage from ModifyResponseException.original_response instead of hardcoding zero, matching the non-streaming _blocked_response_usage path. Shared helper moved to guardrail_translation/utils.py. - streaming_buffer_until_moderated is now forced off when the guardrail has mask_response_content=True, since buffered replay releases the withheld original chunks verbatim -- unsafe for a guardrail that rewrites content (e.g. PII masking). - Fix inverted streaming-flag precedence comment. * style: ruff format after greploop fixes * fix: handle Anthropic streaming guardrail blocks * fix(responses): check terminal event type for streaming guardrail end-of-stream detection _check_streaming_has_ended assumed responses_so_far held ModelResponse objects with .choices, but for the Responses API the accumulated chunks are raw SSE event dicts, causing an AttributeError on every call * fix: preserve Anthropic blocked stream usage --------- Co-authored-by: FERNANDO IZAR <fizar@me.com> Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
mubashir1osmani
|
9b6d0b0e98
|
Merge pull request #31928 from BerriAI/litellm_s3_v2_content_md5 | ||
|
devin-ai-integration[bot]
|
85db18e618
|
feat(prometheus): expose MCP tool metadata in Prometheus metrics (#31899)
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor Agent
|
29cc4e35d6
|
fix: allow S3 Content-MD5 on FIPS hosts | ||
|
Mubashir Osmani
|
e542be17ad |
fix(s3_v2): pass usedforsecurity=False to hashlib.md5 for FIPS envs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mubashir Osmani
|
68a8fc2207 |
feat(s3_v2): send Content-MD5 on PUT and optional server-side encryption
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c4a77bded7
|
fix(prometheus): expose project_alias in custom metadata labels (LIT-3741) (#31784)
Include top-level scalar fields from standard_logging_metadata in the combined metadata dict used by custom_prometheus_metadata_labels. Previously only nested sub-dicts (requester_metadata, user_api_key_auth_metadata, spend_logs_metadata) were spread into combined_metadata, so fields like user_api_key_project_alias were inaccessible and always resolved to None. Co-authored-by: unknown <> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
23af78465c
|
feat: add cache control injection support for v1/messages endpoint (#31778)
* feat: add cache control injection support for v1/messages endpoint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: normalize string content to list for Anthropic-native cache_control injection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor: simplify cache control injection, fix system=[] bug, fix handler system type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor: extract cache control logic into static helper on AnthropicCacheControlHook Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Krrish Dholakia
|
6c21029cb7
|
feat(sandbox): reuse e2b container across requests when metadata.session_id is set (#31688)
* feat(sandbox): reuse e2b container across requests when metadata.session_id is set
When a client passes `metadata.session_id` in a /chat/completions request
alongside a code_interpreter tool, the proxy now routes all requests sharing
that session_id to the same sandbox container. State (variables, imports,
installed packages) persists across requests within the session.
Without a session_id the existing ephemeral behavior is unchanged: one
container per agentic loop, deleted immediately after.
The sandbox key is derived from session_id rather than a per-request UUID.
The cleanup and post-loop hooks skip deletion for session-scoped containers.
TTL-based pruning (15 min idle) still applies and refreshes on every use,
so an active session never expires mid-use. The session_id-scoped key is
registered in all_litellm_params and the proxy strip-list so it never
leaks to the upstream LLM provider.
* fix(sandbox): scope session sandbox key to API key identity; add per-identity LRU cap
Two security issues addressed:
1. Cross-user sandbox isolation: the session_id supplied by the client is now
combined with the server-minted user_api_key_hash to form the cache key
(format: "{hash}:{session_id}" when authenticated, bare session_id for
non-proxy use). Two tenants sharing the same session_id no longer share a
sandbox.
2. Bounded session allocation: each API key identity is capped at
_SESSION_SCOPED_PER_IDENTITY_CAP (10) live session-scoped containers. When
a new session is opened beyond the cap, the least-recently-used entry for
that identity is evicted and its sandbox deleted, preventing unbounded
accumulation via rotating session IDs.
The container cache tuple gains a fourth element (identity: str | None) so
eviction can filter by identity without parsing key formats. Tests added for
both properties.
|
||
|
Krrish Dholakia
|
ada9ef88ac
|
fix(websearch): websearch_interception agentic loop fixes for chat completions and anthropic messages (#31669)
* fix(websearch): wire chat completion agentic loop to correct hooks maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller. Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path. Regression test added. * fix(websearch): strip tool_choice from follow-up request When the original request forces tool_choice to litellm_web_search, the follow-up request after search execution inherited that tool_choice, causing the model to call the search tool again instead of synthesizing an answer from the results. * fix(websearch): inject api_key into agentic hook kwargs for anthropic messages Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's synthesis call after executing Exa/Perplexity searches) were missing api_key because the named api_key param in async_anthropic_messages_handler was never merged into the kwargs dict forwarded downstream. Result: every /v1/messages websearch follow-up failed with "x-api-key header is required" and the caller received the raw tool_use response instead of the synthesized answer. * ci: trigger CI run * fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with _agentic_loop_api_surface to handle both surfaces from one hook. The chat completion loop must also check _gate_overridden so callbacks using the unified hook pattern still fire for chat completions. * fix(websearch): strip tool_choice from legacy chat completion follow-up call The _execute_chat_completion_agentic_loop path merged original optional_params (which includes forced tool_choice) into follow-up params without explicit removal. _build_chat_completion_request_patch already excluded tool_choice from its optional_params output, but dict.update() with a missing key leaves the original value intact. Explicit pop after the merge removes it. * fix(websearch): always strip tool_choice from plan-path follow-up params The tool_choice removal was gated on patch.tools is not None. WebSearch sets tools via patch.optional_params not patch.tools, so the gate was False and forced tool_choice from the original request survived into the synthesis call. Move the pop outside the patch.tools branch so it applies unconditionally. |
||
|
ryan-crabbe-berri
|
468d11f71d
|
feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 (#31525)
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
|
||
|
Yassin Kortam
|
2e575d39f2
|
perf(otel): memoize per-request lazy import of otel runtime hooks (#31707)
The proxy auth path calls phase_span() and seed_request_identity() in litellm/integrations/otel/runtime.py on every request, each doing a try/except lazy import of litellm.integrations.otel.logger. When the OpenTelemetry SDK is not installed (the default), that import raises, and CPython never caches a failed import, so every request re-scanned sys.path and contended on the import lock. At 750 concurrent users this cost about 12% throughput versus v1.85.0. Resolve the hooks once and cache the outcome, absence included, with functools.cache, so the import is attempted a single time instead of per request. Throughput returns to the v1.85.0 baseline. |
||
|
Yassin Kortam
|
70eb4e5d00
|
feat(prometheus): add litellm_total_overhead_latency_metric (SDK overhead + guardrails) (#31593)
litellm_overhead_latency_metric only covers the SDK wrapper window and excludes proxy guardrails. Add a histogram that sums SDK overhead plus pre/post-call guardrail durations (during-call excluded since it runs concurrently with the LLM call, alongside logging_only and MCP modes that never block the response), recorded next to the existing overhead metric with the same labels and buckets. No existing metric's value is changed. |
||
|
ryan-crabbe-berri
|
e195532c14
|
fix(proxy): count only active users toward license seat limit (#31227)
* fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120 |
||
|
Yassin Kortam
|
b2e708d5ae
|
feat(prometheus): add per-team litellm_team_members_metric gauge (#31506)
Emit litellm_team_members_metric on every team member add and delete, labelled by team and team_alias and set to the team's authoritative member count. Because it is set from the current membership rather than incremented or decremented, it tracks the count up and down, never goes negative, and self-corrects on the next change after a proxy restart. Bulk member add is covered for free since it delegates to team_member_add, and the helper no-ops when the Prometheus callback is not registered. Resolves LIT-3082 |
||
|
yucheng-berri
|
0216c969b8
|
fix(otel): point AgentOps OTLP exporter at otlp.agentops.ai (#31490)
The AgentOps preset hardcoded https://otlp.agentops.cloud/v1/traces, a domain that no longer resolves (NXDOMAIN), so every span silently failed to export with a NameResolutionError in the BatchSpanProcessor worker. The live ingest host is otlp.agentops.ai (the auth host api.agentops.ai was already correct). Pin the endpoint to the resolvable host and add a regression test on the constant. |
||
|
Shivam Rawat
|
de82f78e5b
|
fix(websearch): sync tool_choice when converting web_search tools (#31375)
failing test is not related to the pr * fix(websearch): sync tool_choice when converting web_search tools Claude Code forces native web search via tool_choice pointing at web_search while websearch_interception renames the tool to litellm_web_search, causing Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced tool_choice to match the converted tool name. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(websearch): re-wrap agentic loop responses as SSE for streaming clients When websearch interception converts stream=true to false for the agentic loop, dict responses from the loop were returned as application/json even though the client requested SSE. Wrap those responses in FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers (e.g. Claude Code) receive text/event-stream after search completes. Fixes #27721 Co-authored-by: Cursor <cursoragent@cursor.com> * test(websearch): cover tool_choice sync and post-loop SSE wrap; fix UP006 Add regression tests for both websearch interception fixes: _sync_forced_tool_choice repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and _maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for streaming clients (#27721). Switch the new helper annotations to builtin dict/list so the ruff UP006 strict-rule ceiling stays within budget. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(websearch): resolve merge conflict and unify fake stream wrapping Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MBP.localdomain> |
||
|
Krrish Dholakia
|
99b1a323c1
|
feat(guardrails): add headroom guardrail for message compression (#31407)
* feat(guardrails): add headroom guardrail for message compression Adds a headroom guardrail that compresses request messages via POST /v1/compress before they reach the LLM. The guardrail implements apply_guardrail so it runs on the unified guardrail path; it receives pre-built structured_messages (OpenAI format) from the translation layer, calls the headroom compression service, and returns the compressed messages as structured_messages. Set x-headroom-bypass: true on the request to skip compression. Also adds structured_messages write-back support to the OpenAI and Anthropic translation handlers: when apply_guardrail returns structured_messages, those are written to data["messages"] directly (OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic) instead of falling through to the existing text-patch path. This is a prerequisite for any guardrail that needs to replace the full message list rather than patch individual text spans. * fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs * style: fix ruff format violations * fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037) * fix(guardrails): only write back structured_messages when guardrail actually changed them * fix(guardrails/headroom): raise 502 when compression returns empty message list * fix(guardrails/headroom): catch transport errors and fix stale debug log * fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation * fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back * debug(headroom): add INFO logging to trace guardrail execution * debug(headroom): use print() for immediate visibility * debug(headroom): print request_data keys to diagnose metadata dict mismatch * fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log * fix: use model_call_details litellm_params metadata on Logging object * fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy * fix: read slg_info from litellm_metadata when metadata key absent * fix: write slg_info to both litellm_params attr and model_call_details copy * chore: remove debug prints; fix now verified end-to-end * refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py - Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from both async and sync wrappers in @log_guardrail_information, fixing guardrail_information=null in spend logs for all passthrough routes (/v1/messages, /v1/responses, etc.) in one place - Remove the 35-line inline sync block from the anthropic translation handler - Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses - Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase - Add regression tests for _sync_guardrail_info_to_logging_obj * fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold * fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity * fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity * fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity |
||
|
Mateo Wang
|
b9765458ac
|
fix(websearch): wrap agentic loop response in fake stream for streaming requests (#31484)
* fix(websearch): wrap agentic loop response in fake stream for streaming requests When websearch_interception converts stream=True to stream=False internally, the agentic loop returns a plain dict. Previously this dict was returned directly to the client expecting SSE events, resulting in empty streams. Added _maybe_wrap_in_fake_stream() which checks the websearch_interception_converted_stream flag and wraps dict responses in FakeAnthropicMessagesStreamIterator. Applied to all return paths in _call_agentic_completion_hooks: - async_run_agentic_loop (legacy path) - _execute_anthropic_agentic_plan (plan-based path) - plan.response_override - plan.terminate Includes unit tests for _maybe_wrap_in_fake_stream(). * test(websearch): cover agentic-loop wrap paths; gate fake-stream on anthropic_messages surface Guard _maybe_wrap_in_fake_stream on api_surface == anthropic_messages so the responses API surface is never wrapped in an Anthropic SSE iterator, and type logging_obj as Optional to match the None call sites. Adds regression tests that drive the legacy, response_override, and terminate return paths of _call_agentic_completion_hooks end to end. * test(websearch): cover _execute_anthropic_agentic_plan and tail wrap paths Drives the remaining two fake-stream return paths of _call_agentic_completion_hooks (the _execute_anthropic_agentic_plan branch via a stubbed handler, and the tail path when no agentic loop runs) so every converted-stream return path is regression-tested. --------- Co-authored-by: Clawd <fffff.c@gmail.com> |
||
|
yucheng-berri
|
ec4e0146c7
|
feat(prometheus): add requested_model label to spend and requests metrics (#31410)
litellm_spend_metric_total and litellm_requests_metric_total previously exposed only the resolved backend model_id and friendly model name, so operators could not group spend or request counts by the model alias the caller actually asked for when a router fronts multiple deployments behind one name. This adds the existing UserAPIKeyLabelNames.REQUESTED_MODEL to both labelname lists; the value is already populated upstream from standard_logging_payload["model_group"] and flows through the shared _increment_top_level_request_and_spend_metrics call site. The sibling token metrics (input/output/total) already carry the label, so this also restores cross-metric consistency. Resolves LIT-3796 |
||
|
Sameer Kankute
|
133da06aa3
|
chore: litellm oss staging (#31185)
* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano
Three bugs in model_prices_and_context_window.json:
1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
max output, but the values were set as max_input=128000,
max_tokens=272000. This caused token limit errors when sending
prompts over 128K tokens to GPT-5 Pro.
2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
272000, but GPT-5.4 Mini shares the same 1,050,000 token context
window as GPT-5.4. This was inconsistent with the azure/ variants
which already correctly had 1,050,000.
3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
max_input_tokens was 272000 instead of 1,050,000.
Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.
Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)
* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)
Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.
* fix(cost): price gpt-image generated output tokens as image tokens (#31147)
The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.
The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.
Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).
* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)
A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.
Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.
* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)
_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.
Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.
Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com>
* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)
gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.
OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.
Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com>
* fix(deepseek): drop non-function tools before chat completions call (#30910)
* fix(deepseek): drop non-function tools before chat completions call
DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).
Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls
Fixes #30722
* test(deepseek): cover async tool filtering and document tool_choice assumption
Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool
* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)
* feat(ui): surface team budget on key overview when key has no own budget (#30801)
* feat(ui): surface team budget on key overview when key has no own budget
* fix(ui): replace IIFE with derived variable and use find() for team budget display
* fix(anthropic): emit replayable streaming thinking blocks (#31022)
* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)
* feat(proxy): read cold-storage prompts back in the logs detail view
When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.
Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.
Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.
ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.
* Update litellm/proxy/spend_tracking/spend_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure
Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)
* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup
Two bugs fixed:
1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
successful GCS upload, so metricsMarker stayed at 0 and every daily run
re-exported the same dates in an infinite catch-up loop.
Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
file is already committed). A 410 raises consistent with the rest of the
destination.
2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
has triggered lazy instantiation of MavvrikFocusLogger, so it found no
logger instance and silently skipped registering the daily export job.
Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
_init_custom_logger_compatible_class to force instantiation before
the APScheduler job is registered.
* fix(mavvrik): catch up from earliest window when metricsMarker=0
When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.
Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.
* fix(mavvrik): use now as end_time for yesterday's export window
LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.
Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.
Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.
* fix(mavvrik): also use now as end_time for catch-up windows
* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class
Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.
* ci: retrigger CI run
* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)
* Add optional `instruction` passthrough to the rerank API
vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.
Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: thread `instruction` as a typed param + cover rerank_utils
Per PR review (greptile P2 + codecov):
- Make `instruction` a typed, named argument on the rerank provider interface
instead of recovering it from the opaque `non_default_params` blob. Adds
`instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
and every provider override, and forwards it explicitly from
`get_optional_rerank_params`. hosted_vllm now reads the named param directly.
It is still also surfaced in `non_default_params` so providers that read it
there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
previously-uncovered threading line flagged by codecov.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scan rerank `instruction` through request guardrails
The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.
Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.
Addresses the Veria AI security review on PR #30757.
* test: narrow Optional results before len() to satisfy basedpyright budget
The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.
* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget
The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.
It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(github_copilot): synthesize empty choices at the provider seam (#30929)
Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500
Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers
Fixes: https://github.com/BerriAI/litellm/issues/30927
Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com>
* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
yucheng-berri
|
a545c493d7
|
fix(otel): hashable scope for _emit_once when guardrail_mode is list (#31262)
* fix(otel): hashable scope for _emit_once when guardrail_mode is list `_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]` (the shape Presidio expands to with `output_parse_pii: true`, and the shape `event_hook` carries for any `mode: [...]` in config), the tuple contains a list and `spans_logged.get(dedupe_key)` raises `TypeError: unhashable type: 'list'`. On the post-call path this fires inside the logging callback and is swallowed; the request returns 200 but the OTEL `guardrail` span is silently dropped. On the blocking path the same error surfaces as HTTP 500. Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists and tuples into tuples, sets into frozensets, dicts into frozensets of `(key, value)` pairs, and falls back to `repr` for arbitrary unhashables. Applied inside `_emit_once` before the dict lookup, so all three callsites are protected without touching the guardrail-specific callsite. Helper assumes acyclic input; `guardrail_mode` values are built fresh from config (str enums, lists of str enums, TypedDict of str/list-of-str), so no cycle can arise in practice. Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash, distinct-list-scope collision, dict and set scope parts, and an end-to-end `_create_guardrail_span` exercise that confirms exactly one `guardrail` span is emitted across repeated lifecycle entrypoints. Each new test fails on a reverted helper (4/4 mutation kill) * fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector CI's recursive_detector blocks new recursive functions in litellm/ unless they are in the allowlist with a documented bound. Cap the helper at 16 levels and return repr(value) past the cap; this is well past the realistic depth of guardrail_mode (1-3 levels) and means a future caller passing a cyclic container can no longer push the proxy logging path into a RecursionError. Add a regression test that exercises the cycle path. * refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union Per review feedback from @mateo-berri: replace the loose `-> object` annotation with a recursive `HashableScope` union (str | int | float | bool | bytes | None | Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract is visible at the signature. Replace the `try/except hash(value); return value` passthrough with an explicit isinstance check over the hashable-scalar types so the type checker can narrow without requiring `cast(Hashable, value)` on the return. Symmetric: dict keys also flow through the freezer (a TypedDict key is already a string in practice, so behaviorally identical). All 16 regression tests still pass; mutation kill behavior preserved * fix: avoid explicit casting --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> |
||
|
Krrish Dholakia
|
c546b58c09
|
feat: add chat completions code interpreter loop (#31027)
* feat: add chat code interpreter loop * fix: address code interpreter pr checks * fix: satisfy strict lint budget * test: cover chat no-op interception * fix: address code interpreter review * fix: clean up agentic loop helpers * fix: preserve agentic loop controls * fix: generalize agentic loop params * fix: carry agentic state via metadata * fix: restore litellm params helpers * refactor: move chat code-interpreter loop out of provider code Dispatch the chat-completions agentic loop from a provider-agnostic helper (litellm/litellm_core_utils/chat_completion_agentic_loop.py) called from main.acompletion, instead of from OpenAI provider files. Register the agentic loop control fields in all_litellm_params so they stay LiteLLM-level and never become provider payload, removing the need for the OpenAIGPTConfig scrubber. No litellm/llms/ files are modified for this feature. * docs: explain chat agentic loop dispatch and litellm-level param registration * style: drop Any annotations and use PEP585 generics to satisfy ruff strict budget * docs: replace module docstring with one-line patch note |
||
|
Yassin Kortam
|
1322ad7224
|
perf(otel): resolve LITELLM_OTEL_V2 flag once instead of rebuilding settings per call (#30989)
is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every call, which re-scans the process environment and costs ~28us. The flag is read multiple times along the proxy request hot path (auth, logging-callback setup, proxy_server), so the cost compounded into a measurable per-request CPU overhead and a throughput regression visible from v1.87.3 onward. The flag is a process-level setting that is fixed at startup, so resolve it once with lru_cache. Caching it alone restores throughput to the pre-regression baseline in load tests. Tests that toggle the env now call cache_clear(). |
||
|
Krrish Dholakia
|
84c1414aef
|
feat(sandbox): code interpreter interceptor on the Responses API (#30905)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(sandbox): code interpreter interceptor on the Responses API
Route OpenAI's code interpreter to a configured sandbox (e2b) instead of
OpenAI's container, with no client change. A client calls /v1/responses with
a code_interpreter tool; the interceptor converts it to a function tool so the
model emits the code, runs that code in the sandbox via the phase 1 primitive,
feeds the result back, and lets the agentic loop continue.
Reuses the existing agentic-loop hooks (no new hook methods). The anthropic
agentic caller _call_agentic_completion_hooks gains an api_surface argument and
a responses execute path (_execute_responses_agentic_plan re-calls aresponses);
the responses handler invokes it after transforming the response. Web search and
compression interceptors are untouched.
Adds an api_base passthrough to the sandbox SDK and a sandbox_tools registry the
proxy parses, so the interceptor resolves a named tool to provider/key/base.
v0 limitation: no file upload or download yet; stdout and inline results flow
back, attaching input files and downloading produced files do not.
* feat(sandbox): re-inject code_interpreter_call so the response matches OpenAI
The native OpenAI Responses code interpreter returns a code_interpreter_call
output item (id, type, status, code, container_id, outputs) alongside the
message. The interceptor now re-injects an equivalent item via
async_post_agentic_loop_response_hook so a client gets the same response shape
whether the code ran in OpenAI's container or the sandbox: build_plan records
the executed code and the container id per call, and the post hook inserts the
code_interpreter_call before the message in the final response output.
* feat(sandbox): support streaming for the code interpreter interceptor
A stream:true /v1/responses request with code_interpreter previously broke,
because the agentic loop only runs on the non-streaming responses path. The
interceptor now forces stream=False in the pre-call hook (so the loop runs in
the sandbox) and the responses handler wraps the completed response back into a
synthetic stream via MockResponsesAPIStreamingIterator, so the caller still gets
SSE. The follow-up call and nested wrapping are guarded by stripping the
converted-stream flag from the follow-up request and only wrapping at the
outermost call (agentic loop depth 0).
* fix(lint): use builtin generics in code interpreter interceptor to satisfy UP006 budget
* fix(code-interpreter): gate sandbox execution, delete sandboxes, harden registry
Gate the agentic loop on a server-set interception marker and re-check
provider scope so an authenticated caller cannot trigger sandbox code
execution by naming their own function tool litellm_code_execution; the
marker is stripped from client requests at the proxy boundary and only
set when the pre-call hook actually converts a native code_interpreter
tool. Delete the sandbox once the final response is assembled instead of
leaking it until its own timeout, and prune expired cache entries by
deleting their containers too. Resolve sandbox params once at create time
and reuse them for run and delete. Clear the sandbox-tool registry before
re-registering so stale tools do not survive a config reload.
* fix(lint): use PEP 604 X | None unions to satisfy UP045 budget
* test(code-interpreter): cover execution-error and unparseable-argument tool-call paths
* fix(code-interpreter): rewrite forced code_interpreter tool_choice to the function tool
* test(sandbox): cover sandbox-tool registry resolution, reload clearing, and secret lookup
* test(code-interpreter): cover dict-shaped responses and object-attribute tool-call detection
* fix(proxy): strip client-supplied _code_interpreter_interception_converted_stream
A client could inject the converted-stream marker to force the completed
response to be re-wrapped as a synthetic SSE stream it never requested.
Add it to the untrusted root control fields alongside the other agentic
loop markers so the proxy strips it at the request boundary.
* fix(code-interpreter): isolate sandboxes by server-minted key and clear registry on tool removal
Key the per-request sandbox cache on a server-minted random token instead
of the caller-controlled litellm_call_id (sourced from the x-litellm-call-id
header). Two concurrent requests that send a colliding call id can no longer
share a sandbox container and read each other's code or files. The token is
minted in the pre-call hook when interception activates, stripped from client
requests at the proxy boundary, and survives the server-driven followups so a
single request still reuses one sandbox across the agentic loop.
Register sandbox tools unconditionally with an empty-list fallback so a config
reload that removes sandbox_tools clears the previously registered credentials
instead of leaving them resolvable in the process.
* refactor(sandbox): swap the tool registry atomically on reload
Build the new registry and rebind it in one assignment instead of clearing
then repopulating in place, so a concurrent resolve_sandbox_tool can never
observe a transiently empty or half-populated registry during a config
reload. clear_sandbox_tools now delegates to register_sandbox_tools([]).
* fix(code-interpreter): cap caller loop limit and emit OpenAI-shaped outputs
Strip max_agentic_loops at the proxy request boundary so an authenticated
caller cannot raise the agentic-loop ceiling to drive many upstream model
calls and sandbox executions from a single request; the loop stays bounded
by the server default.
Populate the re-injected code_interpreter_call.outputs with an OpenAI-shaped
logs array ([{"type": "logs", "logs": stdout}], or [] when there is no
stdout) instead of None, so clients that iterate over outputs or validate the
response through the OpenAI SDK's Pydantic model do not break.
|
||
|
yucheng-berri
|
1f9323792c
|
fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter (#30590)
* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter The proxy published the OTel global TracerProvider before callbacks were initialized, so no preset logger existed yet and a second generic logger was built that won the global provider. Server spans then exported through a different provider than the preset's gen-ai spans, orphaning the LLM span on the preset backend. Publish after callback init and reuse the already-built logger instead. Separately, per-request tenant OTLP credentials were stamped onto every OTLP exporter, leaking one backend's key onto a co-configured backend. Tag each exporter with the preset that contributed it and apply dynamic credentials only to the matching owner. * fix(otel): satisfy Any-discipline on changed lines Type the logger-selection parameter as Sequence[object] (isinstance narrows it), cast the list[Any] global at the single call site, and pass model_copy a typed dict[str, str] update so no changed line carries an Any value. * fix(otel): annotate the untyped-global boundary with any-ok select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared List[Any] global this change does not own. A cast doesn't satisfy the Any-discipline checker (it inspects the inner expression), and re-annotating the global is out of scope, so mark the single boundary line any-ok. * test(otel): cover the startup global-provider publish via injectable helper The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit tests do not execute), so its lines were uncovered though the selection logic was tested. Extract publish_global_otel_v2_provider, which selects the single v2 logger and publishes its provider through an injected setter, and unit-test that the published provider is the selected logger's. proxy_server delegates to it. * refactor(otel): select global provider from the registered owner, not a list scan The startup publish picked the global TracerProvider by scanning _in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the factory already settled: the first logger built registers itself as proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity seeding, phase spans) routes through that owner via _registered_v2_logger. Pass that owner into select_global_otel_v2_logger so the global provider reuses the same logger instead of an independent, order-dependent guess; the list scan remains the SDK-path fallback. The owner is injected at the proxy call site to keep the helper free of hidden global reads. * refactor(otel): type ExporterSpec.owner as an ExporterOwner enum The owner field carried free-form strings that had to match preset callback names. Introduce a str-based ExporterOwner enum (values equal to the callback names, so per-request credential routing's owner==callback_name comparison still holds) and have each preset tag its exporter with the enum member. * refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX Distinguish the hosted Arize AX backend from Arize Phoenix at the member level while keeping the value 'arize' (the public callback name routing compares against). Add a comment noting AX and Phoenix are separate backends. |
||
|
Yassin Kortam
|
27c1dfbdc7
|
fix(otel): accept UPPER_SNAKE_CASE OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT in v2 (#30562)
V1 read this env var case-insensitively, so SPAN_AND_EVENT enabled content capture. The v2 config compared the value against its lower_snake_case canonical constants without normalizing, so an operator carrying the SPAN_AND_EVENT spelling forward silently left capture off and no gen_ai.input/output.messages reached the span. Normalize the value to lower case at the config boundary so both spellings work. |