* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes#33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in c698b88686 to match upstream litellm_internal_staging.
Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update.
* fix: scan system messages as indirect prompt injection in Singulr guardrail
* chore: restore lint budget files to upstream baseline
* fix: resolve ruff UP006 and I001 violations in singulr guardrail
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* resolve review comments on Singulr guardrail
* fix: scan tool call results as indirect prompt injection in Singulr guardrail
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* minor
* formating fix
* refactor: shift extraction logic to singulr side
* refactor:keep precall hook only
* fix:formatting
* fix:linting
* improve config description
* Trigger CI
* fix
* fix:field description
* fix:errors due to change in field names
* style: apply ruff line-wrap formatting to singulr guardrail
* fix:exception
* fix:formatting
* fix playground
* improved
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix
* fix ci issues
* remove uv.lock from pr
* fix
* fix:resolved comments
* chore: trigger CI
* remove uv.lock
* fix
* fix linting
* fix linting
* fix linting
* remove doc strings
* remove test fixes
* chore: retrigger CI
* change in singulr api contract
* remove some ut
* send litellm call_id to singulr
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Fix non-conformant UUIDv7 generation in native Opik integration (#31294)
create_uuid7() encoded the timestamp in units of 16 seconds instead of
milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's
backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067);
the bad encoding decoded to ~year 2201 and every trace/span batch was
rejected with HTTP 400.
Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms),
using the standard library only so no new dependency is added. Add unit
tests covering UUIDv7 validity and millisecond timestamp encoding.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(proxy): expose uvicorn concurrency limit (#33077)
Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and
LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and
accepted connections and returns HTTP 503 once the configured limit is reached.
Reject non-positive limits at CLI parse time and only add the setting to the
uvicorn startup arguments. Because idle connections also consume capacity,
deployments should use upstream connection/header timeouts and per-client
connection limits.
* test: reorder test_utils tail to keep the daily merge conflict-free (#33788)
The daily OSS branch and litellm_internal_staging each appended an
independent test block at the very end of tests/test_litellm/test_utils.py,
so merging the two collides on that shared end-of-file position even though
the additions are unrelated (this branch adds the vertex embedding
encoding-format tests; staging adds the per-model prompt-cache-minimum
tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class
above test_gemini_image_models_do_not_support_reasoning, which both branches
share, gives the two additions different anchors, so git applies both
without a conflict and without pulling staging into this branch. Pure
reorder; no test bodies change
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
_request_has_cache_control only looked at messages and system, so a client that
marks cache_control on tools alone did not suppress auto-injection. Tool
breakpoints count toward the provider's four-block limit, so three of them plus
the two injected here is five, which Anthropic rejects. Thread tools through
both entry points and treat a client-marked tool as the stand-down signal it
already is for messages and system.
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials
Key-scoped langfuse_otel callbacks only injected Authorization headers into the
init-time exporter, so a proxy without global LANGFUSE_* env vars kept its
fallback exporter and never exported traces to Langfuse. Dynamic params now
build a full per-request OTLP config (endpoint from the key's langfuse_host,
otlp_http, basic auth from the key's credentials).
Resolves LIT-3976
* fix(otel): log dynamic config endpoint in span processor debug output
* fix(otel): redact authorization headers in exporter debug logs
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are
now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and
LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on
without a config file. An unsupported ttl falls back to the provider default
rather than reaching the provider verbatim
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
adds a top-level guard in _increment_remaining_budget_metrics that returns early
when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config),
and per-entity guards in each _set_*_budget_metrics_after_api_request helper for
partial disabling. eliminates four async DB/cache round-trips per successful LLM
request when budget metrics are disabled.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls
The GenAI semantic conventions record failures of a GenAI client operation as
a log-based event named gen_ai.client.operation.exception, carrying the
exception.type / exception.message / exception.stacktrace trio at severity
WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM
call produced only the deprecated error.* span attributes, a generic exception
span event without a stacktrace, and the stacktrace under the vendor key
litellm.provider.error.stack_trace.
Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring
the metrics plumbing) and record the event behind the enable_events flag, which
until now was defined but consumed nowhere. An operator-configured LoggerProvider
global is reused so the events ride their existing logs pipeline; an explicit
NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all.
The existing span-side error surface (error.type, error.message, the exception
span event, and the litellm.provider.error.* detail keys) is untouched for
backwards compatibility.
* fix(otel): always ride the semconv-required exception pair on the GenAI event
Filtering the event attributes on truthiness conflated "absent" with "empty",
so an empty exception.type or exception.message would have been dropped, leaving
an event with neither semconv-required field. Build the attributes so the pair is
unconditional and only the recommended stacktrace is omitted when the payload
carries none.
* docs(otel): document the events plumbing module in the package README
* test(otel): cover the log exporter selection and logs endpoint normalization
The new logs plumbing had no coverage for exporter-kind selection, the
console fallback for an unrecognized kind, the /v1/logs signal-path rewriting
that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch
processor split.
Team-level callback_vars (e.g. langsmith_api_key) get spread into
data["metadata"] as four aliases (user_api_key_metadata,
user_api_key_team_metadata, user_api_key_auth_metadata,
user_api_key_auth). When a guardrail hook echoes that metadata into
its guardrail_response, the plaintext credential landed five times
inside LiteLLM_SpendLogs.metadata.standard_logging_guardrail_information[i].guardrail_response
and every downstream sink that reads it (OTel via emit_guardrail_span,
Langfuse, custom loggers).
Add a purpose-built payload walker (mask_credentials_in_payload) that
only masks strings under sensitive-named keys and preserves every
other value (None, ints, floats, bools, tuples, typed objects) verbatim.
The walker reuses SensitiveDataMasker.is_sensitive_key so the pattern
list stays in one place, and unwraps Pydantic models via model_dump()
so nested UserAPIKeyAuth values reached by the walk get scanned as
plain dicts (they are JSON-serialized downstream anyway).
Apply the walker at add_standard_logging_guardrail_information_to_request_data
after the existing secret_fields pop and match/regex redaction, so
every downstream sink sees masked values from a single seam.
The v2 OTel integration stamped litellm-specific error details as
error.code, error.stack_trace, and error.llm_provider, squatting on the
semconv-owned error.* namespace. They now live at
litellm.provider.error.code, litellm.provider.error.stack_trace, and
litellm.provider.error.llm_provider alongside the other vendor-extension
keys. error.type and error.message stay on the semconv keys.
The v2 emitter has never stamped error.message / error.code /
error.stack_trace / error.llm_provider as span attributes; only error.type
reached the wire. Backends that flatten span attributes into label
indexes (Elastic APM labels.error_*, Datadog span tags) lost these
four fields when v2 became the active integration on v1.90+ for
otel_v2-flagged deployments. The pre-existing exception span event
carrying the full message (LIT-3758) is unchanged; the message now
rides both places at once, matching v1s shape.
SpanError grows three optional detail fields; _parse_error threads
them from StandardLoggingPayloadErrorInformation; the emitters error
branch stamps them via a new module-level helper, guarded per field so
guardrail-shape errors are not polluted with empty attributes. New
semconv constants mirror open_inference.ErrorAttributes byte-for-byte,
so v1 and v2 consumers read the same keys.
Regression tests extend the mapped test files under
tests/test_litellm/integrations/otel/. pytest reports 243 passed.
* feat(prometheus): add api_provider label to token, latency, request and cache metrics
The token (input/output/total), latency (llm_api, time_to_first_token,
request_total, request_queue_time), proxy request (total/failed) and cache
metrics were emitted from the same call sites as litellm_spend_metric and
litellm_requests_metric, which already carry api_provider, yet these were
missing it. That left no way to break tokens, latency, request counts or cache
hits down by upstream provider even though the provider is already on the
payload as custom_llm_provider.
Add api_provider to each metric's label allow-list. The success path already
populates enum_values.api_provider from standard_logging_payload, so those
metrics emit it with no further plumbing. The cache label is added to the
shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and
litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the
provider prompt-cache read/creation token metrics; the label-presence test
asserts all of them. For the client-side failure path, where a deployment may
not have been resolved, derive it best-effort from
litellm_params.custom_llm_provider, a partial standard_logging_object, or
inference from the requested model name via litellm.get_llm_provider, falling
back to empty rather than guessing.
Resolves LIT-4178
* fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions
- Suppress the strict-rule BLE001 budget breach with a justified noqa;
the broad except in the failure-path provider extraction is
intentional defense-in-depth (covered by
test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors),
not dead code to delete
- Update tests/enterprise assertions for litellm_tokens_metric,
litellm_input_tokens_metric, litellm_output_tokens_metric, the three
latency metrics, and the proxy request counters to expect the new
api_provider label, matching what litellm_mapped_enterprise_tests
caught in CI
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
When AzureSentinelLogger is resolved from the string callback name
"azure_sentinel", it is constructed with no arguments, so audit_stream_name is
always None and resolved_audit_stream_name fell back to the standard
resolved_stream_name. Audit logs then ingested into the access-log DCR stream
whose schema is built from StandardLoggingPayload, so Azure Monitor Logs
Ingestion silently dropped the audit-specific columns and audit rows arrived
effectively empty.
Add an AZURE_SENTINEL_AUDIT_STREAM_NAME env var fallback in __init__, mirroring
the AZURE_SENTINEL_STREAM_NAME idiom already used for the standard stream, so
audit logs can target a separate DCR stream without a custom callbacks file.
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Include top-level scalar fields from standard_logging_metadata in the
combined metadata dict used by custom_prometheus_metadata_labels. Previously
only nested sub-dicts (requester_metadata, user_api_key_auth_metadata,
spend_logs_metadata) were spread into combined_metadata, so fields like
user_api_key_project_alias were inaccessible and always resolved to None.
Co-authored-by: unknown <>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: add cache control injection support for v1/messages endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: normalize string content to list for Anthropic-native cache_control injection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: simplify cache control injection, fix system=[] bug, fix handler system type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: extract cache control logic into static helper on AnthropicCacheControlHook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sandbox): reuse e2b container across requests when metadata.session_id is set
When a client passes `metadata.session_id` in a /chat/completions request
alongside a code_interpreter tool, the proxy now routes all requests sharing
that session_id to the same sandbox container. State (variables, imports,
installed packages) persists across requests within the session.
Without a session_id the existing ephemeral behavior is unchanged: one
container per agentic loop, deleted immediately after.
The sandbox key is derived from session_id rather than a per-request UUID.
The cleanup and post-loop hooks skip deletion for session-scoped containers.
TTL-based pruning (15 min idle) still applies and refreshes on every use,
so an active session never expires mid-use. The session_id-scoped key is
registered in all_litellm_params and the proxy strip-list so it never
leaks to the upstream LLM provider.
* fix(sandbox): scope session sandbox key to API key identity; add per-identity LRU cap
Two security issues addressed:
1. Cross-user sandbox isolation: the session_id supplied by the client is now
combined with the server-minted user_api_key_hash to form the cache key
(format: "{hash}:{session_id}" when authenticated, bare session_id for
non-proxy use). Two tenants sharing the same session_id no longer share a
sandbox.
2. Bounded session allocation: each API key identity is capped at
_SESSION_SCOPED_PER_IDENTITY_CAP (10) live session-scoped containers. When
a new session is opened beyond the cap, the least-recently-used entry for
that identity is evicted and its sandbox deleted, preventing unbounded
accumulation via rotating session IDs.
The container cache tuple gains a fourth element (identity: str | None) so
eviction can filter by identity without parsing key formats. Tests added for
both properties.
* fix(websearch): wire chat completion agentic loop to correct hooks
maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller.
Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path.
Regression test added.
* fix(websearch): strip tool_choice from follow-up request
When the original request forces tool_choice to litellm_web_search,
the follow-up request after search execution inherited that tool_choice,
causing the model to call the search tool again instead of synthesizing
an answer from the results.
* fix(websearch): inject api_key into agentic hook kwargs for anthropic messages
Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's
synthesis call after executing Exa/Perplexity searches) were missing api_key
because the named api_key param in async_anthropic_messages_handler was never
merged into the kwargs dict forwarded downstream. Result: every /v1/messages
websearch follow-up failed with "x-api-key header is required" and the caller
received the raw tool_use response instead of the synthesized answer.
* ci: trigger CI run
* fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks
CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with
_agentic_loop_api_surface to handle both surfaces from one hook. The chat
completion loop must also check _gate_overridden so callbacks using the
unified hook pattern still fire for chat completions.
* fix(websearch): strip tool_choice from legacy chat completion follow-up call
The _execute_chat_completion_agentic_loop path merged original optional_params
(which includes forced tool_choice) into follow-up params without explicit
removal. _build_chat_completion_request_patch already excluded tool_choice from
its optional_params output, but dict.update() with a missing key leaves the
original value intact. Explicit pop after the merge removes it.
* fix(websearch): always strip tool_choice from plan-path follow-up params
The tool_choice removal was gated on patch.tools is not None. WebSearch sets
tools via patch.optional_params not patch.tools, so the gate was False and
forced tool_choice from the original request survived into the synthesis call.
Move the pop outside the patch.tools branch so it applies unconditionally.
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
The proxy auth path calls phase_span() and seed_request_identity() in
litellm/integrations/otel/runtime.py on every request, each doing a
try/except lazy import of litellm.integrations.otel.logger. When the
OpenTelemetry SDK is not installed (the default), that import raises, and
CPython never caches a failed import, so every request re-scanned sys.path
and contended on the import lock. At 750 concurrent users this cost about
12% throughput versus v1.85.0.
Resolve the hooks once and cache the outcome, absence included, with
functools.cache, so the import is attempted a single time instead of per
request. Throughput returns to the v1.85.0 baseline.
litellm_overhead_latency_metric only covers the SDK wrapper window and excludes
proxy guardrails. Add a histogram that sums SDK overhead plus pre/post-call
guardrail durations (during-call excluded since it runs concurrently with the LLM
call, alongside logging_only and MCP modes that never block the response),
recorded next to the existing overhead metric with the same labels and buckets.
No existing metric's value is changed.
* fix(proxy): count only active users toward license seat limit
SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat
Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected
* fix(proxy): floor billable user count at zero
count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call
Addresses Greptile P1 on the PR
* refactor(proxy): count teams via TeamRepository in available_users
* style: ruff format changed files at line-length 120
Emit litellm_team_members_metric on every team member add and delete,
labelled by team and team_alias and set to the team's authoritative
member count. Because it is set from the current membership rather than
incremented or decremented, it tracks the count up and down, never goes
negative, and self-corrects on the next change after a proxy restart.
Bulk member add is covered for free since it delegates to
team_member_add, and the helper no-ops when the Prometheus callback is
not registered.
Resolves LIT-3082
The AgentOps preset hardcoded https://otlp.agentops.cloud/v1/traces, a domain
that no longer resolves (NXDOMAIN), so every span silently failed to export with
a NameResolutionError in the BatchSpanProcessor worker. The live ingest host is
otlp.agentops.ai (the auth host api.agentops.ai was already correct). Pin the
endpoint to the resolvable host and add a regression test on the constant.
failing test is not related to the pr
* fix(websearch): sync tool_choice when converting web_search tools
Claude Code forces native web search via tool_choice pointing at web_search
while websearch_interception renames the tool to litellm_web_search, causing
Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced
tool_choice to match the converted tool name.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(websearch): re-wrap agentic loop responses as SSE for streaming clients
When websearch interception converts stream=true to false for the agentic
loop, dict responses from the loop were returned as application/json even
though the client requested SSE. Wrap those responses in
FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers
(e.g. Claude Code) receive text/event-stream after search completes.
Fixes#27721
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(websearch): cover tool_choice sync and post-loop SSE wrap; fix UP006
Add regression tests for both websearch interception fixes: _sync_forced_tool_choice
repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and
_maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for
streaming clients (#27721). Switch the new helper annotations to builtin dict/list so
the ruff UP006 strict-rule ceiling stays within budget.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(websearch): resolve merge conflict and unify fake stream wrapping
Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MBP.localdomain>
* feat(guardrails): add headroom guardrail for message compression
Adds a headroom guardrail that compresses request messages via POST
/v1/compress before they reach the LLM. The guardrail implements
apply_guardrail so it runs on the unified guardrail path; it receives
pre-built structured_messages (OpenAI format) from the translation
layer, calls the headroom compression service, and returns the
compressed messages as structured_messages.
Set x-headroom-bypass: true on the request to skip compression.
Also adds structured_messages write-back support to the OpenAI and
Anthropic translation handlers: when apply_guardrail returns
structured_messages, those are written to data["messages"] directly
(OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic)
instead of falling through to the existing text-patch path. This is a
prerequisite for any guardrail that needs to replace the full message
list rather than patch individual text spans.
* fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs
* style: fix ruff format violations
* fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037)
* fix(guardrails): only write back structured_messages when guardrail actually changed them
* fix(guardrails/headroom): raise 502 when compression returns empty message list
* fix(guardrails/headroom): catch transport errors and fix stale debug log
* fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation
* fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back
* debug(headroom): add INFO logging to trace guardrail execution
* debug(headroom): use print() for immediate visibility
* debug(headroom): print request_data keys to diagnose metadata dict mismatch
* fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log
* fix: use model_call_details litellm_params metadata on Logging object
* fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy
* fix: read slg_info from litellm_metadata when metadata key absent
* fix: write slg_info to both litellm_params attr and model_call_details copy
* chore: remove debug prints; fix now verified end-to-end
* refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py
- Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from
both async and sync wrappers in @log_guardrail_information, fixing
guardrail_information=null in spend logs for all passthrough routes
(/v1/messages, /v1/responses, etc.) in one place
- Remove the 35-line inline sync block from the anthropic translation handler
- Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses
- Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase
- Add regression tests for _sync_guardrail_info_to_logging_obj
* fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold
* fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity
* fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity
* fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity
* fix(websearch): wrap agentic loop response in fake stream for streaming requests
When websearch_interception converts stream=True to stream=False internally,
the agentic loop returns a plain dict. Previously this dict was returned
directly to the client expecting SSE events, resulting in empty streams.
Added _maybe_wrap_in_fake_stream() which checks the
websearch_interception_converted_stream flag and wraps dict responses in
FakeAnthropicMessagesStreamIterator. Applied to all return paths in
_call_agentic_completion_hooks:
- async_run_agentic_loop (legacy path)
- _execute_anthropic_agentic_plan (plan-based path)
- plan.response_override
- plan.terminate
Includes unit tests for _maybe_wrap_in_fake_stream().
* test(websearch): cover agentic-loop wrap paths; gate fake-stream on anthropic_messages surface
Guard _maybe_wrap_in_fake_stream on api_surface == anthropic_messages so the
responses API surface is never wrapped in an Anthropic SSE iterator, and type
logging_obj as Optional to match the None call sites. Adds regression tests
that drive the legacy, response_override, and terminate return paths of
_call_agentic_completion_hooks end to end.
* test(websearch): cover _execute_anthropic_agentic_plan and tail wrap paths
Drives the remaining two fake-stream return paths of
_call_agentic_completion_hooks (the _execute_anthropic_agentic_plan branch via
a stubbed handler, and the tail path when no agentic loop runs) so every
converted-stream return path is regression-tested.
---------
Co-authored-by: Clawd <fffff.c@gmail.com>
litellm_spend_metric_total and litellm_requests_metric_total previously
exposed only the resolved backend model_id and friendly model name, so
operators could not group spend or request counts by the model alias the
caller actually asked for when a router fronts multiple deployments
behind one name.
This adds the existing UserAPIKeyLabelNames.REQUESTED_MODEL to both
labelname lists; the value is already populated upstream from
standard_logging_payload["model_group"] and flows through the shared
_increment_top_level_request_and_spend_metrics call site. The sibling
token metrics (input/output/total) already carry the label, so this
also restores cross-metric consistency.
Resolves LIT-3796
