Making Sentinel follow AZURE_AUTHORITY_HOST is a breaking change for a
deployment that sets that variable for Azure OpenAI or the azure_storage
callback while keeping a commercial Sentinel workspace. That deployment had no
opt-out, because the proxy constructs the logger with no arguments and the
authority_host parameter is reachable only from the SDK.
Resolve the authority from AZURE_SENTINEL_AUTHORITY_HOST before falling back to
AZURE_AUTHORITY_HOST, matching how tenant id, client id and client secret
already resolve in this constructor.
The Azure Sentinel logger hardcoded the commercial Entra authority and the
commercial Azure Monitor audience, so Log Analytics ingestion could not work in
Azure Government even when the ingestion endpoint pointed at a sovereign Data
Collection Endpoint.
Resolve the authority from AZURE_AUTHORITY_HOST and derive the matching Logs
Ingestion audience from it. Moving only the token URL is not enough: sovereign
Entra would then be asked for a token scoped to the commercial audience, which
the sovereign endpoint rejects.
A cached HTTPHandler hands its raw httpx.Client out to consumers that keep it
for the process lifetime. When the shared client cache expires the entry on its
TTL or evicts it under the 200-entry cap, nothing references the handler, so it
is collected and its finalizer closed the client those consumers still hold.
Langfuse ingestion then failed silently on the SDK's background flush thread
until the process restarted.
A finalizer running proves only that nothing references the handler; it proves
nothing about the client. Both handlers now close the client during finalization
only when they built it and are still its sole referrer, so an unshared client is
still released promptly and a handed-out one is left alone. That keeps the
pooled-socket reclamation the finalizer was providing, which measures identical
to base over 2000 handler create-and-drop cycles.
Explicit close() stays, now gated on _owns_client so the wrapper never closes a
caller-injected client, and __aexit__ routes through it.
LangFuseLogger also keeps a reference to the handler whose client it hands the
SDK. Previously that handler was a local that went out of scope immediately,
leaving the client reachable only from the SDK. It still shares the cached
client, so no extra clients are created per logger.
Generic SigV4 double-encodes the canonical URI while S3 canonicalizes the wire path with single encoding, so any object key containing a character that percent-encodes (a team alias, key alias or s3_path with a space) was signed over %2520 while the request carried %20; S3 recomputed a different signature and answered 403.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
The azure_storage logging callback and the azure blob files backend built every
storage URL against the hardcoded commercial host, so an Azure Government account
was unreachable with no way to override it.
Read AZURE_STORAGE_ENDPOINT_SUFFIX (default core.windows.net) once in
AzureBlobStorageLogger and derive the Data Lake and Blob hosts from it, so all
seven previously hardcoded sites follow the configured cloud. Parse stored blob
URLs with urlparse instead of matching the commercial host, so URLs persisted
before the suffix was configured still resolve, and pin the resulting
host-validation boundary with tests.
* feat(otel): stamp service tier attributes on inference spans
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): bound requested service tier to known values
The requested tier is caller-controlled and reaches the span verbatim, so an
arbitrary string lands on every litellm_request span on success and on failure.
A 100k character value was stamped uncapped; safe_set_attribute does not
truncate and no span limits are configured.
Apply KNOWN_REQUEST_SERVICE_TIERS in get_requested_service_tier so both the
span attribute and the Prometheus label bound the value the same way. The
served tier stays unrestricted since it comes from the provider, so a tier a
provider adds later is still reported.
Prometheus label behavior is unchanged.
* fix: derive known service tiers from the ServiceTier enum
The allowlist omitted "fast", which litellm models as a real tier and prices
through the priority cost key, so a request naming it resolved to no tier on
the span and no Prometheus label.
Deriving the set from ServiceTier keeps the two in sync, so a tier added there
for cost calculation cannot go missing here.
Behavior change: a request with service_tier "fast" now carries the tier on the
span and on the Prometheus service_tier label, where it previously resolved to
none. Every other value resolves as before.
* refactor: build the known service tiers without a mutable intermediate
The set comprehension and set literal tripped LIT002, which bounds mutable
collections. Concatenating tuples keeps the derivation from ServiceTier while
every intermediate stays immutable; the resulting frozenset is unchanged.
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
The block event Rubrik receives sourced caller identity from
model_call_details[metadata], where the enriched litellm metadata never
lives; it sits under litellm_params. Every block therefore reported
user_api_key_hash as an empty string, so a security block could not be
traced to a key, user, or team.
Read identity off the authenticated UserAPIKeyAuth the failure hook is
already handed, via the same mapper the success path and the proxy spend
logger use, so a block log and a success log describe their caller with an
identical key set.
* feat(guardrails/rubrik): prompt moderation, response-text blocking, streaming buffer, failure logging (#34019)
* feat(guardrails/rubrik): add prompt moderation, response-text blocking, streaming buffer, failure logging
- Add `pre_call` prompt moderation via `/v1/before_prompt/openai/v1` webhook:
structured messages are flattened and sent before the LLM is called; blocked
prompts surface a `ModifyResponseException` with the refusal text.
- Extend `post_call` response moderation to cover assistant text in addition to
tool calls; text blocks (wholesale replacement) are distinguished from
tool-block explanations (appended) via `startswith` diffing.
- Add `streaming_end_of_stream_only = True` and `streaming_buffer_until_moderated = True`
so streamed responses are withheld until end-of-stream moderation passes
(requires litellm >= BerriAI/litellm#31389; older versions fall back to
detect-only).
- Add `_MalformedToolBlockingResponseError` for structurally invalid service
responses; `_guarded` logs at CRITICAL so operators notice misconfiguration.
- Add `max_queue_size = 10_000`, `_enforce_max_queue_size`, and drop-oldest
backpressure so a webhook outage cannot grow the retry queue unboundedly.
- Add `flush_queue` override that snapshots once for both send and drain,
preventing duplicate delivery on concurrent flush calls.
- Make `_log_batch_to_rubrik` re-raise on error so `flush_queue` preserves
undelivered events for the next retry.
- Add `async_post_call_failure_hook` to log blocked requests
(`ModifyResponseException`) with a best-effort fallback payload for prompt
blocks (where no `standard_logging_object` exists yet).
- Add `_correlation_id` / `_apply_correlation_id` / `_prepend_system_prompt`
helpers; `_prepare_log_payload` now applies them for all providers (not just
Anthropic) so every log correlates by `litellm_call_id`.
- Add `get_supported_event_hooks` classmethod advertising `[pre_call, post_call]`.
- Use dedicated `httpx.AsyncClient` (`moderation_client`) for webhook calls
with explicit pool limits, separate from the shared logging client.
- Drop module-level `rubrik_handler` singleton (inappropriate for a library).
- Update `initialize_guardrail` docstring to explain `pre_call` vs `post_call` mode.
- Update tests: rename `tool_blocking_client` → `moderation_client`,
`tool_blocking_endpoint` → `response_moderation_endpoint`, `_flush_task` →
`_periodic_flush_task`; migrate `TestExtractBlockedTools` to
`TestExtractResponseBlock` for the new combined text+tool block API; add
tests for prompt moderation, text blocking, streaming flags, and failure
payload construction.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* test(guardrails/rubrik): add tests to reach 100% coverage
50 new tests across 18 classes covering previously-untested paths:
- Prompt moderation: passthrough, block, no-messages skip, message
flattening (content-list → string), payload construction with
tools/user/correlation_key/litellm_call_id fallback, refusal extraction
- async_post_call_failure_hook: non-matching exception no-op, missing
stash warning, valid stash → enqueue, AttributeError in payload build,
flush exception handling
- Block payload building: standard_logging_object present vs fallback
path, missing start_time
- async_log_success_event: _rubrik_blocked=True skip path
- aclose: task cancel + moderation_client.aclose()
- Edge cases: sampling rate clamp warning, unknown input_type passthrough,
empty-inputs early return, model_call_details warning, _stash_block_context,
duck-typed tool-call normalization, request_data["tools"] preference over
optional_params, system-prompt exception handler, flush-at-batch-size,
enqueue exception swallowing, queue empty/lock-None guards, non-dict JSON
response TypeError
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): use get_async_httpx_client, ruff format
- Replace bare httpx.AsyncClient with get_async_httpx_client (required
by ensure_async_clients_test; avoids per-request client creation)
- aclose() calls close() (AsyncHTTPHandler interface, not aclose())
- ruff format on rubrik.py and guardrail_hooks/rubrik/__init__.py
- Update 3 tests for AsyncHTTPHandler type (isinstance check, close())
osv-scan and documentation CI failures are pre-existing on the base
branch and unrelated to this PR.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): fix UP006 strict ruff violation
get_supported_event_hooks return type used List[...] (UP006) instead of
list[...]. Replace with the built-in generic and remove the now-unused
List import from typing.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): fix 3 reportArgumentType basedpyright violations
Use `# pyright: ignore[reportArgumentType]` (not `# type: ignore`) to
suppress the three errors basedpyright reports in --outputjson mode:
- convert_content_list_to_str call (dict vs AllMessageValues)
- _apply_correlation_id call (StandardLoggingPayload vs dict[str, Any])
- _prepend_system_prompt call (same)
Also tighten _apply_correlation_id and _prepend_system_prompt signatures
from bare `dict` to `dict[str, Any]`.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): don't close shared HTTP client in aclose()
moderation_client and async_httpx_client both come from LiteLLM's global
HTTP-client cache (get_async_httpx_client keys on llm_provider + params).
Two RubrikLogger instances with the same parameters share the same
underlying AsyncHTTPHandler object. Calling close() in aclose() closed
the shared connection pool for all instances, breaking any subsequent
moderation request on other loggers.
aclose() now only cancels the periodic flush task and lets LiteLLM
manage the shared client lifecycle. Tests updated to assert close() is
NOT called.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): use Counter for duplicate tool-call ID detection
Set-based comparison lost ID multiplicity: two original tool calls with
the same ID both appeared "allowed" even when the service returned only
one (e.g. one allowed + one prohibited sharing an ID). Replace with
Counter so returned_id_counts[id] >= required_id_counts[id] must hold
for every ID. Matches the approach in the original _extract_blocked_tools.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): respect default_on=true when omitted from config
LitellmParams.__init__ converts an omitted default_on to False before
initialize_guardrail receives it, so litellm_params.default_on is always
bool and never None. The is-None guard in RubrikLogger.__init__ therefore
never fired on the proxy path, leaving prompt/response moderation inactive
for any config that omitted default_on.
Fix: read the raw guardrail dict (before LitellmParams coercion) to
distinguish an explicit `default_on: false` from the absent-means-True
default. When the key is absent from the raw config, default_on=True is
used; when it is explicitly set (either True or False), that value wins.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: ruff format rubrik.py after Counter import addition
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): detect ID-less tool call removal; fix UP045
ID-less tool calls (tc.id is falsy) were excluded from required_id_counts,
so the Counter comparison never caught their removal. Add a cardinality
check (len(returned) < len(original)) that fires on any removal regardless
of ID presence, combined with the Counter check for duplicate-ID attacks.
Also fix 5 UP045 violations (Optional[X] → X | None) introduced by our
new code against the daily-branch baseline.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): filter optional_params through ModelParamHelper in fallback payload
_build_fallback_payload forwarded the raw optional_params dict as
model_parameters. optional_params can contain extra_headers, api_key,
and other upstream provider credentials that must not reach the Rubrik
webhook. The normal standard_logging_object path already filters through
ModelParamHelper.get_standard_logging_model_parameters(), which
allowlists only safe LLM API parameters. Apply the same filter here.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): scope failure hook by guardrail_name; moderate text-completions
Guard async_post_call_failure_hook by guardrail_name so multiple Rubrik
instances don't cross-log: the failure hook is called for every registered
callback; without the check the first instance pops the stash and the
originating instance finds None and silently skips logging. Now each
instance only handles blocks raised by itself.
Also moderate /v1/completions prompts: _moderate_prompt returned early
when structured_messages was absent. For text-completion requests litellm
supplies inputs["texts"] with no structured_messages. Added a fallback
that synthesises a user-message from texts so the before_prompt webhook
can evaluate text-completion prompts.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(lint): add reason comments to pyright: ignore suppressions
type-discipline budget requires each # pyright: ignore[...] to carry an
explanatory comment. Add reasons to the three bare suppressions on lines
483, 651, 652.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): include tool-call arguments in prompt moderation
_flatten_messages_for_moderation only sent the content field, silently
dropping tool_calls[].function.arguments and function_call.arguments.
An attacker could embed prohibited text in tool-call arguments inside
assistant history turns and bypass prompt moderation entirely.
Now collects all attacker-controlled text per message: text content via
convert_content_list_to_str, plus all tool_calls[].function.arguments
and the deprecated function_call.arguments, joined with newlines before
being sent to the before_prompt webhook.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): tighten append detection to prevent prefix bypass
startswith(sent_content) allowed any replacement whose text shares the
original as a prefix (e.g. "Hello" → "Hello, blocked.") to be classified
as a tool-block append rather than a text block, bypassing detection.
Use startswith(f"{sent_content}\n\n") to require the exact two-newline
separator the webhook uses between original text and appended tool-block
explanations. Also add `returned_content != sent_content` to text_blocked
so an unchanged passthrough is never classified as a block.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(guardrails/rubrik): default_on=False when omitted (follow existing pattern)
Remove the custom raw-dict lookup that was defaulting default_on to True
when omitted from the guardrail config. Follow the standard litellm
convention: omitted resolves to False (users must explicitly opt in with
default_on: true).
- initialize_guardrail: pass litellm_params.default_on directly
- RubrikLogger.__init__: is-None guard defaults to False not True
- Test updated to assert the correct False default
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* chore(rubrik): keep the ported guardrail within staging lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: credit the original author of the rubrik guardrail work
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: keep this mirror PR's diff limited to the rubrik files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Team-scoped DD credentials (dd_api_key, dd_site) set via POST /team/{id}/callback were silently dropped because _request_blocked_callback_params blocks them from standard_callback_dynamic_params. The security block is correct for request-level injection, but team callback_vars are admin-configured and trusted.
Store the raw init kwargs on the Logging instance and read dd_* params from there in _process_dynamic_callback_list instead of from standard_callback_dynamic_params.
Adds an integration test that exercises the full Logging.__init__ flow with team callback_vars to prevent regression.
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
An import probe proves nothing about the real SDK: it may be absent (it
lives in the proxy-runtime extra) and the tests/test_litellm/llms/anthropic
test package can shadow it once collection puts that path on sys.path,
which made the test order-sensitive across collection sets
The bridge gate compared reasoning_effort against the string "none", so
litellm's dict form ({"effort": "none"}) wrongly bridged; the gate now
reads the effort value from either form and treats a summary inside the
dict as Responses-only regardless of effort. Helicone and lunary
previously skipped custom tool calls entirely; both now serialize them
(helicone as a tool_use block from the custom payload, lunary with the
custom name and input in its function fields, keeping type custom), with
new mapped tests for both integrations
* feat(s3): support SSE-KMS encryption params on both S3 logging paths
* fix(s3): ignore non-string SSE config values instead of crashing logger init
* Update litellm/integrations/s3.py
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): invalidate only the mistyped SSE field instead of dropping both
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name
The GenAI metric attribute builder mapped only chat, text completion, embedding,
responses and MCP tool calls to an operation name, so vector-store searches and
A2A agent sends fell through to the "chat" default. Their duration and cost then
landed in the same series a Grafana GenAI dashboard reads chat latency off, with
no way to tell them apart. Both now map to the operation names the convention
defines for them, retrieval and invoke_agent, and an unmapped call type says so
at debug instead of silently becoming chat.
The provider label used gen_ai.system, which the convention deprecated in favor
of gen_ai.provider.name; the dashboards built on that vocabulary find nothing
under the old key. Metrics now carry gen_ai.provider.name with the semconv
provider value (bedrock -> aws.bedrock) via the resolve_provider helper the span
path already uses, and keep dual-emitting gen_ai.system with its raw value so a
dashboard already querying it keeps matching. A request litellm cannot attribute
to a provider gets no provider label at all rather than a placeholder "Unknown"
that minted a permanent series nobody can act on.
Resolves LIT-4954
Resolves LIT-4959
* fix(otel): map the rest of the vector-store call types off the chat default
Mapping only the search left the store lifecycle (create, retrieve, list,
update, delete) and the file operations (create, list, retrieve, content,
update, delete) falling through to chat, so vector-store admin traffic kept
polluting the same series a dashboard reads chat latency off. A live run
confirmed it: all 20 metric datapoints from a create, retrieve, list, file-list
and delete came out labelled chat.
The convention names no operation for vector-store management, so these take
vendor values under the litellm. prefix, litellm.vector_store_management and
litellm.vector_store_file_management, one per REST resource. Its note on
gen_ai.operation.name directs instrumentation to use a system-specific name
when no predefined value applies, which is the same allowance resolve_provider
already relies on for unmapped providers. Excluding them from the GenAI metrics
altogether was the alternative; it deletes series an operator may be watching
today and is far harder to reverse than a rename, so it stays available as a
follow-up rather than being decided here. Mapping them onto the semconv memory
store family was rejected: litellm vector stores hold documents, not agent
memory records, and borrowing those names would put document admin calls into
whatever charts agent-memory operations, which is the bug this fixes.
/rag/query reaches the same recorder and is the same operation as a vector-store
search, so query and aquery map to retrieval too; leaving them would have left
the defect alive on a second retrieval surface. /rag/ingest is a write with no
semconv equivalent and no retrieval or agent confusion, so it is left for the
RAG owners to name.
Resolves LIT-4954
* fix(otel): give the streaming A2A path a call type so it labels as invoke_agent
The streaming logging object is built by hand and never runs through
update_environment_variables, the only place call_type reaches
model_call_details, so every streamed agent turn arrived at the recorder
with no call type and fell back to chat. Stamp it, and map the streaming
spelling alongside the non-streaming ones.
* feat(otel): record the GenAI duration metric on failed requests
`_record_metrics` ran only from `async_log_success_event`, so
`gen_ai.client.operation.duration` counted only the requests that worked.
Latency read off it during an incident was the latency of the surviving
traffic, and with no error dimension anywhere there was no way to build a
failure-rate panel or a success/failure split per model.
A failed call now records the same duration histogram, tagged with the
semconv `error.type` (the mapped provider exception's class name, bounded by
construction; the message stays on the span). Success attributes are
untouched, so an existing query can still isolate the old series with
`error_type=""`. The other five instruments describe a completed generation
and are skipped rather than filled with a fabricated zero: litellm hands the
failure callback no `response_obj`, so there is no usage to split and no
completion-token count, and it zeroes `response_cost` on failure. A
proxy-gate rejection (auth / rate limit) records nothing, for the same
reason it gets no span; no upstream call happened.
`error.type` is stamped after the cardinality filter, like
`gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot
strip the discriminator and silently merge failures into the success series.
Resolves LIT-4955
* fix(otel): bound the failure metric's attribute set
The failure datapoint reused the success path's full attribute set, which
carries client-supplied fields (`metadata.requester_metadata`,
`metadata.spend_logs_metadata`, the end-user id taken from the request's
`user` field) and per-request ones (the `hidden_params` blob holding the
provider's response headers). A failed request needs no provider spend, so
nothing rate-limits a caller who puts a unique value in a field they control
and mints one histogram series per request.
A failure now carries a bounded allowlist: the operation enum, provider,
request model, framework, the key/alias/team/org/user identifiers, and
`error.type`. Every entry is a fixed enum or an operator-provisioned
identifier, so the failure series count is bounded by the deployment's own
key, team and user count while the labels still answer which team on which
model is failing and how. The user email is left out as PII duplicating the
user id already on the series. The operator's `otel.attributes` filter layers
on top, so it narrows the allowlist further and never widens it.
* fix(otel): cap metric attributes so series count does not grow with traffic (#35166)
`GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object
onto every metric datapoint as one label value. That object is per-request by
construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`,
`usage_object` and the provider's `additional_headers` rate-limit counters all
move on every call. A unique label value is a new time series, and all six GenAI
instruments share those attributes, so one request minted up to six series that
would never be written to again
That is the steady-state behavior of the feature rather than an abuse case, and
it is wrong twice over. Hosted backends bill on series count, so recommending
metrics be enabled would have meant a bill proportional to traffic. And a
histogram whose every datapoint sits in its own series cannot be aggregated, so
the dashboards would have looked populated while answering nothing
Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces
the failure-only allowlist so the two paths cannot drift. The cap runs before the
operator's `otel.attributes` filter, so an operator can narrow it and never widen
it back to an unbounded label. Client-supplied and per-request metadata
(`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`,
`requester_ip_address`) is metric-ineligible and stays on the span, which already
carries it and where cardinality is free. `hidden_params` survives as a label but
carries only `model_id` and `api_base`, which are bounded by the router's own
deployment list and are the part a per-deployment panel reads
Four tests fail against the previous behavior, the load-bearing one being that
two requests differing only in per-request fields must land in one series rather
than two
* fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span
The genai and legacy mappers each spelled out every declared tool as
per-index span attributes. A request declaring hundreds of tools produced
roughly 500 attributes against the OTel SDK's default 128-attribute span
limit, which evicts oldest-first, so the canonical gen_ai.* set written
first was discarded and the span exported with only a tail of tool
schemas. Cap the family at 8 tools, shared by both vocabularies, and
carry the declared total on litellm.request.tools.declared so the
truncation is visible rather than silent.
* fix(otel): apply the tool-definition cap to the OpenInference mapper
The OpenInference vocabulary emits its own unbounded llm.tools.{idx}.*
family, which Arize and Phoenix layer on top of the default two, so those
configurations still overran the span attribute limit and evicted the
core gen_ai.* attributes. Route it through the same shared cap and cover
the layered-mapper path with a test.
* fix(otel): share one span-wide tool-definition budget across vocabularies
Capping the tool-definition family per mapper left each active vocabulary
its own allowance, and several vocabularies write to the same span. With
every vendor vocabulary configured, the three that spell tools out per
index still summed past the SDK's 128-attribute span limit, so the core
gen_ai.* set written first was evicted exactly as before: measured at 128
attributes with 7 dropped and gen_ai.request.model gone.
Reserve a quarter of the span for tool detail and split that ceiling
across the distinct tool-emitting vocabularies at mapper-resolution time,
so the family is bounded span-wide no matter how many are configured. The
same worst case now exports 90 attributes with nothing dropped.
Three defects kept LiteLLM's OTel metrics from reaching an OTLP backend.
OTEL_EXPORTER_OTLP_HEADERS is W3C Baggage encoded per the OTLP spec, so its
values are percent-encoded. litellm split the string on "," and "=" and passed
the raw value straight to the exporter, so a vendor that documents
"Authorization=Basic%20<token>" got a literal "%20" on the wire and the backend
rejected the credential. Grafana Cloud documents exactly that shape, which made
its OTLP gateway unreachable. Header parsing now delegates to the OTel SDK's own
W3C Baggage parser in liberal mode, so percent-encoded values decode and values
that were never encoded keep working. It moves from model/utils.py to
plumbing/providers.py because model/ is deliberately free of opentelemetry
imports; providers.parse_headers was already the entry point every caller used.
The OTLP metric exporters then overrode histogram temporality to delta.
Prometheus and Mimir, which back Grafana Cloud's OTLP gateway, reject delta
histograms outright: the gateway answers 400 "invalid temporality and type
combination" and drops the entire batch, so every GenAI metric was silently lost
while traces kept flowing. Backends that prefer delta still accept cumulative, so
the SDK default is the compatible choice in both directions, and the enterprise
billing exporter already relies on it.
Three GenAI instruments also carried names no convention or backend defines, so
nothing downstream could chart them. Time to first token and time per output
token take their semconv names, gen_ai.server.time_to_first_token and
gen_ai.server.time_per_output_token; the gen_ai.client.response.* spellings
litellm used are not conventions at all. Cost has no semconv instrument, so it
takes gen_ai.usage.cost, the name backends already query for spend. All three are
listed verbatim in Grafana Cloud's AI Observability integration reference, so its
prebuilt panels find them. Both engines now read the names from the shared Metric
constants rather than repeating string literals, so v1 and v2 cannot drift.
The renames are breaking for anyone charting the former names; the docs and the
release changelog carry the migration note.
litellm_provider_cache_creation_input_tokens_metric only read the
Anthropic-style top-level usage.cache_creation_input_tokens and had no
prompt_tokens_details fallback, unlike its cache-read twin. OpenAI models
that bill prompt cache writes report them only in
prompt_tokens_details.cache_write_tokens, so the counter never fired for
them. Resolve provider cache read/write tokens through a shared helper
that falls back to prompt_tokens_details.cache_write_tokens (canonical)
then cache_creation_tokens when the explicit top-level field is absent,
and give litellm_input_cache_creation_tokens_metric the same fallback for
raw usage dicts that only carry cache_write_tokens
Both failures are from this branch, not pre-existing
The component allowlist test asserts the gateway and backend route sets union to
the whole app, so any route on neither is a 404 on both pods. Allowlist the
`/management/v1/` prefix on the backend, next to the other control plane
entries, so every resource that moves under it later is covered without a
per-resource edit
The otel handler test builds its request as a SimpleNamespace carrying only
`state`. The validation handler now reads `request.url.path` to decide whether
the caller is on a surface with its own error contract, so the fake needs a url;
a real Request always has one, which is why the handler does not guard for it
The control plane branch returns early, and nothing covered that it still closes
the dangling SERVER span first, so those requests would have leaked a span
apiece. Added a case that pins it; removing the close call fails it
A failed MCP tool call aimed its error.* attributes at request_root_span(),
a ContextVar written on the ASGI request task. A stateful streamable-HTTP
session runs every message on the single task the session's initialize POST
spawned, so inside the message handler that ContextVar still holds the
initialize request's SERVER span. That span ended long ago, so the SDK
dropped every write (five 'Setting attribute on ended span' warnings plus
set_status and _add_event per failed call) and the POST that actually
failed carried no error at all. The identity attributes seeded onto the
server span went the same way.
Publish the live transport span on the ASGI scope of the request being
handled and read it back in the message handler through req_ctx.request,
the Request the streamable-HTTP transport attaches to each message. That
replaces the session-scoped field with a per-message one: a JSON-RPC
response POST deliberately skips the per-session lock, since it can arrive
while the tool call awaiting it is still in flight, so a field on the
shared auth object could be overwritten mid-call and send the tool call's
telemetry to the response's request. A scope also dies with its request
rather than holding a finished span on idle session state.
Publishing re-anchors the request root for the message so guardrail spans
and identity seeding follow, and only a transport still open for writes is
anchored or stamped: a notification POST can answer before the session task
is done, and moving dropped writes from one finished span to another is no
fix. Live capture goes from seven ended-span warnings and an unmarked
transaction to zero warnings and ERROR on the POST that carried the call.
get_sanitized_user_information_from_key copied UserAPIKeyAuth.metadata
verbatim into user_api_key_auth_metadata, so the key's callback
configuration - including the integration credentials inside callback_vars -
reached the StandardLoggingPayload every integration receives. The two other
sites that stamp key/team metadata into request metadata did the same.
Sanitize at those sources with strip_callback_config, which drops the
`logging` and `callback_settings` slots and leaves everything else (notably
`priority`, read back by the dynamic rate limiter) untouched. Those slots are
resolved from UserAPIKeyAuth during pre-call setup and never read off the
logged copies, so nothing downstream loses input.
This makes the scrub in scrub_sensitive_keys_in_metadata dead - it only
matched the string "logging" under one of the two field names and never
covered callback_settings - so it is removed.
Separately, LangSmith set the run's `inputs` to the raw StandardLoggingPayload
while redacting only `extra`, so redact_user_api_key_info left every
user_api_key_* field in inputs.metadata. Both now go through one
_redact_metadata helper, which also covers the nested requester_metadata copy.
On passthrough requests the shared guardrail plumbing still dispatches
headroom's pre_call apply_guardrail, but the passthrough translation hands it
only `texts` and no `structured_messages`, so it early-returns a no-op. The
@log_guardrail_information decorator then synthesized an "allow"/"success"
StandardLoggingGuardrailInformation entry, and the unified hook added the
guardrail to applied_guardrails, so spend logs reported the compression
guardrail as succeeded even though nothing ran.
Add a records_own_guardrail_information flag for guardrails that log their own
execution (headroom). The decorator skips the synthetic success entry for them,
and the unified hook lists such a guardrail in applied_guardrails only when it
actually recorded a run. A guardrail that owns its logging must record every
outcome it runs, so headroom now records a guardrail_failed_to_respond entry on
the fail_open path (compression attempted, service unreachable, request
forwarded uncompressed) instead of leaving it unlogged; fail_closed is still
recorded by the decorator's error path, and a genuine no-op stays not_run.
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.
Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.
Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.
Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
Under otel_v2 a single MCP tool call surfaced in APM as two disconnected
traces joined only by a span link: the HTTP transport transaction
POST /{mcp_server_name}/mcp and the tools/call span carrying
error.type=MCPToolResultError. resolve_mcp_span_context parented the MCP
span to the W3C trace context the client propagates in params._meta
(SEP-414) and recorded the transport as a link, so with no traceparent
propagated (the common case today, including MCP Inspector) the span
started its own root trace.
Nest the MCP span under the transport span when nothing is propagated, so
the call stays in one trace; the propagated-context path is unchanged and
still parents to the remote context and links the transport per the OTel
GenAI MCP semconv.
The transport has to be resolved per message rather than read from the
request-root ContextVar. A stateful streamable-HTTP session runs every
message on the single task the session's initialize POST spawned, so that
ContextVar is frozen at initialize inside the handler: live capture on
staging showed the tools/call span linking the initialize POST rather than
the POST that carried it, and nesting on that anchor would hang every tool
call of a session off the first request's already-ended span. The gateway
now resolves the current request's span on the ASGI task and carries it to
the handler on the authenticated-user object, the same way per-request auth
already crosses that boundary.
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails
Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.
This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.
The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.
* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails
post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.
This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.
The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.
* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes
Addresses review feedback on the run_in_parallel opt-in.
asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.
The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.
* fix(guardrails): record lifecycle logs for every concurrently-run guardrail
The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.
Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.
* test(guardrails): declare run_in_parallel on post_call guardrail mocks
The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.
* fix(guardrails): harden run_in_parallel reads and address review feedback
Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes#33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in c698b88686 to match upstream litellm_internal_staging.
Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update.
* fix: scan system messages as indirect prompt injection in Singulr guardrail
* chore: restore lint budget files to upstream baseline
* fix: resolve ruff UP006 and I001 violations in singulr guardrail
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* resolve review comments on Singulr guardrail
* fix: scan tool call results as indirect prompt injection in Singulr guardrail
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* minor
* formating fix
* refactor: shift extraction logic to singulr side
* refactor:keep precall hook only
* fix:formatting
* fix:linting
* improve config description
* Trigger CI
* fix
* fix:field description
* fix:errors due to change in field names
* style: apply ruff line-wrap formatting to singulr guardrail
* fix:exception
* fix:formatting
* fix playground
* improved
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix
* fix ci issues
* remove uv.lock from pr
* fix
* fix:resolved comments
* chore: trigger CI
* remove uv.lock
* fix
* fix linting
* fix linting
* fix linting
* remove doc strings
* remove test fixes
* chore: retrigger CI
* change in singulr api contract
* remove some ut
* send litellm call_id to singulr
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Fix non-conformant UUIDv7 generation in native Opik integration (#31294)
create_uuid7() encoded the timestamp in units of 16 seconds instead of
milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's
backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067);
the bad encoding decoded to ~year 2201 and every trace/span batch was
rejected with HTTP 400.
Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms),
using the standard library only so no new dependency is added. Add unit
tests covering UUIDv7 validity and millisecond timestamp encoding.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(proxy): expose uvicorn concurrency limit (#33077)
Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and
LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and
accepted connections and returns HTTP 503 once the configured limit is reached.
Reject non-positive limits at CLI parse time and only add the setting to the
uvicorn startup arguments. Because idle connections also consume capacity,
deployments should use upstream connection/header timeouts and per-client
connection limits.
* test: reorder test_utils tail to keep the daily merge conflict-free (#33788)
The daily OSS branch and litellm_internal_staging each appended an
independent test block at the very end of tests/test_litellm/test_utils.py,
so merging the two collides on that shared end-of-file position even though
the additions are unrelated (this branch adds the vertex embedding
encoding-format tests; staging adds the per-model prompt-cache-minimum
tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class
above test_gemini_image_models_do_not_support_reasoning, which both branches
share, gives the two additions different anchors, so git applies both
without a conflict and without pulling staging into this branch. Pure
reorder; no test bodies change
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
_request_has_cache_control only looked at messages and system, so a client that
marks cache_control on tools alone did not suppress auto-injection. Tool
breakpoints count toward the provider's four-block limit, so three of them plus
the two injected here is five, which Anthropic rejects. Thread tools through
both entry points and treat a client-marked tool as the stand-down signal it
already is for messages and system.
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials
Key-scoped langfuse_otel callbacks only injected Authorization headers into the
init-time exporter, so a proxy without global LANGFUSE_* env vars kept its
fallback exporter and never exported traces to Langfuse. Dynamic params now
build a full per-request OTLP config (endpoint from the key's langfuse_host,
otlp_http, basic auth from the key's credentials).
Resolves LIT-3976
* fix(otel): log dynamic config endpoint in span processor debug output
* fix(otel): redact authorization headers in exporter debug logs
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are
now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and
LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on
without a config file. An unsupported ttl falls back to the provider default
rather than reaching the provider verbatim
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
