* fix(otel v2): restore the Datadog auth span and the last-wins callback merge
Move @tracer.wrap() back onto user_api_key_auth so USE_DDTRACE=true emits the
auth span again, and let a failure entry's callback_vars take part in the
destination merge so the resolver picks the same account the runtime parser does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): drop docstrings from the two regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun proxy-infra after the flaky test_check_migration process-tree test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_sentinel): pin batch_size as a per-request bound under concurrent events
Adds a regression test to the mapped Azure Sentinel test file for the concurrency scenario from LIT-6920: 40 records logged concurrently at batch_size=5 while each ingestion request is still in flight. Asserts no request carries more than batch_size records, every record arrives exactly once in order, and the queue is empty afterwards. Runs for both the standard log queue and the audit log queue.
The test fails on the tree before #39880 (whole shared queue serialized per threshold send, then cleared) and passes on current staging. It is independent of the size-split coverage that #39880 added for LIT-5899.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_sentinel): gate the first send on events so later records provably arrive while it is in flight
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel v2): send a key's or team's whole trace to its own destination
A key or team that configures its own Langfuse, Arize, Weave or New Relic
credentials used to get a single detached span in its account while the rest of
the request trace stayed on the operator's backend, so neither side held a
complete trace. Resolve the destination during auth, forward every span of the
request to it, and hold the same request back from the operator's exporter for
that backend, so the tenant gets the tree the operator would have seen and the
operator gets nothing for that request.
Also let a credential-mandatory preset build without the operator's own env
credentials. Without that, a proxy whose teams each bring their own account fell
back to the legacy integration and never ran a line of the v2 path.
* fix(otel v2): validate tenant destinations and match each backend's own endpoint
Review round on the tenant destination routing.
- A key/team Langfuse host is user-supplied input, so it goes through the
proxy's SSRF guard. A private address is refused, the operator keeps the
trace, and the warning names user_url_allowed_hosts. The operator's own
LANGFUSE_HOST is not checked.
- Arize and Weave destinations now resolve their endpoint and transport
through the backend's own config, so an ARIZE_HTTP_ENDPOINT collector and
a self-hosted WANDB_HOST are honoured instead of the cloud default.
- A half-configured backend no longer resolves: several dynamic header
builders gate each credential separately, so an api key with no space id
produced a non-empty but unusable header set that suppressed the
operator's exporter.
- A callback_type of "failure" no longer takes over the trace. The
destination is resolved during auth, before the outcome is known.
- The fan-out cache evicts without shutting the processor down, matching
ArizePhoenixLogger: a concurrent on_end may still hold it.
- The stdout placeholder is identified by what it does rather than by
equality with an import-time default, so an operator's OTEL_EXPORTER_OTLP_*
collector survives the credential-less path.
* refactor(otel v2): reuse the proxy's own destination allowlist for tenant hosts
A tenant-supplied Langfuse host is the same threat as a URL-valued `model`, so
it now goes through `is_url_destination_allowed_by_host` against
`provider_url_destination_allowed_hosts` instead of a second, DNS-based check
of its own. The DNS lookup would have blocked the asyncio auth path on a
hostname the caller picked, and its cached verdicts could blackhole a real host
after one resolver blip.
Evicting a destination processor now retires it to drain rather than shutting it
down, since `on_end` hands a processor back and exports outside the lock. The
retirees are capped so they cannot accumulate a thread each.
`credential_gated_exporters` tells the synthesized stdout placeholder from a
real exporter by transport rather than by the literal kind `console`, so an
unrecognized kind is not mistaken for a configured collector, and an exporter
the operator did configure survives. That also stops a weave test's env writes
from making this look like a real OTLP exporter later in the same CI worker.
* fix(otel v2): read the tenant's stored callback config the way the sibling parser does
Three divergences between the destination resolver and
`convert_key_logging_metadata_to_callback`, which read the same stored config:
- A key whose callbacks are disabled stores an empty list, and `or` treated that
as "the key configured nothing", so the request inherited the team's
destination. The sibling parser treats an empty list as configured.
- Two entries naming one backend now merge their `callback_vars` last-wins,
matching the sibling, instead of the resolver taking the first entry and the
per-request tracer routing taking the last.
- `credential_gated_exporters` dropped any exporter whose kind had no transport,
which also dropped an `in_memory` exporter the operator asked for. The
placeholder is the spec with every field still at its default, so that is what
the predicate now says.
Arize's `allow_missing_credentials` branch was unreachable: `get_arize_config`
resolves every credential with `os.environ.get` and always supplies an endpoint,
so it never raises. Dropped it and corrected the protocol docstring.
* fix(otel v2): keep the destination merge immutable
The per-backend var merge seeded a plain dict and the gated exporter list a plain
list, both of which the LIT budget counts. Wrap the merge in MappingProxyType and
hand the exporters back as a tuple.
* fix(otel v2): scope the fan-out to its own backend and close shed processors off the export path
Three problems in the fan-out, two of them in the eviction added last round:
- Every v2 logger carries its own provider and emits its own copy of a gen-AI
span, so a proxy running two of them handed the tenant the same model call
twice. A provider now forwards only destinations for the backend it speaks
for; the tenant's own backend always has a logger, since naming it in the key
or team config is what builds one. Reproduced live against a self-hosted
Langfuse on an arize-only proxy and on the bare `otel` callback.
- Eviction could close a processor another thread was still exporting through,
which drops that span. Exports are now counted, and a retired processor is
closed only once its count reaches zero.
- That close ran inside `on_end`, where `shutdown` flushes over the network, so
one unreachable tenant collector stalled every other tenant's spans. It now
runs on a short-lived thread, which also retires the retiree cap: a retiree
drains as soon as its export finishes.
* fix(otel v2): deliver tenant destinations from the published global provider
Scoping the fan-out by callback name in the previous commit left every backend
that is not the canonical logger with a one-span trace: only the published
global provider sees the FastAPI server span, the auth span and the post-call
database spans, so an arize-only proxy handed a team's Langfuse just the model
call. Attach the fan-out once, to that provider, and let it forward every
destination.
An overridden backend now skips per-request tracer routing outright rather than
only clearing its credential headers, since a key or team otel_service_name was
still enough to detach the model call onto a second provider. The destination
carries that service name as a resource attribute instead.
Shed processors drain on a two-thread pool rather than a thread each, so a
tenant cycling its destination config cannot spawn threads as fast as it sends
requests.
* fix(otel v2): drain shed destination processors on daemon workers
A ThreadPoolExecutor joins its workers at interpreter exit, so one unreachable
tenant collector would hold the whole proxy open for its export timeout on the
way down. Two long-lived daemon workers off a queue keep the thread count
bounded without blocking shutdown.
* fix(otel v2): give the fan-out its own drain pool instead of a lazy singleton
functools.lru_cache does not hold a lock across the call it caches, so
concurrent first evictions each finish building a queue and start its
workers, and every queue but the winner is abandoned with two daemon
threads blocked on it forever.
* fix(otel v2): close no destination processor under a span still in flight
The fan-out now refuses new work once shutdown starts and waits out the
spans already being forwarded, so teardown neither drops a trace mid-forward
nor hands the next caller an exporter nothing will ever close. The wait is
bounded so a dead collector cannot hold the proxy open.
* fix(otel v2): retire the drain workers with the fan-out that started them
A proxy that rebuilds its telemetry builds another fan-out, so workers that
outlive the one that started them are two more threads per reload. Shutdown
now retires them once everything queued is closed, and a processor shed
afterwards is closed inline rather than queued to nobody.
* fix(otel v2): guard the fan-out's closed state with the lock that gates it
An Event read on its own leaves room for shutdown to run in the gap. A cache
miss then inserted a live exporter into a map that had been cleared, and a
shed processor landed behind sentinels every drain worker had exited on.
The drain pool takes its queue by injection so both interleavings are
reachable from a test without patching.
* feat(otel v2): let a tenant destination export alongside the operator's own
Override stays the default: a key or team destination replaces the operator's
exporter for that backend. Operators running one org-wide backend across every
team set litellm_settings.otel_tenant_destination_mode to additive, and the
same trace lands in both places. A team that names the operator's own project
is still written once, since the fan-out skips a destination the operator's
exporter is already sending that span to.
* fix(otel v2): let a straggling export close its own destination processor
Shutdown waits out the exports in flight, but the wait has to be bounded or a
tenant collector that stops answering holds the proxy open on the way down.
Past the bound it closed everything anyway, which is the case it was written to
avoid: a processor closed under the span it is carrying loses that span.
Keep the bound and retire the stragglers instead. The thread still exporting one
closes it through the drain as soon as its export returns, so teardown stays
bounded and no span is dropped mid-forward.
* fix(otel v2): identify a destination account by its credentials, not its header names
Under additive the fan-out skips a destination the operator's own exporter
already writes to, so the same account is not written twice. It compared header
names as well as values, and one account answers to more than one spelling:
the operator's Arize exporter sends space_id where a team destination sends
arize-space-id, so every span landed in the operator's own space twice.
The credentials are the identity. Compare those and leave the spelling to each
backend.
* fix(otel v2): keep the credential's role in a destination's account identity
Comparing values alone folds two accounts together whenever they hold the same
strings in different roles, and the second team would then get no trace at all.
Compare the credential under a normalized name instead, and fold the one alias
that actually exists: Arize's space_id and arize-space-id.
* fix(otel v2): build one destination processor per destination, not per racing span
Building outside the cache lock meant a cold cache met by a burst of concurrent
requests constructed an exporter per thread, kept one, and handed the rest to the
drain, so a batch worker and a connection pool per losing thread sat in a queue
two workers service.
Build under the lock that reads the cache. Opening an exporter connects to
nothing, so the lock is held for a constructor, once per destination, and the
race it was avoiding stops existing.
* fix(otel v2): bound the teardown that closes a destination, not the one that never blocks
The five-second bound guarded the wait for spans still inside on_end, but a
batching processor's on_end only queues the span and returns, so that counter is
empty and the bound engaged against nothing. The blocking half was the serial
close, which flushes over the network and joins the SDK's own worker thread with
no timeout of its own, so a single tenant collector that answers and never
finishes held process teardown open for as long as it liked.
Hand every close to the drain, whose workers are daemons, and give the whole
teardown one deadline.
* fix(otel v2): preserve operator spans on destination failure
* fix(otel v2): anchor destinations off the published provider, refuse headerless tenant transports
set_tracer_provider keeps the first provider it is handed, so a process whose
OTel global was claimed before the proxy published (auto-instrumentation, a
legacy logger) had no fan-out on the global and auth anchored no destination.
Auth now reads the fan-out off the registered logger's own provider.
A destination whose protocol maps to a headerless exporter kind is no longer
buildable: the console fallback would drop the tenant's credentials and print
the spans to stdout while the operator's exporter stood down for them.
* fix(otel): anchor tenant fan-out to the published provider
A legacy v1 logger can occupy proxy_server.open_telemetry_logger, in which case
the proxy publishes with registered=None and the fan-out lands on a v2 logger
taken from _in_memory_loggers. Reading the registered slot found no v2 logger
and the OTel global belonged to v1, so auth refused every tenant destination.
* fix(otel): preserve registered provider fallback
* test(otel): cover pre-publish provider fallback
* fix(otel): attach fan-out on fallback provider
* fix(otel): serialize first fan-out attach
* fix(otel): keep the operator's database endpoint out of tenant traces
A database span forwarded to a key or team destination carried the proxy's own
Postgres host, port and schema, and on failure the Prisma error text naming them.
The fan-out now hands tenants a view of each database span without those keys,
its events or its status text, while the operator's own copy is untouched and
model endpoints such as server.address on the LLM span still travel
* fix(otel): keep relabelled spans in the fan-out and honour disabled callbacks for destinations
A key or team otel_service_name used to move a backend's span onto a second
provider even when another backend had a destination, so the fan-out never saw
the model call and the tenant's trace lost it. A service name alone now stays on
the published provider whenever the request has a destination; credential and
project routing to a tenant's own account is unchanged
Destinations now skip a backend the request disabled dynamically, reading the
x-litellm-disable-callbacks header and the key's litellm_disabled_callbacks with
the same precedence and premium gate dispatch applies, so a disabled backend is
neither delivered to nor withheld from the operator
* test(otel): project routing survives a sibling backend destination
* docs(otel): state why a disabled backend still routes its own span
* fix(otel): keep a degraded backend's spans off a collector another v2 logger already serves
* test(otel): a credentialed preset beside another v2 logger keeps every exporter
* fix(otel): keep credentialless fallback on base path
* test(otel): cover legacy callback carrier rejection
* fix(otel): preserve valid exporter beside gated preset
* fix(otel): avoid console export without operator destination
* refactor(otel): share the console placeholder check with the presets
* fix(otel): bound shed destination processors waiting on a dead collector
* fix(otel): preserve explicit console exporters
* fix(otel): avoid mutable field-set construction
* fix(otel): close drain saturation race
* fix(otel): drop captured request headers from tenant spans
* test(otel v2): give the newrelic dispatch tests operator credentials, since a credential-less preset now falls back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): rebuild an anchored destination's processor past drain saturation
A destination deliverable() accepted at auth can be evicted by other tenants' auths
before its request's spans end, and that eviction is what tips the drain over. The
saturation gate then refused the rebuild at on_end, and with the operator's exporter
already stood down for that backend the span went nowhere. The gate now applies only
while a request decides whether to anchor
* fix(otel): hold destination eviction while the drain is saturated
An anchored destination evicted by other tenants' auths is rebuilt on its
next span, and that rebuild evicted another anchored one, so with more
destinations in flight than the cache holds every span cost one more
processor, one more batch thread and one more close queued behind a collector
that never answers. Eviction now holds while the drain is saturated, so the
cache keeps one entry per destination in flight and trims back to its cap on
the next hit or build once the drain has room
* fix(otel): keep the proxy's own error text out of tenant traces
A tenant destination received every span the request produced, error text
included, so a Prisma failure during auth handed a team admin's collector the
operator's Postgres endpoint, and the exception event on any failed span carried
a stack trace naming the proxy's install paths.
Spans the tenant's own call produced (the model call, MCP, guardrails) keep their
error text. Every other span keeps the failure without the prose: its type, its
provider error code and its status code, with the message, the events and the
status description dropped. Stack traces come off every span, attribute and event
alike.
A destination's resource attributes now merge onto the span's resource instead of
rebuilding one per span, which was re-running resource detection on every export.
* fix(otel): redact tenant URL query parameters
* fix(otel): close final tenant routing gaps
* fix(otel): refresh destinations for stateful MCP messages
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): add http/json export protocol for OTel v2 traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/json was accepted but routed to the protobuf
OTLP/HTTP exporter, so collectors that only decode JSON rejected every batch.
Route it to an OTLP/JSON span exporter that reuses the SDK HTTP transport and
expose the protocol as a select field on the OpenTelemetry callback in the
admin UI.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): walk the fixed OTLP shape instead of recursing when hex-encoding ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): map stored callback variables onto their form fields when editing a callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add traces_endpoint (env OTEL_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, per-exporter key traces_endpoint, Admin UI field) as a complete OTLP/HTTP trace URL used verbatim, so collectors that do not serve /v1/traces can receive traces. endpoint keeps its existing base-URL + signal-path normalization.
Resolves LIT-7218
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued
Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.
Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound
* fix(azure_sentinel): retry undelivered records on the flush timer only
Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.
A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.
* fix(azure_sentinel): decide threshold sends under the flush lock
Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.
* test(azure_sentinel): cover successful threshold waiters
* fix(azure_sentinel): preserve cancelled batches for retry
* fix(azure_sentinel): requeue only the undelivered part of a cancelled split
A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush
The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch
* fix(azure_sentinel): drop batches a permanent rejection will never accept
A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule
The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it
Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error
The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry
* chore(logging): document mutable queue ownership
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
* fix(datadog): preserve non-413 retry behavior
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
* fix(batch_utils): requeue by default and let Sentinel opt into dropping
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections
Also drops an explicit return None the strict ruff gate flags in the test helper
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(cloudzero): preserve late resource tags
* style(cloudzero): remove redundant test comment
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
* feat(otel): stamp litellm.request.route on the LLM call span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop redundant comment on REQUEST_ROUTE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): Final-annotate route test locals, drop field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read litellm.request.route off the server span
The LLM call span took the auth-normalized literal path from logging
metadata, which disagrees with the SERVER span wherever FastAPI matched a
template: on /engines/{model:path}/chat/completions the LLM span spelled the
model name while http.route carried the template, so the two spans grouped
into different buckets and the PR's premise did not hold.
Read the value off the span that already holds it. The request's root SERVER
span is anchored per request for parenting, and its attributes stay readable
after it ends, so request_root_http_route() answers from the async close
callback with the same http.route the SERVER span exports: the route template
on a normal route, the literal path where the passthrough hook rewrote it, and
the mount point on an MCP call. Nothing has to re-derive any of that, so the
two spans cannot drift apart.
The route the proxy recorded at auth stays as the backstop for a deployment
whose FastAPI instrumentation never mounted, where there is no server span to
disagree with. Off the proxy the attribute is omitted rather than empty.
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yucheng He <yucheng@berri.ai>
Suppression state moves out of request metadata into a request-scoped ContextVar.
refresh_proxy_server_request_body_snapshot copies metadata into
proxy_server_request.body, which deployments persist to spend logs, so the marker
naming each suppressed guardrail was readable by the caller whose request produced
it. Recovering it was enough to replay {token}:{name} for any CustomGuardrail and
switch off a PII or content-filter guardrail, since the check never verified the
named guardrail was a compression one. Nothing is read from metadata now, so there
is no marker to forge and the per-process token is no longer needed.
Routing-side compression reads the live messages instead of a pre-guardrail copy.
arm_pre_call runs before the pre-call hook, so its snapshot held the prompt as it
was before any masking guardrail rewrote it, and messages_for_routing handed that
to a compression guardrail which POSTs it to an external service. Masked content
left the proxy anyway. The cost is one combination: when the model hop compressed
and the hops differ, routing now classifies on the compressed text, since no
uncompressed copy survives that a masking guardrail has already seen.
policy_for_model no longer falls back to a marker scoped to tags the request does
not carry, which applied an 'eu' policy to a 'us' request on config order alone.
Each fix carries a regression test; all three fail when the fix is reverted.
* fix(shadow_eval): tell a tool-call shadow reply apart from an empty one
Both arrive at the attempt row as the same 'shadow router returned an empty
response', because _chat_final_text returns empty for a tool-final turn by
design and for a reply that genuinely carried no text. Those are different
things: an arm that chose a tool where the real model wrote prose is a
divergence a text judge cannot score, and the sampling side already drops the
real arm's tool-final turns for exactly that reason, so the shadow side reads
as a fault where the real side reads as a filter. A job that is almost all
'empty response' gives no way to tell a tool-happy arm from a broken one.
The error now names which of the two happened, and carries the finish_reason
and the routed model so the row says what the arm was doing. Every varying
part sits behind the first semicolon: operators read these by grouping on the
error text, and interpolating the model into the leading sentence would make
each row its own group.
The outcome stays 'error'. Whether a tool-call reply should instead be its own
non-judged outcome, excluded from the loss rate the way the real arm's
tool-final turns already are, needs the four aggregation predicates that spell
judged as outcome != 'error' rewritten, and a decision on how to surface the
new bucket. That is a separate change.
* fix(shadow_eval): read the tool name of a custom tool call
A custom tool call carries its name under custom.name with no function key,
so every one of them reported as tool=unnamed.
* feat(shadow_eval): judge tool calls instead of dropping the turn
A turn where either arm called a tool was discarded before it could be
compared: the real arm's at sampling, the shadow arm's as an error row. On
agentic traffic that is most of the traffic, so a job set to sample 10% was
sampling 10% of the prose-only slice. Tool calls now serialize to text on
every surface and are judged like any other response, and the judge is told
a tool call is not a defect so it scores the choice rather than the shape.
* feat(shadow_eval): show the judge what tools were available
Both arms were offered the same tools, but the judge only ever saw the
chosen call in isolation, with no way to tell whether a better tool existed
or the arguments matched what the tool expects. Threads the request's tool
definitions (name and description only) into the judge prompt, capped and
omitted entirely on turns that offered none.
* fix(shadow_eval): read a custom tool definition's name from custom, not function
A chat-completions custom tool definition nests name and description under
custom, mirroring how a custom tool call nests them (openai.types.chat.
ChatCompletionCustomToolParam). Reading only function rendered every one as
unnamed, telling the judge nothing about what it was.
* fix(datadog_llm_obs): keep the guardrail audit record under message redaction
Redaction nulled `guardrail_information` on the span whole, so an operator
running `turn_off_message_logging` (or a caller sending
`x-litellm-enable-message-redaction`) lost the record of which guardrails ran,
what they returned, and what they masked. Four of the record's fields can quote
the prompt; the rest report what the guardrail decided without reproducing it.
Replace only those four, the way
`_sanitize_guardrail_information_for_spend_logs` already does for spend logs,
and declare the field list once in `litellm/types/utils.py` so both readers
share it.
* fix(datadog_llm_obs): keep a lone guardrail record, and test through the span
Review round 1.
A guardrail that writes the metadata key itself leaves a single record where
the type says list, which Prometheus already normalizes at
`_guardrail_overhead_seconds`. Redaction dropped that shape and the latency
extraction raised on it, so the span was lost outright. Normalize once and use
it in both places.
The new tests now drive `create_llm_obs_payload` instead of reading the module's
private helpers and the record's declared field names.
The parser message alone cannot separate a judge that answered with nothing
from one truncated mid-object, and the two want opposite fixes. Records the
reply's shape, never its text, since no attempt row carries sampled content.
- Suppression markers now carry the per-process token `_pre_call_marker`
already uses, so a caller cannot switch off an always-on PII, content-filter
or compression guardrail by naming it in its own request metadata.
- Routing set to "none" with the model side compressed now classifies on the
pre-compression snapshot instead of the model-side guardrail's output.
- Both the proxy's pre-call arming and the router's routing hook resolve the
policy through one tag-aware `policy_for_model`, so an alias with several
tag-scoped markers can no longer suppress one marker's guardrail and then
route under another marker's policy.
- The pre-compression snapshot moved from request metadata to a ContextVar:
`refresh_proxy_server_request_body_snapshot` copies metadata into
`proxy_server_request.body`, which deployments persist, and the snapshot
holds the prompt as it was before any masking guardrail rewrote it.
- The compression selector lists Compresr guardrails too, not just Headroom.
An auto router marker deployment can now set auto_router_routing_compression
and auto_router_model_compression in its litellm_params, naming the
compression guardrail each hop should use (or "none" for no compression on
that hop). Neither key set means the request's own compression guardrails
keep applying to both hops unchanged.
Backend: Router.async_pre_routing_hook resolves the marker's policy and
compresses a copy of the messages for the routing decision only when the
policy differs from what the model call already got; when both hops share
the same compression, it reuses what the ordinary pre-call guardrail
pipeline already produced instead of compressing twice. The proxy layer
suppresses every other compression guardrail once a policy is engaged and
arms the model-side guardrail even when it is not default_on.
UI: the auto router's Detailed Configuration gains an Advanced: Compression
section with a routing-decision selector and a same/different toggle for
the model call, matching the same/different address pattern.
The prior commit claimed claude-sonnet-5 reasons invisibly by default and eats
the judge's budget regardless of what the call asks for. Verified against a
live proxy: with no thinking param (what _call_judge sends today), forced
tool-choice json_mode, native structured output, and even an explicit
thinking=adaptive, the model returned 0 reasoning tokens and a clean compact
verdict every time, on prompts up to several thousand characters.
The real mechanism only shows up with an elevated reasoning_effort or
output_config.effort on the request, which happens when the judge_model
deployment is configured with one, e.g. an admin pointing the judge at their
best reasoning model. Reproduced directly: reasoning_effort=max, 300-token
cap, real Anthropic reply came back finish_reason=length, content=None, 299
of 300 tokens spent on reasoning. Same request at 4096 returned a valid
verdict. This is a narrower, verified claim than the one it replaces.
The cap covers reasoning tokens as well as the verdict, and the models people
pick as judges reason before answering whether the call asks them to or not:
Anthropic's 5 family thinks adaptively and cannot be told not to, so the
reasoning bills against max_tokens with nothing in the request to opt out.
At 1500 the reasoning consumed the budget and the reply arrived empty or cut
off mid-object, which the attempt recorded as an unparseable judge verdict
rather than a result. Headroom costs nothing: max_tokens is a ceiling and only
generated tokens bill, so the only movement is that judge calls which used to
bill their full budget and return nothing now return a verdict.
Deliberately not passing reasoning_effort to bound the reasoning instead:
is_thinking_enabled treats any reasoning_effort as thinking-enabled, which
drops the forced tool_choice that json_mode relies on and turns thinking on
with a 1024-token floor for judges that were not reasoning at all.
* fix(azure): restrict the storage credential chain to deployment identities
The keyless Azure Storage path walks the full DefaultAzureCredential chain, so a
proxy with no storage service principal authenticates as whichever identity the
host happens to carry: an operator's az login on a workstation, or the
AZURE_CLIENT_ID/AZURE_CLIENT_SECRET service principal set for Azure OpenAI.
Neither is the identity granted Storage Blob Data Contributor.
Narrow the chain to workload identity and managed identity, the two credentials
a deployment legitimately holds. Azure OpenAI, Postgres IAM auth and the other
callers of get_azure_ad_token_provider keep the full chain.
* test(azure): read the credential chain off the mock instead of an accumulator
* chore: drop a stray launch traceback committed at the repo root
* fix(azure): let the storage chain reach a system assigned managed identity
DefaultAzureCredential keeps one managed identity link and pins it to
AZURE_CLIENT_ID, so a host that sets that variable for Azure OpenAI and runs as
a system assigned identity never got asked for a storage token. Build the chain
from the three credentials a deployment can carry instead of subtracting the
ones it cannot.
One unreachable vector store used to wipe out every store's context on a
chat completion carrying vector_store_ids: the search raised, the blanket
handler returned the original messages, and the request answered with no
retrieved context at all. Each store's search now has its own handler that
warns with the vector store id and moves on to the next store.
The same loop appended every store's results to the original messages
instead of the running copy, so with two healthy stores only the last one
reached the model. It now chains through modified_messages.
The Router is injected through a ProxyRuntime protocol instead of an
in-function litellm.proxy.proxy_server import, so the hook's routing can
be driven in tests without touching proxy globals.
* feat(datadog_llm_obs): cost tag dimensions, router decision fields, reasoning token metric, redaction gating
* test(datadog_llm_obs): satisfy test quality gate
* fix: forward integer parent_id as its string form
* fix(datadog): sanitize redacted message roles
* fix(datadog): keep the A2A agent role on redacted spans
* fix(datadog): merge current staging budget
* style(datadog): format redaction tests
* fix(datadog): handle malformed redacted roles
* test(datadog): put the test quality suppression on the reported line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(azure): support credential chain for storage
* test(azure): clarify credential seam suppressions
* fix(azure): read chain tokens in a worker thread
The credential chain walk (IMDS probe, CLI subprocess) is blocking I/O,
so reading the provider inline in async set_valid_azure_ad_token stalls
every request on the worker's event loop
