The request path used get_end_user_object, which falls back to a database
lookup on a cache miss. Read the LiteLLM_EndUserTable row auth already cached
instead, with the default budget already attached, and leave misses to the
periodic refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The metric attribute filter now removes every attribute whose value is
None, and the content and inference-details events pass their attributes
through drop_none before emitting, so a call with no provider label or no
model name never hands the OTLP exporter a NoneType attribute. This closes
the gen_ai.request.model report on #36759 the same way the gen_ai.system
one was closed, and the regression tests cover both keys.
The pre-call hook passed the proxy's whole per-request metadata dict into the
request identity, so proxy-owned siblings such as requester_ip_address were
promoted alongside the caller's keys. Only the requester_metadata mapping is
read now, keyed under its wrapper, which keeps the default allowlist behaviour
unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mirror the key, team, user and org budget gauges for customer objects with
litellm_remaining_customer_budget_metric, litellm_customer_max_budget_metric
and litellm_customer_budget_remaining_hours_metric. The gauges carry only the
end_user label, are emitted after each request and from the startup budget
refresh for every customer with a budget attached, and reuse the
enable_end_user_cost_tracking_prometheus_only opt-in and end_user series caps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Strip only the proxy's requester_metadata. wrapper from an allowlisted key so
requester_metadata.trace_id lands as litellm.metadata.trace_id while other
dotted keys keep their full path and cannot collide on a shared leaf name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
baggage_metadata_keys entries such as requester_metadata.trace_id now resolve the caller's nested metadata.trace_id and stamp it on the LLM-call span as litellm.metadata.trace_id, in both the OTEL v2 logger and the legacy OpenTelemetry callback. Nested metadata mappings are flattened to dotted paths, only allowlisted leaves are promoted, and the requester_metadata blob itself is never promoted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Invalid or unknown virtual keys were filtered out of the proxy failed and total
request counters entirely. Count them with hashed_api_key unset so caller-chosen
key strings cannot create unbounded label series, and normalize the request
route on the auth failure path so dynamic path ids do not leak into the route
label either.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pre-call limiters reject before a deployment is attached to request_data, so
the failure hook could not resolve api_provider for router aliases and emitted
api_provider="None" on litellm_proxy_failed_requests_metric_total and
litellm_proxy_total_requests_metric_total. Fall back to the provider the
limiter already resolved onto RateLimitError.llm_provider, keeping request
data as the first source and ignoring the proxy placeholder.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): log mask when a guardrail adds request keys
_inputs_were_modified only compared keys present in the pre-hook baseline, so a
guardrail that injected a new key such as tools was logged as allow. Compare over
the union of both key sets, and narrow the pre_call return value to the same
prompt-bearing keys the baseline holds so passthrough stays allow.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): snapshot apply_guardrail inputs before the hook mutates them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline
Auth read each object with its own Redis GET and, on a miss, its own DB
query, then the admission spend counters with one GET each. The prefetch
warms every entry the checks read with one MGET, one raw query for the
Redis misses and one pipeline write, and a per-request batch serves the
spend counter reads from one MGET. The per-object getters stay the
readers and the fallback, so enforcement does not depend on the prefetch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): keep prefetch and spend batch collections immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): prefetch referenced auth objects only after the key's model access check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): give the prefetch-ordering test's patches their test-quality reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read NULL nested permission and budget lists as [] in the prefetch join
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): batch post-call spend counter reads and carry budget state through the request
Post-call warm checks, reservation reads and reconcile reads for one request now go through a task-local spend counter batch: one MGET answers every counter, successful increments write their result back into the batch so no second Redis read follows, and invalidation forgets the key. RedisCache.async_increment sends INCRBYFLOAT and its TTL command in one pipeline round trip.
Auth pins frozen team, user and org budget snapshots on UserAPIKeyAuth, the pre-call setup writes them into the request metadata, and Prometheus reads them back instead of calling get_key_object, get_team_object, get_user_object and get_org_object on the response path. The getters stay as the fallback for requests that carried nothing (custom auth, unauthenticated routes, skipped checks).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): reconcile the budget reservation and the post-call warm checks from one MGET and one pipeline
A scope opened inside an open spend counter batch binds into it instead of starting its own, so the reservation reconcile and the post-call warm checks share the request's single MGET. The reconcile reads every reserved counter concurrently, sends the consistent adjustments in one INCRBYFLOAT+EXPIRE pipeline and settles a flushed or reseeded counter on its own afterwards, keeping the pre-call resize fail-closed. PendingSpendIncrement moves to spend_counter_batch so budget_reservation can build a pipeline without importing a private name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): drop the dataclass import left behind by the PendingSpendIncrement move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): import Self from typing_extensions so the proxy imports on Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use a neutral organization alias in the carried budget state tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover recorded and forgotten spend counter values in the request batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): type the reservation entry carried through reconcile adjustments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): sync logging_obj guardrail info on every record so post_call entries survive streamed chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): hoist regression test imports to module scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): name Langfuse traces from the langfuse_trace_name header or metadata.trace_name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): type the named-request helper in the Langfuse logger tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): stop logging the request payload as guardrail_response on pre_call hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): snapshot pre_call request before the hook so in-place edits log as mask
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): log non-mapping pre_call hook results as mask instead of raising
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): treat legacy functions and tool_choice edits as mask in pre_call logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): log a pre_call rejection string as is instead of "mask"
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
A value length limit clips the input.value blob, so the per-index keys are the
only untruncated copy of a message. Indexing the leading prompt messages dropped
the live user turn from every span attribute on long conversations. Keep message
0 and the most recent turns under the same span-wide budget, original indices
preserved, reply reservation unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(datadog_llm_obs): keep tool call and result structure under redaction and emit tool output tokens
Under datadog_llm_observability_params.turn_off_message_logging the span kept only one role plus "redacted-by-litellm" per message, so Datadog showed Tool Call 0, Tool Result 0 and no tool output token data. The shared CustomLogger hook collapsed the messages before the callback ran, and the Datadog redaction then dropped tool_calls and tool_results.
The Datadog callback now opts out of the shared message collapse (redacts_messages_itself) and redacts its own normalized messages, keeping roles, tool names, ids and types while replacing content, arguments and results. Tool result tokens are counted with litellm.token_counter before redaction and shipped as the tool_output_tokens metric. Other callbacks keep the inherited behavior.
Resolves LIT-7545
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(datadog_llm_obs): drop explanatory docstrings from the redaction change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for the classifier descriptions changed in #40655
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
