mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
31 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
71044bf5ea
|
feat(otel): attribute Prisma database spans to PostgreSQL instead of localhost (#36595)
* feat(otel): attribute Prisma database spans to PostgreSQL instead of localhost Prisma reaches PostgreSQL through a query engine on loopback, so transport instrumentation attributes database waits to localhost and operators cannot tell the work is PostgreSQL or correlate it with the database's own metrics. Datastore service spans now carry db.system.name, db.system, db.operation.name and, for PostgreSQL, server.address, server.port and db.namespace derived from DATABASE_URL, and are emitted as CLIENT spans. Only host, port, database and schema are read, so no credential reaches an exporter. Endpoint attributes are omitted when a read replica is configured, because routing is decided per Prisma call underneath the span. * fix(otel): reject a mis-split DSN authority and name socket-only databases An unencoded '/' in the password truncates the URL authority, so urlparse reports the username as the host and the password tail as the database, which put credential material in db.namespace. Postgres drivers reject that DSN outright, so the only safe reading is no endpoint at all. A hostless 'postgresql:///litellm' is a valid local-socket DSN that Prisma accepts, and it now yields db.namespace with no server address rather than nothing. The default schema is matched case-insensitively, since an unquoted PostgreSQL identifier folds and one deployment must yield one namespace. * fix(otel): keep a non-default schema in db.namespace Prisma quotes the schema name, so a DSN with ?schema=PUBLIC provisions a second schema alongside public rather than reusing it. Observed on a live proxy: the PUBLIC schema came up with its own 70 tables next to public's 78, and a key created under one was not visible under the other. Case-folding the two into a single namespace therefore reported two different schemas as the same database, which is the misattribution this feature exists to remove. Match the default literally. * fix(otel): reject any DSN whose userinfo fell outside the authority An unencoded '#' or '?' in the password sends the tail to the fragment or query, leaving the path empty, so the marker check on the database segment never fired and urlparse's hostname (the database username) was exported as server.address. The stranded userinfo '@' is the general tell for every mis-split, so guard on that instead of enumerating the characters that cause it. * fix(otel): allow an at-sign inside a well-formed DSN query The previous guard rejected any DSN whose userinfo at-sign fell outside the netloc, which also caught libpq parameters that legitimately carry one, so ?application_name=svc@prod and ?user=admin@company.com lost their endpoint attributes. Discriminate instead: a PostgreSQL DSN never has a fragment, its database name cannot hold an unencoded at-sign or slash, and an at-sign in the query is only suspicious when the query did not parse as parameters. * fix(otel): resolve the database endpoint per span instead of once per process The endpoint was cached for the process lifetime on the premise that DATABASE_URL is deployment-static. It is not. The RDS IAM refresh rebuilds the URL from DATABASE_HOST/PORT/NAME/SCHEMA on every rotation, the reconnect path re-reads DATABASE_URL, and the DB-backed environment_variables config overlay sets arbitrary keys post-startup with no blocklist covering DATABASE_*. A process that had genuinely failed over kept exporting the old server. Read the environment per span, which is also what Prisma connects with, so the span can no longer name a different server than the one serving the query; get_secret_str consulted a secret manager first and could diverge from it. Only the parse is memoized, keyed on the URL. * fix(otel): reject a question-mark mis-split whose tail parses as parameters A '?' in a password strands the rest of the authority in the query, and that tail can still parse as key=value, so testing only for an unparseable query let the login through as server.address. One spelling hijacked the host= parameter and put the password suffix there directly. A legitimate at-sign in a query always follows a database path, and a '?'-mis-split never leaves one, so require both. * refactor(otel): drop the DSN parse cache that retained rotated credentials The cache was keyed on the full DATABASE_URL, so up to eight complete DSNs, each carrying a password or a retired IAM token, stayed referenced for the process lifetime and outlived the rotation that replaced them. Nothing reached a span, but a heap dump or crash report would have surfaced them. Parsing costs about four microseconds against a span emission that costs orders of magnitude more, so the cache bought nothing worth that. * fix(otel): avoid a set construction the tightened LIT002 budget rejects * fix(otel): refuse an ambiguous DSN authority instead of guessing at it A password holding both an unencoded slash and a query-like tail defeated all three shape checks: the slash left a clean path carrying the password remainder, the query still parsed as parameters, and no fragment survived. The login went out as server.address, the password's leading digits as server.port and the rest as db.namespace. A DSN whose at-sign sits in a query parameter is indistinguishable from that mis-split by any property of the parse; both leave no userinfo, a host, a port and a path. Guessing wrong publishes a credential fragment, so the ambiguity now resolves to refusing the endpoint. Such a DSN loses server.address and db.namespace and keeps the rest of the span; percent-encoding the at-sign restores them. Also honour port= alongside host=, which libpq pairs and this read ignored. * docs(otel): fix a spliced sentence and a stale cache claim in db_endpoint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d86336a7c6
|
fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads (#36702)
* fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads The langfuse_otel exporter wrote version to langfuse.generation.version and langfuse.trace.version, and release to langfuse.trace.release. Langfuse v4 recognizes neither, so both landed in the generic span attribute bag and every trace reported version and release as null. v4 has a single langfuse.version key, lifted to the trace when it sits on the root span, plus langfuse.release. Also routes the otel v2 preset's per-request headers through the shared builder so key-scoped and team-scoped exports carry x-langfuse-ingestion-version like the other three exporter paths already do. * fix(langfuse): give trace_version precedence over version on the shared v4 key Matches the documented contract in docs/observability/langfuse_integration.md and the legacy langfuse SDK callback, which both treat trace_version as the authoritative trace version with version as its fallback. |
||
|
devin-ai-integration[bot]
|
12aeb53aec
|
fix(otel): mark v2 server spans as failed for pre-call errors (#34546)
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): cover malformed-body rejection when auth error is recovered Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): skip authorization for a request whose body never parsed Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
0a606cb258
|
fix(otel): name the RPC system and upstream on MCP tool-call spans (#35857)
* fix(otel): name the RPC system and upstream on MCP tool-call spans An MCP tool-call span carried only gen_ai.*, mcp.* and litellm.* attributes. A CLIENT span holding none of the http/db/messaging/rpc families is unclassifiable, so Elastic APM indexed these spans as span.type=unknown with no span.subtype at all, and its span-links API then rejected the whole trace with "Missing required fields (span.subtype)". MCP frames every message as JSON-RPC 2.0, so the tool-call span now names rpc.system. It names server.address and server.port alongside it, derived from the already-redacted mcp_server_resource origin: naming the RPC system makes a consumer treat the span as a downstream dependency and key that dependency off the server address, so emitting one without the other labels the dependency ":0". The tools/list span is left alone. It reaches the callbacks with no upstream identity, and a listing can span several upstreams, so it has no address to attach and would produce exactly that ":0" node. The wire is untouched: streamable MCP still returns HTTP 200 with isError: true. * fix(otel): drop rpc.system when no MCP upstream address resolved server.address and server.port come from mcp_server_resource, which is absent whenever the tool name resolves to no registered server, is None for a stdio transport that has no host to log, and parses to no host for an IPv6 origin the redactor rebuilds without its brackets. rpc.system was stamped unconditionally, so each of those paths emitted it alone and named the dependency ":0", the outcome the address pair exists to prevent. Gating the system attribute on a resolved address makes the pairing structural rather than leaving it to the two extractors happening to agree. * fix(otel): require a full MCP destination before naming the RPC system The gate gave rpc.system a resolved address, but not a resolved port. A host-bearing scheme outside the HTTP(S) default-port map resolves an address alone, and mcp_servers[].url is not scheme-validated, so an origin like mcp://host or ws://host reaches the mapper and names the dependency host:0 instead of the :0 the previous commit removed. Gating on the complete pair closes it, and covers a port of 0 as well. _upstream_address_port also gets a direct contract test, including the IPv6 origin the redactor rebuilds without brackets. * fix(otel): do not raise when an MCP origin has an unparseable port _redact_mcp_resource_url rebuilds the origin without its IPv6 brackets, so a zone-scoped address leaves a truthy hostname behind that the host check admits: http://[fe80::1%25eth0]:80 becomes http://fe80::1%25eth0:80, whose hostname is fe80 and whose port raises ValueError. That propagated out of MCPToolCallSpanData.from_standard_logging_payload and cost the span. Reading both halves inside a guard degrades an unparseable origin to no address, which is already how the mapper treats an unresolvable upstream, and matches the guard the redactor puts around the same split. The scheme default port drops the dict literal so the LIT002 ceiling stays put. |
||
|
Yassin Kortam
|
abd239f903
|
fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name (#35151)
* fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name The GenAI metric attribute builder mapped only chat, text completion, embedding, responses and MCP tool calls to an operation name, so vector-store searches and A2A agent sends fell through to the "chat" default. Their duration and cost then landed in the same series a Grafana GenAI dashboard reads chat latency off, with no way to tell them apart. Both now map to the operation names the convention defines for them, retrieval and invoke_agent, and an unmapped call type says so at debug instead of silently becoming chat. The provider label used gen_ai.system, which the convention deprecated in favor of gen_ai.provider.name; the dashboards built on that vocabulary find nothing under the old key. Metrics now carry gen_ai.provider.name with the semconv provider value (bedrock -> aws.bedrock) via the resolve_provider helper the span path already uses, and keep dual-emitting gen_ai.system with its raw value so a dashboard already querying it keeps matching. A request litellm cannot attribute to a provider gets no provider label at all rather than a placeholder "Unknown" that minted a permanent series nobody can act on. Resolves LIT-4954 Resolves LIT-4959 * fix(otel): map the rest of the vector-store call types off the chat default Mapping only the search left the store lifecycle (create, retrieve, list, update, delete) and the file operations (create, list, retrieve, content, update, delete) falling through to chat, so vector-store admin traffic kept polluting the same series a dashboard reads chat latency off. A live run confirmed it: all 20 metric datapoints from a create, retrieve, list, file-list and delete came out labelled chat. The convention names no operation for vector-store management, so these take vendor values under the litellm. prefix, litellm.vector_store_management and litellm.vector_store_file_management, one per REST resource. Its note on gen_ai.operation.name directs instrumentation to use a system-specific name when no predefined value applies, which is the same allowance resolve_provider already relies on for unmapped providers. Excluding them from the GenAI metrics altogether was the alternative; it deletes series an operator may be watching today and is far harder to reverse than a rename, so it stays available as a follow-up rather than being decided here. Mapping them onto the semconv memory store family was rejected: litellm vector stores hold documents, not agent memory records, and borrowing those names would put document admin calls into whatever charts agent-memory operations, which is the bug this fixes. /rag/query reaches the same recorder and is the same operation as a vector-store search, so query and aquery map to retrieval too; leaving them would have left the defect alive on a second retrieval surface. /rag/ingest is a write with no semconv equivalent and no retrieval or agent confusion, so it is left for the RAG owners to name. Resolves LIT-4954 * fix(otel): give the streaming A2A path a call type so it labels as invoke_agent The streaming logging object is built by hand and never runs through update_environment_variables, the only place call_type reaches model_call_details, so every streamed agent turn arrived at the recorder with no call type and fell back to chat. Stamp it, and map the streaming spelling alongside the non-streaming ones. |
||
|
Yassin Kortam
|
8bb8628ab5
|
fix(otel): record the GenAI duration metric on failed requests (#35152)
* feat(otel): record the GenAI duration metric on failed requests `_record_metrics` ran only from `async_log_success_event`, so `gen_ai.client.operation.duration` counted only the requests that worked. Latency read off it during an incident was the latency of the surviving traffic, and with no error dimension anywhere there was no way to build a failure-rate panel or a success/failure split per model. A failed call now records the same duration histogram, tagged with the semconv `error.type` (the mapped provider exception's class name, bounded by construction; the message stays on the span). Success attributes are untouched, so an existing query can still isolate the old series with `error_type=""`. The other five instruments describe a completed generation and are skipped rather than filled with a fabricated zero: litellm hands the failure callback no `response_obj`, so there is no usage to split and no completion-token count, and it zeroes `response_cost` on failure. A proxy-gate rejection (auth / rate limit) records nothing, for the same reason it gets no span; no upstream call happened. `error.type` is stamped after the cardinality filter, like `gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot strip the discriminator and silently merge failures into the success series. Resolves LIT-4955 * fix(otel): bound the failure metric's attribute set The failure datapoint reused the success path's full attribute set, which carries client-supplied fields (`metadata.requester_metadata`, `metadata.spend_logs_metadata`, the end-user id taken from the request's `user` field) and per-request ones (the `hidden_params` blob holding the provider's response headers). A failed request needs no provider spend, so nothing rate-limits a caller who puts a unique value in a field they control and mints one histogram series per request. A failure now carries a bounded allowlist: the operation enum, provider, request model, framework, the key/alias/team/org/user identifiers, and `error.type`. Every entry is a fixed enum or an operator-provisioned identifier, so the failure series count is bounded by the deployment's own key, team and user count while the labels still answer which team on which model is failing and how. The user email is left out as PII duplicating the user id already on the series. The operator's `otel.attributes` filter layers on top, so it narrows the allowlist further and never widens it. * fix(otel): cap metric attributes so series count does not grow with traffic (#35166) `GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object onto every metric datapoint as one label value. That object is per-request by construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`, `usage_object` and the provider's `additional_headers` rate-limit counters all move on every call. A unique label value is a new time series, and all six GenAI instruments share those attributes, so one request minted up to six series that would never be written to again That is the steady-state behavior of the feature rather than an abuse case, and it is wrong twice over. Hosted backends bill on series count, so recommending metrics be enabled would have meant a bill proportional to traffic. And a histogram whose every datapoint sits in its own series cannot be aggregated, so the dashboards would have looked populated while answering nothing Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces the failure-only allowlist so the two paths cannot drift. The cap runs before the operator's `otel.attributes` filter, so an operator can narrow it and never widen it back to an unbounded label. Client-supplied and per-request metadata (`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`, `requester_ip_address`) is metric-ineligible and stays on the span, which already carries it and where cardinality is free. `hidden_params` survives as a label but carries only `model_id` and `api_base`, which are bounded by the router's own deployment list and are the part a per-deployment panel reads Four tests fail against the previous behavior, the load-bearing one being that two requests differing only in per-request fields must land in one series rather than two |
||
|
Yassin Kortam
|
bf8e4af0e2
|
fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span (#34828)
* fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span
The genai and legacy mappers each spelled out every declared tool as
per-index span attributes. A request declaring hundreds of tools produced
roughly 500 attributes against the OTel SDK's default 128-attribute span
limit, which evicts oldest-first, so the canonical gen_ai.* set written
first was discarded and the span exported with only a tail of tool
schemas. Cap the family at 8 tools, shared by both vocabularies, and
carry the declared total on litellm.request.tools.declared so the
truncation is visible rather than silent.
* fix(otel): apply the tool-definition cap to the OpenInference mapper
The OpenInference vocabulary emits its own unbounded llm.tools.{idx}.*
family, which Arize and Phoenix layer on top of the default two, so those
configurations still overran the span attribute limit and evicted the
core gen_ai.* attributes. Route it through the same shared cap and cover
the layered-mapper path with a test.
* fix(otel): share one span-wide tool-definition budget across vocabularies
Capping the tool-definition family per mapper left each active vocabulary
its own allowance, and several vocabularies write to the same span. With
every vendor vocabulary configured, the three that spell tools out per
index still summed past the SDK's 128-attribute span limit, so the core
gen_ai.* set written first was evicted exactly as before: measured at 128
attributes with 7 dropped and gen_ai.request.model gone.
Reserve a quarter of the span for tool detail and split that ceiling
across the distinct tool-emitting vocabularies at mapper-resolution time,
so the family is bounded span-wide no matter how many are configured. The
same worst case now exports 90 attributes with nothing dropped.
|
||
|
Yassin Kortam
|
440b1bcf65
|
fix(otel): make OTLP export work against Grafana Cloud (#35060)
Three defects kept LiteLLM's OTel metrics from reaching an OTLP backend. OTEL_EXPORTER_OTLP_HEADERS is W3C Baggage encoded per the OTLP spec, so its values are percent-encoded. litellm split the string on "," and "=" and passed the raw value straight to the exporter, so a vendor that documents "Authorization=Basic%20<token>" got a literal "%20" on the wire and the backend rejected the credential. Grafana Cloud documents exactly that shape, which made its OTLP gateway unreachable. Header parsing now delegates to the OTel SDK's own W3C Baggage parser in liberal mode, so percent-encoded values decode and values that were never encoded keep working. It moves from model/utils.py to plumbing/providers.py because model/ is deliberately free of opentelemetry imports; providers.parse_headers was already the entry point every caller used. The OTLP metric exporters then overrode histogram temporality to delta. Prometheus and Mimir, which back Grafana Cloud's OTLP gateway, reject delta histograms outright: the gateway answers 400 "invalid temporality and type combination" and drops the entire batch, so every GenAI metric was silently lost while traces kept flowing. Backends that prefer delta still accept cumulative, so the SDK default is the compatible choice in both directions, and the enterprise billing exporter already relies on it. Three GenAI instruments also carried names no convention or backend defines, so nothing downstream could chart them. Time to first token and time per output token take their semconv names, gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token; the gen_ai.client.response.* spellings litellm used are not conventions at all. Cost has no semconv instrument, so it takes gen_ai.usage.cost, the name backends already query for spend. All three are listed verbatim in Grafana Cloud's AI Observability integration reference, so its prebuilt panels find them. Both engines now read the names from the shared Metric constants rather than repeating string literals, so v1 and v2 cannot drift. The renames are breaking for anyone charting the former names; the docs and the release changelog carry the migration note. |
||
|
Yassin Kortam
|
502d3609af
|
fix(otel): stamp an MCP tool failure on the request that carried it (#34551)
A failed MCP tool call aimed its error.* attributes at request_root_span(), a ContextVar written on the ASGI request task. A stateful streamable-HTTP session runs every message on the single task the session's initialize POST spawned, so inside the message handler that ContextVar still holds the initialize request's SERVER span. That span ended long ago, so the SDK dropped every write (five 'Setting attribute on ended span' warnings plus set_status and _add_event per failed call) and the POST that actually failed carried no error at all. The identity attributes seeded onto the server span went the same way. Publish the live transport span on the ASGI scope of the request being handled and read it back in the message handler through req_ctx.request, the Request the streamable-HTTP transport attaches to each message. That replaces the session-scoped field with a per-message one: a JSON-RPC response POST deliberately skips the per-session lock, since it can arrive while the tool call awaiting it is still in flight, so a field on the shared auth object could be overwritten mid-call and send the tool call's telemetry to the response's request. A scope also dies with its request rather than holding a finished span on idle session state. Publishing re-anchors the request root for the message so guardrail spans and identity seeding follow, and only a transport still open for writes is anchored or stamped: a notification POST can answer before the session task is done, and moving dropped writes from one finished span to another is no fix. Live capture goes from seven ended-span warnings and an unmarked transaction to zero warnings and ERROR on the POST that carried the call. |
||
|
Yassin Kortam
|
7263aa0028
|
fix(otel): keep an MCP tool call in one trace, anchored to its own request (#34537)
Under otel_v2 a single MCP tool call surfaced in APM as two disconnected
traces joined only by a span link: the HTTP transport transaction
POST /{mcp_server_name}/mcp and the tools/call span carrying
error.type=MCPToolResultError. resolve_mcp_span_context parented the MCP
span to the W3C trace context the client propagates in params._meta
(SEP-414) and recorded the transport as a link, so with no traceparent
propagated (the common case today, including MCP Inspector) the span
started its own root trace.
Nest the MCP span under the transport span when nothing is propagated, so
the call stays in one trace; the propagated-context path is unchanged and
still parents to the remote context and links the transport per the OTel
GenAI MCP semconv.
The transport has to be resolved per message rather than read from the
request-root ContextVar. A stateful streamable-HTTP session runs every
message on the single task the session's initialize POST spawned, so that
ContextVar is frozen at initialize inside the handler: live capture on
staging showed the tools/call span linking the initialize POST rather than
the POST that carried it, and nesting on that anchor would hang every tool
call of a session off the first request's already-ended span. The gateway
now resolves the current request's span on the ASGI task and carries it to
the handler on the authenticated-user object, the same way per-request auth
already crosses that boundary.
|
||
|
devin-ai-integration[bot]
|
4a297dd611
|
fix(otel): restore proxy-level error.* attributes on v2 failure spans (LIT-4179) (#33664)
* fix(otel): restore proxy-level error.* attributes on v2 failure spans (LIT-4179) * refactor(otel): narrow v2 failure hook return type to drop fastapi import (LIT-4179) --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> |
||
|
Yassin Kortam
|
99b4c5ed3e
|
feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655)
* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. |
||
|
Yassin Kortam
|
1d87084212
|
refactor(otel): move litellm error detail keys under the litellm.* namespace (#32591)
The v2 OTel integration stamped litellm-specific error details as error.code, error.stack_trace, and error.llm_provider, squatting on the semconv-owned error.* namespace. They now live at litellm.provider.error.code, litellm.provider.error.stack_trace, and litellm.provider.error.llm_provider alongside the other vendor-extension keys. error.type and error.message stay on the semconv keys. |
||
|
yucheng-berri
|
85d1fe6e2a
|
fix(otel): restore error.* span attributes on v2 error spans (LIT-4179) (#32524)
The v2 emitter has never stamped error.message / error.code / error.stack_trace / error.llm_provider as span attributes; only error.type reached the wire. Backends that flatten span attributes into label indexes (Elastic APM labels.error_*, Datadog span tags) lost these four fields when v2 became the active integration on v1.90+ for otel_v2-flagged deployments. The pre-existing exception span event carrying the full message (LIT-3758) is unchanged; the message now rides both places at once, matching v1s shape. SpanError grows three optional detail fields; _parse_error threads them from StandardLoggingPayloadErrorInformation; the emitters error branch stamps them via a new module-level helper, guarded per field so guardrail-shape errors are not polluted with empty attributes. New semconv constants mirror open_inference.ErrorAttributes byte-for-byte, so v1 and v2 consumers read the same keys. Regression tests extend the mapped test files under tests/test_litellm/integrations/otel/. pytest reports 243 passed. |
||
|
Yassin Kortam
|
3116ed211b
|
feat(otel): stamp gen_ai.response.time_to_first_chunk on streaming LLM spans (#32236) | ||
|
ryan-crabbe-berri
|
468d11f71d
|
feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 (#31525)
* feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2
Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method
The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path
Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool
* fix(otel): anchor MCP spans to params._meta trace context, not the transport span
MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles
Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports
This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug
* fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing
The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context
extracted the client's W3C Baggage from params._meta into the span's parent context.
The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span,
and the list-tools/tool-call mappers don't set those identity keys, so nothing
overwrites them. A malicious MCP client could send
params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=...
and have those identity attributes attributed to its spans.
Extract trace context only (traceparent/tracestate) in the propagator, and stop
collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the
client's trace context, the actual goal, needs only trace context; remote baggage had
no legitimate consumer here. Regression tests at both layers assert a spoofed
params._meta.baggage never lands as a span identity attribute.
* style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers
The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and
UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead
of `Dict`, and narrow the optional-import guards to `except ImportError` (the only
failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of
a blind `except Exception`.
* fix(otel): stamp authenticated identity baggage onto MCP spans
Parenting MCP spans to the client's params._meta trace context over an empty
Context() meant the tool-call and tools/list spans carried no team/key/metadata
identity at all, so they couldn't be attributed or filtered by team in a traces
backend. The LLM-call span already re-seeds identity from the parsed, authenticated
StandardLoggingPayload rather than trusting ambient/remote context; extract that into
a shared _seed_identity_baggage helper and run both MCP emitters through it.
Identity comes only from the authenticated payload, never the client carrier, so this
keeps the earlier spoofing fix intact while restoring attribution. Regression tests
assert the authenticated team lands on both MCP spans and that a spoofed
params._meta.baggage value can't override it.
* refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
|
||
|
Yassin Kortam
|
2e575d39f2
|
perf(otel): memoize per-request lazy import of otel runtime hooks (#31707)
The proxy auth path calls phase_span() and seed_request_identity() in litellm/integrations/otel/runtime.py on every request, each doing a try/except lazy import of litellm.integrations.otel.logger. When the OpenTelemetry SDK is not installed (the default), that import raises, and CPython never caches a failed import, so every request re-scanned sys.path and contended on the import lock. At 750 concurrent users this cost about 12% throughput versus v1.85.0. Resolve the hooks once and cache the outcome, absence included, with functools.cache, so the import is attempted a single time instead of per request. Throughput returns to the v1.85.0 baseline. |
||
|
yucheng-berri
|
0216c969b8
|
fix(otel): point AgentOps OTLP exporter at otlp.agentops.ai (#31490)
The AgentOps preset hardcoded https://otlp.agentops.cloud/v1/traces, a domain that no longer resolves (NXDOMAIN), so every span silently failed to export with a NameResolutionError in the BatchSpanProcessor worker. The live ingest host is otlp.agentops.ai (the auth host api.agentops.ai was already correct). Pin the endpoint to the resolvable host and add a regression test on the constant. |
||
|
Yassin Kortam
|
1322ad7224
|
perf(otel): resolve LITELLM_OTEL_V2 flag once instead of rebuilding settings per call (#30989)
is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every call, which re-scans the process environment and costs ~28us. The flag is read multiple times along the proxy request hot path (auth, logging-callback setup, proxy_server), so the cost compounded into a measurable per-request CPU overhead and a throughput regression visible from v1.87.3 onward. The flag is a process-level setting that is fixed at startup, so resolve it once with lru_cache. Caching it alone restores throughput to the pre-regression baseline in load tests. Tests that toggle the env now call cache_clear(). |
||
|
yucheng-berri
|
1f9323792c
|
fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter (#30590)
* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter The proxy published the OTel global TracerProvider before callbacks were initialized, so no preset logger existed yet and a second generic logger was built that won the global provider. Server spans then exported through a different provider than the preset's gen-ai spans, orphaning the LLM span on the preset backend. Publish after callback init and reuse the already-built logger instead. Separately, per-request tenant OTLP credentials were stamped onto every OTLP exporter, leaking one backend's key onto a co-configured backend. Tag each exporter with the preset that contributed it and apply dynamic credentials only to the matching owner. * fix(otel): satisfy Any-discipline on changed lines Type the logger-selection parameter as Sequence[object] (isinstance narrows it), cast the list[Any] global at the single call site, and pass model_copy a typed dict[str, str] update so no changed line carries an Any value. * fix(otel): annotate the untyped-global boundary with any-ok select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared List[Any] global this change does not own. A cast doesn't satisfy the Any-discipline checker (it inspects the inner expression), and re-annotating the global is out of scope, so mark the single boundary line any-ok. * test(otel): cover the startup global-provider publish via injectable helper The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit tests do not execute), so its lines were uncovered though the selection logic was tested. Extract publish_global_otel_v2_provider, which selects the single v2 logger and publishes its provider through an injected setter, and unit-test that the published provider is the selected logger's. proxy_server delegates to it. * refactor(otel): select global provider from the registered owner, not a list scan The startup publish picked the global TracerProvider by scanning _in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the factory already settled: the first logger built registers itself as proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity seeding, phase spans) routes through that owner via _registered_v2_logger. Pass that owner into select_global_otel_v2_logger so the global provider reuses the same logger instead of an independent, order-dependent guess; the list scan remains the SDK-path fallback. The owner is injected at the proxy call site to keep the helper free of hidden global reads. * refactor(otel): type ExporterSpec.owner as an ExporterOwner enum The owner field carried free-form strings that had to match preset callback names. Introduce a str-based ExporterOwner enum (values equal to the callback names, so per-request credential routing's owner==callback_name comparison still holds) and have each preset tag its exporter with the enum member. * refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX Distinguish the hosted Arize AX backend from Arize Phoenix at the member level while keeping the value 'arize' (the public callback name routing compares against). Add a comment noting AX and Phoenix are separate backends. |
||
|
Yassin Kortam
|
27c1dfbdc7
|
fix(otel): accept UPPER_SNAKE_CASE OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT in v2 (#30562)
V1 read this env var case-insensitively, so SPAN_AND_EVENT enabled content capture. The v2 config compared the value against its lower_snake_case canonical constants without normalizing, so an operator carrying the SPAN_AND_EVENT spelling forward silently left capture off and no gen_ai.input/output.messages reached the span. Normalize the value to lower case at the config boundary so both spellings work. |
||
|
Yassin Kortam
|
f444539ea9
|
fix(otel): export v2 gen_ai client metrics to the configured meter provider (#30549)
* fix(otel): export v2 gen_ai client metrics to the configured meter provider The V2 OpenTelemetry integration recorded the six gen_ai.client.* histograms into a MeterProvider it built locally in _init_metrics and never published. The recording code ran fine; the metrics simply landed in a provider disconnected from the global pipeline, so an operator's configured readers/exporters (and the server-metric instrumentation bound to the global meter provider) never saw them. Resolve the meter provider the OTel-idiomatic way instead: reuse the operator's globally configured MeterProvider when one is set so its readers receive the GenAI histograms, build and register one as the global only when none is set so V2 owns metrics export (mirroring how V2 owns trace export), and keep the injected meter_provider as an explicit override for DI and tests. * refactor(otel): hoist meter imports and harden global resolution Move the opentelemetry metrics and sdk MeterProvider imports to module top instead of importing inside resolve_meter_provider/build_meter_provider; the SDK is already a top-level dependency for tracing, so the per-call imports added nothing. resolve_meter_provider now reuses an explicit NoOpMeterProvider as well as a real SDK provider, so an operator opt-out is honored, and the built provider is always the one returned so its reader thread is never orphaned. Drive the regression test through the public metrics.get_meter_provider via monkeypatch rather than writing opentelemetry's private _METER_PROVIDER slot, and add focused tests for the injected and no-op resolution branches. * fix(otel): type resolve_meter_provider as the api MeterProvider base mypy flagged the return as incompatible because honoring an explicit NoOpMeterProvider returns a value of the opentelemetry api MeterProvider base rather than the sdk subclass. Annotate the resolver in terms of the api base and keep the sdk class for construction and the reuse isinstance check. |
||
|
Yassin Kortam
|
b8b0d458af
|
fix(otel): stamp gen_ai.input/output.messages on v2 spans (#30548)
The canonical GenAI mapper's _LLM_CALL_ATTRS table had no extractors for gen_ai.input.messages or gen_ai.output.messages, so V2 LLM spans never carried prompt or completion content even when capture was enabled via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_and_event. The request and response bodies were already captured onto LLMCallSpanData.messages_in and choices_out, but the mapper never read them. Add the two extractors, serializing messages_in and output_messages(d) through serialize_messages so the keys are omitted when content capture is off and the spans stay sparse. Resolves LIT-3788 |
||
|
Yassin Kortam
|
45d5153c12
|
feat(otel-v2): emit the 6 gen_ai.client.* metrics at parity with v1 (#30326)
* fix(otel): cap metric attribute cardinality with include/exclude lists OTEL metrics stamped every per-request hidden_params and metadata.* field onto each gen_ai.client.* sample, so near-unique values created one metric time series per request and backends like Splunk Observability Cloud throttled and dropped the data. Add an attributes block under callback_settings.otel with mutually-exclusive include_list (allowlist) and exclude_list (denylist), validated against the known attribute names at startup and applied once to the metric attributes in _record_metrics. Spans are untouched, and with no config every attribute is still emitted so existing setups are unaffected. Resolves LIT-3600 * fix(otel): resolve metric attribute filter from callback_settings The proxy usually constructs the OpenTelemetry logger without forwarding the attributes kwarg, while the filter lives under litellm.callback_settings["otel"]["attributes"]. __init__ only read the kwarg, so the recording instance kept config.attributes=None and shipped metrics at full cardinality even when the filter was configured; a live proxy run exposed this. Fall back to the global at init for the base otel logger, and add a regression test that drives the real success hook through the callback_settings path (the unit tests passed before because they injected the config directly). * fix(otel): reject gen_ai.token.type from metric attribute filter lists gen_ai.token.type was a member of VALID_METRIC_ATTRIBUTE_NAMES, so an operator could list it in include_list or exclude_list and pass startup validation. The attribute is injected into the input/output token series after _filter_metric_attributes runs, so the filter never sees it and the request silently has no effect. Reject it loudly from either list instead, matching the contract that a non-actionable attribute name fails fast rather than falling through to a no-op. It stays a structural discriminator on the token-usage histogram. * fix(otel): resolve metric attribute filter lazily at record time The proxy constructs the OpenTelemetry logger before it populates litellm.callback_settings["otel"]["attributes"], so resolving the filter at __init__ left config.attributes None and shipped metrics at full cardinality. A live proxy run confirmed the leak. Resolve the filter on the first metric record instead, when callback_settings is populated, while still validating an explicit config eagerly so a bad SDK config fails at startup. The regression test now constructs the logger before populating callback_settings to mirror that ordering, so it fails if the filter is resolved too early. * fix(otel): don't cache invalid filter on lazy callback_settings path On the lazy callback_settings resolution path, _ensure_metric_attribute_filter wrote self.config.attributes before validating it. When validation then failed, _metric_attr_filter_resolved stayed False while config.attributes held the bad filter, so the next record skipped the callback_settings re-read and re-raised the stale error indefinitely; fixing the misconfiguration required a restart. Drop the premature write and resolve from the local value. A subsequent record now re-reads callback_settings, so a corrected config takes effect without a restart. The write was dead on the success path anyway, since the resolved frozensets are what the filter reads. * feat(otel-v2): emit the 6 gen_ai.client.* metrics at parity with v1 The v2 OpenTelemetry integration was a span engine: it declared two metric histograms but never created a meter or recorded anything. Bring it to parity with v1 so a v2-default deployment gets bounded metrics. Adds the 4 missing metric names, all 6 histograms, a meter-provider builder that mirrors v1's exporter selection, and a GenAIMetricRecorder that records token usage (split input/output), cost, operation duration, TTFT (streaming), TPOT, and response duration on the success hook. Gated on config.enable_metrics so the default is unchanged. The attribute cardinality filter is reused from v1 by import (no duplication of the valid-name set or validation) and resolved lazily from callback_settings.otel.attributes, matching v1. A misconfigured filter raises out of the recorder; the logger surfaces it once at ERROR and records nothing, rather than silently disabling metrics, and a corrected config recovers without a restart. * test(otel-v2): drop duplicate misconfig logger test (covered in test_otel_v2_logger) |
||
|
Yassin Kortam
|
3b84150137
|
fix(otel): record full error message on standard exception event in otel v2 (#30380)
The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Sameer Kankute
|
079c136742
|
chore(oss): litellm oss staging 120626 (#30292)
* feat(bedrock): add bedrock mantle gemma 4 models (#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request #29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Yassin Kortam
|
8fbdfc7f0d
|
fix: missing mcp otel attributes (#29554) | ||
|
Yassin Kortam
|
08223e1ec3
|
fix: missing span for guardrail passthrough (#29552) | ||
|
Yassin Kortam
|
b98a656254
|
Add MCP semantic conventions to otelv2 (#29468)
* Add MCP semantic conventions to otelv2
Emit OpenTelemetry GenAI MCP tool-call spans from the v2 logger. A closed
call_mcp_tool request now produces a CLIENT span named "tools/call {tool}"
carrying mcp.method.name, gen_ai.operation.name=execute_tool, gen_ai.tool.name,
the upstream server name, and (opt-in, content-gated) tool arguments/result.
Adds the MCP and JSON-RPC attribute vocabulary to the semconv module, an
MCPToolCallSpanData payload built from StandardLoggingMCPToolCall, an
MCP_TOOL_CALL span role, and mapper support.
* Complete the MCP span-attribute vocabulary in otelv2 semconv
Add the remaining OTel GenAI MCP semconv attribute keys: gen_ai.prompt.name,
the network.* transport keys with their well-known NetworkTransport values, and
the client.* peer keys for MCP server spans. A test pins the full vocabulary so
a dropped or renamed key fails loudly.
* Populate mcp.session.id on MCP tool-call spans
Capture the mcp-session-id header (case-insensitively) at the tool-call entry
point and thread it through StandardLoggingMCPToolCall into the span, so spans
for stateful MCP sessions carry mcp.session.id. Stateless calls have no such
header and the attribute is simply absent.
* Test that stateless MCP calls omit mcp.session.id
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Yassin Kortam
|
8190ff4d86
|
feat(otel): allowlist team_metadata sub-keys promoted to baggage (#29442) | ||
|
Yassin Kortam
|
d82eb33a60
|
feat(otel): typed semconv-aligned OpenTelemetry instrumentation (#28909) |