mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
36 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tin-berri
|
ec4f66bb94
|
fix(router): budget classifier input by provider limits (#49306) | ||
|
yucheng-berri
|
8d441ebc15
|
test(otel): give the Langfuse destination tests the otel callback's stdout exporter (#49129)
#48718 added TestLangfuseDestinationFields while #48723 made the unconfigured stdout placeholder opt-in (console_default=True, the otel callback's setting). Both merged within a minute, so main's integrations job fails six tests that read the placeholder's exporter. Build those providers with console_default=True so the operator-exporter checks keep their target. |
||
|
yucheng-berri
|
ffcc4f7a70
|
fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723)
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize holding operator credentials, the proxy printed every span as JSON to stdout, prompts included, on the request path. Nobody set OTEL_EXPORTER, but the console placeholder that OpenTelemetryV2Config folds in for an empty exporter list stayed next to the vendor's own exporter. The provider now skips that placeholder when the config has any other exporter. A console exporter the operator names with OTEL_EXPORTER=console is kept, and so is the placeholder when it is the only exporter. * test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check * fix(otel): print the console default only for the otel callback The proxy's background V2 logger (published when no callback built one, which is what key-only and team-only Langfuse destinations run under) also kept the unconfigured console exporter, so it printed every span with prompts to stdout. The console default now prints only for a listed otel callback or an exporter the operator named. * fix(otel): keep the otel callback's console output on service-name routes A key or team otel_service_name moves the call onto a provider the tenant tracer cache builds, which never got the console choice, so the plain otel callback stopped printing those calls. The no-operator-keys integration rig also drops inherited Langfuse keys. |
||
|
yucheng-berri
|
1e7da87e1e
|
fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and the callback fell back to the legacy logger. Key and team destinations were then served by a generic v2 logger without the Langfuse mapper, so their traces lost tags, user and cost, and the legacy logger added an orphaned raw_gen_ai_request span. The preset now always degrades to the credential-gated exporter when the operator has no keys, the same way arize_preset does, so the startup logger stays on v2, exports nowhere for traffic without a destination, and carries the Langfuse mapper to every key and team destination. * test(otel): type the langfuse tenant-only test helpers * fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback With langfuse_otel only in a key's or team's logging, the startup logger is the generic one and its spans reached the key's Langfuse project without user, session or tags. The emitter now adds the Langfuse mapper for a request with a Langfuse destination, and every non-Langfuse exporter strips the Langfuse request metadata, since any logger can now write it. * fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector With langfuse_otel only in key or team logging, the proxy publishes the generic V2 logger at startup and it already exports to the operator's OTLP collector. The Langfuse logger a key's request builds kept the same base exporter, so the collector got every model span twice. A logger built after the publish now keeps only its preset's own exporters, as it already did beside a configured V2 logger, and a later call that would rebuild one exporting nowhere reuses it. * test(otel): assert the collector gets one copy instead of reading the key logger's exporters * test(otel): count the collector's spans with one comprehension clause * fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries Reverts the preset degrade and the published-provider exporter trim, so every request without a key or team Langfuse destination keeps main's legacy logger. For a request with one, the V2 fan-out already delivers the whole trace with the Langfuse fields, so the legacy logger emits nothing for it. * test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination * fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields At its attribute limit a span sheds the Langfuse fields only a destination asked for first, in the mapper's order, so the generation marker went before the request metadata blob. Put the request metadata first, as a Langfuse-configured logger already does. * fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider The OTel v2 fan-out records no call for a request the proxy rejects before a provider (rate limit, pre-call hook), so standing the legacy logger down there lost the user, session and tags main sent the tenant. Stand it down only for a call the fan-out records, using the same check the v2 logger uses. The tenant and collector integration tests now hold their exact counts for a few seconds, so a late duplicate copy fails them. * fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it. * fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter * fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran * fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys |
||
|
tin-berri
|
54a6a5fb3a
|
fix(otel): suppress spans for excluded proxy requests (#48720) | ||
|
yucheng-berri
|
74392efbd4
|
fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) | ||
|
devin-ai-integration[bot]
|
1171273511
|
fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697)
* fix(langfuse_otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): name trace metadata identity keys after their request metadata fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): type metadata helper and drop null fields so they cannot clobber caller trace metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): annotate new request metadata tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): build rig config and wire batches immutably Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): skip empty trace identity values so teamless requests cannot erase attribution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): drain wire batches per poll instead of accumulating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): cover null stripping and empty trace identities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): set redaction flag directly instead of monkeypatching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): add integration audit matrix for langfuse metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tighten audit chaos and unauthenticated cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): restore contracts.json formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): close SDK clients, pin worker identity, and normalize response id matching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): gate the slow sink reply instead of sleeping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): emit request metadata and identity under langfuse.observation.metadata and langfuse.trace.metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): satisfy type discipline gate for metadata export Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: drop prohibited covers markers from langfuse otel integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): put the mutable-ok suppression on the dict construction line the type-discipline gate reports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse otel): share request metadata export across callbacks * fix(langfuse otel): drop prompt variables, MCP tool I/O and vector store queries from exported request metadata * fix(langfuse_otel): keep span export alive when nested request metadata has non-string keys The content strip validated prompt_management_metadata, mcp_tool_call_metadata and vector_store_request_metadata items as str-keyed mappings, so an SDK caller passing one with a non-string key raised inside exported_request_metadata: OTEL v2 dropped the LLM span for every vendor and v1 langfuse_otel lost the request metadata and every attribute after it. Validate those nested mappings with object keys instead. Also hoists the langfuse_otel imports. * fix(langfuse_otel): send the request metadata blob to Langfuse destinations only - A requester_metadata that is not a mapping no longer raises. The v2 LLM span (every vendor) and the v1 Langfuse attributes survive it again. - Prompt-quoting routing_decision fields stay out of langfuse.observation.metadata. - An exporter or key/team destination for any other backend drops langfuse.observation.metadata and langfuse.trace.metadata.user_api_key_*, as on main. This covers a collector folded into the Langfuse logger. - The v2 Langfuse key-set pin in the arize family test gains the two new keys. * fix(langfuse_otel): export only caller identity under the Langfuse metadata keys - The Langfuse metadata keys carried the whole request metadata. On OTel v2 that sent request headers, caller metadata, user email, user agent and auth metadata to a Langfuse destination that never received them on main; on v1 it doubled each span's metadata payload, so big caller metadata doubled the export size - Both paths now export only the key alias, user, end user, team, team alias and org. On v2 that is the team plus the identity among the operator's `baggage_metadata_keys`, which v2 spans already carry as `litellm.metadata.*`. Everything else stays in the bare `metadata` attribute, as on main - `output_messages` returns a list again, and the arize v2 b3 key pin is back to main's set * fix(langfuse_otel): drop the caller identity first when a span hits its attribute limit v1 wrote the identity attributes after the OpenInference ones, so a span near the 128-attribute limit (about 33+ tools) evicted openinference.span.kind, metadata and llm.model_name instead. The identity now goes first on a fresh span, so the SDK evicts it before anything else, and a caller's active span that already carries attributes does not get it. v2 sheds the Langfuse identity keys before any indexed message attribute when a span is over budget, so a span keeps every attribute it carried without them. * fix(langfuse_otel): keep other spans and dropped counts as main exports them - v1 writes the caller identity only into the room the span's attribute limit leaves next to the error attributes a failed call adds, so the exported dropped-attribute count matches main - v2 strips the Langfuse request metadata only from LiteLLM's own spans; application, instrumentation and server spans keep every key they set - a stripped v2 span keeps the dropped attribute, event and link counts - a config that lists the Langfuse mapper without the langfuse_otel exporter also keeps the request metadata away from other exporters * fix(langfuse_otel): keep the caller identity off the caller's active span v1 writes LiteLLM's attributes onto the caller's active span when one is current instead of starting its own span. That span goes to the caller's own exporter, so the identity keys reached a non-Langfuse collector and could take attribute slots the caller fills later. The identity keys now go only on the span LiteLLM starts, whatever the caller's span carries. * fix(langfuse otel): keep the identity keys on the OTEL v2 Langfuse mapper only The v1 langfuse_otel callback goes back to main: it sends no langfuse.observation.metadata or langfuse.trace.metadata.user_api_key_* keys. The OTEL v2 Langfuse mapper keeps them. The integration cells now run every endpoint, client, redaction, failure and chaos case on v2. * test(langfuse otel): count only accepted batches in the sink outage cell The wire also records the batches the sink answered 503, so an exporter retry made a span look exported twice. The chaos cells now also check the spans that landed carry the caller identity. * test(langfuse otel): split stacked span comprehensions and drop constant comments * test(langfuse otel): split the metadata integration module into focused files * fix(langfuse otel): send the org id in trace metadata too The trace metadata keys now reuse IDENTITY_METADATA_FIELDS, so a key in an org gets langfuse.trace.metadata.user_api_key_org_id next to the generation's org id. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0657ed46cf
|
fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705)
* fix(langfuse_otel): derive trace-level name, input, output and tags on the generation span Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep every reasoning summary in the derived observation output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): leave joined traces untouched by derived trace fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): restore preset cache key fallback and type the derived-field helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep trace tags off joined traces entirely Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tolerate out-of-band upstream requests and add sink outage chaos cell Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build the langfuse rig config without mutating the loaded yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(langfuse_otel): flatten output items with itertools.chain to satisfy LIT014 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): type the _RecordingSpan and _emitted unit helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep joined-trace tags, mask trace io, leave v2 trace io to the root Joined traces (existing_trace_id) keep the caller tags they carried before, and only the derived trace fields are skipped for them. mask_input and mask_output now redact the derived trace input and output, matching the vanilla Langfuse callback. The v2 mapper derives the trace name only, because the root observation already carries the trace input and output there. Uses the public preset cache key helper, moves imports to module top, restores the original formatting of the unit test file, and adds integration cells for the v2 llm_only scope and SDK sync/async calls. * fix(langfuse_otel): split string caller tags the way Langfuse reads them A caller tags value given as a comma list or a JSON array string (body metadata.tags or the langfuse_tags header) used to reach Langfuse raw, where ingestion split it into separate tags. Merging it with the derived tags kept it as one tag, so parse it the same way before the merge * test(otel): expect langfuse.trace.name on the v2 langfuse_otel generation span * fix(langfuse_otel): read string mask flags the way the classic callback does The langfuse_mask_input / langfuse_mask_output headers arrive as strings, so "false" redacted the derived trace io. Parse them with str_to_bool and fall back to truthiness, as the classic langfuse callback does * fix(langfuse_otel): derive trace fields only on traces litellm starts A litellm span that joins a trace an app or caller started (inbound traceparent, the app's active span, a caller-provided parent span) no longer gets a derived trace name, input, output or tags, like a request joining an existing_trace_id. Langfuse applies the later-starting span's trace update, so the derived values overwrote the app's own trace name and io. OTel v2 derives the name on the request root only when it has no parent, and the generation span keeps main's caller-only trace name. * test(langfuse_otel): keep main's OTel v2 generation key set in the arize family cell The OTel v2 generation span no longer carries a derived langfuse.trace.name, so the pinned key set goes back to main's. * fix(langfuse_otel): name the OTel v2 request root once, from the request's own call The v2 Langfuse logger re-derived langfuse.trace.name on the request root for every LLM call in the request. A post_call llm_as_a_judge guardrail's judge call replaced the caller's trace_name on the root with litellm-acompletion, and with another v2 callback (otel, arize_phoenix) listed before langfuse_otel that vendor's root span gained langfuse.trace.name on every request. The derived name now fills only an unnamed root this logger's provider started, and never comes from a call litellm makes for its own features. A caller trace_name still names the root on every call, as on main. * fix(langfuse_otel): keep the derived trace fields from pushing other attributes off a full span The derived trace name, tags, input and output were written after the OpenInference attributes with no room check, so a span near the SDK's attribute limit (about 34 tools at the default 128) evicted keys main exported, such as openinference.span.kind, metadata and llm.model_name, and raised the dropped count. They are now written last and only into the room the limit leaves next to the error attributes a failed call adds, so a full span exports what it would without them. * test(langfuse_otel): give the full-span cells' derived request a caller trace id, as the joined one's traceparent does * chore(langfuse_otel): drop the docstrings and comments the house rule disallows and type the new tests' parameters * test(langfuse_otel): set the span attribute limit the full-span tests run at and keep Final locals out of the app-trace loop --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
38f318e736
|
fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468)
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees * test(otel): cover per-team sampling across pods, workers, endpoints and restarts * fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate * test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one * fix(otel): decide team sampling per request, not per trace Calls sharing a caller's trace id are separate requests, keyed by their root span. Each is decided by the trace hash at the rate for its own outcome, and only its own late spans follow its verdict, so no pod applies an earlier call's verdict to a later call of the trace * fix(otel): forget a span's request root when it ends, not when newer spans start The root of each open span was kept in a first-in-first-out table of 16384 entries, so a span two levels below the root that stayed open while the worker started more spans than that lost its entry and ended under its parent's id. Its request then never closed and was held from every sampled destination until shutdown. Each open span now keeps its entry until it ends, and a span starting under a parent that has already ended reads the root from the parent span itself * fix(otel): decide a request as its spans end once it falls out of the open span table A request whose span was pushed out of the table of open spans, because the worker held more than 16384 open at once, ended that span under another key and was held from every sampled destination until shutdown. Pushing a span out now drops its request's open count too, so the request is decided as each of its spans ends, the way one pushed out of the open request table already is * test(otel): annotate the team sampling test locals as Final * test(otel): build the caller's remote parent in one helper instead of rebinding it per trace * fix(otel): keep each open span's request until it ends, bounded by the open requests The span table had its own bound of 16384, so a worker holding more spans open than that pushed out the oldest, and a failure in such a span no longer reached its request. Each request now holds the ids of its open spans, and a span's entry in the lookup goes when the span ends or when its request is pushed out of the open request table, the same bound main already applies. A failure anywhere in a tracked request reaches its verdict however many spans are open --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Ishaan
|
8f9894fbd6
|
fix(ci): unblock dependency security checks (#48055)
* fix(deps): upgrade vulnerable Python dependencies * fix(deps): lock patched Python dependency versions * fix(otel): preserve JSON export with the updated SDK * test(otel): cover JSON export with the updated SDK * test(otel): update SDK log records and worker discovery * fix(ui): align Lens peer dependency with patched Next.js * fix(otel): hand the AgentOps exporter a session it owns for the JWT header OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a transport, so writing the minted JWT to the exporter's private _session raised, was swallowed, and every AgentOps batch went out unauthenticated. The exporter now creates the session, passes it through the public session= argument and sets the Authorization header on it The tests now drive a real export, serving the token endpoint with respx and capturing the OTLP POST with requests_mock, instead of reading SDK internals * fix(otel): skip FastAPI instrumentation once the app has started opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead of calling add_middleware, so instrumenting an app whose middleware stack is already built no longer raises. The call silently did nothing while we still registered the global response propagator. Check the built stack up front and skip both, which is what the existing test expects * test(otel): assert exported requests and span flags instead of SDK internals OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace and Langfuse tests now export through requests_mock and check the URL and headers the collector receives, and the traceresponse tests expect the server span's own trace flags * fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly, on its own nested copy, so a top-level zod override points every consumer at the root version and the lock carries a single zod 4.6.6 * test(ui): stub the tag summary fetch the client panel retry refreshes Retrying the model breakdown refetches /tag/summary, which this integration test left on the real network. When that request failed before the assertion, the panel switched to its summary error and unmounted the model row, so the test passed or failed by timing * fix(deps): drop the duplicated Lens override from the dashboard package.json * style(ui): format the client panel fetch stub --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
b55024658c
|
fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997)
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update update_cache pins for synchronous write-back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): wrap long assert in update_cache tag test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover refill race and pin TTL with injected clock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count global proxy spend for every request with a user, cached on this worker or not * fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter The global proxy spend bump read the total, awaited, and wrote old + cost back from a background task, so a budget reset that deleted the key in between got the pre-reset total written back, and workers overwrote each other's totals. RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key exists, keeping its TTL, and this worker's copy is then set from the shared total with no await between its read and write. The bump now runs whenever a proxy max_budget is set, matching the DB writer, and the armed post-call read and the batch-read declaration it used are removed. * fix(proxy): report the global spend Redis increment to service metrics and request traces The global proxy spend increment now emits Redis service success and failure events under the spend_counters target and passes the request span through, so it shows as redis.incr spend_counters like the write it replaced. The unit tests run the real RedisCache increment against fakeredis instead of mocks, and a new test checks the span lands under the request span on success and on a Redis failure. * test(proxy): keep the spend counter Redis fake off the network * test(proxy): count increment_if_present as a shared-cache call in the key-family guard * fix(proxy): keep the newer global spend total when Redis replies arrive out of order --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
5145250d83
|
fix(langfuse_otel)!: send the completion start time (#45597)
* fix(langfuse_otel): send the completion start time langfuse_otel never set langfuse.observation.completion_start_time, so Langfuse showed no time to first token for any call, streamed or not. The legacy langfuse callback already sends kwargs["completion_start_time"]. Send the same value as an ISO timestamp with an explicit offset. * fix(langfuse_otel): type kwargs and test against a real span Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any], the declared type of model_call_details, so the new completion start time read adds no unknown-type errors. The tests now check a real SDK span instead of patching safe_set_attribute, and the parameter is typed. * fix(otel): send the completion start time from the v2 Langfuse mapper With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper, which had no completion start time either. Carry the payload's completionStartTime on LLMCallSpanData for streamed calls, the same calls that get time_to_first_chunk_seconds, and map it to langfuse.observation.completion_start_time. * fix(langfuse_otel): take the logging kwargs as a read-only Mapping The new kwargs annotation tripped LIT001. Both helpers only read the logging kwargs, so type them Mapping[str, Any] instead of dict. * fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter * fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON The native Gemini, Interactions and A2A streams never recorded a first chunk, so with completion_start_time now sent both emitters showed TTFT equal to latency on them. They now record it, and Gemini and Interactions take the call's start time like /v1/messages. Both emitters send the completion start time as a JSON string, the form the Langfuse SDK and the langfuse callback send, through one serializer. * fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event A2A streams took their start time when the iterator was built, so their latency left out the work before it. They now use the call's start time like every other route. Cached /v1/messages stream replays recorded no first event, so their time to first token fell back to the end time. The replay now records it on the first event it yields. The tests use the real Interactions config and typed logger hooks. * fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free * test(a2a): type the card fetch test and stop rebinding the seed helper's recorder * test(otel): expect completion_start_time and cache_hit in the B3 span keys --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
77880a0653
|
fix(otel)!: leave out span cost when the price lookup failed (#44187)
* fix(otel): leave out span cost when the price lookup failed A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free. * test(otel): keep the guardrail charge on the guardrail span for unpriced calls * refactor(otel): type the Arize cost writer input and reuse the cache_hit flag * fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost Only the placeholder 0 is left out when the price lookup failed, so a guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel and W&B Weave OTel. Adds unit cases for that fee and a malformed cost. * fix(arize): leave out the span cost on any failed price lookup, matching OTel v2 * fix(logging): clear a stale cost failure flag when a later pricing succeeds Stream-start pricing on the request model name can fail and set the cost failure flag even when the final pricing succeeds (OpenRouter usage.cost, an Azure deployment name priced from the response model). The new span gate then dropped a real cost. Each pricing now resets the flag, and uncached_price restores whatever flag it found. * test(logging): pin the start time in the stale cost flag tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8226b2be9a
|
fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604)
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on
|
||
|
adityajoy-1902
|
20da5b949c
|
feat(proxy): return the W3C traceresponse header on traced responses (#42079)
* feat(proxy): expose trace ID response header * style(otel): format trace correlation middleware * test(otel): cover websocket trace middleware * fix(otel): disable correlation on setup failure * refactor(otel): inject instrumentation setup * fix(proxy): drop a relayed upstream traceresponse when the gateway sends its own * fix(proxy): preserve upstream traceresponse without gateway injection * fix(proxy): replace relayed streaming traceresponse * test(otel): cover trace response fallback paths * test(proxy): cover the streaming pass-through relay when the gateway propagator is off * fix(proxy): keep a relayed upstream traceresponse and stamp the call id on the server span The gateway's response propagator now leaves a response that already carries an upstream traceresponse untouched, so pass-through, file content and plugin proxy routes return one header and untraced pass-through routes keep the upstream's. The per-helper strips are gone. The propagator also sets litellm.call_id on the server span from the response's x-litellm-call-id. * fix(proxy): add traceresponse only to sampled HTTP responses, tuple headers and unhandled 500s included * fix(proxy): write only traceresponse and link a relayed header from a restarted upstream trace * fix(proxy): send traceresponse only from the proxy app and expose it through CORS only with OTel V2 * test(proxy): check the CORS expose list contains the UI headers instead of pinning its identity * test(integration): check traceresponse names the gateway server span across routes, errors, pass-through and chaos * test(integration): read the cache-hit spend row by response id and assert only the traceresponse expose entry --------- Co-authored-by: adityajoy-1902 <bittu19jain@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
1587faeeb9
|
test(unit): wait on the work itself instead of fixed sleeps (#46261)
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files Replace sleeps that raced background logging, cache writes, refreshes and thread work with either awaiting that work directly (logging worker flush, the batch logger's flush_queue, the tasks the call spawned, thread joins) or an Event the test's own fake sets when it runs. Each wait keeps a 5s hang guard that never decides pass or fail. Tests that need a fake clock, real sockets, or measure latency are left unchanged, and no production code moves * test(unit): await the tasks each completion spawns instead of yielding once The logging tests yielded once with asyncio.sleep(0) so the task that enqueues the success log could run before the worker flush. That relied on the enqueue happening before the task's first await. Snapshot the running tasks around each acompletion call and await the ones it spawned, so the guarantee holds however many steps sit before the enqueue |
||
|
mrinal-berri
|
231721f20a
|
feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination (#45880)
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: ruff format otel excluded_services changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: use raw regex for excluded_services match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): reject non-string destination callback vars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): harden destination callback var validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): compare parsed values in callback var conflict check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dc95de5209
|
feat(otel)!: let a team capture_message_content override the global setting for its OTel v2 destination (#44264)
* feat(otel): let a team strip captured message content from its own OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): split the shared callback option conflict check and sync dashboard API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): address review feedback on team capture_message_content Classify capture_message_content as NotSecret in the credential slot registry, inject test exporters through register_exporter_factory instead of patching _exporter_from_spec, type the test helpers, and stop reassigning processor in build_tracer_provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add integration cells for team capture_message_content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): cover weave_otel and newrelic team destinations in capture_message_content cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): stop rejecting client-sent capture_message_content at the request body boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(otel): let a team capture_message_content override the global setting in both directions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep a team's capture_message_content on its own OTel v2 destination The setting no longer rides the flattened team callback vars, so a sibling destination and a routed provider follow the global mode. Request bodies get the base 400 back, a failure-only entry rejects the setting, and entries for one backend must agree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the capture_message_content callback validation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the remaining capture_message_content test signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): mark the new capture_message_content test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): default tenant capture to callback_settings.otel and keep CaptureMessageContent formatting bare Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep CaptureMessageContent a plain str like main The str-to-Enum switch changed the public type: the config field's type, repr and model_dump, a constructor that now raised on unknown values, and unknown env values coerced to no_content. CaptureMessageContent is the plain str subclass main ships again, the field is a str, and a captures_span_content helper replaces the enum property. A test pins that a published preset's capture mode stays the default for an unowned destination, which is what main forwards. * test(otel): drop the module-level CaptureMessageContent import the rewritten test no longer uses * fix(lint): keep the merged branch within main's type budgets Freeze the capture-mode settings model (LIT015) and read stored logging entries through one typed sequence helper, so the conflicting-capture checks add no unknown types. * fix(otel): reject a destination capture_message_content while OTel v2 is off The legacy OTel callbacks never read the per-destination setting, so a team could register no_content and still export prompts and replies under the global capture mode. Registration now refuses the var unless the proxy runs with LITELLM_OTEL_V2=true, the same rule the Arize transport and New Relic routing vars follow. * test(otel): type the OTel v2 fixtures and the v2-off regression test * fix(otel): accept only no_content and span_only on a team or key destination OTel v2 writes captured content only to span attributes, so event_only behaved like no_content and span_and_event like span_only on a destination. The proxy-wide setting keeps all four values * docs(otel): say an omitted capture setting follows the proxy's own callback for that backend * test(otel): describe the general capture fallback in the unowned destination test --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
63d823d28a
|
fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927)
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility * fix: preserve HTTPX compatibility across OpenAI SDK versions * refactor: trim OpenAI SDK compatibility patch * refactor: simplify SDK client defaults * test: cover SDK compatibility across provider routes * fix: use shared names for SDK transport helpers * fix: address SDK compatibility review feedback * test(openai): send requests through the SDK API factory clients * fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2 * ci(base-sdk): count httpx2 as base-only when a base dependency brings it in OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure * test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively * test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10 --------- Co-authored-by: Marcus Wood <marcuswood@openai.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
308c42a4a2
|
fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698)
* fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): shed OpenInference output tool calls individually under the span attribute budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): avoid mutation in Arize OTel v2 integration helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): audit Arize OTel v2 OpenInference spans across endpoints, modes and chaos Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): wait for each exported span before the next request in Arize OTel v2 audit tests * test(otel): make Arize OTel v2 audit absence and outage checks deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): collect Arize OTel v2 outage spans through an in-order sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add skipped BUG cells for pre-existing Arize OTel v2 gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): repair Arize OTel v2 regressions from #43698 (linear fit, metadata slot, repr tool args) (#44488) * fix(otel): keep Arize OTel v2 regressions in check — O(n) fit, metadata slot, repr tool args Three regressions from #43698's OpenInference tool-call/metadata emission: 1. Metadata evicted indexed message attributes: the new `metadata` key competed for the 128-attribute span budget, and the fit sheds whole message groups BEFORE `span.set_attribute`, so the SDK's dropped counter stayed 0 — an invisible eviction (live A/B: input-message attributes 86 -> 84). Two-part fix: the fit pins `metadata` behind every message group (it sheds only once all indexed messages are gone), and the span budget no longer charges pre-set attributes the mappers overwrite in place — a boundary-opened LLM span already carries keys like `gen_ai.request.model`, so the old accounting reserved slots the fit could never spend. Live: 86 input-message attributes with the metadata attribute riding alongside. 2. Quadratic shed on long prompts: `_message_shed_groups` rescanned the full group map once per message (measured on a real acompletion: 0.032/0.128/0.478/1.910s at 1000/2000/4000/8000 messages vs 0.007/0.010/0.022/0.029s at base). Index the tool-call groups once by (family, message index): the fit is linear again (0.008/0.007/0.014/ 0.031s, same rig). 3. Malformed Python tool arguments lost the whole span: provider adapters and `model_construct` responses hand over raw objects, and `json.dumps` raises on tuple-keyed dicts (TypeError) and cycles (ValueError) before the span is exported. Serialize with a repr fallback; both cases now export with a readable arguments attribute. The attribute budget change affects every boundary-opened LLM-call span (strictly more attributes retained, never fewer); the mapper changes only touch the OpenInference vocabulary. * refactor(otel): build the tool-call group index in one shot Review follow-up: the dict.setdefault/append seeding in _tool_call_groups_by_message violated the no-mutation coding convention (AGENTS.md: build values in one shot with comprehensions or generators wrapped in tuple()/MappingProxyType()). Rebuild it as a sorted groupby comprehension; randomized parity harness confirms the shed order is byte-identical to the seeded version (400 trials). Also pin the overflow corner Greptile asked about: a pre-set indexed-message key the fit sheds keeps its earlier value in place, so the span total can never exceed the SDK limit (new emitter test). * fix(otel): key the groupby with an explicit tuple to keep basedpyright at budget The slice-keyed groupby (group[:2]) widened the key to tuple[str | int], adding one reportGeneralTypeIssues over the codebase ceiling. Key by the explicit (family, message index) pair instead; shed order unchanged (300-trial randomized parity harness). * fix(otel): read pre-set span keys through a helper typed for both runtime shapes The SDK annotates ReadableSpan.attributes as a Mapping, but an ended span hands back a tuple of pairs, so the inline isinstance branch narrowed to Never and pushed reportGeneralTypeIssues one over the codebase ceiling. Extract _carried_keys with the runtime union declared on the parameter; behavior unchanged. * test(otel): drop the wall-clock bound from the long-prompt attribute fit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): annotate to_openai_dict as Mapping after the rebase onto main Main's type-discipline budget tightened since the branch point; the plain dict return annotation is the one violation the rebased branch adds. Callers only serialize the result, so the read-only view is accurate. * chore(otel): drop the restating docstring from to_openai_dict --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e2971e0af4
|
refactor(llms): expose public names for private provider helpers (#45037)
* refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
a193c67347
|
feat(otel): trace auto-router configuration and classifier failures (#44926) | ||
|
devin-ai-integration[bot]
|
26a9b02f7b
|
feat(otel): let team and key Arize callbacks choose the OTLP transport (#44492)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
46d2c2a6ea
|
fix(otel): honor per-team Arize sampling rates in OTel v2 fan-out (#44595)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
44d5dacbaa
|
fix(otel): gate the Arize OTel v2 exporter on operator credentials (#44596)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
80f18e1326
|
fix(otel): name an Arize project on every OTel v2 Arize export (#44605)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b4fcc5c1bc
|
fix(otel): read the registered v2 logger without importing the proxy (#44485)
* fix(otel): read the registered v2 logger without importing the proxy phase_span, which the router enters on every deployment pick since #44150, looked up the proxy's OTel logger by importing litellm.proxy.proxy_server. In an SDK process that import loads the whole proxy synchronously on the caller's event loop during its first request, which stalled litellm_router_unit_testing past its 5s wait. Read the module from sys.modules instead: when the proxy was never imported it has no registered logger. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): pin the no-proxy-import guarantee in-process Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4c6c84afb7
|
perf(proxy): stop prompt-cache eligibility from tokenizing the whole conversation (#44221)
is_prompt_caching_valid_prompt ran the full Python token_counter over every message to compare against the deployment's prompt cache minimum, 500 to 1000 ms at 440k to 740k tokens on every request that reaches the prompt_caching pre-call check, Rust on or off. messages_reach_token_count does the same arithmetic as token_counter(...) >= threshold and stops at the first message that reaches the threshold. Groups with one healthy deployment skip the prefix hash and pin lookup, which cannot change the result for them Four fixed name span events make the pre-LLM phases measurable with OTel v2: litellm.request.body_received (with body_bytes) once per body read before parsing, on the JSON, binary and form branches, body_parsed, pre_call_completed, and deployment_selected emitted once per pick inside Router.async_get_available_deployment and get_available_deployment with attempt, reason and model group, so every router surface, retry and fallback is covered. Measured locally on /v1/chat/completions, /v1/messages and /v1/responses at 440k tokens with Rust on and off against a fake upstream Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
797353f13a
|
fix(otel): name postgres service spans by operation and table (#44240)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
564d236985
|
fix(otel): nest cache spans under their operation and name service spans by purpose (#44150)
Response cache reads and writes open cache.get llm_response and cache.set llm_response phase spans with their Redis spans nested underneath, on the Python path and on the native Rust path, and deployment selection runs inside a route {model_group} phase so the cooldown, usage and model-id reads the router issues nest under it before chat {model}. The autorouter classifier call nests under that route phase as well and carries its typed internal origin on litellm.request.purpose, so it is told apart from the provider attempt. Service spans are named {service}.{verb} {target} from a low-cardinality key family the producer declares (llm_response, auth_objects, spend_counters, router_cooldowns, claude_code_session_router_binding, rate_limits, pod_lock, budget_reset, ...) instead of the raw method or a per-request pipeline length; a pipeline flush is targeted by the one family its ops share or by mixed with the sorted families on litellm.redis.families, a batch op keeps the family it was declared under whichever pipeline or standalone read settles it, and the ambient family labels Redis spans only, never the DB write-back a task spawned inside that context performs later. The raw method stays on litellm.service.call_type and on the Prometheus and Datadog labels. Caller attribution is carried across asyncio task boundaries on a ContextVar so forwarder-only chains no longer surface, the raw cache key is dropped from Redis span metadata, pipeline op counts land as an integer attribute, every call_type the Redis cache layer emits maps to a verb, and a scan over litellm/ and enterprise/ fails when a Redis producer, batch reservation included, declares no key family.
A V2 logger built for a key or team logging entry while the operator's V2 logger is already registered keeps only the exporters its own preset contributed, whether or not the operator holds credentials for that backend, so every chat span no longer reaches the operator's collector twice. A span the success callback has to open itself, with no pre-call carrier, starts at the provider handoff (api_call_start_time) instead of the logging object's creation.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
f95446ea39
|
fix(otel): tolerate non-dict callback_settings.otel and ignore bare EXCLUDED_SERVICES env (#44086)
* test(otel): cover non-dict callback_settings.otel and bare EXCLUDED_SERVICES env Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): tolerate non-dict callback_settings.otel and ignore bare EXCLUDED_SERVICES env Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(otel): simplify settings_customise_sources signature Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): poll for present spans instead of waiting the full window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): audit null otel block and bare EXCLUDED_SERVICES across the excluded-services matrix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): assert per-trace datastore spans in the unconfigured burst cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep pydantic-settings runtime options on the OTel v2 env source Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): require post-auth datastore spans at the tenant on the cache-hit twin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): cover pydantic-settings runtime options in callback_settings.otel through the proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
50f5cc9bbb
|
feat(otel v2): excluded_services opt-out for datastore spans on tenant destinations (#43278)
* feat(otel): excluded_services opt-out for datastore spans on tenant destinations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep upstream support unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): excluded_services resolves from the otel callback config only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): name the otel callback logger so excluded_services owner lookup matches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert no aux datastore traces reach the tenant sink Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): read bogus-start proxy log from the results dir Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): read only this invocation's bogus-start proxy log Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert operator kept db spans over the whole recorded window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): split operator db-span asserts by trace scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): build the otel logger after preset callbacks and validate the exclusion env at boot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): tolerate a bogus exclusion env when callback config wins Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep bogus exclusion env fatal when a preset parses it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): hoist the preset check out of the callback loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): use a rule-scoped pyright suppression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): log and drop unknown excluded_services instead of failing boot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the operator spend-writer span before checking the tenant for postgres Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): leave callback init and boot untouched when excluded_services is unset Read callback_settings.otel.excluded_services directly instead of making the otel callback build its own logger, and drop the new boot-time parse of callback_settings.otel, so a proxy without the setting behaves exactly as on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): normalize callback_settings excluded_services without rereading OTel env vars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): log and ignore malformed excluded_services instead of failing startup Lowercase and trim names, drop non-string items, and add an integration matrix over endpoints, clients, cache hits, destination outages and setting shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel v2): cover failed upstream calls in the excluded_services matrix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel v2): pin operator Langfuse credentials in preset-only excluded_services tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mrinal <mrinal@berri.ai> |
||
|
Ankit Jha
|
60fca8298e
|
fix(otel): send cache and reasoning tokens in langfuse usage_details (#43553)
* fix(otel): send cache and reasoning tokens in langfuse usage_details The OTel V2 Langfuse mapper only sent input, output and total, so cache reads, cache writes and reasoning tokens never reached Langfuse. Emit them as input_cached_tokens, input_cache_creation and output_reasoning_tokens, and send input/output net of those buckets so Langfuse does not price the same tokens twice. Fixes #43542 * fix(otel): drop redundant comments from the usage_details change |
||
|
devin-ai-integration[bot]
|
7aba77197d
|
feat(otel): add SigNoz preset for OpenTelemetry v2 (#43296)
Some checks are pending
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests / caching-local (push) Waiting to run
Unit Tests / mcp-integration (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
* feat(otel): add SigNoz preset for OpenTelemetry v2 Adds the signoz callback (OTLP/HTTP exporter, GenAI vocabulary, key and team level dynamic ingestion endpoint and key) as an OpenTelemetry v2 preset, with the preset factory accepting the allow_missing_credentials kwarg the V2 registry always passes so construction no longer falls back silently to legacy OpenTelemetry. Ships the deterministic tests/integration/observability/test_signoz_delivery.py audit suite Absorbs the work from https://github.com/BerriAI/litellm/pull/38206 Co-authored-by: Nagesh Bansal <nageshbansal59@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(otel): drop explanatory comments from the SigNoz preset Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(types): keep signoz dynamic param lines within ruff format width Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(signoz): assert the missing-endpoint boot path directly instead of in an except block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for the signoz health service Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): allowlist SigNoz key/team endpoints and route keyless collectors without the operator key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): terminate the SigNoz shutdown cell before the flush and drop test docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(otel): keep the shared tenant routing untouched and require an ingestion key for SigNoz key/team endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): warn about a keyless SigNoz team endpoint from the header resolver so the shared cache actually reaches it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Nagesh Bansal <nageshbansal59@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d18fcb09d6
|
fix(otel): detach post-response service spans by request phase, name redis spans by operation (#43237)
* fix(otel): detach post-response service spans by request phase, name redis spans by operation Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): mark anthropic messages stream cache write as post-response phase The /v1/messages streaming cache writer awaits async_add_cache inline instead of going through create_cache_write_task, so its redis span stayed parented under the request trace. Wrap the write in post_response_phase so it detaches like the chat completions write. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): write the Messages stream cache in a background task after handoff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
cf491d1df9
|
test: move tests/test_litellm integrations and secret_managers into tests/unit (#43194)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |