* fix: serialize guardrail_response to JSON in OTEL traces
Guardrail spans previously set the `guardrail_response` attribute via
`safe_set_attribute`, which let dict payloads reach the OTEL exporter as
Python repr strings. Downstream log pipelines could not parse those as
JSON, breaking metric creation from guardrail traces.
Serialize `guardrail_response` with `safe_dumps` before setting the
attribute, matching how `masked_entity_count` is already handled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover dict-serialization and None-skip for guardrail_response
Address Greptile feedback on #28362 — add explicit coverage for the
two behavioral guarantees of this fix:
- Dict payloads (the OpenAI moderation case in the report) reach the
span as a JSON string, not a Python repr.
- ``None`` guardrail_response skips the attribute entirely, so no
``"null"`` leaks into traces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `_set_team_attributes_on_span` helper to stamp team_id/team_alias
onto any span, ensuring these attributes are not limited to the root
litellm_request span
- Add `_set_team_attributes_from_kwargs` helper to extract team metadata
from the standard_logging_object in kwargs and apply them to a span
- Apply team attributes to raw request spans via `_maybe_log_raw_request`
so downstream consumers can filter traces by team without needing the
root span
- Apply team attributes to guardrail spans so guardrail activity can be
correlated to teams in tracing backends
- Apply team attributes to exception logging spans to preserve team
context during failure paths
- Add comprehensive unit tests covering all new helpers, including edge
cases where metadata or standard_logging_object is absent
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
The proxy SERVER span ("Received Proxy Server Request") only carried
http.response.status_code on failures (set in _record_exception_on_span),
so success traces had no 2xx bucket — error-ratio and status-breakdown
dashboards were missing their denominator and the span violated the HTTP
semconv (the attribute is required whenever a response is sent). Add a
set_response_status_code_attribute helper and call it from
async_post_call_success_hook with 200, symmetric with the failure path
and the existing route/preprocessing-duration SERVER-span attributes.
* feat(otel): expose http.response.status_code on failure spans
Set the OTel-standard http.response.status_code (integer) on failure
spans alongside the existing OpenInference error.code (kept for
back-compat). error.type is already emitted via ERROR_TYPE.
Crucially, also record structured error attributes on the proxy SERVER
span ('Received Proxy Server Request') from async_post_call_failure_hook
- the only place the SERVER span is in hand. _handle_failure records on
the litellm_request child span (the parent span is not propagated into
its kwargs), so prior to this change the SERVER span that dashboards
query carried only span status, never error.code/error.type. Reuses
_record_exception_on_span + StandardLoggingPayloadSetup.get_error_information
so values match the child span.
Tests: recorder unit coverage + a hook-driven test asserting the SERVER
span is stamped (the gap recorder-only tests missed). Full
test_opentelemetry.py suite: 197 passed.
* feat(otel): set http.route + url.path on the proxy SERVER span
Add the OTel-standard http.route (low-cardinality route template, e.g.
/v1/threads/{thread_id}/runs) and url.path (literal path) to the SERVER
span ('Received Proxy Server Request') so dashboards can group traffic
by endpoint instead of seeing every path param as a unique value.
Same architectural gap as the status-code commit: the success/failure
logging handlers write the litellm_request CHILD span, and
_handle_success explicitly refuses to copy to the SERVER span. Verified
with a console-exporter run that the SERVER span was bare on success.
Unlike error info, route/path are known at request time, so set them
directly on the freshly-created SERVER span in user_api_key_auth (one
edit point, works for success and failure, no hook-ordering risk):
- http.route from the matched FastAPI route (scope['route'].path),
empirically confirmed populated at auth-dependency time.
- url.path from the existing literal-path variable.
New get_request_route_template helper + set_proxy_request_route_attributes
(no-op on None span, so the Langfuse override stays safe).
Tests: route-attribute setter + route-template helper edges. Full
test_opentelemetry.py and test_auth_utils.py green.
* feat(otel): set litellm.preprocessing.duration_ms on the proxy SERVER span
Expose the total time LiteLLM spends before the upstream provider
request begins (auth + parsing + pre-call hooks) as a single number on
the SERVER span ('Received Proxy Server Request'). Window:
proxy-receive -> FIRST provider handoff.
Retry semantics: first attempt only (pure preprocessing, excludes
retry loops + backoff). api_call_start_time is overwritten on every
attempt, so a set-once first_api_call_start_time pins the first handoff.
Same architectural gap as the prior two commits: the success/failure
logging handlers write the litellm_request CHILD span, not the SERVER
span. Set it instead from the post-call hooks on
user_api_key_dict.parent_otel_span.
Failure-path subtlety: request_data.pop('litellm_logging_obj') runs
before the failure-hook loop, so the failure hook can't read the
logging object. litellm_received_at is propagated via the existing
request->metadata channel, and first_api_call_start_time is mirrored
onto litellm_params.metadata, so both anchors survive into request_data
and the OTel helper reads them uniformly for success and failure.
Edits: user_api_key_auth (stash receive instant), litellm_pre_call_utils
(propagate it), litellm_logging (set-once first handoff + metadata
mirror), opentelemetry (constant + set_preprocessing_duration_attribute,
called from both post-call hooks).
Tests: duration helper (both container shapes, missing/negative/None
edges) + set-once invariant (retry doesn't overwrite, metadata mirror).
test_opentelemetry.py + test_auth_utils.py + test_litellm_logging.py:
447 passed. Verified live: SERVER span carries the attribute on success
and failure, coexisting with the status-code and route attributes.
* fix(otel): MyPy type-narrowing for status-code + preprocessing-duration
No behavior change. MyPy (CI lint) flagged:
- error_information["error_code"] is str|None: narrow via a None-checked
local before int().
- _to_timestamp returns Optional[float]: resolve both anchors and return
early if either is None instead of subtracting possibly-None floats.
* fix(otel): stop polluting user request metadata with first_api_call_start_time
The PR3 set-once preprocessing anchor was mirrored into
litellm_params["metadata"] from core litellm_logging.py. That dict is
the caller's request metadata, mutated in place and shared across every
call path including pure SDK (litellm.acreate_batch). It got echoed into
LiteLLMBatch(metadata=...), which the OpenAI batch schema types as
Dict[str, str] -> pydantic ValidationError on a datetime value.
- litellm_logging.py: set first_api_call_start_time only on
model_call_details (success path reads it there directly).
- proxy/utils.py: post_call_failure_hook lifts it off the logging object
into request_data (internal top-level key, same convention as the
other proxy-internal request_data keys) right before the existing
litellm_logging_obj pop. Never touches user metadata.
- opentelemetry.py: read the anchor from the container top level
(model_call_details on success, request_data on failure).
- Tests updated; add TestPostCallFailureHookLiftsFirstApiCallStartTime.
Fixes the batches_testing regression introduced on this branch.
* chore(otel): trim verbose comments to concise rationale
Collapse multi-line why-blocks to one or two lines and drop process/plan references (PR-numbering, "the plan") from test comments. No behavior change.
- Introduce `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` opt-in that switches OTEL traces to conform with the OpenTelemetry GenAI semantic conventions specification
- Extract all semconv behavior into a new `OTELGenAISemconvMixin` class in `gen_ai_semconv.py`, mixed into `OpenTelemetry` to keep concerns separated
- In semconv mode, span name follows `{operation} {model}` pattern (e.g. `chat gpt-4`) and span kind is set to `CLIENT` instead of legacy `litellm_request`
- Replace `gen_ai.system` with `gen_ai.provider.name` and drop `llm.is_streaming` in semconv mode; add `gen_ai.request.{frequency_penalty,presence_penalty,top_k,seed,stop_sequences,stream,choice.count}` and `gen_ai.usage.cache_{creation,read}.input_tokens` attributes
- Replace per-message `gen_ai.content.prompt` / per-choice `gen_ai.content.completion` log events with a single consolidated `gen_ai.client.inference.operation.details` event; omit `gen_ai.input/output.messages` when content capture is disabled
- Suppress the non-standard `raw_gen_ai_request` child span entirely in semconv mode
- Support both programmatic (`OpenTelemetryConfig.semconv_stability_opt_in` field) and environment variable activation; the two sources are unioned so either or both can enable the opt-in
- Extract OTEL SDK `LogRecord` / `SeverityNumber` version-compatibility shim into a reusable `_otel_log_types()` static method to deduplicate the `< 1.39.0` / `>= 1.39.0` import branching
- Add 30+ unit tests covering opt-in gating, span naming, attribute emission/omission rules, stop sequence normalization, cache token attributes, and the consolidated event lifecycle
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
1. Missing litellm_request child span when proxy parent in metadata:
_get_span_context now returns (ctx, None) for the metadata-injected
proxy parent so the primary span is always emitted as a child of ctx.
Proxy span lifecycle managed by new _end_proxy_span_from_kwargs.
2. open_telemetry_logger overwrite by later handlers:
_init_otel_logger_on_litellm_proxy now uses first-registered-wins —
only assigns proxy_server.open_telemetry_logger when currently None.
3. Duplicate litellm_request success spans in streaming paths:
Added _mark_success_span_once with per-handler dedupe key stored in
kwargs metadata, suppressing the second span when both sync and async
success callbacks fire for the same request.
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).
The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
- Replace isinstance(item, dict) with hasattr(item, 'get') so Pydantic
model instances (ResponseOutputMessage, ResponseFunctionToolCall) are
accepted alongside plain dicts (P1)
- Use 'is not None' guards instead of or-chain for system_instructions
coalescing to prevent falsy values (e.g. []) falling through to the
wrong kwarg (P2)
- Emit per-tool-call span attributes (gen_ai.completion.N.function_call.*)
for Responses API function_call items, matching the choices branch
parity with _tool_calls_kv_pair (P2)
- Add 4 new tests: Pydantic-like objects, falsy fallthrough guard,
per-tool-call attribute emission, multiple tool call indexing
Addresses 4 critical OpenTelemetry span issues in LiteLLM:
Issue #3: Remove redundant attributes from raw_gen_ai_request spans
- Removed self.set_attributes() call that was duplicating all parent span
attributes (gen_ai.*, metadata.*) onto the raw span
- Raw span now only contains provider-specific llm.{provider}.* attributes
- Reduces storage and eliminates search confusion from duplicate data
Issue #4: Prevent attribute duplication on litellm_proxy_request parent span
- When litellm_request child span exists, removed redundant
set_attributes() call on the parent proxy span
- Child span already carries all attributes; parent duplication doubles
storage and complicates search
Issue #5: Fix orphaned guardrail traces
- Guardrail spans were created with context=None when no parent proxy span
existed, resulting in orphaned root spans (separate trace_id)
- Added _resolve_guardrail_context() helper to ensure guardrails always
have a valid parent (litellm_request or proxy span)
- Applied fix to both _handle_success and _handle_failure paths
Issue #8: Add gen_ai.response.id for embeddings and image generation
- EmbeddingResponse and ImageResponse types don't have provider response IDs
- Added fallback to standard_logging_payload["id"] (litellm call ID) for
correlation across LiteLLM UI, Phoenix traces, and provider logs
- Completions still use provider ID (e.g. "chatcmpl-xxx") when available
Tests added:
- TestRawSpanAttributeIsolation: Verify raw span has no gen_ai/metadata attrs
- TestNoParentSpanDuplication: Verify parent span doesn't get duplicated attrs
- TestGuardrailSpanParenting: Verify guardrails are children (not orphaned)
- TestResponseIdFallback: Verify response ID set for all call types
All existing OTEL tests pass (73 passed, 14 pre-existing protocol failures).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When OpenTelemetry is configured via the UI, only OTEL_ENDPOINT and
OTEL_HEADERS are set, but OTEL_EXPORTER is not specified. This caused
the exporter to default to "console", meaning traces were printed to
stdout instead of being sent to the configured endpoint.
This fix adds logic in OpenTelemetryConfig.__post_init__ to automatically
infer "otlp_http" as the exporter when an endpoint is specified but the
exporter is still the default "console".
Fixes issue reported by Elastic team where traces weren't being sent
to their OTEL endpoint when configured through the LiteLLM UI.
* feat: change guardrail_information to list type to support displaying multiple guardrails
* fix: add missing commit and revert auto-format changes in utils.py
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* fix(opentelemetry.py): fix issue where headers were not being split correctly
* feat(bedrock/image): Support bedrock titan image generation
Closes https://github.com/BerriAI/litellm/issues/361
* build(model_prices_and_context_window.json): track titan image gen pricing
enables cost tracking per request
* feat(amazon_titan_transformation.py): support titan image generation cost tracking
* docs: document new model
* docs: update docs to indicate cost tracking + refactor rerank into separate doc
* fix: fix mypy linting error
* fix: fix type ignore
* fix(opentelemetry.py): initial commit adding support for OTEL_RESOURCE_ATTRIBUTES
* test: update unit tests
* feat(ui/): initial commit with revamped model hub
* fix(model_hub_table.tsx): generic 'supports_' rendering as capabilities
* feat(model_hub_table.tsx): indicate if a model is publicly available
* refactor(model_hub_table.tsx): refactor to use common model data table component
* fix(model_hub_table.tsx): fix box sizes
* fix(model_hub_table.tsx): enable selecting / deselecting models in columns
* feat(public_endpoints.py): initial commit adding `/public/model_hub` endpoint
enables sharing public models
* feat(public_endpoints/): instrumentation for public model hub route
* feat(proxy_server.py): support request access form for model groups
allows user to request access to a model
* refactor: use a dictionary of text + link instead of 1 hardcoded request access form
* fix(proxy_server.py): prevent duplicates in model_group info
* fix: fix linting error
* fix(__init__.py): fix linting error
