* fix(otel): hashable scope for _emit_once when guardrail_mode is list
`_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a
guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]`
(the shape Presidio expands to with `output_parse_pii: true`, and the
shape `event_hook` carries for any `mode: [...]` in config), the tuple
contains a list and `spans_logged.get(dedupe_key)` raises
`TypeError: unhashable type: 'list'`. On the post-call path this fires
inside the logging callback and is swallowed; the request returns 200 but
the OTEL `guardrail` span is silently dropped. On the blocking path the
same error surfaces as HTTP 500.
Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists
and tuples into tuples, sets into frozensets, dicts into frozensets of
`(key, value)` pairs, and falls back to `repr` for arbitrary
unhashables. Applied inside `_emit_once` before the dict lookup, so all
three callsites are protected without touching the guardrail-specific
callsite. Helper assumes acyclic input; `guardrail_mode` values are
built fresh from config (str enums, lists of str enums, TypedDict of
str/list-of-str), so no cycle can arise in practice.
Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash,
distinct-list-scope collision, dict and set scope parts, and an
end-to-end `_create_guardrail_span` exercise that confirms exactly one
`guardrail` span is emitted across repeated lifecycle entrypoints. Each
new test fails on a reverted helper (4/4 mutation kill)
* fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector
CI's recursive_detector blocks new recursive functions in litellm/ unless they
are in the allowlist with a documented bound. Cap the helper at 16 levels and
return repr(value) past the cap; this is well past the realistic depth of
guardrail_mode (1-3 levels) and means a future caller passing a cyclic
container can no longer push the proxy logging path into a RecursionError.
Add a regression test that exercises the cycle path.
* refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union
Per review feedback from @mateo-berri: replace the loose `-> object` annotation
with a recursive `HashableScope` union (str | int | float | bool | bytes | None
| Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract
is visible at the signature. Replace the `try/except hash(value); return value`
passthrough with an explicit isinstance check over the hashable-scalar types so
the type checker can narrow without requiring `cast(Hashable, value)` on the
return. Symmetric: dict keys also flow through the freezer (a TypedDict key is
already a string in practice, so behaviorally identical). All 16 regression
tests still pass; mutation kill behavior preserved
* fix: avoid explicit casting
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* fix(otel): cap metric attribute cardinality with include/exclude lists
OTEL metrics stamped every per-request hidden_params and metadata.* field
onto each gen_ai.client.* sample, so near-unique values created one metric
time series per request and backends like Splunk Observability Cloud throttled
and dropped the data.
Add an attributes block under callback_settings.otel with mutually-exclusive
include_list (allowlist) and exclude_list (denylist), validated against the
known attribute names at startup and applied once to the metric attributes in
_record_metrics. Spans are untouched, and with no config every attribute is
still emitted so existing setups are unaffected.
Resolves LIT-3600
* fix(otel): resolve metric attribute filter from callback_settings
The proxy usually constructs the OpenTelemetry logger without forwarding the
attributes kwarg, while the filter lives under
litellm.callback_settings["otel"]["attributes"]. __init__ only read the kwarg,
so the recording instance kept config.attributes=None and shipped metrics at
full cardinality even when the filter was configured; a live proxy run exposed
this. Fall back to the global at init for the base otel logger, and add a
regression test that drives the real success hook through the callback_settings
path (the unit tests passed before because they injected the config directly).
* fix(otel): reject gen_ai.token.type from metric attribute filter lists
gen_ai.token.type was a member of VALID_METRIC_ATTRIBUTE_NAMES, so an
operator could list it in include_list or exclude_list and pass startup
validation. The attribute is injected into the input/output token series
after _filter_metric_attributes runs, so the filter never sees it and the
request silently has no effect.
Reject it loudly from either list instead, matching the contract that a
non-actionable attribute name fails fast rather than falling through to a
no-op. It stays a structural discriminator on the token-usage histogram.
* fix(otel): resolve metric attribute filter lazily at record time
The proxy constructs the OpenTelemetry logger before it populates
litellm.callback_settings["otel"]["attributes"], so resolving the filter at
__init__ left config.attributes None and shipped metrics at full cardinality. A
live proxy run confirmed the leak. Resolve the filter on the first metric record
instead, when callback_settings is populated, while still validating an explicit
config eagerly so a bad SDK config fails at startup. The regression test now
constructs the logger before populating callback_settings to mirror that
ordering, so it fails if the filter is resolved too early.
* fix(otel): don't cache invalid filter on lazy callback_settings path
On the lazy callback_settings resolution path, _ensure_metric_attribute_filter
wrote self.config.attributes before validating it. When validation then failed,
_metric_attr_filter_resolved stayed False while config.attributes held the bad
filter, so the next record skipped the callback_settings re-read and re-raised
the stale error indefinitely; fixing the misconfiguration required a restart.
Drop the premature write and resolve from the local value. A subsequent record
now re-reads callback_settings, so a corrected config takes effect without a
restart. The write was dead on the success path anyway, since the resolved
frozensets are what the filter reads.
* Fix incorrect agent API request example payload structure (#29556)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs
On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span
is stored in litellm_params['litellm_metadata'] instead of
litellm_params['metadata']. When the request body contains a native
'metadata' field (e.g. Anthropic's {"user_id": "..."}),
litellm_params['metadata'] gets overwritten and the parent span is lost,
producing orphan root spans with a different trace_id.
Add fallback checks to litellm_metadata in:
- _get_span_context(): so child spans find the correct parent
- _end_proxy_span_from_kwargs(): so the proxy span gets closed
Fixes: https://github.com/BerriAI/litellm/issues/27934
* test(otel): tighten assertions per Greptile review
- test_span_context_metadata_takes_priority: assert litellm_metadata
span is never accessed, proving metadata takes priority
- test_span_context_no_parent_when_neither_has_span: assert both ctx
and detected_span are None
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: remove premature end-user budget check from get_end_user_object (#29420)
* fix(proxy): remove premature end-user budget check from get_end_user_object
Problem:
- `_check_end_user_budget()` was called inside `get_end_user_object()`
- This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated
- Zero-cost models (e.g., local vLLM) were incorrectly blocked when
end-users exceeded their budget, even though they should bypass budget checks
Solution:
- Remove `_check_end_user_budget()` calls from `get_end_user_object()`
- Budget enforcement now happens exclusively in `common_checks()` where
`skip_budget_checks` context is available
- `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation.
* refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object
- test_get_end_user_object() verifies data fetching
- test_check_end_user_budget() verifies enforcement
- test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget()
- test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object()
* Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534)
* Fix Gemini image config mapping
* Address Gemini image config review
* Format Gemini image generation transform
* Fix Gemini image token usage logging
* Share Gemini image request helpers
* Fix Gemini Imagen model routing
* Fixes as per self code review
* Fixes per internal code review
* Stop gating Imagen imageSize forwarding
* Document Gemini image size mapping source
* chore: retrigger lint
* Clarify Gemini candidate count precedence
* Add Inception provider (#29522)
* add inception as provider (chat, fim)
* linting
* seperate test suite for chat and fim
* fix test coverage
* fix: model hub custom pricing model info (#29293)
* Opik user auth key metadata extractors (#28397)
* fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic
* test: add unit tests for OPik metadata extraction logic
* fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy
* fix(ci): clarified comments and edited unit tests
* test: add unit tests for OPik metadata extraction with auth and requester overrides
* fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532)
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
* fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561)
`_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls`
so a following tool result can be matched back to its tool call. The assignment
was inside a branch guarded by
`assistant_msg.get("tool_calls", []) is not None`, which is also True for a
text-only assistant message (an empty list is not None). As a result, an
assistant message with no tool calls that appears between a tool call and its
tool result overwrote the reference, and conversion failed with:
Exception: Missing corresponding tool call for tool response message.
This shape is common: a model emits a short narration/assistant message after a
tool call before the tool result is appended.
Only update `last_message_with_tool_calls` when the assistant message actually
carries tool_calls (or a function_call). Adds a regression test.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models
The 1-hour prompt-cache write tier
(`cache_creation_input_token_cost_above_1hr`) was added to the
us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but
the eu./au./jp. cross-region inference profiles were left without it.
AWS Bedrock pricing applies the same +10% regional premium across all
geo profiles, so eu./au./jp. should carry the same 1-hour rates as
us. (1.6x the 5-minute regional rate).
Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL
prompt caching falls back to the 5-minute write rate and undercounts
spend by ~60% for European, Australian, and Japanese tenants.
Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where
AWS publishes one) to 14 regional Bedrock entries in both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- eu./au. Opus 4.6 ($11.00 / MTok)
- eu./au. Opus 4.7 ($11.00 / MTok)
- eu./au./jp. Sonnet 4.6 ($6.60 / MTok)
- eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC)
- eu./au./jp. Haiku 4.5 ($2.20 / MTok)
Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py`
with a `REGIONAL_EXPECTED` parametrized block covering all 13 new
entries plus the existing 1.6x ratio invariant.
Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the
wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06),
which would break the 1.6x ratio check. It is intentionally left out
of this PR so the scope stays "1-hour cache tier addition" — a
separate follow-up should correct the EU 5m rates for Opus 4.5.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models
GCP Vertex AI publishes a separate 1-hour cache write column for the
Claude family (1.6x the 5-minute write rate, matching the documented
Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the
5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}`
on Vertex AI Claude is undercounted in cost tracking by ~60%.
The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig`
extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and
`_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`.
Only the price registry was missing data.
Adds the field to 19 vertex_ai/claude-* entries across both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- Haiku 4.5 ($1.25 -> $2.00 / MTok)
- Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok)
- Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok)
- Opus 4 / 4.1 ($18.75 -> $30.00 / MTok)
Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py`
mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model
and asserts the 1.6x ratio across the family.
Fixes#27781.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Fix Gemini multimodal function responses (#29325)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* address greptile review: add _transform_image_usage method and model-map supports_image_size flag
- Add _transform_image_usage instance method to GoogleImageGenConfig that
delegates to transform_gemini_image_usage, fixing the regression test
- Replace hardcoded "2.5-flash" string check in supports_gemini_image_size
with a get_model_info lookup on supports_image_size (default true)
- Add supports_image_size: false to all gemini-2.5-flash model entries in
model_prices_and_context_window.json so capability is controlled via the
model map rather than embedded in code
* fix test failures: schema validation, mypy type, model info plumbing, pricing test
- Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it
- Pass supports_image_size through _get_model_info_helper constructor call
- Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True)
- Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid
- Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values
* Add Azure AI Kimi K2.6 metadata (#27052)
* Add Azure AI Kimi K2.6 metadata
* Scope Kimi metadata test cost map setup
* fall back to substring check for models not in model_prices_and_context_window.json
Models like gemini-2.5-flash-image-preview are not in the pricing JSON,
so get_model_info raises. Fall back to "2.5-flash" not in model when the
JSON has no explicit supports_image_size entry for the model.
* fix(inception): don't forward global litellm.api_key to Inception FIM
Match the Inception chat config: resolve only an Inception-specific key
(param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion
FIM path. The global litellm.api_key (often an OpenAI key) was both leaking
to api.inceptionlabs.ai and taking precedence over the configured Inception
key when set.
* fix(auth): enforce end-user budget on custom-auth path that skips common_checks
get_end_user_object() no longer raises BudgetExceededError, so custom-auth
deployments with custom_auth_run_common_checks unset (which skip the
centralized common_checks gate) stopped enforcing the end-user budget,
letting an over-budget end user keep making requests. Re-enforce the
budget in _run_post_custom_auth_checks on that path.
---------
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com>
Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com>
Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com>
Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk>
Co-authored-by: Lovro Seder <vrovro@gmail.com>
Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com>
Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: serialize guardrail_response to JSON in OTEL traces
Guardrail spans previously set the `guardrail_response` attribute via
`safe_set_attribute`, which let dict payloads reach the OTEL exporter as
Python repr strings. Downstream log pipelines could not parse those as
JSON, breaking metric creation from guardrail traces.
Serialize `guardrail_response` with `safe_dumps` before setting the
attribute, matching how `masked_entity_count` is already handled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover dict-serialization and None-skip for guardrail_response
Address Greptile feedback on #28362 — add explicit coverage for the
two behavioral guarantees of this fix:
- Dict payloads (the OpenAI moderation case in the report) reach the
span as a JSON string, not a Python repr.
- ``None`` guardrail_response skips the attribute entirely, so no
``"null"`` leaks into traces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `_set_team_attributes_on_span` helper to stamp team_id/team_alias
onto any span, ensuring these attributes are not limited to the root
litellm_request span
- Add `_set_team_attributes_from_kwargs` helper to extract team metadata
from the standard_logging_object in kwargs and apply them to a span
- Apply team attributes to raw request spans via `_maybe_log_raw_request`
so downstream consumers can filter traces by team without needing the
root span
- Apply team attributes to guardrail spans so guardrail activity can be
correlated to teams in tracing backends
- Apply team attributes to exception logging spans to preserve team
context during failure paths
- Add comprehensive unit tests covering all new helpers, including edge
cases where metadata or standard_logging_object is absent
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
The proxy SERVER span ("Received Proxy Server Request") only carried
http.response.status_code on failures (set in _record_exception_on_span),
so success traces had no 2xx bucket — error-ratio and status-breakdown
dashboards were missing their denominator and the span violated the HTTP
semconv (the attribute is required whenever a response is sent). Add a
set_response_status_code_attribute helper and call it from
async_post_call_success_hook with 200, symmetric with the failure path
and the existing route/preprocessing-duration SERVER-span attributes.
* feat(otel): expose http.response.status_code on failure spans
Set the OTel-standard http.response.status_code (integer) on failure
spans alongside the existing OpenInference error.code (kept for
back-compat). error.type is already emitted via ERROR_TYPE.
Crucially, also record structured error attributes on the proxy SERVER
span ('Received Proxy Server Request') from async_post_call_failure_hook
- the only place the SERVER span is in hand. _handle_failure records on
the litellm_request child span (the parent span is not propagated into
its kwargs), so prior to this change the SERVER span that dashboards
query carried only span status, never error.code/error.type. Reuses
_record_exception_on_span + StandardLoggingPayloadSetup.get_error_information
so values match the child span.
Tests: recorder unit coverage + a hook-driven test asserting the SERVER
span is stamped (the gap recorder-only tests missed). Full
test_opentelemetry.py suite: 197 passed.
* feat(otel): set http.route + url.path on the proxy SERVER span
Add the OTel-standard http.route (low-cardinality route template, e.g.
/v1/threads/{thread_id}/runs) and url.path (literal path) to the SERVER
span ('Received Proxy Server Request') so dashboards can group traffic
by endpoint instead of seeing every path param as a unique value.
Same architectural gap as the status-code commit: the success/failure
logging handlers write the litellm_request CHILD span, and
_handle_success explicitly refuses to copy to the SERVER span. Verified
with a console-exporter run that the SERVER span was bare on success.
Unlike error info, route/path are known at request time, so set them
directly on the freshly-created SERVER span in user_api_key_auth (one
edit point, works for success and failure, no hook-ordering risk):
- http.route from the matched FastAPI route (scope['route'].path),
empirically confirmed populated at auth-dependency time.
- url.path from the existing literal-path variable.
New get_request_route_template helper + set_proxy_request_route_attributes
(no-op on None span, so the Langfuse override stays safe).
Tests: route-attribute setter + route-template helper edges. Full
test_opentelemetry.py and test_auth_utils.py green.
* feat(otel): set litellm.preprocessing.duration_ms on the proxy SERVER span
Expose the total time LiteLLM spends before the upstream provider
request begins (auth + parsing + pre-call hooks) as a single number on
the SERVER span ('Received Proxy Server Request'). Window:
proxy-receive -> FIRST provider handoff.
Retry semantics: first attempt only (pure preprocessing, excludes
retry loops + backoff). api_call_start_time is overwritten on every
attempt, so a set-once first_api_call_start_time pins the first handoff.
Same architectural gap as the prior two commits: the success/failure
logging handlers write the litellm_request CHILD span, not the SERVER
span. Set it instead from the post-call hooks on
user_api_key_dict.parent_otel_span.
Failure-path subtlety: request_data.pop('litellm_logging_obj') runs
before the failure-hook loop, so the failure hook can't read the
logging object. litellm_received_at is propagated via the existing
request->metadata channel, and first_api_call_start_time is mirrored
onto litellm_params.metadata, so both anchors survive into request_data
and the OTel helper reads them uniformly for success and failure.
Edits: user_api_key_auth (stash receive instant), litellm_pre_call_utils
(propagate it), litellm_logging (set-once first handoff + metadata
mirror), opentelemetry (constant + set_preprocessing_duration_attribute,
called from both post-call hooks).
Tests: duration helper (both container shapes, missing/negative/None
edges) + set-once invariant (retry doesn't overwrite, metadata mirror).
test_opentelemetry.py + test_auth_utils.py + test_litellm_logging.py:
447 passed. Verified live: SERVER span carries the attribute on success
and failure, coexisting with the status-code and route attributes.
* fix(otel): MyPy type-narrowing for status-code + preprocessing-duration
No behavior change. MyPy (CI lint) flagged:
- error_information["error_code"] is str|None: narrow via a None-checked
local before int().
- _to_timestamp returns Optional[float]: resolve both anchors and return
early if either is None instead of subtracting possibly-None floats.
* fix(otel): stop polluting user request metadata with first_api_call_start_time
The PR3 set-once preprocessing anchor was mirrored into
litellm_params["metadata"] from core litellm_logging.py. That dict is
the caller's request metadata, mutated in place and shared across every
call path including pure SDK (litellm.acreate_batch). It got echoed into
LiteLLMBatch(metadata=...), which the OpenAI batch schema types as
Dict[str, str] -> pydantic ValidationError on a datetime value.
- litellm_logging.py: set first_api_call_start_time only on
model_call_details (success path reads it there directly).
- proxy/utils.py: post_call_failure_hook lifts it off the logging object
into request_data (internal top-level key, same convention as the
other proxy-internal request_data keys) right before the existing
litellm_logging_obj pop. Never touches user metadata.
- opentelemetry.py: read the anchor from the container top level
(model_call_details on success, request_data on failure).
- Tests updated; add TestPostCallFailureHookLiftsFirstApiCallStartTime.
Fixes the batches_testing regression introduced on this branch.
* chore(otel): trim verbose comments to concise rationale
Collapse multi-line why-blocks to one or two lines and drop process/plan references (PR-numbering, "the plan") from test comments. No behavior change.
- Introduce `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` opt-in that switches OTEL traces to conform with the OpenTelemetry GenAI semantic conventions specification
- Extract all semconv behavior into a new `OTELGenAISemconvMixin` class in `gen_ai_semconv.py`, mixed into `OpenTelemetry` to keep concerns separated
- In semconv mode, span name follows `{operation} {model}` pattern (e.g. `chat gpt-4`) and span kind is set to `CLIENT` instead of legacy `litellm_request`
- Replace `gen_ai.system` with `gen_ai.provider.name` and drop `llm.is_streaming` in semconv mode; add `gen_ai.request.{frequency_penalty,presence_penalty,top_k,seed,stop_sequences,stream,choice.count}` and `gen_ai.usage.cache_{creation,read}.input_tokens` attributes
- Replace per-message `gen_ai.content.prompt` / per-choice `gen_ai.content.completion` log events with a single consolidated `gen_ai.client.inference.operation.details` event; omit `gen_ai.input/output.messages` when content capture is disabled
- Suppress the non-standard `raw_gen_ai_request` child span entirely in semconv mode
- Support both programmatic (`OpenTelemetryConfig.semconv_stability_opt_in` field) and environment variable activation; the two sources are unioned so either or both can enable the opt-in
- Extract OTEL SDK `LogRecord` / `SeverityNumber` version-compatibility shim into a reusable `_otel_log_types()` static method to deduplicate the `< 1.39.0` / `>= 1.39.0` import branching
- Add 30+ unit tests covering opt-in gating, span naming, attribute emission/omission rules, stop sequence normalization, cache token attributes, and the consolidated event lifecycle
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
1. Missing litellm_request child span when proxy parent in metadata:
_get_span_context now returns (ctx, None) for the metadata-injected
proxy parent so the primary span is always emitted as a child of ctx.
Proxy span lifecycle managed by new _end_proxy_span_from_kwargs.
2. open_telemetry_logger overwrite by later handlers:
_init_otel_logger_on_litellm_proxy now uses first-registered-wins —
only assigns proxy_server.open_telemetry_logger when currently None.
3. Duplicate litellm_request success spans in streaming paths:
Added _mark_success_span_once with per-handler dedupe key stored in
kwargs metadata, suppressing the second span when both sync and async
success callbacks fire for the same request.
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).
The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
- Replace isinstance(item, dict) with hasattr(item, 'get') so Pydantic
model instances (ResponseOutputMessage, ResponseFunctionToolCall) are
accepted alongside plain dicts (P1)
- Use 'is not None' guards instead of or-chain for system_instructions
coalescing to prevent falsy values (e.g. []) falling through to the
wrong kwarg (P2)
- Emit per-tool-call span attributes (gen_ai.completion.N.function_call.*)
for Responses API function_call items, matching the choices branch
parity with _tool_calls_kv_pair (P2)
- Add 4 new tests: Pydantic-like objects, falsy fallthrough guard,
per-tool-call attribute emission, multiple tool call indexing
Addresses 4 critical OpenTelemetry span issues in LiteLLM:
Issue #3: Remove redundant attributes from raw_gen_ai_request spans
- Removed self.set_attributes() call that was duplicating all parent span
attributes (gen_ai.*, metadata.*) onto the raw span
- Raw span now only contains provider-specific llm.{provider}.* attributes
- Reduces storage and eliminates search confusion from duplicate data
Issue #4: Prevent attribute duplication on litellm_proxy_request parent span
- When litellm_request child span exists, removed redundant
set_attributes() call on the parent proxy span
- Child span already carries all attributes; parent duplication doubles
storage and complicates search
Issue #5: Fix orphaned guardrail traces
- Guardrail spans were created with context=None when no parent proxy span
existed, resulting in orphaned root spans (separate trace_id)
- Added _resolve_guardrail_context() helper to ensure guardrails always
have a valid parent (litellm_request or proxy span)
- Applied fix to both _handle_success and _handle_failure paths
Issue #8: Add gen_ai.response.id for embeddings and image generation
- EmbeddingResponse and ImageResponse types don't have provider response IDs
- Added fallback to standard_logging_payload["id"] (litellm call ID) for
correlation across LiteLLM UI, Phoenix traces, and provider logs
- Completions still use provider ID (e.g. "chatcmpl-xxx") when available
Tests added:
- TestRawSpanAttributeIsolation: Verify raw span has no gen_ai/metadata attrs
- TestNoParentSpanDuplication: Verify parent span doesn't get duplicated attrs
- TestGuardrailSpanParenting: Verify guardrails are children (not orphaned)
- TestResponseIdFallback: Verify response ID set for all call types
All existing OTEL tests pass (73 passed, 14 pre-existing protocol failures).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When OpenTelemetry is configured via the UI, only OTEL_ENDPOINT and
OTEL_HEADERS are set, but OTEL_EXPORTER is not specified. This caused
the exporter to default to "console", meaning traces were printed to
stdout instead of being sent to the configured endpoint.
This fix adds logic in OpenTelemetryConfig.__post_init__ to automatically
infer "otlp_http" as the exporter when an endpoint is specified but the
exporter is still the default "console".
Fixes issue reported by Elastic team where traces weren't being sent
to their OTEL endpoint when configured through the LiteLLM UI.
* feat: change guardrail_information to list type to support displaying multiple guardrails
* fix: add missing commit and revert auto-format changes in utils.py
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* fix(opentelemetry.py): fix issue where headers were not being split correctly
* feat(bedrock/image): Support bedrock titan image generation
Closes https://github.com/BerriAI/litellm/issues/361
* build(model_prices_and_context_window.json): track titan image gen pricing
enables cost tracking per request
* feat(amazon_titan_transformation.py): support titan image generation cost tracking
* docs: document new model
* docs: update docs to indicate cost tracking + refactor rerank into separate doc
* fix: fix mypy linting error
* fix: fix type ignore
* fix(opentelemetry.py): initial commit adding support for OTEL_RESOURCE_ATTRIBUTES
* test: update unit tests
* feat(ui/): initial commit with revamped model hub
* fix(model_hub_table.tsx): generic 'supports_' rendering as capabilities
* feat(model_hub_table.tsx): indicate if a model is publicly available
* refactor(model_hub_table.tsx): refactor to use common model data table component
* fix(model_hub_table.tsx): fix box sizes
* fix(model_hub_table.tsx): enable selecting / deselecting models in columns
* feat(public_endpoints.py): initial commit adding `/public/model_hub` endpoint
enables sharing public models
* feat(public_endpoints/): instrumentation for public model hub route
* feat(proxy_server.py): support request access form for model groups
allows user to request access to a model
* refactor: use a dictionary of text + link instead of 1 hardcoded request access form
* fix(proxy_server.py): prevent duplicates in model_group info
* fix: fix linting error
* fix(__init__.py): fix linting error
