* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and
ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the
dict-gated marker write skipped them and the served hit still stamped the full
price. The marker write is now container-agnostic, which it already was for the
partial-embedding stamp, and a regression test exercises the carrier end to end.
_uncached_response_cost also restores billed_token_rates, zero_cost_warned, and
the response_cost_failure_debug_information key alongside cost_breakdown, so the
probe leaves no pricing state on the call it reads.
* test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler
* fix(logging): keep the uncached price probe's restores to the state it can leak
* fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read cache payload fields defensively without casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel): clarify pyright suppression reason on cache payload reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11
* fix(otel): report cache hits at zero cost everywhere and mark misses
Cache hits now carry a zero cost breakdown and a zero hidden response cost,
the /v1/messages hit cost header follows chat and responses hits, every span
carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps
the provider-prefixed model name, and the V2 attribute budget counts preset keys
* fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM
Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and
/v1/messages, while a fresh call whose cost is 0 still sends no header.
The async hit path stamps the same model name as main again. Stamping the
provider-prefixed name made a hit's tokens count toward the deployment's
TPM limit, so repeated cached requests hit 429s under usage-based routing.
Drop comments that restated fields and annotate new test values as Final.
* fix(otel): keep the served pricing basis on a cache hit's cost breakdown
A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path.
* fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys
A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0.
* test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans
* test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double
* test: annotate the new cache-hit unit test locals with Final
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
8226b2be9a