The async gate already checks every string for the data URI markers before it copies a few large strings inline, so the copy now keeps those strings as they are instead of scanning each one again. SSE-shaped tool output, which holds both ';' and 'data:', was paying for that scan twice on the event loop.
Check for ';', 'data:', and ';base64,' before running the data URI regex, and
offload to a worker thread only when a request holds a data URI. Tests now watch
every regex call and confirm plain text is copied without yielding to the loop.
* fix(proxy): list TypeSafe Jev models in the admin model picker
* fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids
Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones.
* fix(typesafe): match only published Jev ids and keep picker loads off the secret manager
Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file.
* test(typesafe): type the picker test fixture and tests
* fix(typesafe): match only published Jev versions and cover non-version suffixes in tests
* fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker
* fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers
The dashboard sends include_model_access_groups on every /models call, so
gate the TypeSafe names on wildcard routes too and skip access-group-only
listings. Also say the typesafe-jev-ids rule matches case-insensitively.
* fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main
A typesafe/* router deployment registers typesafe/* in the cost map, so the
allowlist pickers offered a bare *, which allows every model. The pickers now
list only priced names the rule sends to TypeSafe. The non-expand list also
stops de-duplicating every entry and only skips TypeSafe names already listed.
* fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost
The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287.
* fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates
* style: format the chat-over-Responses bridge helper signature
* refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain
* fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress
* fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies
* test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes
* refactor(stream): drop the docstring on _stamp_usage_cost, the body says it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream
The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.
SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF
Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed
Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.
Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.
Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.
* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests
Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.
* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines
* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter
* test: type the iterator class parameters in the line-separator stream tests
* fix(responses): stream tool call arguments opened in a text chunk
* fix(responses): hold a tool call that shares a chunk with text until the text ends
* test: type the a2a SSE handler helper and the sandbox fake responses
---------
Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them
* fix(streaming): surface upstream abort finish reasons as errors
* fix(streaming): keep SDK chunks with null choices streaming past the abort check
* fix(streaming): treat null choices as empty in the raw SSE abort check
* refactor(streaming): bind raw SSE choices as Final instead of rebinding
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust-bridge): require the native extension
_native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader
The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ci): build Rust bridge in Windows smoke tests
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): derive Responses session id from prompt_cache_key
ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.
Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.
Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
per-request and are skipped, checked in both "metadata" and
"litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
to a stable conversation key
Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.
Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.
* style: ruff format
* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments
- hash prompt_cache_keys containing header-unsafe chars instead of
mapping them through _safe_header_value, whose replacement char could
collapse distinct keys onto one cache shard and serialize unrelated
requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
file per AGENTS.md test-placement rule
* fix(chatgpt): keep session typing and test diff scoped
* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests
* test(chatgpt): record upstream requests with a mock instead of appending to a list
* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys
* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid
* test(chatgpt): type the session id test parameters
* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key
* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence
* fix(chatgpt): keep the trace-id session marker out of provider params
The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(fallback-generalizations): remove the legacy extends and mixed-rule shim
Main's remote cost map, the bundled backup, and the root map all ship the split
routing/capability rule format, so the shim for the old format is dead code. A rule
mixing litellm_provider with other model_info keys is now warned about and skipped.
* test(fallback-generalizations): fail when a shipped cost map rule would be skipped at install
* test(fallback-generalizations): cover ignored extends and annotate the shipped-rules test
* fix(fallback-generalizations): warn about old-format rules with a pointer to a current cost map and skip extends rules
* fix(fallback-generalizations): tell local cost map users to replace the bundled backup in the old-format warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): dict-usage arm no longer drops token detail fields
In _dispatch_provider_chunk's openai/azure else-branch, a plain-dict
usage was rebuilt into litellm.Usage from only prompt_tokens /
completion_tokens / total_tokens. prompt_tokens_details (cached_tokens),
completion_tokens_details (reasoning_tokens) and cache token counts were
silently dropped — while both sibling arms (Usage passthrough, BaseModel
via Usage(**model_dump())) preserve them.
Every in-repo chunk producer currently normalizes usage to
Usage/BaseModel before this point, so the dict arm is reachable today
only via chunks litellm did not construct (e.g. a custom
streaming_decoder feeding CustomStreamWrapper). When it does fire on a
usage-only final chunk (stream_options.include_usage), the stripped
Usage is exactly what the stream consumer receives — the early-return
path has no later aggregation to repair it.
Fix: pass the full dict through Usage(**usage_dict). Usage.__init__
coerces nested detail dicts into their wrapper types and tolerates
extra keys. Side effect: a legitimate 0 count is no longer coerced to
None by the old 'or None' expressions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop explanatory source comment per repo comment policy
Rationale lives in the PR body and the regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(streaming): type the dict-usage unpack for the basedpyright gate
Usage(**response_obj["usage"]) unpacks a dict whose value type pyright
cannot see, and the delta gate reports that once per typed constructor
parameter: five, against the three .get() calls the line replaced. Moving
the unpack behind _usage_from_payload keeps the same Usage(**payload)
call and is reported once, at the call site, so the rule count ends below
its merge base
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials
* fix(proxy): drop caller metadata from proxy-only failure logs
metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata.
* fix(proxy): keep allowlisted metadata in proxy-only failure logs
* test(proxy): isolate proxy-only failure log metadata tests
* fix(logging): ignore non-list request tags when building the logging payload
A guardrail-rejected request with metadata tags set to a non-list value crashed
the standard logging payload, so its spend row lost model attribution.
* fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs
A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend.
* fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata
allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata.
* refactor(proxy): keep the proxy-only failure-log fixes inside the type gates
Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change.
* fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent
The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(logging): cover extra spend tag header casing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): match spend tag headers case-insensitively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): validate the guardrail trace with one pydantic model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): record the native guardrail entry when litellm_params metadata is None
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): import the renamed messages invoke transformation module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): walk the guardrail trace iteratively when redacting it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type the native guardrail trace fixtures and locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages
Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): keep Final variables out of loops in the rename shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): match the warning text in the rename shim test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): mark the alias loader spec restore for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): format the renamed package shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): keep the rebind-ok marker on the spec restore line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): carry the new package search path on alias specs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): declare the shim test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): drop the legacy max_tokens field in favor of max_input_tokens and max_output_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop assertions that pinned the retired cost-map max_tokens field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): stop re-raising and re-logging a failed stream on every next()
A stream that raised keeps its provider iterator alive, so a consumer that iterates past the error re-detected, re-raised and re-logged the same failure on every call, flooding failure callbacks for one request. Latch the failure and re-raise it without logging again.
* fix(streaming): drop the retained traceback before re-raising a failed stream
* test(streaming): count failures through a real logging callback and require a clean stop on later reads
* test(streaming): require the stuck stream to close early and release its socket
The provider stream now records a GeneratorExit and caps its chunks, so draining it or detecting the loop late fails. Two wire cases check a sync and an async stuck stream release their TCP session.
* test(streaming): avoid failure recorder name collision
---------
Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
* feat(cost): parse 12-hour cache write rates for batch and ultrafast tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill 12-hour batch cache writes once when only the 12hr batch key exists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill OpenCode Go requests from the reported list_cost
OpenCode returns usage.opencode.list_cost on every response. Read it into the provider-reported cost the cost calculator checks before the price map, for chat and Responses, streamed and not. Keep the static rows as the fallback, and stop the generic usage.cost path from billing Go requests at 0 (usage.cost is what hit the prepaid balance, 0 inside the plan).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(core): resolve reported cost breakdowns without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(opencode): cover reported list_cost spend through the proxy
Replaces the recursive _reported_cost_value with a flat _cost_number helper
so the code-quality gate passes, and adds an integration test that runs the
proxy against a scripted OpenCode upstream for a non-stream and a stream
chat completion, asserting the response cost header and the spend log row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill native /v1/messages from the reported list_cost
The native Anthropic Messages route priced OpenCode Go bodies from the price
rows twice over: the non-stream body is a TypedDict the response-cost header
path priced as-is, and the stream assembled its ModelResponse with the rows
cost already stamped. Both now read usage.opencode.list_cost, the non-stream
body through billable_result and the stream from the final message_delta.
The integration test now covers chat, responses and messages, stream and
non-stream, through the proxy against a scripted OpenCode upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): keep the stream cost stamp inside the lint gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): read list_cost off dict chunks and trust it only from OpenCode on /v1/messages
Stream builders receive chunks both as mappings and as ModelResponseStream
objects, so the list_cost scan now reads usage through chunk_usage. The
native Messages route only trusts a reported cost from opencode_go, every
other upstream keeps the price map, which also keeps a zero usage.cost from
being billed as free. Served bodies without stop_sequence still price, and
the new code stays inside the basedpyright gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(typing): replace pass-through Any with object in 13 annotations
* docs(client): say dict[str, object] in the models get docstring
* refactor(typing): drop the unused Span aliases in base_cache and levo
* refactor(typing): take a Mapping in delete_nested_value and cover the fresh copy
* fix(typing): deep-copy a read-only Mapping in delete_nested_value through a dict
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)
* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)
* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts
* fix: restore pass-through behavior on paths the typing pass tightened
MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites
* fix(runwayml): keep a malformed 2xx task body on the 500 path
* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report
EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.
`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.
The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.
* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed
* test: annotate the new regression tests' locals with Final
* test: annotate the remaining new test locals with Final
* feat: add OpenAI 12-hour cache write pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: omit empty cache-write response fields
* style: format cache-write fields
* fix(cost): bill openai 30m and 12h cache writes from the usage split, add gpt-5.6 rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): apply the 12h long-context write rate only above the token threshold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): key cache write rates by TTL instead of growing the rate tuple
_get_token_base_cost returned a positional 6-tuple and every cache lifetime added a slot to it, to the off-peak helper and to every caller. It now returns a TokenBaseCosts dataclass whose cache write rates live in one mapping keyed by TTL. The TTLs come from the ephemeral_*_input_tokens fields on CacheCreationTokenDetails, so supporting a new lifetime only means adding that usage field and its rate key
A lifetime with no rate of its own now always falls back to the cache creation rate after off-peak pricing is applied. Before this the non-tiered 12h rate and the tiered 1h rate fell back to the rate from before off-peak, so the same request billed differently depending on the pricing path
* refactor(cost): hardcode the cache write TTLs as a Literal instead of parsing usage field names
Parsing ephemeral_*_input_tokens names saved nothing, since a new TTL still needs a usage field and provider parsing code. The TTLs are now a CacheWriteTTL Literal with an explicit rate key per TTL
* refactor(cost): resolve cache write lifetime rates to floats at lookup
Decide the 1h and 12h cost map key in the threshold loop, then resolve each
rate once after off-peak with the base write rate as the default, so no None
flows through the cost code. Add an integration case for a 12h write on a
row without a 12h rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): name every cache write TTL, marking 5m and 30m as base rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): list every cache write TTL in one rate key table, None meaning the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): drop base rate entries from the cache write rate table and rely on fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): key cache write rates by plain string TTL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): fill missing cache write TTL rates with the base write rate after off-peak
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): default missing cache write TTL rates to the base write rate inline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): rename cache write rate key table to make clear it only covers long TTLs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): list every cache write TTL and bill writes of an unknown TTL at the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): resolve tiered cache write TTL rates after off-peak instead of passing None
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): carry cache write TTL rates on TokenRates and drop TokenBaseCosts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): define TokenRates before its first use to drop the string annotation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): build the TokenRates TTL default with default_factory and type tiers as Mapping
MappingProxyType({}) as a dataclass default is rejected on Python 3.11, where
mappingproxy is unhashable, so import litellm failed there. The tier type
moves from dict[str, object] to Mapping[str, object] (LIT001), with tier_rate
accepting a Mapping to match.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): type select_tier_for_input tiers as Mapping so tier_rate callers stay typed
The tier returned as a bare dict made every tier_rate call site a partially
unknown argument once tier_rate took Mapping[str, object]. The range bounds are
now validated with a TypeAdapter instead of indexing an unknown list.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own
* fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced
* test: isolate the offline-mode tokenizer tests from the local Hugging Face cache
* test: cover the offline policy templates and the cached Kimi K3 tokenizer
* test(integration): count Kimi K3 tokens in the offline-mode egress test
* refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(deps): upgrade vulnerable Python dependencies
* fix(deps): lock patched Python dependency versions
* fix(otel): preserve JSON export with the updated SDK
* test(otel): cover JSON export with the updated SDK
* test(otel): update SDK log records and worker discovery
* fix(ui): align Lens peer dependency with patched Next.js
* fix(otel): hand the AgentOps exporter a session it owns for the JWT header
OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it
The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals
* fix(otel): skip FastAPI instrumentation once the app has started
opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects
* test(otel): assert exported requests and span flags instead of SDK internals
OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags
* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it
zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6
* test(ui): stub the tag summary fetch the client panel retry refreshes
Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing
* fix(deps): drop the duplicated Lens override from the dashboard package.json
* style(ui): format the client panel fetch stub
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(responses): round-trip Anthropic compaction blocks as compaction items
/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.
Fixes#41456
* fix(responses): replay compaction blocks natively only when the request compacts
* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails
Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.
* fix(responses): count, route, guard, and redact compaction items consistently
A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.
* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it
* fix(responses): keep a stored compaction summary whole in spend logs
* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning
A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.
* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order
Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.
* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes
A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.
* fix(spend_logs): lift only the redaction opt-in header into failure rows
The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main
* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes
* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary
A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.
* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id
* fix(responses): count only message items with a string role as compaction instructions
* test(responses): flatten the last two-for comprehension in the compaction wire test
* fix(responses): replay from a compaction in stored Responses output and SDK compaction items
A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.
* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history
* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary
A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.
* fix(responses): replay a compacted session only through the referenced response
A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item
* fix(spend-logs): keep main's stored value for a prompt length limit under 2
At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value
* fix(spend-logs): keep main's stored tail for a negative prompt length limit
At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start
* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages
* test(responses): run the compaction wire tests on the worker database under isolated workers
* fix(router): keep the encrypted content affinity pin out of provider requests
* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction
* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message
* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses
* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call
* test(proxy): run the dispatched-call-type failure test through the real request setup
* fix(proxy): count a failed Cursor stream's input from the newest compaction item
The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.
* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item
* fix(responses): keep a tool call a later output answers when trimming before a compaction item
* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes
* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time
* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests
* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item
* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item
* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content
* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks
* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once
* fix(responses): count input_tokens after a compaction item the way inference sends it
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* fix(llms): join api_base and route with one shared helper
A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base
* refactor(llms): move per-provider api_base strips onto join_api_base
Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join
* ci: run the trailing-slash URL builder test in the llms shard
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
* fix(llms): send every provider route through join_api_base
Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it
Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call
The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash
* fix(llms): keep the route skip only where a builder already had it
join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path
already ends in it, and it is used only where the merge base or the trailing-slash test PR head
already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI
model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the
path so a host named anthropic-* builds /anthropic/v1/messages
* fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL
* test(llms): run the Gemini RAG trailing-slash case through aingest
The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears
* test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base
* test(integration): cover api_base joins across providers, query strings and named hosts
* feat(url_utils): add join_api_base and ensure_api_route helpers
* fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host
join_api_base and ensure_api_route now keep the // of a base left with only a scheme,
so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client
base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it
* fix(azure_ai): build the count_tokens URL from the messages URL
* fix(azure_ai): match anthropic as a whole path segment in the messages URL
* fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints
Azure AI chat and messages go back to cutting the base at the first
/anthropic, as on main, and count_tokens keeps its path check, so a
gateway path like /anthropic-proxy builds the same URLs as main. A base
that already ends in /v1/messages still gets count_tokens next to it.
A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is
used as its text again in AgentCore, Bedrock responses and the RAGFlow
base, instead of falling back to the default host.
get_volcengine_base_url and GigaChat's get_api_base strip a trailing
slash again for outside callers. The cache-hit wire test is skipped as a
known proxy bug, since one cache hit writes two cache_hit spend rows.
* fix(llms): drop a slash before the api_base query and tighten the route test checks
* test(llms): annotate new test locals as Final and flatten the builder generator
* fix(url_utils): keep a scheme-only api_base hostless so it fails before sending
---------
Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>
A guardrail hook plants a litellm_metadata key on a chat completions request after the proxy
stamped litellm_received_at into metadata. get_litellm_metadata_from_kwargs prefers that key
when both exist and only copied the user_api_key_* fields across, so the response timing
headers lost their receive anchor and x-litellm-overhead-duration-ms reported 2 ms on
/v1/chat/completions against 58 ms on /v1/responses for the same proxy work. The receive stamp
now rides along with the user_api_key_* fields.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage
A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object
across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception,
and a second drop never reached the failure handlers because the first had already marked the object as
logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the
router discards the stashed usage right before it opens the next attempt, on same-group retries and on
every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage.
* fix(router): dispatch chat stream failures through the attempt-scoped logging copy
The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types.
* test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy
* test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader
The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead.
* test(router): use real Logging in fallback cooldown test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): preserve superseded retry usage for billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill retry input after prior stream usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): track retry dispatch for terminal billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): use deterministic retry timestamp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): avoid real clock in fallback tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): remove duplicate streaming logger import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): retain error-frame usage for exhausted retries
* test(proxy): fix retry billing expectations and helper annotations
* fix(proxy): preserve failure estimate without carried usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): audit retry helpers and fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fix retry fixture timestamps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe retry stash at attempt boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): keep terminal failure attribution when dropped attempts log on copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish a dropped attempt's payload only while it is still the current attempt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover spend-log encoding error propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): cool down the final fallback attempt when its failure callbacks were blocked
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): import Final in adapter regression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop spend-log encoding expectation superseded by main's poison isolation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): finish the wrapped stream's failure log before surfacing the error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin pass-through stream fixture start times
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload before awaiting failure callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload from the failure handler after baseline invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging): drop unused discard_partial_usage_for_failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover billing every round of a streamed MCP auto-execute
* fix(responses): bill every round of a streamed MCP auto-execute on one spend row
* fix(responses): bill every round of a streamed MCP auto-execute on bridged providers
A streamed /v1/responses MCP auto-execute on a provider bridged through chat
completions (for example Anthropic) billed only round 1, because each round's
chat stream dispatched on the shared logging object and the first dispatch won.
Bridged rounds now go through the same one-row billing as native rounds. A
round is billed as soon as its chat stream ends, before its closing events
reach the client. The row carries the final response, usage and cost summed
over every round, and the fixed margin once. A round served from the response
cache is billed at zero. A follow-up chat stream that breaks bills the paid
rounds on its failure row. A single-round stream logs exactly as before.
* fix(responses): charge one fixed margin and keep bridged usage fields on streamed MCP rounds
A follow-up round that fails after earlier rounds were billed now charges the
per-request fixed margin once, and its failure row carries no cost breakdown
that would disagree with the summed cost.
A bridged multi-round stream logs a plain cache_hit flag like a single-round
stream does, and its usage keeps the summed provider usage fields of every
round.
A new test covers a stream dropped after its event loop closed, which must
release its parked log without an unawaited coroutine warning.
* fix(responses): price each streamed MCP round on its own and bill every round when a guardrail blocks
Each round of a streamed MCP auto-execute is now priced from its own
uncached cost, so a provider that states its cost (xAI) charges the stated
cost of every round plus the fixed margin once, and the cost breakdown
always agrees with the billed cost.
Summed usage goes through a JSON-safe dump, so a nested provider field such
as xAI's context_details no longer breaks a failed stream's response.failed
event.
A post-call guardrail that blocks a multi-round bridged stream now drops the
parked success log and bills every round on the failure row, and a failed
follow-up round's failure row carries the summed breakdown of both rounds.
* fix(responses): keep earlier MCP rounds on their success row when a later round fails
When a later round of a streamed MCP auto-execute fails, the rounds before
it are now logged on their success row, and the failure row bills only the
failed round with no second fixed margin. That failure log runs on its own
copy of the request's logging state, so its usage and cost never leak into
the earlier rounds' row.
A post-call guardrail that blocks a native multi-round stream keeps the
summed success row beside the $0 failure row, as at the merge base, while a
bridged stream still bills every round on its failure row.
* fix(responses): keep a failed later MCP round on its own failure row
A streamed MCP auto-execute whose later round failed wrote only the earlier
rounds' success row. The failure was logged on a copy of the logging state,
while the proxy builds its failure row from the request's own state, which by
then held the success payload, so the failure row took the success row's id
and was dropped.
Now the failure is logged on the request's own state as at the merge base,
and once a later round starts, the success row for the rounds before it is
logged from a copy of the state taken at that moment, so the two rows never
mix.
* fix(proxy): let an MCP tool call reuse its request's max_parallel_requests slot
* test(proxy): create the other-request task in a way Python 3.10 supports and mark its bindings Final
* fix(proxy): keep a reused max_parallel slot until the request's last round
An MCP tool call that reuses its request's max_parallel_requests slot no
longer loses it to the first round's late success callback, so the key
stays counted for the rounds still to run. The reused call also stops
replacing the request's rate limit headers.
* fix(responses): keep an unpriced streamed MCP round unpriced instead of billing it as free
* fix(responses): log a bridged MCP stream as a chat completion with every round's usage
* fix(router): unregister strategy selectors from global callbacks on discard
Router.discard only removed its own bound methods from litellm's callback
lists, so the strategy selectors it registered (least-busy, usage-based,
latency-based, cost-based and routing group selectors) stayed behind and
kept receiving events after the router was gone. Reuse the identity based
unregister that update_settings already uses so a discarded router takes
only its own selectors with it
* fix(router): purge discarded selectors from every event callback list
function_setup copies litellm.callbacks into the success and failure
lists on every request, so a router that served traffic still leaked its
selectors there after discard. Route the cleanup through
remove_callback_from_all_lists, which now also covers input_callback, and
fold the selector gathering into discard so the router coverage check
sees no untested helper
* fix(router): keep live routers, in-flight hooks and user_config limits intact on discard
discard() now rebuilds every global callback list without the router's selectors, optional pre-call checks and own usage callbacks instead of removing them in place, so a callback loop already running never skips a hook. It then re-registers what the remaining live routers need, so a same-strategy survivor keeps its deduped handler. The proxy's per-request user_config router is discarded only after its call returns.
* fix(router): bind user_config handlers to the request and tighten discard
When a user_config request carries a logging object, its router is
discarded before the request runs and its handlers ride on that
logging object, so v1 rpm and v2 tpm limits count every request and
concurrent requests count once. Without one, the router is discarded
after the request as before. The router's pre-call loops also read
the request-scoped handlers.
discard() now refills only the handler keys it vacated, guards the
callback lists and the live-router set with a lock, covers routers
built without a model_list, and drops its Slack alerting, adaptive
router hook and budget limiter sync task
* fix(proxy): reuse one user_config router per config and bind its handlers to each request
Every user_config request now goes through a cached router built once per
config. Its strategy handlers leave the global callback lists and are bound
to the request's logging object instead, so limits count each request once
and nothing is left behind after the request.
* fix(router): keep the proxy budget limiter on user_config requests
A user_config router now swaps in only its strategy selectors, so the proxy's
RouterBudgetLimiting stays on the request and keeps counting its spend. Usage
logged after a strategy is retired, such as a request still in flight when its
user_config router is evicted, now schedules one more push to Redis.
* fix(proxy): init the retire fields on the key model budget limiter
The key model budget limiter builds its own state instead of calling the base
init, so with the proxy's Redis attached every spend increment raised on the
missing retire flag and the key's model budget never counted spend.
* fix(router): bind the deployment callbacks to user_config requests
A cached user_config router serves requests through each request's logging
object but bound only its selectors and policies there, so a failing
deployment was never cooled down and its usage counters missed requests.
* fix(router): retry an evicted router's budget writes and keep the adaptive header on user_config requests
A retired budget limiter now keeps retrying its queued Redis increments every
sync interval until Redis accepts them, instead of giving up after one failed
write.
The x-litellm-adaptive-router-model header now comes from one stateless hook in
litellm.callbacks that holds no router, so user_config adaptive routers, whose
per-router hooks are request-scoped, still answer with the header.
* fix(router): lock request callback setup against router cleanup and drain in-flight budget writes on retire
function_setup now edits the global callback lists under the same lock that
router cleanup uses, so a router dropping its callbacks on another thread can
no longer shift the indexes function_setup pops or put a dropped handler back.
A budget limiter retired while a Redis write is still in flight now keeps
retrying that batch if the write fails, instead of leaving the spend queued
with no task left to send it.
* fix(utils): set up loggers after releasing the callback lists lock
function_setup held the callback lists lock while set_callbacks ran its one-time
logger setup, which can block on a Slack package install, so other threads and
Router.discard waited on it. The lock now covers only the list reads and writes.
* test(router): annotate the retire test's locals as Final
* fix(logging): skip replaced global callbacks on the input path too
replace_global_callback already kept a same-key global logger out of
success and failure dispatch. pre_call and post_call still sent it the
input events, so a logger with an input hook could count a request start
it never saw finish. Both now share one filter with the success path
* test(logging): type the input-path regression test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): stop re-downloading a HuggingFace tokenizer after a failed download
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tokenizer): isolate HuggingFace tokenizer caches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): clear the traceback before re-raising a cached tokenizer load error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): retry failed HuggingFace tokenizer loads after the cache window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token_counter): start tokenizer retry TTL after a failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): lock every HuggingFace tokenizer load and keep load failures as values
Concurrent llama-2, llama-3 and cohere counts each started their own failed
download because only the bundled tokenizers took a load lock. Every tokenizer
kind now has one, so a burst during an outage makes a single attempt and two
counts can no longer read the same expiring entry at once.
A failed load is now remembered as a small value with the error message
instead of the error object, so re-raising it no longer keeps the first
failing request's frames alive. Adds tests for the concurrent outage and for
fork-guard errors staying raised and unremembered.
* fix(utils): keep Hugging Face tokenizer downloads unlocked and warn once per failed download
Reverts the per-kind load lock so concurrent first downloads still run in parallel, as #47353 intends. A short lock around the failure cache read and write stops two counts racing on an expiring entry, and the tiktoken fallback warning now fires only when a download fails, not on every count that reuses the remembered failure.
* fix(caching): read an entry another reader just expired as a cache miss
Two threads reading the same expiring key in InMemoryCache could both pass the membership check, and the second then raised KeyError after the first removed it. get_cache now treats that key as a miss. This replaces the extra lock the tokenizer failure cache took, and the regression test forces the interleaving without timing.
* fix(caching): treat a TTL another reader just removed as not expired
InMemoryCache's expiry check read the TTL after a separate membership check, so a reader racing another reader's eviction raised KeyError. It now reads the TTL once and treats a missing one as not expired, and get_cache already reads the vanished entry as a miss.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
