Commit graph

238 commits

Author SHA1 Message Date
devin-ai-integration[bot]
444423a5ca
fix(cost): bill a deployment's per-second rates on native Anthropic /v1/messages (#42417)
* fix(cost): honor per-second custom pricing on chat completions for every provider

* test(cost): assert a per-second priced deployment bills instead of staying at $0

The zero-cost diagnostic tests from #42345 used a per-second-only entry as their
misconfigured fixture, which this branch now bills. Switch that fixture to a
per-query-only entry, which is still selected as the deployment's own pricing and
still prices chat usage at $0, and add a per-second test asserting the call
duration is billed with no diagnostic

Also let a caller's explicit total_time outrank the logging window in
completion_cost, so the SDK precedence stays stamped response, caller, logging

* test(response_metadata): move the per-second pricing regression into the mapped tests/unit file

* fix(cost): bill a deployment's per-second rates on /v1/responses and /v1/messages

The proxy builds the request's logging state before the router picks a
deployment, and only the chat path re-read the deployment's
input_cost_per_second / output_cost_per_second afterwards. The shared
update step every other route runs (Logging.update_from_kwargs) now
seeds those custom pricing keys from the router-merged kwargs too, so
custom_pricing flips on and the cost calculator prices the call against
the router-registered deployment id on every surface.

* fix(cost): price a /v1/messages call off the response time the proxy reports

A /v1/messages result is a plain dict, so the proxy keeps its response time on
the logging object. The cost step now reads it there for a non-streamed call,
so the cost header and the spend log bill the same duration.

* fix(anthropic): bill a deployment's per-second rate only on the native /v1/messages route

Move the custom pricing copy from the shared logging refresh into the native
Messages handler, reusing the helper the Responses handler already had, so
generateContent and other routes keep their current billing

* fix(cost): pass only per-second rates to native /v1/messages cost tracking

Other deployment overrides keep billing as on main, matching chat's per-second gate

* test(anthropic): pin per-second-only pricing on native /v1/messages with a long-context-rate deployment

* fix(cost): bill native /v1/messages from logging setup, as main did, to the end the proxy reports

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 05:29:03 -07:00
tavren27
e7d3b24d7c
fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs (#48888)
* fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs and logging payloads

* fix(router): keep the mirror flag when litellm_metadata is set and strip a client-sent is_silent_experiment

Spend rows read router metadata from litellm_metadata when a call carries it, so the router now sets the flag there too. The proxy drops a client-sent is_silent_experiment with the other router-owned keys, so real traffic cannot be recorded as mirrored.

* fix(logging): mark mirrored calls with a router-only key so clients cannot set is_silent_experiment

Spend rows and the standard logging payload now derive is_silent_experiment from _litellm_silent_experiment, which only the router writes and the proxy and pass-through routes strip from client metadata. A caller's own is_silent_experiment key stays untouched, so it keeps its current effect on mirroring and on provider metadata, and unmirrored calls log false in both places.

* test(langfuse): expect is_silent_experiment false on every logged generation

* fix(logging): mark mirrored calls with a router-only enum value so no client route can set is_silent_experiment

The router stamps SilentExperimentMarker.MIRRORED, which client JSON can never produce, so batches, Responses WebSocket frames and pass-through need no strip. Langfuse layers the flag after its requester-metadata dedupe.

* fix(logging): set is_silent_experiment only on mirrored calls so real calls keep their payloads

Real calls no longer gain an is_silent_experiment key in spend rows or the standard logging payload, so OTEL, S3, GCS and every other logger receive what they did before. The Langfuse regression test now runs the real prompt helper.

* fix(logging): keep the mirror marker out of callback metadata and read the spend-row flag from the logging payload

Logging now strips the router's mirror marker from the request metadata buckets before any callback or vendor sees it, without touching the router's own dicts, so retries keep it. Spend rows read is_silent_experiment from the standard logging payload, which survives the collector's JSON round trip, so collector-mode rows keep the flag too.

* fix(logging): mark mirrored calls through a context variable so router usage counts them once

* fix(router): leave callback-started calls unmarked and stop client flags from turning mirroring off

Logging handlers now run outside the mirrored-call context, so an LLM call a success or failure callback starts while logging a mirrored call is not flagged as mirrored. The guard against mirroring a mirror reads the same context variable instead of request metadata, so a client-sent is_silent_experiment no longer skips the mirrors. The unused module-level get_standard_logging_metadata stops copying resolved keys from raw metadata.

* fix(router): keep main's client flag mirroring skip and the other logging metadata keys

A client's is_silent_experiment: true skips mirroring again as on main, the module-level get_standard_logging_metadata drops only the flag, and the silent experiment tests wait on a logged-call signal instead of polling a deadline

* fix(logging): take the mirror flag in Prometheus labels and Datadog cost tags only from the router

* test: type the new parametrized tests and keep the recorder from changing its argument

* fix(datadog): keep client request tags from setting the silent experiment cost tag

A request tag is_silent_experiment:true set the Datadog cost tag on a direct call.
Only the router flag sets it now. Also covers the sync failure handler's
mirrored-call reset with a failure function callback test.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 05:12:50 -07:00
Priyansh Nandwana
67ae3fb527
fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885)
* fix(websearch): keep the provider prefix on the agentic follow-up model

The agentic follow-up rebuilt its model string with "a slash means it is already
qualified". Handlers are handed the provider-stripped model, and for a provider
that routes through a sub-path the remainder still holds a slash, so
bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5
and stayed that way. The follow-up acompletion could not resolve a provider from it
and raised "LLM Provider NOT provided".

Prefix unless the model is already qualified with that same provider, in one shared
helper used by both the handler and the websearch patch builder, which carried
copies of the same heuristic. This also fixes openrouter, whose stripped model
keeps a slash for the same reason, and the old startswith test that read openai as
a prefix of openai_like.

Fixes #38829

* test(websearch): cover the agentic follow-up call site

The helper had unit tests but the call site did not, so the two changed lines in
_execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with
mock_response instead of patching litellm, which keeps the test on the behaviour:
against the old code the sub-path case raises "LLM Provider NOT provided", and the
ordinary-model case passes either way as a guard.

* refactor(websearch): trim the comments on the provider-qualify helper

* refactor(websearch): cut the helper and test docstrings to one line

* fix(websearch): leave a hook-qualified follow-up model on its own provider

A hook that patches in its own model owns that choice, so re-qualifying it sent a
cross-provider follow-up to the original request's provider instead. Only a bare
patched model and the request's provider-stripped model take the prefix now, and
the sibling agentic-loop path resolves the model the same way.

* fix(websearch): fall back to the request model when a patch carries no model

An empty patched model dispatched a provider-only name instead of falling back, and a
provider-stripped sub-path copied into the patch lost its prefix. A patched model now
keeps its own provider only when its leading segment names one.

* fix(websearch): keep the provider prefix on the responses follow-up model too

The responses patch builder still used the old "a slash means already qualified" test, so
a provider-stripped sub-path model reached the follow-up unqualified on that surface. It
now resolves the model the same way the chat completion builder does.

* fix(websearch): read custom_llm_provider without a member call on untyped kwargs

* fix(websearch): type the chat patch builder kwargs instead of working around them

* test(websearch): run the real search path and flatten the qualify helper tests

* fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model

* fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider

* fix(websearch): send the agentic follow-up to the model the caller requested

completion() and responses() record the caller's model before routing strips
it, and the chat and Responses follow-up builders prefer that model. This keeps
bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/*
deployments on their own route for the follow-up.

* test(websearch): cover the recorded caller model in Headroom's retrieval follow-up

* test(websearch): cover the recorded caller model in Compresr's retrieval follow-up

Also merges the repeated provider-prefix check in core_helpers into one helper
and drops a test docstring

* test(websearch): drive the real search path through a mocked Perplexity HTTP route

* refactor(websearch): read the provider from kwargs inside _caller_model

* test(http-handler): type the agentic follow-up provider tests

* refactor(websearch): record agentic loop params through a Logging method

Moving the recording out of completion() keeps it under basedpyright's
code-flow complexity limit after the merge with main.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 04:47:18 -07:00
quietjules
d037d86a94
fix(token_counter)!: keep a loaded Hugging Face tokenizer when a slower download fails (#48916)
* fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window

A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window

* fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL

The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend

* fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time

The loaded outcome now holds the tokenizer, so an LRU eviction can no longer
send every count back to an unguarded download. A retry blocks further retries
until it ends, even when it runs past the window, and a download that fails
after another count loaded the tokenizer counts with that tokenizer

* fix(token_counter): retry a failed Hugging Face tokenizer download in the background

After the retry window, the count that finds the stored failure starts one
background download and answers with tiktoken, so no live count waits on a
retry. A failure from a download that started before the first failure no
longer resets a running retry, and a failed retry never replaces a tokenizer
another download loaded. The outcome store takes its clock and retry starter
as dependencies, so the tests inject them instead of patching time.monotonic.

* fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start

If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart.

* fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again

A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load

* fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 03:50:58 -07:00
devin-ai-integration[bot]
b93d26bc85
fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path (#45330)
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path

* refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge

* fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper

* fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints

Also drop a marker set on the message item itself, sanitize before the
cache_control conversion so a client typo no longer erases that breakpoint,
and skip items that carry no marker so unmarked history costs no extra walk

* fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it

A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a
litellm_proxy alias) kept its valid markers before this PR and must keep them under
drop_params. base_model is read from litellm_params first, then model_info, as chat
completions does.

* fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged

* fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers

* fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied

* fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one

Under drop_params the marker step now hands back the caller's own objects when every marker is
already wire-safe, so a request with valid markers is no longer copied and walked a second time.

A deployment that turns cache_control into markers keeps the caller's own markers too, whatever
its base_model says, so one request never gets two answers about breakpoint support.

On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion,
so the block gets the cache hint cache_control alone would give, as on /v1/responses.

* test(responses): keep a null marker and the original key order on a model that reads breakpoints

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 03:37:54 -07:00
zoveki
310c96162b
fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893)
* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions

Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema

* fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens

Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema.

* fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models

Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them.

* fix(bedrock): drop tool custom object on the Mantle count_tokens fallback

Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way.

* fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages

* test(vertex): annotate count_tokens tool test locals with Final

* fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names

* fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 03:36:20 -07:00
paveln83
032342d2b7
fix(responses): resume background response streams with stream and starting_after (#48819)
* fix(responses): resume background response streams with stream and starting_after

Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}

Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally

* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams

A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.

* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve

A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.

* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params

* fix(responses): keep deployment stream off retrieves and reject oversized cursors

* fix(responses): keep router default stream off plain retrieves

* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing

* test(responses): replay a real stream in the billing test and type the fake wrapper chunks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 02:31:22 -07:00
simonp9
dff474e450
fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912)
* fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn

* fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards

* fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set

* fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit

* fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum

* fix(responses): leave a turn without a phase when its content message cannot be read

* test(responses): add black box tests for chat turn history replay

* test(responses): expect the replay record on bridged chat replies

* test(responses): expect the replay record in bridged chat translation cases

* fix(streaming): keep the served service tier on the finish chunk the stream adds

* test(responses): read the expected chat message without JSON validation so the replay matcher stays

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 00:56:02 -07:00
devin-ai-integration[bot]
3c1eb73a06
fix(proxy)!: list every priced Vertex family under a vertex_ai/* grant (#41959)
* fix(proxy): list every priced Vertex family under a vertex_ai/* grant

A vertex_ai/* key listed none of the 68 cost map rows priced under the
Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels
because their sets were never part of the vertex_ai union, and Qwen rows
had no ingestion branch at all. Wildcard expansion also dropped a meta/
or openai/ publisher segment as if it were a provider prefix, and the
wildcard health check would have picked $0-per-token rows first once the
union grew, so it now sorts paid rows ahead of unpriced ones. The
VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone
so the cost map guard enforces every label from now on.

* fix(health): route every wildcard health-check pick through the listed provider

* fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick

* fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick

* fix(health): only health-check ids the router sends to the provider

The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id.

* fix(health): match wildcard health picks against the provider's own route

The previous check kept a bare id only when get_llm_provider routed it to the
exact provider name, which dropped every ai21 row because bare ai21 ids route
to the ai21_chat alias, so ai21/* health checks sent no request.

Compare the bare id's route with the route of the provider-prefixed id
instead, so aliases match. Resolve routes with inferred_provider, which
returns the declared provider for github_copilot and chatgpt without
starting their device-code login.

* fix(health): keep wildcard health picks that only route through a saved custom_llm_provider

Rank picks routed to the checked provider first and ids routed to no provider
second, each by cost, and skip only ids routed to a different provider, so a
wildcard saved with custom_llm_provider still sends its health request

* refactor(vertex_ai): mark the new Vertex registry sets Final

* chore: re-run CI against current main

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 00:31:04 -07:00
rj-cairn
5eb1042bbf
fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728)
* fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(token_counter): move the o200k provenance note into the assert message

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust_bridge): azure/gpt-4o now resolves to o200k_base

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know

* fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer

* fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer

A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model.

* fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model

* fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k

A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count

base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response

Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model

* fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written

Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted.

* test: type the new token-counting tests' parameters

* fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it

* fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test

* perf(token-counter): classify each distinct served model once when walking a model group

Budgeted requests walk every deployment in their model group, since the
router's cached copy is outside the router surface the proxy may use.
Classifying each distinct served model once instead of once per
deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms
with the same result.

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 00:08:12 -07:00
devin-ai-integration[bot]
a66dc50efb
test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio (#43370)
* test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio

The drain regression guard compared the wall-clock time of draining 80,000 values against 40,000 and tripped its 3x bound whenever the CI runner was busy. A cursor drain allocates only per-value decode scratch while every re-slice of the remainder copies it, so the guard now bounds peak allocation during the drain, which does not move with runner load and still fails on every quadratic variant

* test(json_fragment_accumulator): widen the drain's peak bound to a quarter of the payload

* test(json_fragment_accumulator): guard copying drains by peak memory and rescanning drains by CPU-time scaling

* test(json_fragment_accumulator): move the drain's timing guard from the unit suite into a CodSpeed benchmark

* test(json_fragment_accumulator): measure only the pops after the buffer is joined, so a drain that copies once passes

* test(json_fragment_accumulator): separate every other streamed value with a space, so a copy after whitespace fails the drain test

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-28 22:04:47 -07:00
Mateo Wang
f46adf80e4
fix(proxy): let typesafe/* allowlists cover /typesafe Jev calls and list Jev in the model picker (#47697)
* fix(proxy): list TypeSafe Jev models in the admin model picker

* fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids

Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones.

* fix(typesafe): match only published Jev ids and keep picker loads off the secret manager

Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file.

* test(typesafe): type the picker test fixture and tests

* fix(typesafe): match only published Jev versions and cover non-version suffixes in tests

* fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker

* fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers

The dashboard sends include_model_access_groups on every /models call, so
gate the TypeSafe names on wildcard routes too and skip access-group-only
listings. Also say the typesafe-jev-ids rule matches case-insensitively.

* fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main

A typesafe/* router deployment registers typesafe/* in the cost map, so the
allowlist pickers offered a bare *, which allows every model. The pickers now
list only priced names the rule sends to TypeSafe. The non-expand list also
stops de-duplicating every entry and only skips TypeSafe names already listed.
2026-10-28 20:19:58 -07:00
Kerry Lu
4fc2bba1f2
fix(stream): the last usage frame carries litellm's bill in usage.cost (#48955)
* fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost

The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287.

* fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates

* style: format the chat-over-Responses bridge helper signature

* refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain

* fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress

* fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies

* test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes

* refactor(stream): drop the docstring on _stamp_usage_cost, the body says it

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 19:32:44 -07:00
Nate Armstrong
7650f766b0
refactor(router): fence the router surface and document the router-core boundary (#48839)
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.

Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
2026-10-29 00:01:52 +00:00
rj-cairn
d06b889d41
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream

The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.

SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF

Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.

* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator

* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed

Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.

Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.

Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.

* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests

Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.

* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines

* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter

* test: type the iterator class parameters in the line-separator stream tests

* fix(responses): stream tool call arguments opened in a text chunk

* fix(responses): hold a tool call that shares a chunk with text until the text ends

* test: type the a2a SSE handler helper and the sandbox fake responses

---------

Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
2026-10-28 17:00:29 -07:00
Kerry Lu
e21de0e418
fix(logging): bill each SDK call its own native Bedrock guardrail trace when calls share one metadata dict (LIT-10239) (#48923)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 15:11:26 -07:00
moe-berri
cf7c35a117
fix(anthropic): map model_context_window_exceeded stop reason to length (#48874)
* test(anthropic): cover model_context_window_exceeded finish reason on chat and responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(anthropic): map model_context_window_exceeded stop reason to length

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rust): mirror model_context_window_exceeded finish reason mapping

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 13:48:37 -07:00
Mateo Wang
dfd9ad4103
fix(responses)!: bill partial usage on mid-stream failures and report bridged /v1/messages failures to the proxy (#47537)
A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them
2026-10-28 13:32:09 -07:00
Simon
aa769fbc69
fix(streaming): surface upstream abort as server errors (LIT-10159) (#48328)
* fix(streaming): surface upstream abort finish reasons as errors

* fix(streaming): keep SDK chunks with null choices streaming past the abort check

* fix(streaming): treat null choices as empty in the raw SSE abort check

* refactor(streaming): bind raw SSE choices as Final instead of rebinding

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-28 13:18:05 -07:00
yujonglee
1f68cc5649
refactor(rust-bridge): require the native extension (#48852)
* refactor(rust-bridge): require the native extension

_native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader

The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ci): build Rust bridge in Windows smoke tests

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 13:16:56 -07:00
tin-berri
ca6c830221
fix(proxy): account for media in combined TPM reservations (#48865) 2026-10-28 12:39:41 -07:00
moe-berri
365d3d3729
feat(decisions): support Ollama decision models (#48871)
* test: cover local Ollama Decisions wire contracts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): support ollama providers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(decisions): account for Ollama model-info requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 12:35:26 -07:00
Onur C. Cakmak
d9aae590e3
fix(chatgpt): derive the upstream session id from prompt_cache_key (#42014)
* fix(chatgpt): derive Responses session id from prompt_cache_key

ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.

Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.

Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
  per-request and are skipped, checked in both "metadata" and
  "litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
  to a stable conversation key

Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.

Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.

* style: ruff format

* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments

- hash prompt_cache_keys containing header-unsafe chars instead of
  mapping them through _safe_header_value, whose replacement char could
  collapse distinct keys onto one cache shard and serialize unrelated
  requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
  file per AGENTS.md test-placement rule

* fix(chatgpt): keep session typing and test diff scoped

* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests

* test(chatgpt): record upstream requests with a mock instead of appending to a list

* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys

* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid

* test(chatgpt): type the session id test parameters

* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key

* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence

* fix(chatgpt): keep the trace-id session marker out of provider params

The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-28 10:57:51 -07:00
devin-ai-integration[bot]
3afe779125
fix(cost-map): rolling provider pricing, capability and retirement audit (#48188)
* fix(cost-map): registry audit 2026-10-24, add provider retirement dates and fix mistral-large-4 and azure_ai sonnet 5.5 prices

Co-authored-by: Hannes Brantner <desktop.stilted-9t@icloud.com>
Co-authored-by: Aasif-Simpplr <aasif.multani@simpplr.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mistral): derive mistral-large-4 cost expectations from the registry row instead of pinned prices

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): add fireworks spot routers and reprice together Qwen3.7-Max

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): absorb open registry PRs and update azure_ai mistral-small-2503 retirement date

Co-authored-by: kerry <kerry@berri.ai>

Co-authored-by: Aryan <aryan.m@kelpglobal.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(cost-map): credit absorbed registry PR authors

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Aryan <aryan.m@kelpglobal.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): add azure gpt-6.1-terra rows and fireworks models/rune path

Co-authored-by: ceng-p <christianpetersen98@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): add azure_ai gpt-6.1-terra global standard row

Co-authored-by: Ahmet Selim FEDAKAR <122402435+selimfedakar@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): price mistral-large-4 cache reads at the published $0.14

Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* revert(cost-map): leave mistral-large-4 rows to #48825

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): refresh provider pricing and retirement metadata

Absorb the verified GPT-5.6 Terra and Luna price fields from #48709. Add official Groq caching, xAI batch pricing, Moonshot metadata, and retirement dates without replacing existing model entries.

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Hannes Brantner <desktop.stilted-9t@icloud.com>
Co-authored-by: Aasif-Simpplr <aasif.multani@simpplr.com>
Co-authored-by: Aryan <aryan.m@kelpglobal.com>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: ceng-p <christianpetersen98@gmail.com>
Co-authored-by: Ahmet Selim FEDAKAR <122402435+selimfedakar@users.noreply.github.com>
Co-authored-by: zoveki <349028329+zoveki@users.noreply.github.com>
2026-10-28 16:07:22 +00:00
norkelbyte
940b865424
refactor(fallback-generalizations)!: remove the legacy extends and mixed-rule shim (#48737)
* refactor(fallback-generalizations): remove the legacy extends and mixed-rule shim

Main's remote cost map, the bundled backup, and the root map all ship the split
routing/capability rule format, so the shim for the old format is dead code. A rule
mixing litellm_provider with other model_info keys is now warned about and skipped.

* test(fallback-generalizations): fail when a shipped cost map rule would be skipped at install

* test(fallback-generalizations): cover ignored extends and annotate the shipped-rules test

* fix(fallback-generalizations): warn about old-format rules with a pointer to a current cost map and skip extends rules

* fix(fallback-generalizations): tell local cost map users to replace the bundled backup in the old-format warning

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-28 05:17:28 -07:00
Kerry Lu
7acdf9f00d
fix(anthropic): count search results and reject unsupported token-count blocks (#48699)
* fix(anthropic): count search results and reject unsupported token-count blocks

Co-authored-by: Prasanna Jain <prassu017@gmail.com>

* fix(anthropic): avoid a Final assignment inside the token-count loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Prasanna Jain <prassu017@gmail.com>
2026-10-28 02:33:38 +00:00
tin-berri
b9e4298506
fix: preserve media across Responses and Gemini conversions (#48484) 2026-10-27 18:07:10 -07:00
Ambuj Upadhyay
cb70bc560e
fix(streaming): dict-usage arm drops prompt_tokens_details/completion_tokens_details its sibling arms preserve (#36678)
* fix(streaming): dict-usage arm no longer drops token detail fields

In _dispatch_provider_chunk's openai/azure else-branch, a plain-dict
usage was rebuilt into litellm.Usage from only prompt_tokens /
completion_tokens / total_tokens. prompt_tokens_details (cached_tokens),
completion_tokens_details (reasoning_tokens) and cache token counts were
silently dropped — while both sibling arms (Usage passthrough, BaseModel
via Usage(**model_dump())) preserve them.

Every in-repo chunk producer currently normalizes usage to
Usage/BaseModel before this point, so the dict arm is reachable today
only via chunks litellm did not construct (e.g. a custom
streaming_decoder feeding CustomStreamWrapper). When it does fire on a
usage-only final chunk (stream_options.include_usage), the stripped
Usage is exactly what the stream consumer receives — the early-return
path has no later aggregation to repair it.

Fix: pass the full dict through Usage(**usage_dict). Usage.__init__
coerces nested detail dicts into their wrapper types and tolerates
extra keys. Side effect: a legitimate 0 count is no longer coerced to
None by the old 'or None' expressions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: drop explanatory source comment per repo comment policy

Rationale lives in the PR body and the regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(streaming): type the dict-usage unpack for the basedpyright gate

Usage(**response_obj["usage"]) unpacks a dict whose value type pyright
cannot see, and the delta gate reports that once per typed constructor
parameter: five, against the three .get() calls the line replaced. Moving
the unpack behind _usage_from_payload keeps the same Usage(**payload)
call and is reported once, at the call site, so the rule count ends below
its merge base

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-27 15:21:11 -07:00
yucheng-berri
788868a52f
fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599)
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials

* fix(proxy): drop caller metadata from proxy-only failure logs

metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata.

* fix(proxy): keep allowlisted metadata in proxy-only failure logs

* test(proxy): isolate proxy-only failure log metadata tests

* fix(logging): ignore non-list request tags when building the logging payload

A guardrail-rejected request with metadata tags set to a non-list value crashed
the standard logging payload, so its spend row lost model attribution.

* fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs

A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend.

* fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata

allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata.

* refactor(proxy): keep the proxy-only failure-log fixes inside the type gates

Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change.

* fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent

The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-27 12:40:59 -07:00
moe-berri
2dc4847579
fix(token_counter): count Anthropic-format tool_choice without KeyError (#48624)
* test(token_counter): cover Anthropic tool_choice counting

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(token_counter): count Anthropic-format tool_choice without KeyError

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 12:33:48 -07:00
moe-berri
32297c9177
fix(logging): match extra_spend_tag_headers case-insensitively (#48620)
* test(logging): cover extra spend tag header casing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(logging): match spend tag headers case-insensitively

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 12:16:34 -07:00
Kerry Lu
0b66a27528
feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278)
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(bedrock): validate the guardrail trace with one pydantic model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): record the native guardrail entry when litellm_params metadata is None

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): import the renamed messages invoke transformation module

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): walk the guardrail trace iteratively when redacting it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): type the native guardrail trace fixtures and locals

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 12:10:45 -07:00
Mateo Wang
14193435b8
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages

Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): keep Final variables out of loops in the rename shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): match the warning text in the rename shim test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): mark the alias loader spec restore for the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): format the renamed package shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): keep the rebind-ok marker on the spec restore line

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): carry the new package search path on alias specs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): declare the shim test locals Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 10:08:25 -07:00
ishaan-j-berri
a0d0ce1aa5
fix(token-counter): safely count inline auto image tiles (#48422) (#48526) 2026-10-27 02:31:31 +00:00
Kerry Lu
db1c871c12
refactor(cost-map): drop the legacy max_tokens field in favor of max_input_tokens and max_output_tokens (#48471)
* refactor(cost-map): drop the legacy max_tokens field in favor of max_input_tokens and max_output_tokens

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: drop assertions that pinned the retired cost-map max_tokens field

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:03:05 -07:00
devin-ai-integration[bot]
2e27a02a9f
feat(chatgpt): forward client subscriptions through the gateway (#41522)
* feat(chatgpt): forward each Codex user's ChatGPT OAuth login through the proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): cover _model_group_llm_provider metadata resolution

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(chatgpt): keep the gateway ChatGPT login bound to the authenticator api_base

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(chatgpt): move client credential header selection into the chatgpt provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(chatgpt): cover credential recognition and stored account id fallbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(chatgpt): support per-model chatgpt_auth_file for multiple ChatGPT accounts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(chatgpt): resolve bare chatgpt_auth_file paths and drop redundant test docstring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(chatgpt): pass per-model auth file to early provider detection

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(chatgpt): normalize chatgpt_auth_file before caching the authenticator

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(chatgpt): type account selection parameters

* fix(chatgpt): preserve account selection through saved prompts

* fix(chatgpt): allow administrator account configuration through the proxy

* test(chatgpt): construct immutable request parameters

* fix(router): preserve typed deployment credential resolution

* fix(chatgpt): allow admins to probe unsaved account configurations

* fix(chatgpt): retain SDK login headers during provider resolution

* fix(chatgpt): allow admins to probe unsaved account configurations

* fix(chatgpt): normalize subscription health probe input

* fix(chatgpt): normalize health probe input and cover credential destinations

* test(chatgpt): keep destination rejection assertions focused

* feat(proxy): forward client ChatGPT subscriptions on native Codex routes

* docs: lead ChatGPT forwarding with standard model setup

* fix(chatgpt): preserve selected credentials in health provider lookup

* fix(chatgpt): preserve gateway credentials and workspace selection

* refactor(chatgpt): avoid rebinding streaming metadata inputs

* fix(chatgpt): keep account lookup token argument immutable

* fix(chatgpt): scope forwarded credential routing to ChatGPT

* refactor: keep forwarded credential isolation opt-in by provider

---------

Co-authored-by: Moe Khalil <moe@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Urbanonymous <ugutitej@gmail.com>
Co-authored-by: mateo <mateo@berri.ai>
2026-10-26 15:44:05 -07:00
Arsh Goyal
96792d0913
test(streaming): pin that a stream stuck on one chunk closes its provider stream and logs one failure (#38753)
* fix(streaming): stop re-raising and re-logging a failed stream on every next()

A stream that raised keeps its provider iterator alive, so a consumer that iterates past the error re-detected, re-raised and re-logged the same failure on every call, flooding failure callbacks for one request. Latch the failure and re-raise it without logging again.

* fix(streaming): drop the retained traceback before re-raising a failed stream

* test(streaming): count failures through a real logging callback and require a clean stop on later reads

* test(streaming): require the stuck stream to close early and release its socket

The provider stream now records a GeneratorExit and caps its chunks, so draining it or detecting the loop late fails. Two wire cases check a sync and an async stuck stream release their TCP session.

* test(streaming): avoid failure recorder name collision

---------

Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-10-26 18:03:37 +00:00
Kerry Lu
5171b3fbd8
feat(cost): parse 12-hour cache write rates for batch and ultrafast tiers (#48259)
* feat(cost): parse 12-hour cache write rates for batch and ultrafast tiers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): bill 12-hour batch cache writes once when only the 12hr batch key exists

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 17:17:09 -07:00
Kerry Lu
9b7f2c8dea
fix(opencode): bill OpenCode Go requests from the reported list_cost (#48254)
* fix(opencode): bill OpenCode Go requests from the reported list_cost

OpenCode returns usage.opencode.list_cost on every response. Read it into the provider-reported cost the cost calculator checks before the price map, for chat and Responses, streamed and not. Keep the static rows as the fallback, and stop the generic usage.cost path from billing Go requests at 0 (usage.cost is what hit the prepaid balance, 0 inside the plan).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(core): resolve reported cost breakdowns without recursion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(opencode): cover reported list_cost spend through the proxy

Replaces the recursive _reported_cost_value with a flat _cost_number helper
so the code-quality gate passes, and adds an integration test that runs the
proxy against a scripted OpenCode upstream for a non-stream and a stream
chat completion, asserting the response cost header and the spend log row

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): bill native /v1/messages from the reported list_cost

The native Anthropic Messages route priced OpenCode Go bodies from the price
rows twice over: the non-stream body is a TypedDict the response-cost header
path priced as-is, and the stream assembled its ModelResponse with the rows
cost already stamped. Both now read usage.opencode.list_cost, the non-stream
body through billable_result and the stream from the final message_delta.

The integration test now covers chat, responses and messages, stream and
non-stream, through the proxy against a scripted OpenCode upstream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): keep the stream cost stamp inside the lint gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): read list_cost off dict chunks and trust it only from OpenCode on /v1/messages

Stream builders receive chunks both as mappings and as ModelResponseStream
objects, so the list_cost scan now reads usage through chunk_usage. The
native Messages route only trusts a reported cost from opencode_go, every
other upstream keeps the price map, which also keeps a zero usage.cost from
being billed as free. Served bodies without stop_sequence still price, and
the new code stays inside the basedpyright gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 16:44:29 -07:00
Mateo Wang
d0671f75a4
refactor(typing): widen delete_nested_value to Mapping and drop two unused Span aliases (#48001)
* refactor(typing): replace pass-through Any with object in 13 annotations

* docs(client): say dict[str, object] in the models get docstring

* refactor(typing): drop the unused Span aliases in base_cache and levo

* refactor(typing): take a Mapping in delete_nested_value and cover the fresh copy

* fix(typing): deep-copy a read-only Mapping in delete_nested_value through a dict
2026-10-24 19:39:27 +00:00
Mateo Wang
464932b1d1
refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)

* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)

* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts

* fix: restore pass-through behavior on paths the typing pass tightened

MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites

* fix(runwayml): keep a malformed 2xx task body on the 500 path

* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report

EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.

`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.

The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.

* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed

* test: annotate the new regression tests' locals with Final

* test: annotate the remaining new test locals with Final
2026-10-24 11:31:09 -07:00
Kerry Lu
dc0237ccd7
fix(cost): bill gemini audio input and cached audio at flex, priority and batch rates (#48086)
* fix(gemini): price tiered audio token rates

* fix: correct Gemini audio pricing catalog

* fix(gemini): preserve cached modality pricing

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 09:12:31 -07:00
Kerry Lu
455923014a
feat(cost): bill openai 12-hour prompt cache writes at their own rate (#48088)
* feat: add OpenAI 12-hour cache write pricing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: omit empty cache-write response fields

* style: format cache-write fields

* fix(cost): bill openai 30m and 12h cache writes from the usage split, add gpt-5.6 rates

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): apply the 12h long-context write rate only above the token threshold

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(cost): key cache write rates by TTL instead of growing the rate tuple

_get_token_base_cost returned a positional 6-tuple and every cache lifetime added a slot to it, to the off-peak helper and to every caller. It now returns a TokenBaseCosts dataclass whose cache write rates live in one mapping keyed by TTL. The TTLs come from the ephemeral_*_input_tokens fields on CacheCreationTokenDetails, so supporting a new lifetime only means adding that usage field and its rate key

A lifetime with no rate of its own now always falls back to the cache creation rate after off-peak pricing is applied. Before this the non-tiered 12h rate and the tiered 1h rate fell back to the rate from before off-peak, so the same request billed differently depending on the pricing path

* refactor(cost): hardcode the cache write TTLs as a Literal instead of parsing usage field names

Parsing ephemeral_*_input_tokens names saved nothing, since a new TTL still needs a usage field and provider parsing code. The TTLs are now a CacheWriteTTL Literal with an explicit rate key per TTL

* refactor(cost): resolve cache write lifetime rates to floats at lookup

Decide the 1h and 12h cost map key in the threshold loop, then resolve each
rate once after off-peak with the base write rate as the default, so no None
flows through the cost code. Add an integration case for a 12h write on a
row without a 12h rate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(cost): name every cache write TTL, marking 5m and 30m as base rate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): list every cache write TTL in one rate key table, None meaning the base write rate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): drop base rate entries from the cache write rate table and rely on fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): key cache write rates by plain string TTL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): fill missing cache write TTL rates with the base write rate after off-peak

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): default missing cache write TTL rates to the base write rate inline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): rename cache write rate key table to make clear it only covers long TTLs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cost): list every cache write TTL and bill writes of an unknown TTL at the base write rate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): resolve tiered cache write TTL rates after off-peak instead of passing None

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): carry cache write TTL rates on TokenRates and drop TokenBaseCosts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): define TokenRates before its first use to drop the string annotation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cost): build the TokenRates TTL default with default_factory and type tiers as Mapping

MappingProxyType({}) as a dataclass default is rejected on Python 3.11, where
mappingproxy is unhashable, so import litellm failed there. The tier type
moves from dict[str, object] to Mapping[str, object] (LIT001), with tier_rate
accepting a Mapping to match.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): type select_tier_for_input tiers as Mapping so tier_rate callers stay typed

The tier returned as a bare dict made every tier_rate call site a partially
unknown argument once tier_rate took Mapping[str, object]. The range bounds are
now validated with a TypeAdapter instead of indexing an unknown list.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:07:02 -07:00
moe-berri
9fab92672a
fix(opencode): forward stable sessions automatically across inference routes (#48121)
Forward caller-supplied stable session metadata to recognized OpenCode destinations across Chat Completions, Responses, and native Messages. Preserve explicit headers, configured affinity headers, and existing transports.

Fixes #39503
2026-10-23 17:38:47 -07:00
yucheng-berri
f7e6ddec66
feat(proxy): add LITELLM_OFFLINE_MODE, one switch for the fetches LiteLLM makes on its own (#48096)
* feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own

* fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced

* test: isolate the offline-mode tokenizer tests from the local Hugging Face cache

* test: cover the offline policy templates and the cached Kimi K3 tokenizer

* test(integration): count Kimi K3 tokens in the offline-mode egress test

* refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-23 16:54:19 -07:00
Ishaan
8f9894fbd6
fix(ci): unblock dependency security checks (#48055)
* fix(deps): upgrade vulnerable Python dependencies

* fix(deps): lock patched Python dependency versions

* fix(otel): preserve JSON export with the updated SDK

* test(otel): cover JSON export with the updated SDK

* test(otel): update SDK log records and worker discovery

* fix(ui): align Lens peer dependency with patched Next.js

* fix(otel): hand the AgentOps exporter a session it owns for the JWT header

OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it

The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals

* fix(otel): skip FastAPI instrumentation once the app has started

opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects

* test(otel): assert exported requests and span flags instead of SDK internals

OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags

* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it

zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6

* test(ui): stub the tag summary fetch the client panel retry refreshes

Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing

* fix(deps): drop the duplicated Lens override from the dashboard package.json

* style(ui): format the client panel fetch stub

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
2026-10-23 14:05:46 -07:00
shrey-berri
bc18e10f6b
fix: recognize OpenAI context_length_exceeded errors (#47954) 2026-10-23 10:09:39 -07:00
Mateo Wang
a8113b0dd7
fix(responses)!: round-trip Anthropic compaction blocks as compaction items (#46754)
* fix(responses): round-trip Anthropic compaction blocks as compaction items

/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.

Fixes #41456

* fix(responses): replay compaction blocks natively only when the request compacts

* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails

Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.

* fix(responses): count, route, guard, and redact compaction items consistently

A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.

* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it

* fix(responses): keep a stored compaction summary whole in spend logs

* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning

A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.

* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order

Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.

* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes

A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.

* fix(spend_logs): lift only the redaction opt-in header into failure rows

The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main

* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes

* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary

A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.

* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id

* fix(responses): count only message items with a string role as compaction instructions

* test(responses): flatten the last two-for comprehension in the compaction wire test

* fix(responses): replay from a compaction in stored Responses output and SDK compaction items

A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.

* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history

* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary

A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.

* fix(responses): replay a compacted session only through the referenced response

A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item

* fix(spend-logs): keep main's stored value for a prompt length limit under 2

At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value

* fix(spend-logs): keep main's stored tail for a negative prompt length limit

At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start

* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages

* test(responses): run the compaction wire tests on the worker database under isolated workers

* fix(router): keep the encrypted content affinity pin out of provider requests

* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction

* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message

* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses

* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call

* test(proxy): run the dispatched-call-type failure test through the real request setup

* fix(proxy): count a failed Cursor stream's input from the newest compaction item

The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.

* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item

* fix(responses): keep a tool call a later output answers when trimming before a compaction item

* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes

* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time

* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests

* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item

* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item

* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content

* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks

* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once

* fix(responses): count input_tokens after a compaction item the way inference sends it
2026-10-21 08:50:19 +00:00
Mateo Wang
66c8f5b1a9
fix(llms): join api_base and route with one shared helper (#47535)
* fix(llms): sanitize trailing slashes in provider get_complete_url

* test(llms): move trailing-slash url tests into each provider's mapped unit file

* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI

* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders

* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests

* fix(gigachat): strip a trailing slash from the pass-through api_base

* fix(llms): join api_base and route with one shared helper

A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base

* refactor(llms): move per-provider api_base strips onto join_api_base

Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join

* ci: run the trailing-slash URL builder test in the llms shard

* test(integration): cover trailing-slash api_base across provider URL builders

Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell

* fix(llms): send every provider route through join_api_base

Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it

Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call

The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash

* fix(llms): keep the route skip only where a builder already had it

join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path
already ends in it, and it is used only where the merge base or the trailing-slash test PR head
already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI
model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the
path so a host named anthropic-* builds /anthropic/v1/messages

* fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL

* test(llms): run the Gemini RAG trailing-slash case through aingest

The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears

* test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base

* test(integration): cover api_base joins across providers, query strings and named hosts

* feat(url_utils): add join_api_base and ensure_api_route helpers

* fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host

join_api_base and ensure_api_route now keep the // of a base left with only a scheme,
so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client
base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it

* fix(azure_ai): build the count_tokens URL from the messages URL

* fix(azure_ai): match anthropic as a whole path segment in the messages URL

* fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints

Azure AI chat and messages go back to cutting the base at the first
/anthropic, as on main, and count_tokens keeps its path check, so a
gateway path like /anthropic-proxy builds the same URLs as main. A base
that already ends in /v1/messages still gets count_tokens next to it.

A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is
used as its text again in AgentCore, Bedrock responses and the RAGFlow
base, instead of falling back to the default host.

get_volcengine_base_url and GigaChat's get_api_base strip a trailing
slash again for outside callers. The cache-hit wire test is skipped as a
known proxy bug, since one cache hit writes two cache_hit spend rows.

* fix(llms): drop a slash before the api_base query and tighten the route test checks

* test(llms): annotate new test locals as Final and flatten the builder generator

* fix(url_utils): keep a scheme-only api_base hostless so it fails before sending

---------

Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>
2026-10-21 06:59:32 +00:00
Kerry Lu
7ae46f2768
fix(timing): carry litellm_received_at across to a planted litellm_metadata bucket so the overhead header stays anchored at receive time (#47790)
A guardrail hook plants a litellm_metadata key on a chat completions request after the proxy
stamped litellm_received_at into metadata. get_litellm_metadata_from_kwargs prefers that key
when both exist and only copied the user_api_key_* fields across, so the response timing
headers lost their receive anchor and x-litellm-overhead-duration-ms reported 2 ms on
/v1/chat/completions against 58 ms on /v1/responses for the same proxy work. The receive stamp
now rides along with the user_api_key_* fields.

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 22:37:21 -07:00