* fix(otel): leave out span cost when the price lookup failed
A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free.
* test(otel): keep the guardrail charge on the guardrail span for unpriced calls
* refactor(otel): type the Arize cost writer input and reuse the cache_hit flag
* fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost
Only the placeholder 0 is left out when the price lookup failed, so a
guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel
and W&B Weave OTel. Adds unit cases for that fee and a malformed cost.
* fix(arize): leave out the span cost on any failed price lookup, matching OTel v2
* fix(logging): clear a stale cost failure flag when a later pricing succeeds
Stream-start pricing on the request model name can fail and set the cost
failure flag even when the final pricing succeeds (OpenRouter usage.cost,
an Azure deployment name priced from the response model). The new span gate
then dropped a real cost. Each pricing now resets the flag, and
uncached_price restores whatever flag it found.
* test(logging): pin the start time in the stale cost flag tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(sagemaker): parse every streamed record however SageMaker cuts the payload parts
SageMaker cuts a container's response stream into PayloadParts at arbitrary
byte offsets, so one part can hold several SSE events or JSON lines, or only
part of one. The decoder treated each part as one record, so vLLM's [DONE]
was logged at ERROR on every stream and a split at an unlucky byte dropped
the rest of the answer.
Peel records off a running buffer with JSONDecoder.raw_decode, skipping SSE
framing and [DONE] between them.
* test(sagemaker): cover streamed records cut or merged across payload parts
* fix(sagemaker): decode UTF-8 across parts, skip SSE comment lines, and log container error records
A UTF-8 character split across two PayloadParts now decodes whole through one incremental decoder per stream.
SSE comment lines (TGI's keep-alive) and event, id and retry field lines are skipped as framing.
A container error record is logged at ERROR with its message and skipped, and any other record that is not a
chunk is logged and skipped, the same way on the sync and async paths.
* fix(sagemaker): raise a container's error record and ModelStreamError on both stream paths
A record with a non-null error now raises a 500 SagemakerError carrying the record after the chunks before it, on sync and async. Async no longer swallows a non-200 event, so a ModelStreamError raises as it does on sync. A chunk with a null error field is delivered, and any other record that is not a chunk is logged and skipped on both paths.
* fix(sagemaker): answer a streamed error with the container's status and fail a stream cut mid-record
A container's error record now raises with the status the container gave it (vLLM's error.code, the older flat vLLM shape, TGI's http_status_code or error_type) instead of a fixed 500 that the router retried as a 503. A ModelStreamError raises a SagemakerError naming its ErrorCode and Message, a stream that ends inside a record raises instead of ending as a stop, and a skipped record is logged by its keys only
* fix(sagemaker): trust only 422 and 429 from a streamed error record and fail LMI generation errors
* test(sagemaker): drive an LMI failed-generation stream through the proxy on a sagemaker/ deployment
* test(sagemaker): fall back to the next model when a stream fails before any content
* fix(responses): keep strong refs to stream and websocket logging tasks
* test(responses): patch _spawn_logging_task instead of asyncio.create_task
The success-logging dispatch in _log_completed_response now goes through
_spawn_logging_task, so tests that patched asyncio.create_task no longer
observed it (and the sync ones hit get_running_loop with no loop).
* test(responses): assert stream and websocket logging tasks survive garbage collection
Adds regression tests that force a collection while the logging task waits
on an otherwise unreferenced future, and types the new test parameters
* test(responses): drive drain and GC tests through a real logging object with gated callbacks
* test(responses): use asyncio.wait_for so the logging tests run on Python 3.10
* test(guardrails): type the spawn_logging_task tracking wrapper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
* feat(harness): add pi harness for litellm.agent()
Adds Harness.PI, which drives the pi coding agent CLI (`pi --mode json`) once per turn
against the per-session model endpoint. The endpoint is declared as a `litellm`
openai-completions provider in the models.json of a LiteLLM-owned PI_CODING_AGENT_DIR, so
the user's own pi settings, auth, extensions and MCP servers are never read, and the token
travels only through an env var referenced as `$LITELLM_HARNESS_TOKEN`
pi exits 0 after a failed provider call, so a turn fails when the last assistant message
ends with stopReason error/aborted. Permission modes map to a `--tools` allowlist,
disable_tools to `--exclude-tools`, and sessions resume with `--session` from a persisted
session dir. build_instructions and turn_prompt move from the OpenCode config into the
shared harness utils so both configs use them
Unit tests run on JSONL recorded from pi 1.1.0
* test(harness): align pi harness tests with the other CLI harness suites
Mirror the OpenCode suite's layout, fakes and helpers, and add the cases every CLI harness
covers: turn before start, early close kills the process, persist failure, endpoint model
fallback, long JSONL lines, skills without SKILL.md and a missing endpoint. Add an
endpoint_requests.jsonl fixture recording what pi sends to the session endpoint, and
re-record the turn fixtures in one run so their ids agree
* fix(harness): parametrize harness e2e only over harnesses with a model group
harness_params() iterated every Harness member, so Harness.TOOL_LOOP, which has no entry in
BINARIES or DEFAULT_MODEL_GROUPS, raised KeyError and the whole suite failed to collect
* feat(harness): accept pi config and MCP servers through PiOptions.config
PiOptions gains `config`, like the other harness options. `mcpServers` is written to the
private agent dir's mcp.json and every other key to its settings.json. Keys that would reroute
model calls, bypass permissions=, or load code as the host user (defaultProvider, defaultModel,
defaultTools, defaultProjectTrust, extensions, packages, ...) raise OptionsMismatch
A plain-name `--tools` allowlist drops MCP tools in pi 1.1.0, so when servers are configured
the allowlist also gets `mcp__*` plus the tool each exposure is reached through (`codemode` by
default, `tool_search` for `deferred`). disable_tools still excludes individual MCP tools
* refactor(harness): give harness options immutable mapping defaults
`field(default_factory=dict)` left every config/env default typed as
`dict[Unknown, Unknown]`, which basedpyright reports as
reportUnknownVariableType under strict mode: 9 errors across the five
options classes, two of them added by PiOptions. A shared module-level
`MappingProxyType({})` default is accepted by frozen dataclasses, matches
the no-mutation rule the rest of the harness follows, and drops the file
to zero errors so the rule shrinks against the merge base instead of
growing.
* fix(harness): reject managed env keys and a missing model in the pi harness
PiOptions.env could name PI_CODING_AGENT_DIR or LITELLM_HARNESS_TOKEN and
the value was dropped in silence, because transform_session_setup merges
the managed keys last. The ordering stays as defense in depth, but
validate_environment now rejects those keys up front the way the Claude
Code config does, so a bad option is an error rather than a setting that
appears to apply and does not.
transform_turn_request also stringified the model it resolved, so a call
with neither model= nor an endpoint model would have put the literal
"None" on pi's --model. Both call sites now go through session_model(),
which raises the same ValueError transform_session_setup already raised.
* test(harness): escape the regex dot in the pi skill manifest match
* fix(harness): keep harness options importable on Python 3.11
Python 3.11 rejects an unhashable dataclass default, and MappingProxyType
only became hashable in 3.12, so the shared MappingProxyType({}) defaults
made importing litellm.harness.options raise ValueError and broke every
harness on 3.11. The defaults now come from typed default factories that
still return read-only mappings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): address review feedback on the pi harness tests
Drop the endpoint-request fixture test, which only read a recorded file and
could not fail when PiHarnessConfig changed, along with its now unused
fixture. Remove the section-divider comments and a comment that restated
MANAGED_ENV_KEYS, and type the test helpers and fakes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): cover the last uncovered pi harness lines
Assert that a tool_execution_end event without a result yields an empty
ToolResult output, and that a PiOptions.config value JSON cannot encode
raises TypeError. The Python 3.11 default check now asserts on the list of
unhashable defaults instead of relying on hash() raising, which the
test-quality gate flagged as an assertion-free test
* test(harness): annotate harness option tests and stop rebinding helper params
Mark the option test locals Final and check read-only defaults through the
MutableMapping interface instead of writing into them. In the pi suite,
type the parametrized test arguments, and have started() and parse_all()
bind new names instead of reassigning their sandbox and state parameters
* test(harness): test harness option defaults by constructing the options
Replace the field-introspection check with one that builds each options
class twice and asserts every mapping default is empty, read-only and not
shared between instances. Sharing one MappingProxyType across instances is
what broke class creation on Python 3.11, and that sharing is observable on
every Python version, so the test fails on the old code without inspecting
dataclass declarations
* test(harness): stop mutating test helper inputs and assert whole values
make_ctx took model and endpoint out of **kwargs with pop, which mutated
its own parameter; they are explicit parameters now, with a frozen default
endpoint. assistant_end builds its message in one expression instead of
adding errorMessage afterwards. Two docstrings that explained an assertion
became that assertion's message, and the options test compares whole
values so a failure names the offending fields
* ci(unit): run the pi harness unit tests in the llms-h-to-z shard
main now lists every tests/unit/llms provider directory per shard, so the new
tests/unit/llms/pi directory was claimed by no shard and assert_ci_coverage
failed on both the shard and the test-file checks
* test(harness): mark pi test locals Final and freeze the test output model
Every single-assignment local and module constant in the pi suite is now
Final. The name-mapping loop became one comprehension compared as a whole
list, test_start_persists_sessions_dir takes the sandbox by index instead of
binding _ twice, and the Answer output model is frozen like every other
pydantic model
* fix(harness): send PiOptions.thinking to the gateway as reasoning_effort
pi clamps --thinking to "off" for a model that does not declare reasoning,
and the generated models.json only declared {"id": model}, so every
thinking level was dropped and the gateway never saw reasoning_effort.
xhigh and max are also only offered when thinkingLevelMap names them. The
model now declares reasoning plus that map whenever a level other than
"off" is requested, and nothing when thinking is unset, so pi does not send
its default effort on every call
The pi tests also drop **kwargs from make_ctx and started in favour of
explicit parameters, and annotate their helpers and the fake sandbox with
read-only types; the fake records calls by rebinding tuples instead of
appending to lists
* docs(harness): list pi among the harnesses in the module and agent() docstrings
* fix(harness): keep pi calls provider-agnostic and harness option defaults copyable
Declare supportsStore false so pi stops sending store, take contextWindow and maxTokens from the model map, treat a length stop or a silent exit as an error, and give every options class plain dict defaults again so deepcopy, pickle and asdict work
* fix(harness): look up pi gateway groups as litellm_proxy and ask pi for cache_control marks
A gateway session's model is the bare group name, so the model map lookup
now passes custom_llm_provider=litellm_proxy, the way the SDK prices it.
That keeps an unmapped group from printing get_llm_provider's provider
list banner into the caller's stdout.
pi only adds Anthropic cache_control marks for OpenRouter Anthropic models,
so every Claude call through the harness missed the prompt cache. Models
the map says support prompt caching now declare cacheControlFormat
anthropic, which LiteLLM translates for each provider.
* fix(harness): cap pi's maxTokens at its own default and fork sessions saved in another directory
pi sends maxTokens as max_completion_tokens on every call, and the gateway reserves that many
tokens against the key's TPM limit, so passing the model's full output cap (128000 for
claude-haiku-5-5) made keys with lower limits answer 429. maxTokens is now the lower of the
model's cap and pi's own 16384 default.
agent_resume() into another sandbox directory stopped at pi's interactive fork prompt. The
native session id now carries the directory pi saved the session in, and a resume from a
different directory passes --fork, so the conversation continues in the new sandbox.
* fix(harness): leave pi room for the thinking budget and price cached prompt tokens
pi's maxTokens now adds LiteLLM's thinking budget for the requested level to pi's 16384 default, within the model's output cap, so Anthropic budget thinking at max no longer answers 400. Harness turns now pass cache read and write tokens to cost_per_token, so cached prompts are priced at their cache rates on every harness.
* test(harness): cover cached prompt cost on the gateway and non-stream routes
* test(harness): mark the cost test constants Final
* test(harness): type the cached cost test helpers without Any
* fix(harness): price OpenAI cache write tokens in harness cost
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): restore the token estimate for streams that send no usage
A stream without stream_options got a made-up Usage(0, 0) written into its
final chunk's hidden params. Since #42323 the chunk builder treats that as a
provider-reported zero, so these streams were logged and billed at 0 tokens
and $0 instead of the tokenizer estimate. Only attach the summed usage when
some chunk actually carried usage.
* test(streaming): use a fixed start time and check the estimated cost
* fix(streaming): keep reported usage on the synthesized finish chunk and test it
* test(streaming): cover explicit zero usage reported by the provider
* fix(streaming): set the token estimate on a copy of the final chunk and ignore usage with no counts
* fix(streaming): keep final chunk edits in the logged response when the provider sends no usage
* fix(streaming): mark the unreported usage placeholder so every chunk rebuild falls back to the estimate
* test(streaming): wait on the success callback event instead of sleep-polling
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path
* fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format
* fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params
A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did.
A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper.
Adds integration cases for the json_schema envelope and the cut stream.
* fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates
Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded
* fix(anthropic): join a split answer when the stream ends in its last block
A cut during the second json tool block now runs the held block through
the same flush as a block stop, so it joins the streamed keys or the
earlier split blocks instead of sending two objects back to back. The
Vertex output_format fold drops its unreachable drop_params branch,
since both parent configs drop a non-object value first.
* test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages
* fix(proxy): drop router_settings_override before routing and apply it on every branch
route_request only popped router_settings_override inside the branch that
runs after the api_key/api_base, comma-separated batch and user_config
branches. Requests taking one of those earlier branches forwarded the field
to the provider, which Anthropic rejects with a 400, and silently lost the
key/team num_retries, fallbacks and timeout. The override is now popped and
merged into the request before any branch is chosen, keeping the existing
precedence where request values win and None values are skipped
Fixes#43582
* fix(proxy): apply key/team fallbacks when the request carries api_key or api_base
The previous commit skipped fallbacks, context_window_fallbacks and
content_policy_fallbacks from router_settings_override whenever the request
had api_key or api_base. That broke the scenario from #43582, where a
trusted pre-call hook sets api_key and the key/team router_settings still
need to apply. All per-request router settings are merged again on every
branch, with None values skipped and request values still winning
The tests now check that fallbacks run on the api_key and api_base paths
and that the provider never receives router_settings_override
* fix(proxy): answer 400 for a non-object router_settings_override and treat null as absent
* fix(proxy): drop a malformed router_settings_override under drop_params
* fix(proxy): route key and team router settings like any request, honor user_config and global drop_params
* fix(proxy): keep Router-only override settings out of direct provider calls
Calls that route_request sends straight to litellm (evals, endpoints that need
no model, pass_through_all_models, the video fallback and the no-router paths)
now leave out context_window_fallbacks, content_policy_fallbacks and
model_group_retry_policy merged from key or team router settings. Only the
Router reads them, and providers reject them as unknown parameters.
* test(proxy): drop a comment from the router settings override test
* fix(proxy): keep key router settings on the model read-through retry
route_request now merges router_settings_override once, before the first
attempt. The retry after a model read-through used to pop the merged
enable_tag_filtering again and recompute an empty Router-only set, so a
model created on another instance was served without the key's tag
filtering
* test(proxy): check the read-through retry keeps key tag filtering through a real router
The regression test now routes through a real Router with two tagged deployments from the database instead of replacing Router.acompletion. Before the fix the retry picks the lower-order untagged-for-request deployment, after it the tagged one
* fix(proxy): keep key Router-format fallbacks out of direct provider calls
Key and team fallbacks use the Router's model-group mapping. litellm.acompletion reads each mapping as a fallback config, so a direct call retried the same model with the mapping merged into the provider body. The override's fallbacks now stay out of direct calls like the other Router-only settings, while num_retries and timeout still pass
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): schedule default_priority requests once and reject a non-integer priority
A router configured with default_priority failed every request that carried no
priority with a 500 naming a missing 'priority' argument, and an explicit
priority re-entered acompletion and hit the same error. acompletion now resolves
the priority once (the request's integer, else the default, with 0 kept as a
real priority) and schedules through _schedule_factory directly, while
schedule_acompletion delegates to acompletion so the two paths share one queue
pass. A non-integer priority answers 400 naming the param, or is dropped under
drop_params, instead of leaking to the provider or crashing the scheduler.
* fix(router): honor router-level drop_params and schedule prompt-management models under a priority
* fix(router): drop a non-integer priority when the model's deployments set drop_params
* fix(router): resolve the drop_params deployment scope for team routes and deployment ids
* fix(router): read the request and router drop_params flags without a loop-assigned Final
* refactor(router): type the prompt-management wrapper's return and mock the provider in its test
* perf(router): consult drop_params for the priority only when it is not an integer
* test(integration): assert the new priority contract for non-integer and zero priorities
* docs(router): describe default_priority as the priority every request without one takes
* fix(router): parse default_priority once and apply it to text completions and the queue route
A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig.
* fix(router): type the priority dispatch so the lint budgets hold
* fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order
* fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses
* fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority
A request whose model is a deployment id now counts that deployment as healthy in the
scheduler, since routing by id already ignores cooldown. Before, the scheduler found no
healthy deployment for an id, so the request waited behind every entry ahead of it.
A boolean default_priority (true, or YAML off) is ignored at startup with a warning,
like any other value that is not a whole number.
The scheduler integration tests now share worker state for the defaulted pair, cool a
group through /v1/responses so no stale in-memory queue is left behind, and use "high"
instead of "1" for the non-integer cases. The concurrent burst test no longer asserts
that the queue drains, matching the explicit-priority burst test, since the scheduler's
read-modify-write queue can race across workers.
* test(router): keep the queue drain check as a BUG-skipped integration test
* fix(router): queue default_priority chat requests once instead of failing them
* test(router): open with the default_priority regression tests alone
* fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set
With router_settings.tag_routing_prefix configured, an unprefixed caller tag
(for example a spend tag such as user_id:234) no longer drives tag routing, so
it can no longer refuse a model group whose deployments are untagged.
Key/team/project tags (metadata.inherited_tags) keep applying with or without
the prefix. Rows of batches the proxy runs itself now carry those tags as
inherited_tags too, so they keep routing on them once the prefix is set.
Takes over #39903.
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* test(integration): pin that proxy-run batch rows keep key/team tag protection
* refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells
* test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst
* test(integration): proxy-run batch rows keep key and project tags too
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* feat(context_management): honor clear_at_least, exclude_tools and clear_tool_inputs in the clear_tool_uses polyfill
The clear_tool_uses_20250919 polyfill honored only trigger and keep and
ignored the other three fields with a warning. It now follows the Anthropic
API, as measured against it:
- keep counts clearable tool uses only; uses of exclude_tools are never
cleared and count toward neither keep nor clear_at_least.
- clear_at_least clears oldest first in batches of at least that many
tokens and leaves an incomplete last batch, so the cleared prefix only
grows between requests and each batch costs one prompt-cache miss.
- clear_tool_inputs (true or a list of tool names) also empties the input
of the cleared tool uses.
Invalid values are ignored with a <field>_invalid warning on the applied
edit.
* refactor(context_management): build the clear_tool_uses selection without mutable state
* test(context_management): cover a history made only of excluded tools
* fix(context_management): leave tiny tool results out of clear_tool_uses like the API
The API neither counts nor clears a tool use whose result is 1 to 46 bytes
of text, whatever the tool or error flag; empty results still count. Probed
against the API with crafted histories. Test data under 46 bytes is
lengthened so those tests keep exercising clearing
* fix(context_management): count tiny tool results when clear_tool_inputs covers their tool
The API leaves a use whose result is 1 to 46 bytes out of clear_tool_uses
only while its input is kept. When clear_tool_inputs is true or names the
use's tool, the API counts it toward keep and clears it like any other use.
* fix(context_management): count cleared inputs and images toward clear_at_least, answer malformed fields with a 400
* fix(context_management): validate clear_tool_uses edits before any editor runs, drop only the malformed edit under drop_params
* fix(context_management): drop only the malformed clear_tool_uses fields under drop_params
A malformed clear_at_least, exclude_tools or clear_tool_inputs no longer drops the whole edit under drop_params. The edit still runs with its well-formed fields, so the clearing that worked before is kept. Also types the new tests and swaps the hand-rolled spy router for an AsyncMock
* test(context_management): pin the 46 byte cutoff in UTF-8 bytes and a batch closing at exactly clear_at_least
* fix(context_management): leave a non-string tool_use id for the provider, size lone surrogates, and read drop_params strings
* fix(context_management): read a drop_params string the same way on count_tokens and both /v1/messages paths
count_tokens still checked drop_params with is True, so a body drop_params of "true" answered 400 there while /v1/messages dropped the malformed field. Both routes and the sync path now use drops_unsupported_params.
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): expose supported_endpoints on /model_group/info
* feat(router): route batch creates to mode batch deployments in mixed model groups
* fix(router): prefer batch deployments after eligibility filters and in group-name credential lookups
* fix(router): keep group-name credential lookups on the first deployment and prefer batch deployments before cooldown
Model-encoded ids carry only the group name, so preferring the batch deployment there moved ids created before the upgrade onto its credentials. Batch creates now pick batch-capable deployments before health checks and cooldown, routing and group info share one batch-capable check, the cost map's /v1/batch is dropped only for Azure, and /model/info applies the same per-deployment rule
* test(router): map _listed_endpoints to its /model/info tests in the router coverage check
* fix(router): skip only Azure chat deployments for batch creates and list batch support per caller team
Narrow the batch create filter to Azure and azure_ai deployments that are not mode batch, so OpenAI groups keep
their chat deployments during cooldown and under tag filters. Mark batch deployments by mode batch only, since
stored endpoint lists from older Admin UI saves copy the cost map. Leave /model/info unchanged so the Admin UI
never saves a computed endpoint list, and pass the caller's team into /model_group/info.
* test(proxy): let the model hub router fakes take the caller's team_id
* fix(router): gate batch-create routing on deployments the caller's team may use and list only /v1/batch for Azure batch deployments
* fix(router): skip a malformed model_info.supported_endpoints instead of failing group info and chat
* fix(router): apply the batch create filter after cooldown, managed files, and tags
The batch filter ran before health checks and cooldown, so a batch create on a mixed
Azure group answered with a router 429 or 401 when cooldown, tags, or a managed file's
targets left no batch deployment, where main sends the create on and returns Azure's
400. The filter now runs right before order filtering, after every eligibility filter,
so those cases answer as main does and creates still skip Azure chat deployments
whenever a batch deployment is left
* test(router): annotate the batch routing test assignments with Final
* fix(otel): put the error message on the failed span status
The legacy OTel logger (used by otel, langfuse_otel, arize and others) ended a failed call with Status(StatusCode.ERROR) and no description, so backends that read the span status, such as Langfuse's status message, showed an empty error. The text was only in the error.message attribute. Use the same message as that attribute, as the OTel v2 emitter already does.
* fix(otel): take the failure status message from _record_exception_on_span
Instead of reading the logging payload a second time in _handle_failure,
_record_exception_on_span returns the error message it already records
as error.message, and _handle_failure puts it on the ERROR status. The
other callers ignore the return value, so their spans are unchanged.
Its input is now a typed view of what it reads (exception,
error_information, error_str) instead of a bare dict.
* fix(otel): put the error message on every failed span status
Covers the proxy SERVER span, management endpoint spans, service spans, and Arize Phoenix spans, so no ERROR span ships with an empty status description
* fix(otel): cut the error status to the span attribute length limit and describe dangling server spans
The status description now uses the error text as stored on the span, so OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT applies to it too. A server span the proxy closes on a 4xx or 5xx outside the request path gets the same description, and an empty error message falls back to the error class
* test(otel): type the validation-handler status test
* test(otel): cover the Phoenix parent span and error_str length-limit status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(search): price Tavily and Linkup searches at their per-mode cost map entries
* test(search): read the reported cost through the public hidden_params property
* fix(search): keep the type gate flat on the cost map membership check
* test(search): drop vendor price guards and a redundant case, type helper params as mappings
* fix(search): keep a registered search model's own price, cover every interception endpoint, hostile modes and a failing burst
* fix(search): keep a runtime-registered base search price for modes that have no runtime price of their own
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content
Provider transforms encode returned video ids with an empty model_id, so
GET /v1/videos/{id} and /content cannot resolve the deployment (and its
per-model api_key) and fail with invalid_api_key.
Re-stamp the returned VideoObject id with the resolved model_id in
video_generation, video_status, video_remix, video_edit and
video_extension.
Fixes#33423
* fix(videos): return a copy instead of mutating the response, read hidden params via helper
* test(videos): fully type the new video id re-stamping tests
Annotate every parameter and return added by this PR (hidden_params,
harness, response, -> None), drop Any from the test helper, mark locals
Final, and narrow the endpoint/helper results with isinstance(VideoObject)
instead of reading .id off an untyped value. Route the mocked processor
response through a typed Harness.respond_with, read NotRequired
DecodedVideoId keys with .get, and name the rule and reason on the
_hidden_params suppressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): fill only an empty model in returned video ids
Keep an id that already names a model as is, so the asked-for name,
routing group and served group in it still drive access checks and
routing. Fill an empty model from the model the caller asked for
before the routed deployment id, and read the request model through a
typed view so the type gate does not grow
* fix(videos): route video calls to the deployment that created the video
Video ids now carry the id of the deployment that created them. Status,
content, remix, edit and extension calls go to that deployment when it
belongs to the requested group, so a group spread over two provider
accounts no longer polls the wrong one. A provider model shared by
several groups resolves to the creating deployment's group, and the
asked-for model is written into an id only when the router can route it.
* fix(videos): pin video calls only to deployments the caller may use
Pin a video id's creating deployment at deployment selection, and only when the caller may use that
deployment in the requested group, so aliases, routing groups and access groups still apply and the
spend label and fallbacks keep the requested group.
A status poll keeps the deployment its id names and takes the deployment that answered when the id
names none. An alias that routes to the creating deployment's group is kept on status and content.
* test(router): reset module fallback globals between unit tests
* fix(videos): check and route polls by the name the router serves
A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does
* fix(videos): pin polls only among the deployments routing allows
Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests
* fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s
* fix(videos): stamp the requested group only when one of its router fallbacks served the create
* fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router
* test(router): list the video pin helper with its sibling in the router coverage exemptions
* refactor(auth): walk chained router fallbacks without recursion
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions
stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt
Read chunk fields as typed attributes instead of untyped subscripts, which
clears the reportUnknownArgumentType breach the lint gate flagged
* fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): rebuild dict-form text completion chunks again
stream_chunk_builder routes dict chunks holding TextChoices to the text
completion builder, which now reads attributes. Convert them to
TextCompletionResponse at the routing point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path
Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage
* fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild
* fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions
* fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null
* fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path
Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`.
The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text`
next to `delta`, so streamed /v1/responses calls to Together ended without response.completed.
Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about
the same as rebuilding an n=1 stream of the same size.
* fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs
A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising
Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals
* fix(streaming): rebuild text completion choices without an index at index 0
Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base
* fix(streaming): rebuild text completion choices with a string index under that int
A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string
* test(streaming): drop a docstring that repeats its test
* fix(streaming): fall back to choice 0 for an infinite text completion index
A float infinity index made int() raise OverflowError, which the index
coercion did not catch, so the rebuild failed where the merge base
returned choice 0.
---------
Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover gzip and zstd request bodies on /v1/systemone
* fix(proxy): decode zstd and gzip request bodies before auth reads them
* fix(proxy): bound request body decompression and keep it inside CORS
Limit the decoded size by an expansion ratio and the operator's max_request_size_mb, pass bodies without the coding's magic number through, answer 415 when zstd is unavailable, and register the middleware innermost so errors carry CORS and security headers
* fix(proxy): budget decoded request bodies per worker and forward plugin bodies as sent
Each worker holds at most MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB of
decoded bodies and answers 503 past it. /plugin-proxy/ bodies reach the
plugin as sent, a zstd skippable frame counts as zstd, the decoded-size 413
names the decoded limit, and an integer max_request_size_mb stays an integer
in the 413 text.
* fix(proxy): read the magic number before size limits and reserve declared decoded sizes
* test(proxy): pin gzip mtime so parametrized test ids match across xdist workers
* fix(proxy): parse max_request_size_mb the same way in the auth size check
* fix(proxy): pass Claude Code OTLP export bodies through undecoded and skip the size parse without premium
* test(proxy): cover client disconnects, chunked bodies past the cap, and body-less request messages in the decompression middleware
* test(proxy): name the decompression test cases so CI logs stay small
* feat(dashscope): price Qwen implicit context-cache reads separately
* fix(dashscope): normalize implicit cache price strings
* fix(dashscope): price explicit cache hits through an explicit_ override key
Keep cache_read_input_token_cost as the default (implicit) Qwen cache-read
rate, which the cost map already holds, and add
explicit_cache_read_input_token_cost for hits DashScope tags with
cache_type "ephemeral". Untyped and unknown cache types keep the base rate,
so older releases reading the remote cost map keep billing unchanged.
Add explicit cache-read and cache-creation rates to the qwen3.7-max rows.
* fix(types): accept explicit_cache_read_input_token_cost as a constructor keyword
* fix(dashscope): inherit the explicit cache price on custom deployments and keep tier rates all-or-nothing
Deployments that set only input and output prices now inherit
explicit_cache_read_input_token_cost from the cost map, so explicit hits bill
at the explicit rate. A pricing tier without its own explicit rate bills its
own cache_read rate instead of the model-level explicit one, matching how an
off-peak block falls back.
The DashScope calculator now delegates the explicit rate to the shared
resolver, and the register_model warning is back to main's wording.
* fix(dashscope): keep a deployment's own cache-read price for explicit hits and ignore a non-string cache_type
* fix(dashscope): keep cache_type through Responses usage and let a registered cache-read price cover explicit hits
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(mcp): cover one spend row billing both rounds of a chat completions MCP auto-execute
* fix(mcp): bill both model calls of a chat completions MCP auto-execute on one spend row
* fix(mcp): keep provider usage fields when summing both chat rounds of an MCP auto-execute
* fix(mcp): log each auto-executed MCP tool call under its own spend-log id
A chat MCP auto-execute logged its tool calls under the request's call id, which is also the id of the request's failure row. When the follow-up call failed, the database kept the tool row and dropped the failure row that bills the paid first call
* test(mcp): cover the margin, cache, failure row and parallel tool rows of MCP auto-execute billing
* feat(router): apply each deployment's reasoning_effort_levels on every provider and route
A deployment's declared reasoning_effort_levels now decide the effort sent upstream on /v1/chat/completions, /v1/messages and /v1/responses for every provider. Requests round up by default, litellm.reasoning_effort_rounding and model_info.reasoning_effort_rounding switch to down, a declaration lets providers like azure_ai accept reasoning_effort, and each remap logs one INFO line with structured fields.
* fix(router): round reasoning effort only on a deployment's own declared levels
Cost map levels no longer round requests or force-allow reasoning_effort, so undeclared sibling deployments and catalog-only models behave as on main. A declaring deployment lets azure_ai accept any level the resolver sends, and a hosted_vllm chat thinking budget is rounded too.
* fix(router): drop undeclared reasoning effort under drop_params and round native /v1/messages effort
Under drop_params a level outside the deployment's reasoning_effort_levels is
dropped instead of sent. Providers that take /v1/messages natively now round
output_config.effort onto the deployment's declaration. The rounding setting
ignores case and surrounding spaces.
* fix(router): strip line breaks from every field in the reasoning effort remap log
* fix(router): round thinking-budget efforts onto a declaring deployment on every provider
* fix(router): keep the caller's thinking display when rounding a thinking budget on adaptive Claude
Warn on a reasoning_effort_rounding value other than up or down instead of silently rounding up
* test(integration): cover deployment reasoning_effort_levels rounding on chat, messages and responses
Wire cells assert the exact body each provider receives when a deployment
declares reasoning_effort_levels, plus undeclared controls, malformed input,
drop_params, rounding settings, fallbacks, model groups, live updates and
the response cache. Chaos cells cover a killed worker, a proxy restart, an
upstream outage, a slow stream and the INFO remap log.
* test(integration): cut held streams when the upstream stops in the reasoning effort chaos cell
The upstream-stop cell waited for the held streams before stopping the upstream, so they only ended on a client or proxy timeout and the result depended on which fired first. The upstream now stops while the streams are held, and each cut stream must carry an error and no answer
* fix(router): round thinking budgets only on declaring deployments and round Converse output_config
An undeclared hosted_vllm deployment serving a Claude model name now maps a thinking budget the same as any other undeclared hosted_vllm deployment. A model_info that echoes a cost map key no longer borrows that entry's levels. A non-numeric thinking.budget_tokens answers 400, or is dropped under drop_params, where main answered 500. A declaring Bedrock Converse Claude deployment now rounds output_config.effort, which /v1/messages sends for Claude Code
* test(integration): cover output_config rounding and non-numeric thinking budgets in the reasoning effort cells
* test(integration): keep an explicit reasoning_effort winning over a malformed thinking budget on hosted_vllm
* test(integration): cover output_config rounding on openai, azure and azure_ai chat
* fix(hosted_vllm): let an explicit effort win over a malformed budget and reject one on /v1/messages
An explicit reasoning_effort beside a non-numeric thinking.budget_tokens keeps main's
behavior again. A declared hosted_vllm deployment with a Claude model name maps a
budget through the generic table instead of the adaptive Claude one. /v1/messages to a
non-Claude target hands a malformed budget to completion(), which answers 400 or drops
it under drop_params, where it answered 500
* fix(router): drop thinking beside an explicit effort when a declaring deployment's provider cannot read it
* test(integration): cover malformed thinking budgets on /v1/messages to non-Claude targets
* fix(anthropic): answer 400 for a malformed thinking budget on /v1/messages to any non-Claude model, or drop it under drop_params
* test(integration): cover an explicit effort beside thinking on azure_ai and drop_params on the /v1/messages bridges
* fix(anthropic): answer 400 for a malformed thinking budget through adapter_completion without drop_params
* fix(anthropic): type the adapter_completion request so the drop_params read adds no unknown argument
* fix(router): a default effort beside a thinking budget sends no effort on chat
Chat now leaves the budget-derived effort out when the caller's effort resolves to none or asks output_config for one, matching /v1/messages. The chat thinking decision moves into resolve_chat_reasoning so completion() stays under basedpyright's complexity limit.
* test: type respx_mock and mark the adapter result Final in the reasoning effort tests
* fix(router): keep the caller's thinking beside a kept effort and the budget effort beside a dropped one
A non-default output_config.effort beside a thinking budget now leaves the thinking as sent, so Bedrock
Invoke keeps display updates. A level drop_params drops counts as never sent, so the budget still sets the
effort. output_config.effort replaces a thinking budget the provider cannot read, and a non-numeric string
budget on adaptive Claude answers 400 in place of 500, or drops under drop_params.
* fix(router): send a kept output_config effort as reasoning_effort where the provider cannot read thinking
On a declaring hosted_vllm or azure_ai deployment, a thinking budget goes out as reasoning_effort. An
output_config.effort beside it now takes that slot, rounded onto the declared levels, instead of leaving
only output_config, which those backends ignore.
* fix(reasoning-effort): honor global drop_params for a non-numeric budget and keep the budget effort beside a dropped /v1/messages level
* fix(reasoning-effort): send a level a gpt-5 deployment declares even where the cost map does not list it
* fix(reasoning): apply declared levels on azure_ai gpt-5 and reject a bool thinking budget on adaptive Claude
* fix(router): drop a thinking budget's effort that a deployment declaring no ranked levels excludes
A deployment declaring [] or ["none"] skipped the budget check, so adaptive Claude
still sent the budget-derived output_config.effort under drop_params on chat,
Bedrock Converse, and native /v1/messages. The adaptive thinking now goes out
without that effort
* fix(anthropic): reject a false, null or empty thinking budget on adaptive Claude
* test(proxy): keep answering requests while a cold /openapi.json builds
* fix(proxy): build the OpenAPI schema off the event loop so a cold /openapi.json no longer stalls the worker
* perf(proxy): reuse the rendered /openapi.json body until the schema changes
* fix(proxy): scope the openapi build's pydantic patch to the build
The schema build swapped a process-wide pydantic hook, which live requests saw once the build moved to a thread, and overlapping builds could leave it installed. The hook now checks a context variable set only inside the build. The proxy test counts liveliness answers instead of timing them.
* test(proxy): hold the cold openapi build on an event instead of pacing on a clock
* fix(proxy): serve a replaced app.openapi like FastAPI's own docs route
* test(proxy): build the cold-docs test on the proxy module's current app
* test(proxy): freeze the model that holds the cold docs build
* feat(proxy): spend log size settings, storage probe and the reclaimable-bytes count
Adds maximum_spend_logs_size, maximum_spend_logs_cleanup_target_size and
maximum_spend_logs_size_check_interval, size_in_bytes, the pg_stat_user_tables
storage probe, and SizeCleanupState / advance_size_state, the running count of
reclaimable bytes kept in LiteLLM_Config that the size pass (next PR) compares
against the limit and the target. Nothing is scheduled or deleted yet.
* refactor(proxy): keep the spend log size settings out of the state layer
The three general_settings keys, their schema.d.ts rows, the SizeRule
parsing and the scheduler constants move up to the PR that acts on them,
so merging this layer alone changes nothing an operator can set
* test(proxy): cover the storage probe and the size state round trip
The probe, the LiteLLM_Config load and save and SpendLogStorage.after_deleting
had no caller in this layer, so codecov/patch sat at 67 percent
* fix(proxy): a stale storage probe carries the size state forward instead of reseeding
Bugbot on #47449: with zero live and dead rows the reseed made
bytes_per_row the whole file. advance_size_state now keeps the prior
count on a stale probe and, with no prior count, treats the files as
fully reclaimable so nothing is deleted on its say-so
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing
* fix(router): key prompt caching affinity on tools and the /v1/messages system prompt
Co-authored-by: nuernber <benjamin.nuernberger@jpl.nasa.gov>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read a 1h tool cache_control nested under function for the pin ttl
---------
Co-authored-by: nuernber <benjamin.nuernberger@jpl.nasa.gov>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): match partial wildcards against prefix-stripped model ids (issue #46384)
Refactor model checks to improve filtering logic and type hints.
* test(proxy): partial wildcard listing regression coverage (issue #46384)
* fix(proxy): drop unneeded annotation on suffix list (LIT001)
* fix(proxy): annotate suffix list with reasoned mutable-ok (basedpyright)
Change suffix_appended_wildcard_models to a typed list.
* Add lock timeout before creating telemetry report table
Set local lock timeout to 10 seconds before creating the telemetry report table.
* fix(proxy): list alias wildcards as routable ids, not raw catalog ids
An alias whose suffix collides with a catalog sub-prefix (e.g. public
"cloudflare/@cf/*" over deployment "cloudflare/*") took the partial-filter
branch and listed raw catalog ids. Routing substitutes the captured suffix
into the deployment pattern, so those listed ids routed to targets that do
not exist. Only use filter semantics when the deployment repeats the
wildcard suffix (or no deployment is available); aliases prepend their
prefix to provider-stripped catalog ids so every listed id routes back to
a real catalog model. Addresses Greptile review on #46390.
* fix(proxy): derive partial wildcard listings from the deployment's routing pattern
List an id for a partial wildcard only when routing it through the deployment's
pattern lands on a catalog model, so aliases, catch-alls, and key filters over a
wildcard deployment list ids that answer when called. /model/info expands a bare
* row the same way /v1/models does.
* fix(proxy): drop fragment ids for a catch-all over a partial wildcard and route bare aliases through the deployment pattern
A * catch-all over a partial deployment now lists no catalog ids, a bare alias like my-cf/* over cloudflare/@cf/* names ids by the deployment pattern, an alias whose wildcard count differs from its deployment lists nothing, and /model/info rows show the model each listed id routes to
* fix(proxy): keep listed ids on /model/info alias rows and clean up test-registered models
* fix(proxy): list catalog models through routing-prefix wildcard deployments
* fix(proxy): keep namespace and bare catch-all wildcards on routable listings
A leading deployment segment the model name repeats, or the provider name
itself, is no longer read as a routing segment, so same-name namespace
wildcards and a key filter over `*` -> `openai/*` stop listing ids that
route to no model. A bare `*` deployment under a prefixed name keeps the
legacy listing. `*` over a partial deployment now lists the names that
route through it, dropping empty ids.
* fix(proxy): list nothing for a catch-all over a mid-word wildcard deployment
* fix(proxy): list only wildcard ids that route back to their deployment and catalog model
A listed id now has to round-trip through the alias pattern to the catalog model it names, and the router's wildcard match for it has to pick this deployment, so an id a more specific sibling wildcard serves is no longer listed under the catch-all on /v1/models or /model/info
* fix(proxy): list wildcard aliases that route at a word boundary and ids a sibling serves with the same model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(streaming): skip full-chunk dump and delta rebuild on openai-compatible chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(streaming): reuse the built choice and delta on plain openai-compatible chunks
Co-authored-by: yassin <yassin@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): copy upstream chunk attributes in upstream key order
The scalar fast path iterated a frozenset of keys, so the SSE key order of chunk extras like service_tier and obfuscation changed with each worker's hash seed
* test(streaming): count chunk_creator calls with a profiler instead of patching classes
* test(streaming): check service_tier still passes through chunk_creator
* fix(streaming): reuse the built choice for chunks with upstream choice extras
* test(streaming): cover ModelResponseStream chunks on the choice reuse path
* fix(streaming): keep main's delta rebuild for deltas with nested values
Clearing role in place on a delta that holds tool calls left its nested
fields unset, so chunk.model_dump() dropped keys like a custom tool call's
name on later chunks. Deltas holding only scalars keep the in-place path.
* perf(streaming): build each plain OpenAI chunk's delta once and skip pydantic attribute misses
Later plain ChatCompletionChunks now build one Delta and one StreamingChoices, with no full-chunk model_dump, and read stored values instead of attribute probes that raise and catch AttributeError. Nested-value chunks no longer pay the dict(model) misses that made them slower than the merge base.
* fix(streaming): read hidden params a chunk carries as a pydantic extra
* test(streaming): pin logprobs, Azure filter extras, and audio and image deltas on later OpenAI chunks
* test(streaming): cover role, reasoning, choice extras and billing on plain OpenAI SDK chunks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): stream Claude json_schema answers as text on every streaming path
Async and proxy streams on Databricks Claude json_schema returned the
internal json_tool_call tool call with empty text, because the async
stream factory dropped json_mode. Databricks now passes it through, the
json tool's argument fragments stream verbatim as content (a lone {}
fragment was blanked before, corrupting nested empty objects), and the
stream ends with finish_reason stop like the non-stream answer.
* test(databricks): keep the json_schema stream regression tests ahead of the fix
Holds the Databricks transformation at the merge base so the first approval covers the regression tests; the fix commit follows.
* fix(databricks): stream Claude json_schema answers as text on every streaming path
* fix(databricks): keep finish_reason tool_calls when a json_schema stream also calls a user tool
* fix(databricks): drop text beside the json tool and renumber user tools in Claude json_schema streams
With reasoning_effort set, Claude can write prose before json_tool_call or call it next to a user tool.
Streams now hold text until the end and drop it when the json tool was called, renumber user tool
deltas past the dropped json tool so indexes start at 0, and non-stream returns the json tool's
arguments as content next to the user tools.
* fix(databricks): stream JSON-looking text at once and keep prose ahead of user tools in Claude json_schema streams
* fix(databricks): hold a user tool call behind held prose so a later json tool answer does not glue to it
A json_schema stream with thinking on that wrote prose, called a user tool, then called the json tool sent the prose at the user tool call and appended the JSON answer to it, so content did not parse. User tool deltas behind held prose are now held with it and sent merged per index at the json tool or the finish, and a json tool after text already sent is dropped so the answer is not repeated.
* test(databricks): cover json_schema answers on every endpoint in the integration wire cells
* test(perf): gate the gateway's own overhead header, redis latency and postgres load in the perf cases
* test(perf): keep the gateway_added_seconds limits next to the header ones
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row
* fix(proxy): read the provider handoff before the proxy-only error handler stamps it so a pre-handoff 499 is not priced
* fix(proxy): drop the pre-logging handoff read now that pre_call no longer stamps proxy-only failures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): avoid mutating default routes payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): apply default allowed_routes to key_type presets it covers
* fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy
* fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes
* fix(proxy): keep owners from changing routes a key got from default_key_generate_params
* docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off
* test(proxy): type the default_key_generate_params allowed_routes tests
* fix(proxy): let key owners send unchanged allowed_routes back on /key/update
* fix(proxy): ignore a route default with a non-string entry and match resent routes exactly
A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths.
* test(proxy): cover a non-string allowed_routes default in the key integration tests
* test(proxy): read key rows from the owned proxy's database in the default routes integration tests
* fix(proxy): leave allowed_routes out of a key update that resends them unchanged
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool
completion(), responses() and anthropic messages imported the MCP gateway handler
whenever tools were passed, and responses() did so on every call. Since the handler
began importing the MCP catalog at module level in #42568, that import needs the
optional mcp package, so a base SDK install failed with ModuleNotFoundError on any
function-tool call and on every responses() call
A new import-safe pre-check skips the handler unless some tool is a dict of type mcp,
which is the only shape the handler's gateway decision can accept. MCP gateway tools
still reach the handler as before
Adds in-process regression tests for the three entry points and a function-tool
completion to the base SDK install check
* fix(mcp): leave one-shot tool iterables unread in the gateway pre-check
responses() declares tools as an Iterable and the handler scans the same
object a second time, so a pre-check that read a generator would hand the
handler one tool fewer. The pre-check now scans only collections and reports
anything else as possibly naming the gateway, which keeps the previous path
The Anthropic messages regression test now mocks the httpx layer with respx
and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler
* fix(mcp): scan only lists and tuples in the gateway pre-check
The pre-check read any Collection, but a Collection that yields its items only
once would still be exhausted by the handler's second scan. Only the list and
tuple builtins are guaranteed to iterate the same way twice, so they are the
only inputs read now. Everything else keeps the previous path and is reported
as possibly naming the gateway
The unread-input test now covers a generator, an iterator, a one-shot
collection and a deque
* fix(mcp): scan only exact list and tuple types in the gateway pre-check
isinstance accepted list subclasses, and a subclass that iterates only once
would still be consumed by the handler's second scan. The fast path now reads
only objects whose type is exactly list or tuple. Every other input, subclasses
included, keeps the previous path and is reported as possibly naming the gateway
The unread-input test gains a one-shot list subclass and a tuple subclass
* fix(sdk): decide MCP gateway routing without importing the proxy handler
The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool.
* fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load
A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch.
* test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check
* style(sdk): drop docstrings from the MCP gateway pre-check and its tests
* test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request
An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's
litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper
now reads the referenced file or env var on each call and never writes the token back, so a rotated
token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as
server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback
targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400
* fix(secret_managers): strip only the leading oidc/ from a secret reference
A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed
* fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them
The advisor tool's own api_key now answers a 400 when it holds an oidc/
reference, since its sub-call runs through the client wrapper that resolves
one. Stored fallback targets, auto-router tiers and user_config refuse only an
oidc/ api_key, inline or through the credential they name, so their workload
identity federation handling stays as it was. Provider model discovery for a
wildcard deployment resolves the reference before calling the provider.
* fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments
* fix(utils): resolve oidc api_key refs only from admin-configured sources
Mark an oidc/ api_key read from a proxy-level deployment or a stored
credential, and resolve only that marked value. Team deployments, prompt
templates, nested configs and a direct SDK api_key send the text as
written. An unsupported reference kind raises a 401 so the router fails
over to the group's next deployment.
* fix(utils): send a credential's oidc token only to its own api_base host
A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the
credential's own api_base or base_url, so a team model or vector store that names the
credential with another api_base gets a 401 and that host gets nothing. Saving a team model
with an inline oidc/ api_key answers 400, since team rows never resolve one.
* fix(proxy): send an admin oidc api_key only to the hosts configured beside it
Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths.
* fix(proxy): redact the token path from token count auth errors
* test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs
* fix(proxy): redact the token path from secret reference errors where they are raised
* fix(proxy): keep the secret reference's path out of the error a caller sees
The 401 for an unreadable oidc api_key or pass-through credential now names only
the reference kind and a fixed reason, so a token path under /run/secrets or a
custom credential dir never reaches the caller, while the server log keeps the
full reference for the operator
* fix(secret_managers): strip line breaks from the logged secret reference
A reference holding a newline can no longer forge a server log line
* fix(secret_managers): label an unknown oidc reference kind instead of echoing it
An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in
full in the caller's 401 and in the spend log, since the kind was whatever
text followed oidc/. A kind get_secret does not know now reads
oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>,
which the client-message redaction on chat routes leaves whole.
* test(health): run the real model auth check in the oidc test connection test
* test(secret_managers): read an oidc/file token stored under a directory named oidc
* fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references
A huggingface/* wildcard call could name its own endpoint URL as the model and
receive the resolved token. The origin check now treats a URL in the model as
a host the reference must allow. The integration contract also reads the
proxy's own database instead of the pytest worker's.
* fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin
A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443,
though both reach the same server. The origin check now compares scheme, host
and effective port, as assert_same_origin does, and an unparseable api_base is
refused instead of raising. The advisor test sets the real proxy flag instead of
replacing our own helpers.
* test: type the oidc request-body helpers and drive the tier test through mock_response
* fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: []
Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix#47410 made for the stream stall cooldown cases
* fix(friendliai): drop litellm-internal assistant fields before sending
Claude Code replays thinking blocks on every turn after the first. The
Anthropic adapter turns them into thinking_blocks plus reasoning_content
on the chat assistant message, and FriendliAI rejects any message key it
does not know, so multi-turn Claude Code sessions on friendliai/ models
failed with a 500 after the first tool call
FriendliAI accepts reasoning_content and GLM reads it back, so only
thinking_blocks and provider_specific_fields are dropped. together_ai
already had this exact helper as a private function, so it moves to
common_utils next to strip_litellm_internal_message_fields and both
providers share it
Related to #32188
* test(friendliai): cover a replayed tool call turn
The regression fixture is now the shape Claude Code replays after a tool
call: an assistant turn with thinking_blocks, tool_calls and no content,
followed by the matching tool result. The test asserts both survive the
sync and async request paths unchanged
* ci: retrigger checks
* ci: retrigger benchmarks after runner timeout
* chore(friendliai): mark inherited list signatures as mutable-ok
The type-discipline gate on main now caps LIT001 at the merge-base count,
so the _transform_messages overloads need the same inherited-contract
marker together_ai uses for its identical override
* ci(test-unit): add tests/unit/llms/friendliai to the llms-providers shard
* test(friendliai): check the outbound FriendliAI body through the proxy for chat and messages replays
Both replay shapes go through a real proxy to a scripted FriendliAI upstream, and the test reads the
request body FriendliAI received. It fails at the merge base, where thinking_blocks and
provider_specific_fields reach the upstream, and passes with the field stripping in place.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: pin the validation lines #44798 kept with failing cases
* fix(containers): accept endpoint table entries that omit their param lists
* test: keep the endpoint table and xAI adapter as on main
The endpoint table ships inside the package, so optional param lists fixed no
reachable state and turned a loud failure into silently dropped params. Drops
the Gemini case that main's tests already cover.
* test: cover the cost map backup and Gemini image config validation lines
The cost map backup adapter's hide flag, and the validation itself, could be
removed with every test still green. The Gemini imageConfig adapter could be
narrowed with only the integration suite noticing, which does not run on PRs.
The endpoint table cases now read a real file through the module's location
instead of replacing open, so reading the file another way keeps them green.
* test: pin the container handler's loud table failure and loosen two pinned drops
A container endpoint table entry without path_params or query_params fails
the call on main, and no test held that. The new case trims each key from the
packaged table and expects the validation error.
The Gemini imageConfig case now accepts a drop or UnsupportedParamsError for a
non-object value, and the Bedrock beta case checks only that numbers come back
as strings, so neither pins behavior the drop_params convention may change.
* test: check a numeric beta array is never split as text and use the latest Gemini image model
The Bedrock case now also fails when a JSON array of numbers falls through to
the comma split, which returns bracketed fragments, while a drop or a string
coercion both stay allowed.
