#47525 renamed _deployment to _fallback_deployment; #47363 was written against the old name and merged after it, leaving three F821 undefined-name errors that fail the python lint job on main
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(completion): send prompt_cache_retention to OpenAI instead of silently dropping it
litellm.completion(), Router.completion() and the proxy's /v1/chat/completions
accepted prompt_cache_retention and listed it as a supported OpenAI param, but
completion() never handed it to get_optional_params, and the leftover-kwargs
paths skip known OpenAI params, so it never reached the request body. Make it a
named argument of completion() and acompletion() like prompt_cache_key and pass
it on.
A unit guard sends every OpenAI body param through completion() and fails when a
param added to OPENAI_CHAT_COMPLETION_PARAMS has no case proving it reaches the
wire, the same hole store and prompt_cache_key fell into in #33184. Live e2e
tests capture the body OpenAI receives for the proxy, the SDK and the Router on
both openai/gpt-5.5 and openai/responses/gpt-5.5.
* chore(completion): type the param-forwarding unit test and keep the basedpyright count flat
Every new completion() kwarg adds one unknown-argument error at each call
site that splats an untyped dict into it; typing print_verbose's argument
and Chat's params removes more than prompt_cache_retention added.
* test: freeze the outbound-body model and read captured requests from the mock
Addresses review: the e2e body model is frozen, the backend params are a
tuple, and the unit test reads the request from the respx route instead of
appending to a list.
* fix(cost): price base_model deployments past rule-only cost map entries and warn when base_model has no rates
* fix(router): warn on unpriced base_model from add_deployment too and honour model_info prices
* refactor(cost): drop the unpriced base_model startup warning, keep the one-line ladder fix
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): measure Admin UI journey latency in a browser on the 1TB restore
* test(perf): settle on rendered rows, chart bars and the usage view trigger
* test(perf): cover the UI journey summaries in the e2e harness tests
* test(perf): fail UI journeys on failed samples and p50/p90 ceilings
* test(perf): add Usage, Models and Keys journeys and seed realistic keys, teams, users and models on the disposable restore
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate model cost maps in runtime registration tests
* test: restore cost map entries and provider model lists after unit tests
* test: restore provider model sets after unit tests and cover the catalog restore
* test: snapshot litellm globals before scanning provider model sets
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request
_billed_earlier_attempts reads the router's previous_models records, keeps the Timeout ones other than the final attempt (matched on deployment_id plus attempted_retries, since the count restarts after a fallback hop and the fallback handler re-logs the exhausting attempt) and prices each at the deployment that served it. post_call_failure_hook lifts the tuple as billed_failed_attempts; nothing writes it yet. previous_models joins the router-reserved metadata fields so a client-planted record never reaches it
* fix(proxy): price earlier timed-out attempts from the prompt estimate only
The final row may lift a mid-stream recovered usage (prompt plus delivered chunks) or a failed_attempts_usage sum; earlier attempts never produced output, so they are priced from _estimated_prompt_usage, and when failed_attempts_usage already bills them into the final row none is lifted again
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count-tokens): translate chat tool use before Claude token counts
Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent.
* test(count-tokens): compare Claude count bodies with inference for Responses tool use
* fix(responses): forward the caller's Responses body to OpenAI token counts
OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it
* fix(responses)!: count Claude Responses bodies the way inference sends them
Non-native Responses counts now run the body through the inference bridge, so
text.format, tool_choice, reasoning, built-in tools, and file_search count the
way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The
OpenAI native count body uses the drop_params-resolved tool fields.
* fix(responses): honor drop_params and replay previous turns in Claude counts
Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation
* fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts
* fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params
* fix(responses): leave tool_choice out of counts when drop_params drops every tool
* fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send
The Bedrock Mantle fallback now counts thinking, tool_choice and output_config
like the runtime count. Claude Responses counts leave out cache_control marks
again, so more than four marks count instead of failing. Counts answer 400
before any provider call for user blocks chat inference refuses and for
Responses tools with no Chat Completions equivalent, instead of dropping the
block or counting a placeholder tool. Adds the wire tests for these paths
* fix(count-tokens): count Responses tools the way the inference bridge sends them
A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request
* fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id
* test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler
* fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic, bedrock): cover a redacted_thinking content item on every Claude route
* fix(anthropic, bedrock): keep a redacted_thinking content item on every Claude route
* fix(anthropic, bedrock): send each replayed thinking block once, without cache_control or blank data
* fix(anthropic, bedrock): keep thinking order and gate replay key drops on drop_params
A thinking block that only one of content and thinking_blocks carries now goes
back in its original place instead of being prepended. cache_control on a
thinking block and blank redacted data are dropped only under drop_params and
otherwise reach Claude as sent. Converse drops a cachePoint after reasoning
under drop_params, skips a thinking_blocks entry that is not a block, and the
cache control hook no longer marks a thinking part.
* fix(anthropic, snowflake): match a signed thinking block on its signature and send each block once on Snowflake Cortex
A signed thinking block in both content and thinking_blocks is the same block when the signatures match, even if
the thinking text differs, so it is sent once from content instead of twice. Snowflake Cortex now merges
thinking_blocks into content the same way the Anthropic routes do. A redacted_thinking item with blank data no
longer counts as sent for system message placement.
* fix(anthropic, bedrock): send a thinking block once when consecutive assistant messages both carry it
Consecutive assistant messages merge into one Claude turn, so a block one message holds in content and the next
holds in thinking_blocks went out twice and Claude answered 400. The merged turn now keeps the first copy of each
redacted or signed block on the Anthropic routes and both Bedrock Converse builders.
Snowflake Cortex goes back to its previous message building, since its double send cannot be proven live without a
Cortex account and is tracked on its own.
* test(anthropic, bedrock): cover a redacted block with null data and a non-block part beside a cache breakpoint
* fix(anthropic, bedrock): match only text-valued thinking items when merging replayed blocks
* refactor(types): replace Any with proven types in 39 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use valid response fixtures for strict parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): revert anthropic chat response seam
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): revert validation seams that changed tolerated inputs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): revert remaining validation seams to keep behavior unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): drop no-op search router annotation and helper docstring
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep ip address utils at base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): send the completion start time
langfuse_otel never set langfuse.observation.completion_start_time, so
Langfuse showed no time to first token for any call, streamed or not.
The legacy langfuse callback already sends kwargs["completion_start_time"].
Send the same value as an ISO timestamp with an explicit offset.
* fix(langfuse_otel): type kwargs and test against a real span
Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any],
the declared type of model_call_details, so the new completion start time
read adds no unknown-type errors. The tests now check a real SDK span
instead of patching safe_set_attribute, and the parameter is typed.
* fix(otel): send the completion start time from the v2 Langfuse mapper
With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper,
which had no completion start time either. Carry the payload's
completionStartTime on LLMCallSpanData for streamed calls, the same calls
that get time_to_first_chunk_seconds, and map it to
langfuse.observation.completion_start_time.
* fix(langfuse_otel): take the logging kwargs as a read-only Mapping
The new kwargs annotation tripped LIT001. Both helpers only read the
logging kwargs, so type them Mapping[str, Any] instead of dict.
* fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter
* fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON
The native Gemini, Interactions and A2A streams never recorded a first chunk, so with
completion_start_time now sent both emitters showed TTFT equal to latency on them. They
now record it, and Gemini and Interactions take the call's start time like /v1/messages.
Both emitters send the completion start time as a JSON string, the form the Langfuse SDK
and the langfuse callback send, through one serializer.
* fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event
A2A streams took their start time when the iterator was built, so their latency left out
the work before it. They now use the call's start time like every other route.
Cached /v1/messages stream replays recorded no first event, so their time to first token
fell back to the end time. The replay now records it on the first event it yields.
The tests use the real Interactions config and typed logger hooks.
* fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free
* test(a2a): type the card fetch test and stop rebinding the seed helper's recorder
* test(otel): expect completion_start_time and cache_hit in the B3 span keys
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost-map): add volcengine doubao-seed-2-1-pro-260915 and doubao-seed-2-1-lite-260915
Pro row absorbed from #47372, lite row added from the same Volcengine Ark pricing and model list docs
Co-authored-by: donghongfei <1247543758@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-map): cover volcengine doubao-seed-2-1-pro-260915 row
Co-authored-by: donghongfei <1247543758@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): add undated azure/eu and azure/us rows for o1, o3, o3-mini, o4-mini, gpt-4o, gpt-4o-mini
Absorbed from #47499
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: donghongfei <1247543758@qq.com>
* fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): use lane AWS credentials for the Kimi K3 messages test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bypass the proxy cache and declare test metadata for the Kimi K3 probe
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit ded4f084c5.
* fix(responses): bill a background response under the model name the client sent, with its project alias
* fix(responses): name the team's public model as a billed background response's model group
The model group lookup now passes the creating key's team, so a team model or team wildcard bills under the public name a normal call logs and per-model budgets on that name move
Vertex AI and Gemini context caching now remember each cache they create or find until 10s before its expireTime, so a repeat cache-marked request goes straight to generateContent instead of listing every cachedContents first. Concurrent async requests for one prefix share one lookup or create
When generation rejects a remembered cache with a 400, 403 or 404 that names the cached content, the request rebuilds the cache and sends once more. Any other 400, 403 or 404 drops the record without a retry
Fixes#36395
* feat(embeddings): add max_batch_inputs pre-call validation
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* add unit tests for max embedding batch size
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* feat(embeddings): enforce batch limit on extra_body input override
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* chore(dashboard): update schema.d.ts for embedding batch size model field
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): close Bedrock batch bypass and satisfy ruff PT012
Count max(top-level input, extra_body.input) for batch limits so providers
that ignore extra_body cannot bypass max_embedding_batch_size. Add router
tests and refactor pytest.raises blocks for ruff-tests.
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): drop unchecked casts in router batch-size check
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): route oversized batches to a group deployment that admits them
* fix(embeddings): count extra_body input for any top-level input and prefer admitting deployments before cooldown
* fix(embeddings): log cap rejections to failure callbacks and scope batch routing to embedding calls
* fix(embeddings): apply the batch cap after tag and pre-call filters and fall back on cap rejections
* fix(router): keep embedding cap rejections out of deployment health and hold only for routable cooling deployments
* fix(router): clear the embedding cap rejection mark before a fallback's provider call
A fallback or retry shares the request's logging object, so the mark a rejected attempt set made telemetry skip the fallback deployment's real call. Unit tests now pass a fixed logging start time instead of reading the clock.
* fix(router): log embedding batch rejections once, before a fallback runs
* refactor(router): drop docstrings that restate the code
---------
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let proxy admin keys read raw cloud storage file ids on GET /files/{id}/content
Since #30595 the endpoint answered 400 for every raw gs:// or s3:// id, which broke the
documented unmanaged Vertex batch flow at its last step, while a double-encoded id slipped
past the guard entirely. Raw ids are now readable by proxy admin keys only, every other key
gets 403, and the guard decodes the id fully before deciding, so encoding depth no longer matters
* fix(proxy): decode nested percent-encoded file ids iteratively
An id percent-encoded a thousand or more times sent the recursive decoder past the
interpreter's recursion limit, so a proxy admin key got a 500 instead of a verdict on
the id. The decoder now walks the encodings in a loop, so any depth resolves
* fix(proxy): cap the decode passes the raw cloud storage file id guard makes
Any authenticated key could send a file id wrapped in tens of thousands of percent-encoding layers and the guard would unwrap every one of them, each pass walking the whole id, so a 120 KB request line held a worker for about five seconds and a 2 MB one for minutes. The guard now unwraps at most eight layers, and an id still changing after that is treated as a raw cloud storage id, so it stays admin-only instead of being decoded any further
* refactor(proxy): type the decode step the raw cloud storage file id guard folds over
* test(proxy): return a streaming result from the admin raw id test mock now that vertex_ai file content streams
* test(proxy): send raw cloud ids to the content route the way uvicorn delivers them
Starlette's TestClient decodes the path twice, so the twice-encoded cases reached the route fully decoded and never hit the decode loop. The cases now name the id the route receives, and the helper tests take a typed quote helper and Final constants.
* test(proxy): mark the raw cloud id test locals Final
* test(integration): follow #47441 metadata-routed Mantle counting in count_tokens wire tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build the runtime count body without mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): leave out span cost when the price lookup failed
A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free.
* test(otel): keep the guardrail charge on the guardrail span for unpriced calls
* refactor(otel): type the Arize cost writer input and reuse the cache_hit flag
* fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost
Only the placeholder 0 is left out when the price lookup failed, so a
guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel
and W&B Weave OTel. Adds unit cases for that fee and a malformed cost.
* fix(arize): leave out the span cost on any failed price lookup, matching OTel v2
* fix(logging): clear a stale cost failure flag when a later pricing succeeds
Stream-start pricing on the request model name can fail and set the cost
failure flag even when the final pricing succeeds (OpenRouter usage.cost,
an Azure deployment name priced from the response model). The new span gate
then dropped a real cost. Each pricing now resets the flag, and
uncached_price restores whatever flag it found.
* test(logging): pin the start time in the stale cost flag tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(sagemaker): parse every streamed record however SageMaker cuts the payload parts
SageMaker cuts a container's response stream into PayloadParts at arbitrary
byte offsets, so one part can hold several SSE events or JSON lines, or only
part of one. The decoder treated each part as one record, so vLLM's [DONE]
was logged at ERROR on every stream and a split at an unlucky byte dropped
the rest of the answer.
Peel records off a running buffer with JSONDecoder.raw_decode, skipping SSE
framing and [DONE] between them.
* test(sagemaker): cover streamed records cut or merged across payload parts
* fix(sagemaker): decode UTF-8 across parts, skip SSE comment lines, and log container error records
A UTF-8 character split across two PayloadParts now decodes whole through one incremental decoder per stream.
SSE comment lines (TGI's keep-alive) and event, id and retry field lines are skipped as framing.
A container error record is logged at ERROR with its message and skipped, and any other record that is not a
chunk is logged and skipped, the same way on the sync and async paths.
* fix(sagemaker): raise a container's error record and ModelStreamError on both stream paths
A record with a non-null error now raises a 500 SagemakerError carrying the record after the chunks before it, on sync and async. Async no longer swallows a non-200 event, so a ModelStreamError raises as it does on sync. A chunk with a null error field is delivered, and any other record that is not a chunk is logged and skipped on both paths.
* fix(sagemaker): answer a streamed error with the container's status and fail a stream cut mid-record
A container's error record now raises with the status the container gave it (vLLM's error.code, the older flat vLLM shape, TGI's http_status_code or error_type) instead of a fixed 500 that the router retried as a 503. A ModelStreamError raises a SagemakerError naming its ErrorCode and Message, a stream that ends inside a record raises instead of ending as a stop, and a skipped record is logged by its keys only
* fix(sagemaker): trust only 422 and 429 from a streamed error record and fail LMI generation errors
* test(sagemaker): drive an LMI failed-generation stream through the proxy on a sagemaker/ deployment
* test(sagemaker): fall back to the next model when a stream fails before any content
* fix(responses): keep strong refs to stream and websocket logging tasks
* test(responses): patch _spawn_logging_task instead of asyncio.create_task
The success-logging dispatch in _log_completed_response now goes through
_spawn_logging_task, so tests that patched asyncio.create_task no longer
observed it (and the sync ones hit get_running_loop with no loop).
* test(responses): assert stream and websocket logging tasks survive garbage collection
Adds regression tests that force a collection while the logging task waits
on an otherwise unreferenced future, and types the new test parameters
* test(responses): drive drain and GC tests through a real logging object with gated callbacks
* test(responses): use asyncio.wait_for so the logging tests run on Python 3.10
* test(guardrails): type the spawn_logging_task tracking wrapper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
* feat(harness): add pi harness for litellm.agent()
Adds Harness.PI, which drives the pi coding agent CLI (`pi --mode json`) once per turn
against the per-session model endpoint. The endpoint is declared as a `litellm`
openai-completions provider in the models.json of a LiteLLM-owned PI_CODING_AGENT_DIR, so
the user's own pi settings, auth, extensions and MCP servers are never read, and the token
travels only through an env var referenced as `$LITELLM_HARNESS_TOKEN`
pi exits 0 after a failed provider call, so a turn fails when the last assistant message
ends with stopReason error/aborted. Permission modes map to a `--tools` allowlist,
disable_tools to `--exclude-tools`, and sessions resume with `--session` from a persisted
session dir. build_instructions and turn_prompt move from the OpenCode config into the
shared harness utils so both configs use them
Unit tests run on JSONL recorded from pi 1.1.0
* test(harness): align pi harness tests with the other CLI harness suites
Mirror the OpenCode suite's layout, fakes and helpers, and add the cases every CLI harness
covers: turn before start, early close kills the process, persist failure, endpoint model
fallback, long JSONL lines, skills without SKILL.md and a missing endpoint. Add an
endpoint_requests.jsonl fixture recording what pi sends to the session endpoint, and
re-record the turn fixtures in one run so their ids agree
* fix(harness): parametrize harness e2e only over harnesses with a model group
harness_params() iterated every Harness member, so Harness.TOOL_LOOP, which has no entry in
BINARIES or DEFAULT_MODEL_GROUPS, raised KeyError and the whole suite failed to collect
* feat(harness): accept pi config and MCP servers through PiOptions.config
PiOptions gains `config`, like the other harness options. `mcpServers` is written to the
private agent dir's mcp.json and every other key to its settings.json. Keys that would reroute
model calls, bypass permissions=, or load code as the host user (defaultProvider, defaultModel,
defaultTools, defaultProjectTrust, extensions, packages, ...) raise OptionsMismatch
A plain-name `--tools` allowlist drops MCP tools in pi 1.1.0, so when servers are configured
the allowlist also gets `mcp__*` plus the tool each exposure is reached through (`codemode` by
default, `tool_search` for `deferred`). disable_tools still excludes individual MCP tools
* refactor(harness): give harness options immutable mapping defaults
`field(default_factory=dict)` left every config/env default typed as
`dict[Unknown, Unknown]`, which basedpyright reports as
reportUnknownVariableType under strict mode: 9 errors across the five
options classes, two of them added by PiOptions. A shared module-level
`MappingProxyType({})` default is accepted by frozen dataclasses, matches
the no-mutation rule the rest of the harness follows, and drops the file
to zero errors so the rule shrinks against the merge base instead of
growing.
* fix(harness): reject managed env keys and a missing model in the pi harness
PiOptions.env could name PI_CODING_AGENT_DIR or LITELLM_HARNESS_TOKEN and
the value was dropped in silence, because transform_session_setup merges
the managed keys last. The ordering stays as defense in depth, but
validate_environment now rejects those keys up front the way the Claude
Code config does, so a bad option is an error rather than a setting that
appears to apply and does not.
transform_turn_request also stringified the model it resolved, so a call
with neither model= nor an endpoint model would have put the literal
"None" on pi's --model. Both call sites now go through session_model(),
which raises the same ValueError transform_session_setup already raised.
* test(harness): escape the regex dot in the pi skill manifest match
* fix(harness): keep harness options importable on Python 3.11
Python 3.11 rejects an unhashable dataclass default, and MappingProxyType
only became hashable in 3.12, so the shared MappingProxyType({}) defaults
made importing litellm.harness.options raise ValueError and broke every
harness on 3.11. The defaults now come from typed default factories that
still return read-only mappings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): address review feedback on the pi harness tests
Drop the endpoint-request fixture test, which only read a recorded file and
could not fail when PiHarnessConfig changed, along with its now unused
fixture. Remove the section-divider comments and a comment that restated
MANAGED_ENV_KEYS, and type the test helpers and fakes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(harness): cover the last uncovered pi harness lines
Assert that a tool_execution_end event without a result yields an empty
ToolResult output, and that a PiOptions.config value JSON cannot encode
raises TypeError. The Python 3.11 default check now asserts on the list of
unhashable defaults instead of relying on hash() raising, which the
test-quality gate flagged as an assertion-free test
* test(harness): annotate harness option tests and stop rebinding helper params
Mark the option test locals Final and check read-only defaults through the
MutableMapping interface instead of writing into them. In the pi suite,
type the parametrized test arguments, and have started() and parse_all()
bind new names instead of reassigning their sandbox and state parameters
* test(harness): test harness option defaults by constructing the options
Replace the field-introspection check with one that builds each options
class twice and asserts every mapping default is empty, read-only and not
shared between instances. Sharing one MappingProxyType across instances is
what broke class creation on Python 3.11, and that sharing is observable on
every Python version, so the test fails on the old code without inspecting
dataclass declarations
* test(harness): stop mutating test helper inputs and assert whole values
make_ctx took model and endpoint out of **kwargs with pop, which mutated
its own parameter; they are explicit parameters now, with a frozen default
endpoint. assistant_end builds its message in one expression instead of
adding errorMessage afterwards. Two docstrings that explained an assertion
became that assertion's message, and the options test compares whole
values so a failure names the offending fields
* ci(unit): run the pi harness unit tests in the llms-h-to-z shard
main now lists every tests/unit/llms provider directory per shard, so the new
tests/unit/llms/pi directory was claimed by no shard and assert_ci_coverage
failed on both the shard and the test-file checks
* test(harness): mark pi test locals Final and freeze the test output model
Every single-assignment local and module constant in the pi suite is now
Final. The name-mapping loop became one comprehension compared as a whole
list, test_start_persists_sessions_dir takes the sandbox by index instead of
binding _ twice, and the Answer output model is frozen like every other
pydantic model
* fix(harness): send PiOptions.thinking to the gateway as reasoning_effort
pi clamps --thinking to "off" for a model that does not declare reasoning,
and the generated models.json only declared {"id": model}, so every
thinking level was dropped and the gateway never saw reasoning_effort.
xhigh and max are also only offered when thinkingLevelMap names them. The
model now declares reasoning plus that map whenever a level other than
"off" is requested, and nothing when thinking is unset, so pi does not send
its default effort on every call
The pi tests also drop **kwargs from make_ctx and started in favour of
explicit parameters, and annotate their helpers and the fake sandbox with
read-only types; the fake records calls by rebinding tuples instead of
appending to lists
* docs(harness): list pi among the harnesses in the module and agent() docstrings
* fix(harness): keep pi calls provider-agnostic and harness option defaults copyable
Declare supportsStore false so pi stops sending store, take contextWindow and maxTokens from the model map, treat a length stop or a silent exit as an error, and give every options class plain dict defaults again so deepcopy, pickle and asdict work
* fix(harness): look up pi gateway groups as litellm_proxy and ask pi for cache_control marks
A gateway session's model is the bare group name, so the model map lookup
now passes custom_llm_provider=litellm_proxy, the way the SDK prices it.
That keeps an unmapped group from printing get_llm_provider's provider
list banner into the caller's stdout.
pi only adds Anthropic cache_control marks for OpenRouter Anthropic models,
so every Claude call through the harness missed the prompt cache. Models
the map says support prompt caching now declare cacheControlFormat
anthropic, which LiteLLM translates for each provider.
* fix(harness): cap pi's maxTokens at its own default and fork sessions saved in another directory
pi sends maxTokens as max_completion_tokens on every call, and the gateway reserves that many
tokens against the key's TPM limit, so passing the model's full output cap (128000 for
claude-haiku-5-5) made keys with lower limits answer 429. maxTokens is now the lower of the
model's cap and pi's own 16384 default.
agent_resume() into another sandbox directory stopped at pi's interactive fork prompt. The
native session id now carries the directory pi saved the session in, and a resume from a
different directory passes --fork, so the conversation continues in the new sandbox.
* fix(harness): leave pi room for the thinking budget and price cached prompt tokens
pi's maxTokens now adds LiteLLM's thinking budget for the requested level to pi's 16384 default, within the model's output cap, so Anthropic budget thinking at max no longer answers 400. Harness turns now pass cache read and write tokens to cost_per_token, so cached prompts are priced at their cache rates on every harness.
* test(harness): cover cached prompt cost on the gateway and non-stream routes
* test(harness): mark the cost test constants Final
* test(harness): type the cached cost test helpers without Any
* fix(harness): price OpenAI cache write tokens in harness cost
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): restore the token estimate for streams that send no usage
A stream without stream_options got a made-up Usage(0, 0) written into its
final chunk's hidden params. Since #42323 the chunk builder treats that as a
provider-reported zero, so these streams were logged and billed at 0 tokens
and $0 instead of the tokenizer estimate. Only attach the summed usage when
some chunk actually carried usage.
* test(streaming): use a fixed start time and check the estimated cost
* fix(streaming): keep reported usage on the synthesized finish chunk and test it
* test(streaming): cover explicit zero usage reported by the provider
* fix(streaming): set the token estimate on a copy of the final chunk and ignore usage with no counts
* fix(streaming): keep final chunk edits in the logged response when the provider sends no usage
* fix(streaming): mark the unreported usage placeholder so every chunk rebuild falls back to the estimate
* test(streaming): wait on the success callback event instead of sleep-polling
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path
* fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format
* fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params
A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did.
A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper.
Adds integration cases for the json_schema envelope and the cut stream.
* fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates
Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded
* fix(anthropic): join a split answer when the stream ends in its last block
A cut during the second json tool block now runs the held block through
the same flush as a block stop, so it joins the streamed keys or the
earlier split blocks instead of sending two objects back to back. The
Vertex output_format fold drops its unreachable drop_params branch,
since both parent configs drop a non-object value first.
* test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages
* fix(proxy): drop router_settings_override before routing and apply it on every branch
route_request only popped router_settings_override inside the branch that
runs after the api_key/api_base, comma-separated batch and user_config
branches. Requests taking one of those earlier branches forwarded the field
to the provider, which Anthropic rejects with a 400, and silently lost the
key/team num_retries, fallbacks and timeout. The override is now popped and
merged into the request before any branch is chosen, keeping the existing
precedence where request values win and None values are skipped
Fixes#43582
* fix(proxy): apply key/team fallbacks when the request carries api_key or api_base
The previous commit skipped fallbacks, context_window_fallbacks and
content_policy_fallbacks from router_settings_override whenever the request
had api_key or api_base. That broke the scenario from #43582, where a
trusted pre-call hook sets api_key and the key/team router_settings still
need to apply. All per-request router settings are merged again on every
branch, with None values skipped and request values still winning
The tests now check that fallbacks run on the api_key and api_base paths
and that the provider never receives router_settings_override
* fix(proxy): answer 400 for a non-object router_settings_override and treat null as absent
* fix(proxy): drop a malformed router_settings_override under drop_params
* fix(proxy): route key and team router settings like any request, honor user_config and global drop_params
* fix(proxy): keep Router-only override settings out of direct provider calls
Calls that route_request sends straight to litellm (evals, endpoints that need
no model, pass_through_all_models, the video fallback and the no-router paths)
now leave out context_window_fallbacks, content_policy_fallbacks and
model_group_retry_policy merged from key or team router settings. Only the
Router reads them, and providers reject them as unknown parameters.
* test(proxy): drop a comment from the router settings override test
* fix(proxy): keep key router settings on the model read-through retry
route_request now merges router_settings_override once, before the first
attempt. The retry after a model read-through used to pop the merged
enable_tag_filtering again and recompute an empty Router-only set, so a
model created on another instance was served without the key's tag
filtering
* test(proxy): check the read-through retry keeps key tag filtering through a real router
The regression test now routes through a real Router with two tagged deployments from the database instead of replacing Router.acompletion. Before the fix the retry picks the lower-order untagged-for-request deployment, after it the tagged one
* fix(proxy): keep key Router-format fallbacks out of direct provider calls
Key and team fallbacks use the Router's model-group mapping. litellm.acompletion reads each mapping as a fallback config, so a direct call retried the same model with the mapping merged into the provider body. The override's fallbacks now stay out of direct calls like the other Router-only settings, while num_retries and timeout still pass
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): schedule default_priority requests once and reject a non-integer priority
A router configured with default_priority failed every request that carried no
priority with a 500 naming a missing 'priority' argument, and an explicit
priority re-entered acompletion and hit the same error. acompletion now resolves
the priority once (the request's integer, else the default, with 0 kept as a
real priority) and schedules through _schedule_factory directly, while
schedule_acompletion delegates to acompletion so the two paths share one queue
pass. A non-integer priority answers 400 naming the param, or is dropped under
drop_params, instead of leaking to the provider or crashing the scheduler.
* fix(router): honor router-level drop_params and schedule prompt-management models under a priority
* fix(router): drop a non-integer priority when the model's deployments set drop_params
* fix(router): resolve the drop_params deployment scope for team routes and deployment ids
* fix(router): read the request and router drop_params flags without a loop-assigned Final
* refactor(router): type the prompt-management wrapper's return and mock the provider in its test
* perf(router): consult drop_params for the priority only when it is not an integer
* test(integration): assert the new priority contract for non-integer and zero priorities
* docs(router): describe default_priority as the priority every request without one takes
* fix(router): parse default_priority once and apply it to text completions and the queue route
A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig.
* fix(router): type the priority dispatch so the lint budgets hold
* fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order
* fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses
* fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority
A request whose model is a deployment id now counts that deployment as healthy in the
scheduler, since routing by id already ignores cooldown. Before, the scheduler found no
healthy deployment for an id, so the request waited behind every entry ahead of it.
A boolean default_priority (true, or YAML off) is ignored at startup with a warning,
like any other value that is not a whole number.
The scheduler integration tests now share worker state for the defaulted pair, cool a
group through /v1/responses so no stale in-memory queue is left behind, and use "high"
instead of "1" for the non-integer cases. The concurrent burst test no longer asserts
that the queue drains, matching the explicit-priority burst test, since the scheduler's
read-modify-write queue can race across workers.
* test(router): keep the queue drain check as a BUG-skipped integration test
* fix(router): queue default_priority chat requests once instead of failing them
* test(router): open with the default_priority regression tests alone
* fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set
With router_settings.tag_routing_prefix configured, an unprefixed caller tag
(for example a spend tag such as user_id:234) no longer drives tag routing, so
it can no longer refuse a model group whose deployments are untagged.
Key/team/project tags (metadata.inherited_tags) keep applying with or without
the prefix. Rows of batches the proxy runs itself now carry those tags as
inherited_tags too, so they keep routing on them once the prefix is set.
Takes over #39903.
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* test(integration): pin that proxy-run batch rows keep key/team tag protection
* refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells
* test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst
* test(integration): proxy-run batch rows keep key and project tags too
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* feat(context_management): honor clear_at_least, exclude_tools and clear_tool_inputs in the clear_tool_uses polyfill
The clear_tool_uses_20250919 polyfill honored only trigger and keep and
ignored the other three fields with a warning. It now follows the Anthropic
API, as measured against it:
- keep counts clearable tool uses only; uses of exclude_tools are never
cleared and count toward neither keep nor clear_at_least.
- clear_at_least clears oldest first in batches of at least that many
tokens and leaves an incomplete last batch, so the cleared prefix only
grows between requests and each batch costs one prompt-cache miss.
- clear_tool_inputs (true or a list of tool names) also empties the input
of the cleared tool uses.
Invalid values are ignored with a <field>_invalid warning on the applied
edit.
* refactor(context_management): build the clear_tool_uses selection without mutable state
* test(context_management): cover a history made only of excluded tools
* fix(context_management): leave tiny tool results out of clear_tool_uses like the API
The API neither counts nor clears a tool use whose result is 1 to 46 bytes
of text, whatever the tool or error flag; empty results still count. Probed
against the API with crafted histories. Test data under 46 bytes is
lengthened so those tests keep exercising clearing
* fix(context_management): count tiny tool results when clear_tool_inputs covers their tool
The API leaves a use whose result is 1 to 46 bytes out of clear_tool_uses
only while its input is kept. When clear_tool_inputs is true or names the
use's tool, the API counts it toward keep and clears it like any other use.
* fix(context_management): count cleared inputs and images toward clear_at_least, answer malformed fields with a 400
* fix(context_management): validate clear_tool_uses edits before any editor runs, drop only the malformed edit under drop_params
* fix(context_management): drop only the malformed clear_tool_uses fields under drop_params
A malformed clear_at_least, exclude_tools or clear_tool_inputs no longer drops the whole edit under drop_params. The edit still runs with its well-formed fields, so the clearing that worked before is kept. Also types the new tests and swaps the hand-rolled spy router for an AsyncMock
* test(context_management): pin the 46 byte cutoff in UTF-8 bytes and a batch closing at exactly clear_at_least
* fix(context_management): leave a non-string tool_use id for the provider, size lone surrogates, and read drop_params strings
* fix(context_management): read a drop_params string the same way on count_tokens and both /v1/messages paths
count_tokens still checked drop_params with is True, so a body drop_params of "true" answered 400 there while /v1/messages dropped the malformed field. Both routes and the sync path now use drops_unsupported_params.
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): expose supported_endpoints on /model_group/info
* feat(router): route batch creates to mode batch deployments in mixed model groups
* fix(router): prefer batch deployments after eligibility filters and in group-name credential lookups
* fix(router): keep group-name credential lookups on the first deployment and prefer batch deployments before cooldown
Model-encoded ids carry only the group name, so preferring the batch deployment there moved ids created before the upgrade onto its credentials. Batch creates now pick batch-capable deployments before health checks and cooldown, routing and group info share one batch-capable check, the cost map's /v1/batch is dropped only for Azure, and /model/info applies the same per-deployment rule
* test(router): map _listed_endpoints to its /model/info tests in the router coverage check
* fix(router): skip only Azure chat deployments for batch creates and list batch support per caller team
Narrow the batch create filter to Azure and azure_ai deployments that are not mode batch, so OpenAI groups keep
their chat deployments during cooldown and under tag filters. Mark batch deployments by mode batch only, since
stored endpoint lists from older Admin UI saves copy the cost map. Leave /model/info unchanged so the Admin UI
never saves a computed endpoint list, and pass the caller's team into /model_group/info.
* test(proxy): let the model hub router fakes take the caller's team_id
* fix(router): gate batch-create routing on deployments the caller's team may use and list only /v1/batch for Azure batch deployments
* fix(router): skip a malformed model_info.supported_endpoints instead of failing group info and chat
* fix(router): apply the batch create filter after cooldown, managed files, and tags
The batch filter ran before health checks and cooldown, so a batch create on a mixed
Azure group answered with a router 429 or 401 when cooldown, tags, or a managed file's
targets left no batch deployment, where main sends the create on and returns Azure's
400. The filter now runs right before order filtering, after every eligibility filter,
so those cases answer as main does and creates still skip Azure chat deployments
whenever a batch deployment is left
* test(router): annotate the batch routing test assignments with Final
* fix(otel): put the error message on the failed span status
The legacy OTel logger (used by otel, langfuse_otel, arize and others) ended a failed call with Status(StatusCode.ERROR) and no description, so backends that read the span status, such as Langfuse's status message, showed an empty error. The text was only in the error.message attribute. Use the same message as that attribute, as the OTel v2 emitter already does.
* fix(otel): take the failure status message from _record_exception_on_span
Instead of reading the logging payload a second time in _handle_failure,
_record_exception_on_span returns the error message it already records
as error.message, and _handle_failure puts it on the ERROR status. The
other callers ignore the return value, so their spans are unchanged.
Its input is now a typed view of what it reads (exception,
error_information, error_str) instead of a bare dict.
* fix(otel): put the error message on every failed span status
Covers the proxy SERVER span, management endpoint spans, service spans, and Arize Phoenix spans, so no ERROR span ships with an empty status description
* fix(otel): cut the error status to the span attribute length limit and describe dangling server spans
The status description now uses the error text as stored on the span, so OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT applies to it too. A server span the proxy closes on a 4xx or 5xx outside the request path gets the same description, and an empty error message falls back to the error class
* test(otel): type the validation-handler status test
* test(otel): cover the Phoenix parent span and error_str length-limit status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(search): price Tavily and Linkup searches at their per-mode cost map entries
* test(search): read the reported cost through the public hidden_params property
* fix(search): keep the type gate flat on the cost map membership check
* test(search): drop vendor price guards and a redundant case, type helper params as mappings
* fix(search): keep a registered search model's own price, cover every interception endpoint, hostile modes and a failing burst
* fix(search): keep a runtime-registered base search price for modes that have no runtime price of their own
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content
Provider transforms encode returned video ids with an empty model_id, so
GET /v1/videos/{id} and /content cannot resolve the deployment (and its
per-model api_key) and fail with invalid_api_key.
Re-stamp the returned VideoObject id with the resolved model_id in
video_generation, video_status, video_remix, video_edit and
video_extension.
Fixes#33423
* fix(videos): return a copy instead of mutating the response, read hidden params via helper
* test(videos): fully type the new video id re-stamping tests
Annotate every parameter and return added by this PR (hidden_params,
harness, response, -> None), drop Any from the test helper, mark locals
Final, and narrow the endpoint/helper results with isinstance(VideoObject)
instead of reading .id off an untyped value. Route the mocked processor
response through a typed Harness.respond_with, read NotRequired
DecodedVideoId keys with .get, and name the rule and reason on the
_hidden_params suppressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): fill only an empty model in returned video ids
Keep an id that already names a model as is, so the asked-for name,
routing group and served group in it still drive access checks and
routing. Fill an empty model from the model the caller asked for
before the routed deployment id, and read the request model through a
typed view so the type gate does not grow
* fix(videos): route video calls to the deployment that created the video
Video ids now carry the id of the deployment that created them. Status,
content, remix, edit and extension calls go to that deployment when it
belongs to the requested group, so a group spread over two provider
accounts no longer polls the wrong one. A provider model shared by
several groups resolves to the creating deployment's group, and the
asked-for model is written into an id only when the router can route it.
* fix(videos): pin video calls only to deployments the caller may use
Pin a video id's creating deployment at deployment selection, and only when the caller may use that
deployment in the requested group, so aliases, routing groups and access groups still apply and the
spend label and fallbacks keep the requested group.
A status poll keeps the deployment its id names and takes the deployment that answered when the id
names none. An alias that routes to the creating deployment's group is kept on status and content.
* test(router): reset module fallback globals between unit tests
* fix(videos): check and route polls by the name the router serves
A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does
* fix(videos): pin polls only among the deployments routing allows
Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests
* fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s
* fix(videos): stamp the requested group only when one of its router fallbacks served the create
* fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router
* test(router): list the video pin helper with its sibling in the router coverage exemptions
* refactor(auth): walk chained router fallbacks without recursion
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions
stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt
Read chunk fields as typed attributes instead of untyped subscripts, which
clears the reportUnknownArgumentType breach the lint gate flagged
* fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): rebuild dict-form text completion chunks again
stream_chunk_builder routes dict chunks holding TextChoices to the text
completion builder, which now reads attributes. Convert them to
TextCompletionResponse at the routing point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path
Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage
* fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild
* fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions
* fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null
* fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path
Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`.
The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text`
next to `delta`, so streamed /v1/responses calls to Together ended without response.completed.
Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about
the same as rebuilding an n=1 stream of the same size.
* fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs
A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising
Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals
* fix(streaming): rebuild text completion choices without an index at index 0
Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base
* fix(streaming): rebuild text completion choices with a string index under that int
A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string
* test(streaming): drop a docstring that repeats its test
* fix(streaming): fall back to choice 0 for an infinite text completion index
A float infinity index made int() raise OverflowError, which the index
coercion did not catch, so the rebuild failed where the merge base
returned choice 0.
---------
Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover gzip and zstd request bodies on /v1/systemone
* fix(proxy): decode zstd and gzip request bodies before auth reads them
* fix(proxy): bound request body decompression and keep it inside CORS
Limit the decoded size by an expansion ratio and the operator's max_request_size_mb, pass bodies without the coding's magic number through, answer 415 when zstd is unavailable, and register the middleware innermost so errors carry CORS and security headers
* fix(proxy): budget decoded request bodies per worker and forward plugin bodies as sent
Each worker holds at most MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB of
decoded bodies and answers 503 past it. /plugin-proxy/ bodies reach the
plugin as sent, a zstd skippable frame counts as zstd, the decoded-size 413
names the decoded limit, and an integer max_request_size_mb stays an integer
in the 413 text.
* fix(proxy): read the magic number before size limits and reserve declared decoded sizes
* test(proxy): pin gzip mtime so parametrized test ids match across xdist workers
* fix(proxy): parse max_request_size_mb the same way in the auth size check
* fix(proxy): pass Claude Code OTLP export bodies through undecoded and skip the size parse without premium
* test(proxy): cover client disconnects, chunked bodies past the cap, and body-less request messages in the decompression middleware
* test(proxy): name the decompression test cases so CI logs stay small
