* test(nvidia_riva): audit #41021
Adds an owned Riva ASR gRPC server and provider integration cells for the riva_offline request-body ban, the offline and streaming paths, the per-deployment caller opt-in, and the 413 and 408 error mappings
* test(nvidia_riva): check word timing requests and audio samples in the Riva double
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* test(streaming): cover reasoning text and error frames that mention [DONE]
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(fallback-generalizations): remove the legacy extends and mixed-rule shim
Main's remote cost map, the bundled backup, and the root map all ship the split
routing/capability rule format, so the shim for the old format is dead code. A rule
mixing litellm_provider with other model_info keys is now warned about and skipped.
* test(fallback-generalizations): fail when a shipped cost map rule would be skipped at install
* test(fallback-generalizations): cover ignored extends and annotate the shipped-rules test
* fix(fallback-generalizations): warn about old-format rules with a pointer to a current cost map and skip extends rules
* fix(fallback-generalizations): tell local cost map users to replace the bundled backup in the old-format warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(router): cover silent_model shadow fan-out and streaming end to end
Proxy cells check that each silent_model target gets one request in the
caller's stream mode, that the caller sees only the primary, and that every
shadow lands its own spend row under its own model group. SDK cells check the
sync Router.completion thread path streams each shadow and fires its success
callback.
* test(router): flatten the spend row lookup with itertools.chain
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47358
Add integration cells for pass-through bodies and settings-endpoint mappings across workers
* test(proxy): wait for every worker to hold the saved mapping before probing
* test(proxy): compare passthrough bodies to the client body and probe each worker on its own connection
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Pin the queue route contracts #47498 changed: the priority header on streamed answers, the 400 for a request with no priority to queue at, a provider bad request keeping its own status, and a deployment-id request served ahead of the queue
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): avoid assistant prefill in Responses fallbacks
* fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations
A Responses fallback to a Claude model that takes assistant prefill now gets
the partial response as prefill again, so it continues mid-word without the
prompt-shape drift. Every other fallback keeps the request to continue.
A stream cut after text no longer continues when the request constrains its
output (a json_schema or json_object text format, or a forced tool_choice),
since a continued JSON document or tool call would not parse. The client gets
the primary's error, as on the chat path.
The fallback's re-sent leading space at the seam is dropped, and the prefill
shaping runs per deployment after the encrypted reasoning strip, so stripped
reasoning is never replayed.
* fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output
Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's
error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools
in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its
prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted.
* fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call
* fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls
A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main
* fix(responses): count a stream of tool arguments as a started tool call
* fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls
A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Covers the json_schema envelope edge cases #47478 fixed on the Anthropic json tool path: a key listed only in
`required`, a tool-named wrapper on a stream, non-ASCII and lone-surrogate answers, a stream cut before the tool
block closes (proxy and sync SDK, Anthropic and Bedrock Invoke), and a string `output_format` with and without
`drop_params` on Anthropic and Vertex chat and /v1/messages. The scripted Anthropic peer now refuses a non-object
`output_format` the way Anthropic does
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): let drop_params drop speed on claude-opus-4-7 after Anthropic removed its fast mode
* test(anthropic): cite the Claude Code fast-mode page for the Opus 4.7 removal date
* test(anthropic): cover the drop_params speed drop for claude-opus-4-7
* test(anthropic): type the model parameter of the speed drop test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic, bedrock): audit #47489 thinking replay across Claude routes and Bedrock Converse
* test(bedrock): compare the whole replayed Converse turn in the merged-message and cache point cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(files): audit #40621
Pin the raw cloud storage file id gate on /v1/files content and delete against a running proxy: twice-encoded gs:// ids stay blocked for non-admin keys, proxy admin keys read s3:// objects, and ordinary provider ids keep working
* test(files): check the full SigV4 signature on the admin raw s3 read
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params
* test(bedrock): cover the strict tools 400 retry under drop_params
* fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict
* test(bedrock): return captured requests from the strict tools test helper
* fix(bedrock): retry without strict only when a tool sent strict to Bedrock
A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list
* fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts
An application inference profile with base_model sends strict on the wire, so the retry check now resolves the
capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a
strict-grammar rejection too, so it takes the same single retry under drop_params.
* test(bedrock): annotate the strict-tools retry test helper's values as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A DeepSeek V4 stream that ends without a usage block now has a spend-row cell proving its prompt_tokens are counted with DeepSeek's tokenizer, the one path #47353's own cells did not reach
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): apply weighted-failover exclusion before deployment-affinity callback
* fix(router): let affinity pins yield to deployments that already failed this request
Moving the weighted failover exclusion ahead of the affinity check also changed what tag, order, plugin and managed-file filters saw. Restore the original filter order and have the affinity check ignore session and API key pins that point at a deployment already excluded or retry-skipped in this request. previous_response_id pins still win.
* fix(router): release affinity pins on deployments that failed this request
Delete a session or API key pin whose deployment already failed the request, so
the deployment that serves it claims the pin. Prompt caching pins skip failed
deployments too, and the proxy strips the internal failed-id fields from client
bodies.
* fix(router): keep affinity pins in place and skip failed origins in encrypted content affinity
A pin now only yields for the request whose pinned deployment failed. Deleting it moved pins after one 429 or a request-caused 400 and added a Redis call on the routing path. Encrypted content affinity also ignores deployments that failed this request, so a failed origin takes the strip-and-dispatch path.
* fix(router): keep encrypted reasoning when a retry can only go back to its origin
The encrypted content check now drops failed deployments the way the router does: a weighted failover exclusion always holds, and a retry skip that would leave nothing falls back to the whole group. The proxy also drops client-sent metadata._failover_excluded_ids, and the encrypted content tests use a real Router
* fix(router): keep encrypted reasoning when previous_response_id binds the failed origin
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47418
* test(proxy): hold the stream open past its first chunk and require cost headers in the audit cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): count an explicit allowed_fails equal to the default as set
Router(allowed_fails=3) stored the same value as an unset Router, so
_is_allowed_fails_set_on_router treated it as "not set" and kept the
rate-based cooldown logic. Keep the passed value in explicit_allowed_fails,
like _explicit_timeout, and count it as set.
* chore(router): drop the comment on explicit_allowed_fails
* fix(router): keep the fallback deployment out of cooldown when no deployment can be picked
When every deployment of the requested model is in cooldown, the router raises
"No deployments available" and schedules the failure callbacks on the request's
logging object. By the time they run, the fallback has already written its own
deployment into that object, so the 429 counted as a failure of the healthy
fallback deployment and cooled it down too, leaving the model group with nothing
to serve.
Tag the selection error and skip cooldown accounting for it, the same way
advisor orchestration failures are skipped.
* test(router): annotate the new cooldown test variables with Final
* test(router): pin that a saved allowed_fails of 3 stays unset and type the fallback helper
* test(router): use a fixed start time in the fallback helper
* test(router): cover a fallback group staying up after the primary's selection fails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add aiand provider
* feat: expose aiand provider in admin UI
* feat(aiand): add ai& model pricing to the cost map
* feat(aiand): list ai& models for aiand/* wildcards and mark /v1/messages supported
* feat(aiand): send Anthropic /v1/messages to ai& natively
* fix(aiand): drop the retired max_tokens field and regenerate endpoint support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds an integration cell proving a failed /v1/responses request counts its input from the newest compaction item on its failure spend row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
gpt-5-nano and gpt-5-nano-2025-08-07 carried gpt-5's priority input price, 50x the nano standard rate. OpenAI sells no priority tier for nano and answers a nano priority request with service_tier default, so proxy spend on real calls was already billed at the standard rate. The wrong price showed in /model/info, in the tier pricing a custom-priced nano deployment inherited, and in SDK cost estimates like litellm.cost_per_token with service_tier priority. The key is removed, and a new test keeps every priority rate between 1x and 5x of its standard rate in both cost map files
* test(proxy): cover the max_parallel_requests slot a streamed /v1/messages MCP auto-execute keeps
A streamed /v1/messages request that auto-executes MCP tools replays its answer
as a stream after its model rounds already logged success. The stream path frees
the request's max_parallel_requests slot only on a client disconnect, so after
message_stop the slot stays taken and the key's next request answers 429
The test streams such a replay to the end and expects the slot to be free. It
fails until the fix lands
* fix(proxy): free the max_parallel_requests slot when a replayed /v1/messages MCP stream completes
A streamed /v1/messages request that auto-executes MCP tools runs its model
rounds without streaming and replays the final answer as a stream. The first
round claims the request's only success log, and its limiter callback keeps the
slot because the tool call reused it afterwards. The stream path released the
slot only on a client disconnect, so after message_stop the key stayed at its
max_parallel_requests limit and its next request answered 429
Stream cleanup now releases the stashed slot when the stream completed and its
logging object already claimed the success log, which only happens when the
stream replays a response that was logged before it started. A real stream
still releases through its own success callback
* test(mcp): free the key's only parallel slot after a streamed auto-execute ends
Run a streamed MCP auto-execute on a key with max_parallel_requests 1 on
every LLM surface, then send the next request on that key and expect 200.
On main both /v1/messages surfaces answer 429 here, while /v1/responses and
/v1/chat/completions already pass
* test(proxy): keep only the MCP stream regression tests, since #48628 already frees the slot at stream end
* test(proxy): require the MCP stream regression test's response to stream and reach its end event
* test(gemini): audit #47542
Proxy integration cells for the Gemini and Vertex default output budget: the env budget reaches Google on chat and responses, turns on long decode, yields to the caller's own limit, caps per deployment, skips resumes, and a camelCase two-candidate request is never continued
* test(gemini): annotate the budget wire module's pytestmark as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Checks that a SageMaker stream failing after its first chunks reaches /v1/chat/completions, /v1/messages and /v1/responses callers as an error with no success end event, and that each writes a failed spend log row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Check that cancelling a LocalSandbox run stops the grandchild process it started, using a real child process
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): reject or drop Responses params the chat completions bridge cannot translate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep additional_drop_params working for Responses fields the bridge cannot translate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Update litellm/responses/main.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* style(responses): drop the blank line ruff format rejects after the dataclass header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(responses): skip failed spend-log turns when rebuilding a session from previous_response_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): guard the failed-turn status read and annotate the regression test locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Fixes#46977: Cohere chat API reference expects stop_sequences to be a list of strings. Convert string stop parameter to a single-element list for CohereChatConfig and CohereV2ChatConfig while preserving lists.
Co-authored-by: Praising Harris Ratnam <praisingharris@gmail.com>
* fix(langfuse_otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): name trace metadata identity keys after their request metadata fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): type metadata helper and drop null fields so they cannot clobber caller trace metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): annotate new request metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): build rig config and wire batches immutably
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): skip empty trace identity values so teamless requests cannot erase attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): drain wire batches per poll instead of accumulating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): cover null stripping and empty trace identities
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): set redaction flag directly instead of monkeypatching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): add integration audit matrix for langfuse metadata keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): tighten audit chaos and unauthenticated cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): restore contracts.json formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): close SDK clients, pin worker identity, and normalize response id matching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): gate the slow sink reply instead of sleeping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): emit request metadata and identity under langfuse.observation.metadata and langfuse.trace.metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): satisfy type discipline gate for metadata export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop prohibited covers markers from langfuse otel integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): put the mutable-ok suppression on the dict construction line the type-discipline gate reports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse otel): share request metadata export across callbacks
* fix(langfuse otel): drop prompt variables, MCP tool I/O and vector store queries from exported request metadata
* fix(langfuse_otel): keep span export alive when nested request metadata has non-string keys
The content strip validated prompt_management_metadata, mcp_tool_call_metadata
and vector_store_request_metadata items as str-keyed mappings, so an SDK
caller passing one with a non-string key raised inside
exported_request_metadata: OTEL v2 dropped the LLM span for every vendor and
v1 langfuse_otel lost the request metadata and every attribute after it.
Validate those nested mappings with object keys instead. Also hoists the
langfuse_otel imports.
* fix(langfuse_otel): send the request metadata blob to Langfuse destinations only
- A requester_metadata that is not a mapping no longer raises. The v2 LLM span
(every vendor) and the v1 Langfuse attributes survive it again.
- Prompt-quoting routing_decision fields stay out of langfuse.observation.metadata.
- An exporter or key/team destination for any other backend drops
langfuse.observation.metadata and langfuse.trace.metadata.user_api_key_*, as on
main. This covers a collector folded into the Langfuse logger.
- The v2 Langfuse key-set pin in the arize family test gains the two new keys.
* fix(langfuse_otel): export only caller identity under the Langfuse metadata keys
- The Langfuse metadata keys carried the whole request metadata. On OTel v2 that
sent request headers, caller metadata, user email, user agent and auth metadata
to a Langfuse destination that never received them on main; on v1 it doubled
each span's metadata payload, so big caller metadata doubled the export size
- Both paths now export only the key alias, user, end user, team, team alias and
org. On v2 that is the team plus the identity among the operator's
`baggage_metadata_keys`, which v2 spans already carry as `litellm.metadata.*`.
Everything else stays in the bare `metadata` attribute, as on main
- `output_messages` returns a list again, and the arize v2 b3 key pin is back to
main's set
* fix(langfuse_otel): drop the caller identity first when a span hits its attribute limit
v1 wrote the identity attributes after the OpenInference ones, so a span near the
128-attribute limit (about 33+ tools) evicted openinference.span.kind, metadata
and llm.model_name instead. The identity now goes first on a fresh span, so the
SDK evicts it before anything else, and a caller's active span that already
carries attributes does not get it.
v2 sheds the Langfuse identity keys before any indexed message attribute when a
span is over budget, so a span keeps every attribute it carried without them.
* fix(langfuse_otel): keep other spans and dropped counts as main exports them
- v1 writes the caller identity only into the room the span's attribute
limit leaves next to the error attributes a failed call adds, so the
exported dropped-attribute count matches main
- v2 strips the Langfuse request metadata only from LiteLLM's own spans;
application, instrumentation and server spans keep every key they set
- a stripped v2 span keeps the dropped attribute, event and link counts
- a config that lists the Langfuse mapper without the langfuse_otel
exporter also keeps the request metadata away from other exporters
* fix(langfuse_otel): keep the caller identity off the caller's active span
v1 writes LiteLLM's attributes onto the caller's active span when one is
current instead of starting its own span. That span goes to the caller's
own exporter, so the identity keys reached a non-Langfuse collector and
could take attribute slots the caller fills later. The identity keys now
go only on the span LiteLLM starts, whatever the caller's span carries.
* fix(langfuse otel): keep the identity keys on the OTEL v2 Langfuse mapper only
The v1 langfuse_otel callback goes back to main: it sends no
langfuse.observation.metadata or langfuse.trace.metadata.user_api_key_*
keys. The OTEL v2 Langfuse mapper keeps them. The integration cells now
run every endpoint, client, redaction, failure and chaos case on v2.
* test(langfuse otel): count only accepted batches in the sink outage cell
The wire also records the batches the sink answered 503, so an exporter
retry made a span look exported twice. The chaos cells now also check the
spans that landed carry the caller identity.
* test(langfuse otel): split stacked span comprehensions and drop constant comments
* test(langfuse otel): split the metadata integration module into focused files
* fix(langfuse otel): send the org id in trace metadata too
The trace metadata keys now reuse IDENTITY_METADATA_FIELDS, so a key in an org gets langfuse.trace.metadata.user_api_key_org_id next to the generation's org id.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search
Tavily search dropped every field except title, url and content, so
include_raw_content, include_answer, include_images and include_favicon
reached Tavily but their output never reached the caller. Each result now
carries raw_content, score and favicon when Tavily returned them, and the
top-level fields (answer, images, request_id, usage, ...) come back as
provider_metadata, same as the Exa fix in #47423
* test(tavily): move the Tavily search wire test's sample data to module constants, like the Exa test
* feat(passthrough): let upstreams report the model name via x-litellm-upstream-model
A pass-through upstream that prices its own requests already reports cost and
tokens in x-litellm-response-cost / x-litellm-total-tokens. Product endpoints
that take no `model` in the body still landed under `unknown`, so their spend
could not be broken down.
When the upstream sends x-litellm-upstream-model, record its value as the
request's model on success and failure rows, overriding any body `model`. The
name is a label only: it never marks the request as upstream-priced and is not
used to look up a price. A blank or missing header changes nothing.
Refs LIT-9910
* fix(passthrough): keep the upstream-reported model name bounded
The upstream model name becomes a spend row, a daily rollup key and a metric
label. Use only the first element of a repeated header (RFC 9110 field
combining, which is how httpx joins it) and cut the name to
MAX_SPEND_LOG_MODEL_NAME_LENGTH, the limit the spend log already applies to
model names.
Refs LIT-9910
* test(passthrough): cover x-litellm-upstream-model through spend rows and rollups
Drives a config-defined pass-through against a scripted upstream and reads
back the spend row, the daily team rollup and /team/daily/activity. Covers the
header winning over a body model on buffered and streamed responses, failure
rows, a model-only header keeping cost_per_request, the no-header and blank
header cases, and repeated or long header values.
Refs LIT-9910
* test(passthrough): annotate new test locals with Final
Refs LIT-9910
* test(integration): update stale expectations to intended provider behavior
Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request
* test(e2e-ui): type the guardrail name before deleting it
GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.
* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body
OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01
#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400
* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test
The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated
* test(guardrails): count the Responses disconnect spend rows #47611 added
Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent
The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes
* test(observability): size K4's timeout from the harness boot budget
K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
* fix(guardrails): redact secrets in tool traffic and Anthropic system
* fix(guardrails): bound tool payload scans and stop nested tool_result recursion
Scan each tool value beside its key instead of an indented dump, read
tool_result content one level deep, keep non-finite tool arguments as
raw text, and give read-only guardrails one fragment per tool payload
* test(guardrails): cover read-only guardrail consumers of tool and system text
* fix(guardrails): keep tool argument parsing, scanning and tests within bounds
Treat tool arguments that decode to a lone surrogate as raw text, decode
JSON-escaped detections back to the leaf value, quote unkeyed leaves
without repeating a parent key per list item, drop cycle tracking that
parsed JSON cannot reach, and use the CI license instead of a fake
license server
* fix(guardrails): scan list items beside a capped parent key
List items keep their parent key in the scanned JSON line, capped to its
last 64 characters, so a credential key over a list is still detected
while a long key over many items no longer multiplies the scan text
* fix(guardrails): cap the scanned key after JSON escaping
A key of control characters escaped to six times its length after the
cap, so every list item's scan line grew with it
* fix(guardrails): collapse whitespace in the scanned key context
Space-heavy keys made every inherited list item scan several times slower
than a short key
* fix(guardrails): build each key's scan context once per payload
Repeating the collapse, escape and cut for every list item made a long key
over many items cost key length times item count
* fix(guardrails): cover Responses tool call items and instructions
Responses carries the same tool call arguments and system prompt as chat
and messages under function_call.arguments, custom_tool_call.input and
instructions, so secrets there reached the provider unscanned
* perf(guardrails): skip rescanning text hide-secrets already found clean
Agents resend their whole conversation on every turn, so each turn
rescanned every earlier tool result. Each guardrail now keeps a bounded
set of digests of scan inputs that held no secret and skips those; text
with a secret is always rescanned and no secret material is kept
* fix(guardrails): size the clean-scan cache for many sessions and replace longest secrets first
4096 digests held about 20 long agent sessions per worker, so interleaved
sessions evicted each other and every resend rescanned its history;
65536 digests (about 10 MB) hold a few hundred. Tool payload secrets are
now replaced longest first after JSON unescaping, so a shorter overlapping
match cannot leave part of a secret behind. The key-context test checks
the work it bounds instead of wall-clock time, which CI could not meet
* fix(guardrails): keep unmasked tool argument text and redact keyword matches cut inside an escape
* fix(guardrails): write a tool visitor's whole payload when it changes more than string values
* fix(guardrails): read message text before tool and system text so a vendor error on the new text cannot skip it
* fix(guardrails): moderate tool and system text in its own call so a long payload cannot hide a flagged message
* fix(guardrails): skip the empty moderation call when only tool or system text is present
* fix(guardrails): scan long multi-line tool strings as their own lines so the scan stays linear
* fix(guardrails): keep the key beside every line of a long multi-line tool string
* fix(guardrails): confine tool-traffic coverage to hide-secrets
Restore iter_message_text to its main behavior, so banned keywords, Azure
Content Safety, IBM detectors and the OpenAI and Google moderation hooks
read exactly what they read on main. Only walk_user_text, which
hide-secrets alone calls, still reaches system, instructions and tool
payloads.
* test(guardrails): stop the tool-traffic integration tests from flaking on a busy host
Use the shared per-case timeout for the rig client, and open a new
connection per call in the worker-kill burst so a pooled keep-alive
socket to the killed worker is never reused.
* test(guardrails): drop the wall-clock check from the deep tool arguments unit test
The integration test test_deeply_nested_tool_arguments_do_not_stall_the_worker
already covers the timing; the unit test keeps the redaction check.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(morph): support open-weight chat models and cached input pricing
* chore(morph): preserve existing catalog number formatting
* test(morph): include provider regressions in CI shard
* fix(morph): satisfy catalog and test package checks
* fix(morph): declare abliterated model output limit
* fix(cost): a provider slug the price map lacks no longer bills the turn at zero
Anthropic names the dated build in `message_start`, so a streamed turn carries
`provider_response_model="claude-opus-5-20250930"` while the price map holds
`claude-opus-5`. `_select_model_name_for_cost_calc` prefers the reported name
over `response.model`, and `get_model_info` answers a model it has never heard
of with zero rates rather than raising -- so the turn is billed 0.0 with nothing
in the logs to explain it.
Same usage, only that field varying, on main:
provider_response_model absent -> 0.00079
provider_response_model=claude-opus-5 -> 0.00079
provider_response_model=claude-opus-5-20250930 -> 0.0
Missing is harmless; the damage is a name that is present and unpriced.
The reported name still wins whenever it resolves to a rate -- it is the most
specific truth about what served the turn. It is only passed over when charging
by it would produce a silent zero, and then the response's own model is used:
the name the deployment resolved to, and the one an unstreamed turn is already
priced by.
This is not specific to Anthropic or to streaming. Any provider reporting a more
specific slug than the map carries -- dated, regional, build-suffixed -- is
affected, and only streamed turns carry the field, which is why it reads as a
streaming bug from the outside.
Reported in #42161.
* address review: drop explanatory comments, stop pinning vendor slugs in tests
Three points from the review, two of which were right.
The helper docstrings explained behaviour the code already shows, which AGENTS.md
rules out. Removed.
The tests asserted that `claude-opus-5` is in the price map and
`claude-opus-5-20250930` is not. Both are facts we do not own: if Anthropic registers
the dated build, the test breaks without any litellm change. They now use a name that
is ours, and assert the invariant instead, that what the provider reports cannot change
the price of a turn the map cannot quote it by.
Picking that name needed care. An arbitrary string makes `get_model_info` raise, which
a caller upstream already handles, so the test passed with or without the fix and
defended nothing. A `claude-` prefixed name reaches the zero-rate fallback that this
bug rides on, so `claude-test-42161` reproduces it and stays ours.
The third point, that lines 801 and 828 exceed 120 characters, does not hold: they are
98 and 39.
* Pass completion_response_model as str or None to _priced_provider_response_model
basedpyright counted the bare variable as partially unknown (reportUnknownArgumentType +1 over the budget).
* test(cost): isolate runtime registration replay cases
---------
Co-authored-by: Eduardo Pessin <eduardopessin@users.noreply.github.com>
