* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* test(cost): add Azure AI Claude Haiku 5.5 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): declare service tier cache price fields on CostMapEntry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): drop CostMapEntry reorder now declared on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): open faked stream items empty so tool arguments and text arrive once
* fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty
* fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once
* test(integration): cover the empty-open Responses stream items on every fake, replay and block path
Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL
* feat(providers): add Inference APIs as OpenAI-compatible provider
Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests
- provider_create_fields.json entry so the Add Model dropdown test passes
- provider_endpoints_support.json (+ backup) entry so the provider docs check passes
- tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations
- drop a redundant comment
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(providers): inject a recording HTTP session instead of patching httpx.Client
Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is
monkeypatched on a class.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* chore(providers): drop redundant comment on the inferenceapis entry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route
Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models.
Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries.
* feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported
* fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider
* fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError
* fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring
* fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only
Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families.
* test(inferenceapis): hold upstream replies across the worker kill in the chaos cell
A reply the killed worker sent before SIGKILL can lose its queued spend row, which the
product never promised to keep. Holding the upstream until after the kill means only the
surviving worker answers, so every 200 must log exactly once.
* test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies
The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown
---------
Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep provider headers on MidStreamFallbackError
A 429 raised while opening a stream is wrapped in MidStreamFallbackError,
which dropped the litellm_response_headers exception_type() attaches to
the wrapped error. retry-after and x-request-id reached callers on the
non-streaming path but not on the streaming one, so retry backoff,
provider request id logging and proxy error headers lost them.
* refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy
* test(streaming): cover sync and async header carry on MidStreamFallbackError
Parametrizes the provider-header test over the sync and async stream
wrappers, and drops the hasattr test, which pinned how the attribute is
stored rather than what any caller observes
* test(streaming): add integration cells for stream-open provider headers
Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the
eager providers), the Router stream retry and fallback paths, and the proxy on
both the router and the config-less serving paths, including a mixed burst, a
worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less
proxy owned by the test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): route Model Garden models through the generateContent adapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vertex_ai): annotate generateContent route test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): type litellm_params in get_vertex_ai_model_route
* fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions
Streamed Model Garden calls through the generateContent adapter now request
stream usage where the deployment supports it, keep cached-token details, and
look up the provider-relative cost key, so they bill like the non-stream call.
The adapter forwards router metadata, passes upstream errors through with their
status, countTokens falls back to local counting for non-Gemini routes, and the
native Gemini config is limited to the Gemini routes.
* fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing
Streamed generateContent through the chat adapter now ends with the upstream usage, honors
always_include_stream_usage false and the caller's stream_options, counts one failure per
failed call, and routes Vertex deployments by their base_model like chat does
* fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage
* test(google_genai): expect the trailing usage chunk on bridged OpenAI streams
The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone.
* fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk
* fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent
* test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire
* test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: align token counting with inference context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve Vertex Gemini count message shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: forward Google countTokens tools in shapes every provider counter accepts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: add package marker for Vertex count-token unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): ignore deployment system_instruction when counting tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(count_tokens): ruff format gemini count handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): return Mapping from count request body builders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): simplify Gemini count context and tests
Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default
Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see
Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table
* fix(count_tokens): reuse inference tool translation for Vertex Gemini count
Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped
Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep model capability flags out of provider requests
A supports_* flag or another model map field set on a deployment's litellm_params was forwarded
to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on
chat traffic and on the background health check alike. The shared owned-kwarg predicate now
treats every ProviderSpecificModelInfo field as litellm-owned.
Builds on the health-check-only approach in #40952 by leilei3167.
* fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params
* fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments
* fix(router): strip model capability flags from the router's deployments instead of every request
* fix(router): keep capability flags out of health probes and read them from the loaded catalog
* perf(router): read the catalog's supports_ field set once per cost map load
Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove an OpenAI-format box answer reaches the caller as a box
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the box wire test a deployment that advertises supports_box_questions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(images): preserve explicit provider for async generation
* test(images): prove a bare model name reaches its explicit provider on the async path
* fix(images): validate async provider hints instead of reading untyped kwargs
Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling
* fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription
* fix: keep the explicit provider for realtime and cover the audio bad-hint 400
* fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio
* fix: stop async image generation from rejecting a base_url it never reads
* fix(audio): skip base_url validation when api_base is set on async speech and transcription
* test(integration): cover a declared provider with a bare model on image, audio, and realtime routes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(vertex_ai): keep yield outside the JSON decode guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): surface BaseLLMException raised while streaming a request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(http_handler): only follow __cause__ when unwrapping streamed body errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): avoid recursive streamed error unwrapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows
Keep main's error path for a directly raised provider error and only unwrap
one found behind a transport error. Add the parser column to invalid JSON
errors, and cover the real aiohttp chain in the _handle_error test
* fix(vertex_ai): report every row conversion error as a 400 naming its line
A message with no role, a non-string role, a bad temperature, or a litellm
BadRequestError raised while converting a row still returned a 500, and the
async upload lost the 400 for litellm's own exceptions. The stream now turns
any error from one row's conversion into a 400 naming that line.
_handle_error only unwraps an httpx.TransportError, which is what the aiohttp
transport raises around a failing request body, so other callers keep their
own status. A file that starts with a UTF-8 byte order mark now uploads, and
provider messages start with 'Batch input file line N' like the proxy check.
* fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line
A chat row whose body is missing or not an object now raises a 400 naming
its line instead of uploading a blank prompt, and a non-object body on
line 1 no longer crashes the GCS object name lookup. Rows too deeply nested
or too long to parse return a 400 from both the SDK and the proxy's own
batch file check. JSON error columns now count leading whitespace.
* fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites
The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line
* test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos
* fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact
* test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex): report an unreadable credentials file as a read failure
Tell a path apart from inline JSON by shape, not os.path.exists().
Fixes#40166
* test(vertex): narrow pytest.raises to the expected message
* fix(vertex): log the credentials path instead of returning it to the caller
* refactor(vertex): log the credential source decision where it is made
* fix(vertex): handle paths open() rejects with a bare ValueError
* fix(vertex): redact the credentials path before it reaches the log
* test(vertex): cover the brace fallback and non-text credentials file
* test(vertex): exclude the unreachable exhaustiveness guard from coverage
* refactor(vertex): read the failure's fields off the matched object
CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.
* fix(vertex): accept a read-only mapping in extract_aws_params
* fix(vertex): keep encoded credentials out of the log and cover the legacy route
* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust
* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file
* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message
* test(vertex): integration cells for credentials file and inline value failures
* test(vertex): drive the Rust messages cell through Vertex Claude
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema
`translate_anthropic_tools_to_openai` assigned the caller's `input_schema`
straight into `function_chunk["parameters"]`, then merged the tool's remaining
top-level keys into that same dict. Because the assignment aliased rather than
copied, keys such as `display_width_px` on a computer tool were written back
into the caller's schema.
That matters because callers reuse the tool list across passes: the pass-through
guardrail translates the tools, and the request is then translated again for the
real call. The first pass pollutes the source and the second forwards a schema
carrying non-schema keys.
Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the
same file already copies its schema for this reason, so this brings the two into
line.
Rebased onto current litellm_internal_staging; the original branch had gone to
conflict on surrounding drift while the line it fixes is unchanged.
* fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects
* fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool
A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR
* fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools
Copying only input_schema left the extra keys merged into parameters, such as
input_examples, and OpenAI-format function tools shared with the caller, so
Gemini's strict strip still edited the caller's tool before a fallback to Claude
* fix(anthropic/adapter): keep translating tools that deepcopy cannot copy
A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before
* fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema
* test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback
* test(anthropic/adapter): close the connection after the malformed tools request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): cache the two-turn guardrail replays and derive the thinking pass-through timeout
* test(integration): keep the thinking pass-through timeout as on main
The derived timeout had no failing run behind it, so this PR now only fixes the guardrail cache cells
Add live-captured cache-read fixtures and align replay usage blocks with current provider responses
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id
* fix(responses): answer 400 to a tool_search execution that is not server or client
OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed
execution is dropped like the other malformed tool flags, which leaves a server search
* test(responses): cover the 400 and drop_params paths for a malformed tool_search execution
* fix(responses): map Anthropic tool search to OpenAI server tool search
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): sync generated tool search API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): type tool search mapping predicates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): validate tool-search mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search
* fix(responses): keep deferred chat tools beside a tool search on the responses bridge
Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the
supported floor) has no ToolSearchToolParam, and carry defer_loading from chat
function tools onto the Responses request when a tool search tool is present,
which OpenAI requires for tool_search
* fix(responses): leave the bridge's defer_loading carry to its own PR
#46439 owns keeping defer_loading on chat function tools sent through the
Responses bridge, so this PR keeps only the tool search mapping and the
openai 2.20.0 compatible tool type
* fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it
* fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host
* fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params
* perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent
* test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos
* test(integration): define the fallback cell's deployments in the gateway config so both workers load them
* test(integration): keep the worker-kill burst held past the replacement wait
* fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge
main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the chat bridge to map the search tool now that defer_loading survives
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(xai): report a completed xAI batch's results as its output file details
* fix(xai): keep a short results download and per-call timeouts out of the reported file details
* fix(xai): describe a batch's results file from the batch alone and let the batch job fill its size
Reading an xAI batch id as a file now costs one batch read and never downloads
the results: the file reports a 0 byte placeholder, the results filename and
the registration time. The managed proxy fills in the size the batch cost job
measures when it downloads the results, keeps a known size when a provider
reports none, and drops the per-process results download cache.
* fix(managed_files): keep a measured batch output size when a refresh races the batch job
A refresh that read the row before the batch cost job stored the measured size
could save the provider's unsized details over it. The refresh now reads the
row back from the writer and saves only while the row still holds what it read,
retrying when another write landed in between. The new tests drive the real
xAI and OpenAI retrieve paths through a fake HTTP upstream.
* fix(managed_files): guard the refresh save with the row the request already read
The conditional save first compares against the row the request read, so a
refresh adds no database read unless another write landed in between; only
then does it read the row back from the writer and try again.
* fix(managed_files): save a refreshed file object with one guarded write and no extra database reads
A refresh that loses its guarded write to a newer one now leaves that row
alone and drops this worker's cached copy, so the next read loads the newer
row instead of retrying against the database on the request path
* fix(managed_files): retry the batch cost job's measured size after a read refreshes the row first
The batch cost job re-reads the row from the writer and saves again when its
guarded write loses to a concurrent refresh, so its measured size is never
dropped. Reads without a measured size still make one write and no re-read
* fix(managed_files): size a concurrently refreshed row instead of restoring the batch job's placeholder copy
When the batch cost job loses its guarded write and the row it re-reads holds
refreshed details, it applies its measured size to those details, so the
results filename and any size the provider reported are kept
* fix(managed_files): keep a batch output's earlier created_at when a refresh replaces its placeholder
A refresh saved over a stored row now keeps the earlier of the two creation
times, so an xAI placeholder keeps the time the proxy registered it instead of
the read time, while a provider's own earlier creation time still wins
* test(integration): an xAI placeholder row keeps its registration time after the proxy refreshes it
* test(integration): drop the cached output row so the placeholder cell reads the seeded database row
* test(integration): drop the cached output row without requiring it to still exist
* fix(managed_files): limit the batch output size fill, guarded save and kept created_at to xAI rows
* fix(xai): keep the batch id as the batch output filename
Managed xAI batch output files keep the filename main already returns
(the batch id) instead of renaming them to <batch id>_results.jsonl.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at 9150723c84: 58 cells across the Claude routes, Converse, Databricks, the drop_params sources, replayed history on openai, azure, Converse and gemini, and a burst, an outage and a worker kill
* fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id
* fix(responses): answer 400 to a tool_search execution that is not server or client
OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed
execution is dropped like the other malformed tool flags, which leaves a server search
* test(responses): cover the 400 and drop_params paths for a malformed tool_search execution
* test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks
* fix(anthropic): map only a server or unset tool_search execution to Anthropic tool search, so a malformed value answers 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): judge the chat tool_search gate with a strict server check and keep the Responses bridge classification
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(responses): sort the tool_search __all__
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): give the defer_loading casts a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep tool_search items on a turn with no message or tool call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep every reasoning item on a search-only turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails
* fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses
Under drop_params, defer_loading is dropped when no tool_search tool is sent,
so the 400 it would cause upstream goes away as before. A non-bool value answers
400 without drop_params and is dropped with it. A null tool-level value falls
back to the nested function value. A request carrying a tool_search tool now
bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort
none, since chat completions rejects the tool_search type.
* fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy
Store the ordered output prefix through the last tool search item, reasoning included, so a
second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses
converter now replays tool search items and namespace only when the bridge opts in, so Noma,
the complexity router and guardrail payloads keep their shape. The tool policy guardrail and
the allowlist count replayed tools only when the request can send them.
* fix(responses bridge): forward defer_loading only beside a tool_search tool
defer_loading only means something with tool_search, and backends without it
reject the key (xAI answers 403). A true value with no tool_search tool now
answers 400 naming drop_params and is dropped under drop_params; a false value
there is a no-op and is dropped.
* fix(tool_loop): keep tool search items and namespace between rounds
* fix(responses bridge): check replayed tool search loads when a deployment adds tool_search
* fix(chat): drop litellm response fields before strict OpenAI-compatible providers
A chat reply from the Responses bridge carries provider_specific_fields on the
message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral
and Together reject unknown fields, so sending that reply back to them as
history failed with a 400 or 422. Strip them in the shared helper and use it
in those providers.
* fix(caching): keep reasoning_items and tool search items on response cache hits
A cached bridge response lost reasoning_items on a non-streaming hit and lost
both reasoning_items and provider_specific_fields (tool_search_items) on a
streamed hit, so the next turn could not replay the search or the reasoning
around it
* test(integration): cover tool search item replay on the chat to Responses bridge
Proxy, SDK, authorization and chaos cells for returning and replaying
tool_search_call and tool_search_output items and the tool call namespace
* test(integration): add response cache twins for the tool search guardrail cells
Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache.
* fix(responses bridge): forward defer_loading beside a client-executed tool_search
The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one
* test(responses bridge): pin the client tool_search turn and its replay, streamed and not
* fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search
Forwarding defer_loading beside a client tool_search made GPT stop with a
search call the chat client has to answer by hand, so a plain question that
used to get tool calls got none. Keep main's behavior of dropping it there.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Providers like Mistral put every parallel tool call, with full arguments, into one
chat-completions chunk. The /v1/messages adapter opened one tool_use block for the
first call and concatenated every call's arguments into it, so clients got invalid
JSON like {...}{...}. _CombinedChunkSplitter now splits such a chunk into one chunk
per call. The first call streams live and calls opened behind it are held, gathering
their fragments by index, until a non-fragment chunk arrives or the stream ends
Fixes#44029
* fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl
A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise.
* fix(router): give prompt cache pins their own tier and honour any declared ttl
Pins now live in a DualCache of their own instead of the router's 200-slot
in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows.
The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not
only the literal "1h".
* fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache
Compute the cache_control ttl scan through offload_token_count beside the
prefix hash, clear the dedicated pin cache in Router.flush_cache, and point
the order-fallback and priority-group pin tests at the pin cache they read
* fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots
* fix(router): match the 1h pin gate to what Bedrock forwards
Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock,
the same check that decides whether the 1h ttl reaches Bedrock, so an
application inference profile ARN keeps a 5 minute pin whatever its
base_model prices
* test(router): give the anthropic_messages pin fixture its provider
* fix(router): let each pin write set its own lifetime and require a positive 1h price
In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support.
* test(router): integration cells for the one-hour prompt cache pin lifetime
Cover every surface the pin lifetime touches: chat and /v1/messages through
both SDKs, the responses bridge, response-cache hits, each provider family's
1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed
and cross-group routing, concurrency, every Redis attach path, the in-memory
slot pressure without Redis reads, a Redis outage and a killed worker
* fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes
* test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap
* fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry
* fix(caching): keep the in-memory cache bounded when entries have no usable expiry
* test(caching): cover the response cache bound when cached entries have no usable ttl
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): follow #44788 and #46101 in the openrouter translation and cache mark census expectations
Expect prompt_cache_breakpoint for opted-in OpenAI and omit assistant tool-call marks from the capped census
Match OpenRouter refusal, logprobs, and system fingerprint nulls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): follow #46341 blank env and guardrail param handling in keepalive and scope expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read the first stream chunk through the stream's iterator so MCP streams keep working
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): follow #46493 retry and fallback semantics for streams that fail before content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the openrouter expectations #46510 already landed on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): restore blank lines lost in the main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id
* fix(responses): answer 400 to a tool_search execution that is not server or client
OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed
execution is dropped like the other malformed tool flags, which leaves a server search
* test(responses): cover the 400 and drop_params paths for a malformed tool_search execution
* test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add rich anthropic claude haiku 5.5, opus 5.5 and fable 5.1 translation cases
Capture-backed basic cases on /v1/chat/completions, /v1/messages and /v1/responses for claude-haiku-5-5, claude-opus-5-5 and claude-fable-5-1, reusing the shared client templates and ANTHROPIC_MESSAGES_REQUEST/RESPONSE with model-specific id, content and usage overrides. The minimal Opus 5.5 hello cases are replaced by the rich conversation. Sonnet 5.5 and legacy models are unchanged
* test(translation): rich OpenAI captures for gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(translation): rich Azure gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra cases
* test(translation): rich azure_ai Claude cases for haiku-5-5 and fable-5-1
Bind the shared chat_completions, messages and responses client templates and the anthropic provider templates to azure_ai/claude-haiku-5-5 and azure_ai/claude-fable-5-1 from real Azure AI Foundry captures, keeping the Azure dialect exact (/anthropic/v1/messages, x-api-key plus api-key on chat_completions and responses, system as a text block list on /v1/messages). Add prompt_cache_hit variants for both models from the second call inside the cache window. The two deployments need rows in tests/integration/proxy_config.yaml
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(translation): rich Gemini bases for gemini-3.8-flash, gemini-3.1-pro-preview and gemini-4-argon
Add the shared generateContent provider templates in templates/gemini.py from
real captures and bind the three target models on chat completions, messages
and responses with explicit per-model response overrides. gemini-4-argon is
registered in the three basic lists. gemini-3.5-flash is unchanged
* test(translation): enrich vertex_ai claude and gemini cases from live captures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): rich Bedrock converse, invoke and mantle translation cases from real captures
* test(integration): register the rich translation deployments and follow the rich Gemini 3.8 Flash cases
Add the synthetic deployments the rich provider cases bind (anthropic, openai, azure, azure_ai, gemini, vertex_ai, bedrock converse, invoke and mantle) to tests/integration/proxy_config.yaml, all pointed at the fake provider on 127.0.0.1:8191 with synthetic keys. The Gemini omitted-temperature contract now takes its expected generationConfig and reply text from the rich GEMINI_3_8_FLASH_TEST_CASE literals instead of the retired minimal hello cases
* test(translation): reuse bridged and native response templates
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent
Replaying an assistant turn that carried a redacted_thinking block to Bedrock
Converse turned it into an empty unsigned reasoningText, which the assistant
content builder then dropped as blank text. Bedrock returned 200 so callers
never noticed. Map redacted_thinking to redactedContent in both the
thinking_blocks and inline content paths of the sync and async builders
* fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay
* fix(bedrock): replay redacted_thinking to Converse only when the target is Claude
Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude
history replayed to them through a Router fallback or model switch keeps dropping the redacted block
the way main does. Claude is resolved from the Bedrock base model of the target, and an application
inference profile ARN counts as Claude when its deployment's base_model names a Claude model.
Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a
tool call, and for non-Claude and ARN targets.
* test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy
Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live
edge between the proxy and Bedrock checks the replayed Converse body carries the same
reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip
* fix(bedrock): validate litellm_params before reading the deployment base_model
litellm_params arrives as an untyped dict, so passing it straight into the typed
helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first
keeps the read typed without growing the type-check gate
* fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model
* test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path
* fix(bedrock): match bare claude- base_model names without reading the untyped cost map
* fix(bedrock): keep redacted_thinking data as Claude's own string on every route
Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data,
so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now
decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap
histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to
Claude on Converse, which rejected it with a 400
* fix(anthropic): keep tool_search_tool_result so the search call replays with its result
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): drop a client tool_result for a search call replayed with its server result
* fix(anthropic): replay a server tool call the client answered as the client sent it
* fix(anthropic): keep a server tool result under modify_params, count client-answered marks, and replay it from run_tool_loop
modify_params no longer adds a placeholder answer for a server tool call whose result is kept, unless the call ends the request. The cache control hook counts a mark on a server tool call the client answered, since that call goes out as a plain tool_use. run_tool_loop sends provider_specific_fields back and runs only the client's tool calls.
* fix(anthropic): keep a server tool result when another tool answer follows, scope run_tool_loop's replay, and keep code_interpreter_results off tool search turns
* test(anthropic): audit cells for tool search result round trips, replay, cache marks, pass-through logging and chaos
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): give the defer_loading casts a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses
Under drop_params, defer_loading is dropped when no tool_search tool is sent,
so the 400 it would cause upstream goes away as before. A non-bool value answers
400 without drop_params and is dropped with it. A null tool-level value falls
back to the nested function value. A request carrying a tool_search tool now
bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort
none, since chat completions rejects the tool_search type.
* fix(responses bridge): forward defer_loading only beside a tool_search tool
defer_loading only means something with tool_search, and backends without it
reject the key (xAI answers 403). A true value with no tool_search tool now
answers 400 naming drop_params and is dropped under drop_params; a false value
there is a no-op and is dropped.
* fix(responses bridge): drop defer_loading beside a client-executed tool_search
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Brings integration tests in line with intentional behavior changes from
#46331, #46291, #46341, #46355, #46368 and #46493, moves the Databricks
non-Claude cases off llama-3 names so they no longer download a Hugging Face
tokenizer, drops the unrealistic count_tokens tools=5 case and skips the
Mantle region-prefixed v1 base case as unrealistic input
* feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs
When the setting is on, spend log rows store requester_ip_address as null in
the column, the metadata and the stored request (which also drops client-IP
forwarding headers), and IP addresses in stored error messages are masked, so
the Logs page shows no caller IP. The setting is read the way the settings API
validates it. Capture is unchanged, so guardrails, logging callbacks and
metrics still receive the IP. Toggleable from the Admin UI General Settings.
Resolves LIT-7442
* fix(proxy): mask IPv6 addresses that end or start with :: across eight groups
The masking pattern allowed seven colons, so valid forms like 1:2:3:4:5:6:7::
and ::1:2:3:4:5:6:7 stayed in stored error text with
disable_requester_ip_address_logging on. Also lists the request metadata walker
in the recursion check, since it only walks the tree the spend log sanitizer
already returned.
* fix(proxy): mask IPs in every stored header value and after labels or underscores
With disable_requester_ip_address_logging on, client IPs carried in headers outside the forwarding list (do-connecting-ip, incap-client-ip, custom names) and addresses written as host:2001:db8::2, peer_2001:db8::2, socket_192.0.2.1 or an unbracketed IPv6 with a port were still stored in spend logs.
* fix(proxy): mask request header IPs before long values are truncated
A header value longer than the spend log string limit kept the part of an IP that fell before the truncation cut. Request headers in metadata and requester_metadata are now masked before the body is truncated, like error text already was. IPv4 followed by a dotted port (192.0.2.1.443) is masked again.
* fix(proxy): mask the IP piece a header truncation cut leaves instead of pre-masking headers
Masking header values before truncation scanned the whole client value,
including text the cut drops, and walked hook-made header cycles. Mask after
truncation instead, then mask the hex and digit run touching each side of the
truncation marker when it is short enough to be part of an IP address. Every
nested headers mapping gets the same treatment.
* docs(proxy): say the requester IP setting masks IPs in other stored header values
* fix(proxy): mask stored error text after truncation and harden the cut-piece pass
Error messages and tracebacks were regex-scanned in full before the DB cap,
which costs seconds on multi-megabyte provider errors. Mask them after the
cap like header values, with the same pass over the run beside each
truncation marker. That pass now handles every marker, skips long hex tokens
so an address beside one is still masked, keeps at most one address length,
and also masks colon-only IPv6 groups.
* fix(proxy): drop the requester IP from the originating request an auto router classifier row stores
* fix(proxy): drop the requester IP from the classifier input an auto router classifier row stores
* refactor(proxy): mask stored IPs with the content filter's ipv4 and ipv6 patterns
The requester IP setting now reuses the content filter guardrail's prebuilt
ipv4 and ipv6 patterns instead of its own address regexes, so spend logs mask
what that guardrail masks. Addresses glued to a label or underscore stay as
the content filter leaves them.
* fix(proxy): mask stored IPs with the spend-log matcher again
The content filter's ipv4 and ipv6 patterns left labelled IPv6 (host:2001:db8::2),
unbracketed IPv6 with a port and IPv4 after an underscore in stored error text and
header values, and masked parts of text like std::io::Error. Restore the
address-validating matcher and its regression cases.
This reverts commit 79537a8aa5.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(proxy): expose trace ID response header
* style(otel): format trace correlation middleware
* test(otel): cover websocket trace middleware
* fix(otel): disable correlation on setup failure
* refactor(otel): inject instrumentation setup
* fix(proxy): drop a relayed upstream traceresponse when the gateway sends its own
* fix(proxy): preserve upstream traceresponse without gateway injection
* fix(proxy): replace relayed streaming traceresponse
* test(otel): cover trace response fallback paths
* test(proxy): cover the streaming pass-through relay when the gateway propagator is off
* fix(proxy): keep a relayed upstream traceresponse and stamp the call id on the server span
The gateway's response propagator now leaves a response that already carries an upstream traceresponse untouched, so pass-through, file content and plugin proxy routes return one header and untraced pass-through routes keep the upstream's. The per-helper strips are gone. The propagator also sets litellm.call_id on the server span from the response's x-litellm-call-id.
* fix(proxy): add traceresponse only to sampled HTTP responses, tuple headers and unhandled 500s included
* fix(proxy): write only traceresponse and link a relayed header from a restarted upstream trace
* fix(proxy): send traceresponse only from the proxy app and expose it through CORS only with OTel V2
* test(proxy): check the CORS expose list contains the UI headers instead of pinning its identity
* test(integration): check traceresponse names the gateway server span across routes, errors, pass-through and chaos
* test(integration): read the cache-hit spend row by response id and assert only the traceresponse expose entry
---------
Co-authored-by: adityajoy-1902 <bittu19jain@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
