Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys
Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models
A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays.
* feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog
A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it
* fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window
* test(proxy): stub the max_input_tokens group field and type the Codex window tests
* fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates
Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat
template does, and add a per-deployment token_count_tool_framing next to
token_count_framing for the hidden tool preamble a host bills. The Azure AI
Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry
the measured framing, overridable in model_info
* fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing
* fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice
* fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing
* fix(token_counter): count an explicit null strict like false on Azure Responses
* fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments
Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302
* fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts
* fix(proxy): frame Responses counts for the api_base router defaults send inference to
* fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): resolve Invoke Claude default max_tokens from the base model id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): build Invoke Claude request params without mutating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): type tool-search normalizer return to clear unknown-type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row
* fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig
Read the called deployment's model_info.max_output_tokens before the base model's
cost row, skip the shared regional key the router fills from every deployment, and
leave the default to litellm.AnthropicConfig(max_tokens) when it is set.
* fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat
* fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap
* test(bedrock): annotate the new Mantle default max_tokens tests with Final
* refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): join plain text blocks for non-Claude json_object requests
* fix(databricks): join only the last message for json_object and drop cache_control under drop_params
Databricks checks for "json" only in the last message, so earlier turns now go out as sent and only the
last message, plus any system messages merged into it, is joined into a string. Under drop_params a
text block's cache_control is dropped so that message can be joined too, and without it the request
still goes out as a list.
* fix(databricks): skip dropped assistant turns and blank parts when joining for json_object
The json_object join picked the last message before the request transform drops bare assistant turns,
so a trailing thinking-only or reasoning-only turn (a Responses input ending in a reasoning item included)
left the real last user message as a list and Databricks still returned its 400. The pick now skips the
turns that transform drops.
The join also ran when only one part had text, which sent whitespace-only parts the existing empty block
filter used to drop on requests that already worked. It now joins only two or more non-blank parts, so a
request with one non-blank part goes out exactly as before. Without drop_params, a message with a
message-level cache_control now goes out as sent instead of being rewritten into one cached block.
* fix(databricks): join json_object text parts only on models Databricks hosts
Databricks checks for the word "json" only on the models it hosts
(databricks-* pay per token names and system.ai.* AI Gateway names). An
external model endpoint skips that check and takes a list of text parts
today, so its request now goes out as sent instead of joined.
The drop_params read now goes through drops_unsupported_params, and a
test pins that a text block with a truthy non-string text is kept.
* fix(databricks): reject a text part holding a non-string locally with a clear 400
* refactor(databricks): reject a non-string text part at the transform_request boundary
* fix(databricks): read pydantic message objects in the non-string text check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions
* fix(openai): leave a non-object content part for OpenAI to reject under drop_params
* fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params
* fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params
Azure OpenAI chat now gets the same content part trim as OpenAI chat, and
the Azure and Foundry gates also read a deployment's declared base_model.
Each Responses config keeps only the ttl its API takes: Azure keeps 30m,
and Mantle keeps no ttl. A trimmed ttl is logged at debug.
* fix(responses): trim prompt_cache_breakpoint only on APIs known to read it
Other Responses providers get the marker whole or not at all, as on main.
Azure chat now trims for a deployment with no base_model unless the cost
map marks the model as not reading breakpoints, and Azure AI Foundry
Responses keeps Azure's 30m-only shape.
* fix(azure): trim prompt cache breakpoints only for deployments known to read them
Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params
* fix(providers): keep main's marker handling where the target won't trim it
A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6.
* fix(providers): read a non-version hyphen suffix like -32k as part of the model name
* fix(providers): keep main's reading of a dotted minor version like gpt-5.6o
* fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Scope the Prisma migration lock test's advisory lock check to its own database, so a lock another xdist worker holds in its database no longer trips it
Keep in-process MCP peers' idle connections open for 30s instead of 1s, so the gateway's pooled HTTP client doesn't reuse a connection the peer is closing and hang the tool call
Delete database guardrails in the logging-only scope tests through DELETE /guardrails/{id} on the shared gateway instead of raw SQL, so a default-on guardrail stops applying to the next test on the same worker right away
Bound the wedged coordination Redis /customer/update check by the shortest Redis wait the proxy uses, like the burst check, so a slow runner can't fail it while a handler that waits on Redis still does
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept
* test(e2e): send the azure_ai beta check without the code execution tool
* test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta
* fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept
* test(e2e): send Vertex's header-only betas in the beta acceptance case
* fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them
A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two
computer-use betas for the computer tool, so dropping both broke it. The e2e
acceptance test leaves those two out of its all-at-once Vertex call, and the
integration and unit tests that relied on a beta being dropped now use one
the platform still lacks.
* test(anthropic): point unsupported-beta tests at betas the platforms still drop
The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the
Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it,
so both now use values that stay dropped or unknown.
* fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool
The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the
provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now
forwards on anthropic and azure_ai
* fix(anthropic): send no computer use beta for an unknown computer tool version
An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included
* test(azure_ai): read the branch's beta map in the auto-injected beta wire test
* fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool
* test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep encrypted reasoning on a hop between same-key deployments on the provider's default endpoint
* test(router): type the default-endpoint boundary test helpers
* fix(guardrails): restore the google_text_moderation embeddings fix an earlier main merge dropped
* fix(router): match encryption boundaries on the key, account, and region the request really uses
A deployment's named credential now fills only the values the deployment leaves out, through the
same helper the request path uses, so the boundary names the key and base the request sends.
A deployment with no api_key takes the provider's default key, the boundary carries the OpenAI
organization and project a deployment sends, and a Bedrock Mantle region named in the model
routes through the provider resolver before the Responses URL builder
* fix(router): keep header-signed accounts apart on a shared boundary and skip boundary work no reasoning needs
Headers the request logger masks as secrets (Authorization, api-key, x-api-key) now join the
boundary's account scope, with extra_headers overriding headers the way the request merges them,
so deployments that sign in through a header on one base never share encrypted reasoning.
Target boundaries are built only when a reasoning item from another deployment needs the
comparison, and the Responses input strip asks only about encrypted reasoning items, so a pinned
turn or fallback hop with nothing to compare makes no secret-manager reads.
* fix(router): resolve default boundaries only for openai and drop the unsent organization param
Aggregators and Bedrock Mantle serve several vendors on one key, so a missing key or base there no longer yields a shared boundary. The Responses request never sends the organization param, so it no longer splits the boundary, while a Cookie header now does.
* fix(router): match explicit pairs as on main and key default boundaries by model
An explicit api_base plus api_key pair is the whole boundary again, so tag
headers no longer split deployments that main kept together. The default
OpenAI boundary now carries the model, since one OPENAI_BASE_URL can front
several vendors, and a deployment that declares another provider gets no
boundary before any secret lookup.
* fix(router): match an explicit key and the environment key on one api_base
A deployment with an explicit api_base and api_key now shares a boundary with one on the same api_base that takes the same key from OPENAI_API_KEY, when the model and account headers also match. Two explicit pairs still match on base and key alone, as on main.
* fix(router): skip a deployment whose params cannot be read in boundary checks
A named credential with a value of the wrong type, or an extra header whose name is not a string, made the boundary check raise and fail the whole request. That deployment now has no boundary, so the others still keep the reasoning they can decrypt. Two explicit api_base plus api_key pairs also compare without resolving the provider, so they read no secrets, as on main.
* fix(router): ask a secret manager for default boundaries at most once a minute per deployment
* fix(router): resolve a changed named credential at once in the cached boundary and take an injected clock
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Deletes the A2A e2e test that registered a third-party agent, which depended on a site the project does not control, along with its helpers and its coverage registry row. Adds an integration test where a scripted A2A agent receives a skill call as a data part and returns a data artifact, asserting the agent and the caller each get theirs unchanged
* test: move Sail and Weave e2e tests to integration tests
The Sail service_tier and completion_window cases now run against the scripted upstream in tests/integration/pricing, asserting the outbound body and the billed rates for chat, responses and messages
The key-scoped Weave delivery cases now run against an OTLP collector in tests/integration/observability. The case where only the otel callback is configured is skipped with a BUG reason
Removes the e2e copies, the Weave read-back client, the Sail registry rows, the Weave docs paragraph and the unused devin MCP server entry from the stage mirror config
* test: move Presidio masking e2e tests to integration tests
Moving them to integration tests. The pre_call, post_call and default scope cases now run against a mock Presidio analyzer and anonymizer that follow the real wire contract, and a scripted model upstream, on chat completions, messages and responses, with and without streaming
The spend log cases check the stored response and the guardrail record with a proxy that stores prompts in spend logs. The e2e copy, its Presidio guardrail params body, its registry rows and the doc paragraphs naming the two PRESIDIO variables are removed
* fix(proxy): support reload with multiple workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop the reload worker pool on a broken edit instead of respawning it
Workers now load the app through a factory that exits with uvicorn's
STARTUP_FAILURE code on any import error, so the pool stops and the
reloader waits for the next save, the same as single-worker reload.
Multi-worker reload needs uvicorn>=0.50.0 for that, so older versions
keep the single-worker reload with a warning.
* test(integration): stop the reload test's proxy through the harness shutdown helper
Main's TQ010 rule rejects a raw .terminate() under tests/integration, so the multi-worker reload test now calls OwnedProxy.begin_shutdown(), which sends the same SIGTERM
* test(proxy): cover the reload worker pool end to end and edit the config the proxy loads
* docs(agents): scope the two-worker reload QA rule to uvicorn proxies that use Redis
* fix(proxy): keep the original traceback when a reloaded worker fails to import the app
* docs(agents): point both router and coordination Redis at the QA Redis and cover single-worker trees
* fix(proxy): stop the reload worker pool when the reloader exits and give reloads a longer worker healthcheck
Closing the terminal or pressing Ctrl-\ killed the reloader and left the pool parent holding the port while it respawned workers that could not start. The pool now stops once its parent reloader is gone. The reload pool also defaults the worker healthcheck to 30 seconds unless --timeout_worker_healthcheck is set, since uvicorn's 5 second default killed workers that were still booting on a reload.
* fix(proxy): watch the reloader through its process sentinel so the pool also stops on Windows
* fix(proxy): carry the graceful shutdown timeout into the reload worker pool
* fix(proxy): stop the reload worker pool on Ctrl-\ instead of waiting out a dying worker's healthcheck
* test(proxy): skip the startup-failure reload tests on uvicorn without a startup failure code
* docs(agents): keep the QA proxy rule to two workers, Redis, and reload
* fix(proxy): keep Windows on single-worker reload and pass unlisted uvicorn args to the reload pool
* fix(proxy): stop reload workers when their pool parent dies
* test(proxy): move the reload pool process tests to the integration suite
The unit test guide keeps tests/unit in-process, so the subprocess pool cases now live in tests/integration/configuration and use the harness port allocator and process helpers
* docs(agents): keep main's proof rules and inline the multi-worker QA command
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): exit reload workers as startup failures when the app factory raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): check the reload pool serves the overridden worker app
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ovalix): file and tool scanning, app discovery from the key alias
Mirror of BerriAI/litellm#34132 at 5ffc268585, rebuilt on main.
* fix(ovalix): keep inspecting requests whose key has no user
The tracker allows a checkpoint call with an empty actor without inspecting
it, so a key with no user (or a /v1/messages or /v1/responses request that
carries its own metadata object) got no Ovalix enforcement. Fall back to
"unknown" as the guardrail did before, so every request is inspected.
* refactor(ovalix): check texts without mutation and test discovery through tracker requests
- _check_texts builds its result from per-text verdicts checked newest first by _check_text
- discovery and routing cache unit tests answer the tracker routes instead of replacing private methods
- integration test adds streaming cells and a tracker outage during a concurrent burst
* fix(ovalix): resolve tool result names in one pass
Each tool result rescanned every earlier message for its call id, so a request with thousands of
tool results spent seconds in the event loop. Declarations are now indexed once per request and
each result takes the nearest earlier one by bisection
* test(ovalix): annotate new test bindings and build the burst plan with product
* fix(ovalix): drop app discovery from the key alias
A config with no application_id no longer boots in discovery mode, where
the [Name] prefix of the key alias picked the Ovalix application and so its
policy. Anyone who could rename a key could move it to a weaker app, or to
an unknown name the tracker created with its default policy. The guardrail
now needs application_id at startup again, as on main, and every checkpoint
call goes by the configured application and checkpoint ids.
Removes enable_routing_cache and fail_if_no_application, the
app_name_regex and resolve_application tracker calls, and the per-alias
routing cache. Restores main's startup check that a requested pre_call or
post_call hook has its checkpoint id.
---------
Co-authored-by: Shalom Jamil <shalom@ovalix.io>
* test(agents): cover empty search embedding model settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(agents): treat an empty search embedding model as not configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): sort search integration imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): format search embedding tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(agents): annotate search test locals and isolate upstream call checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): expect JSON-RPC errors for proxy refusals on /a2a POST routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): build JSON-RPC refusal data from typed fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): stamp the OTel server span on JSON-RPC refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover JSON-RPC refusal route in unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): return the server tool call id instead of a TypeGuard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Gemini cache write carries the system prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(watsonx): read passthrough credentials from use_in_pass_through deployments
/watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever
worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide
watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env
fallback when none is flagged.
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(watsonx): keep env key precedence when no deployment is flagged
Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to
WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
* fix(streaming): fail on clean EOF without finish reason
* fix(streaming): cover shared OpenAI-compatible EOF paths
* test(router): complete mocked streaming responses
* fix(streaming): keep the EOF error message and cover Anthropic clean EOF
The request-less RemoteProtocolError tripped exception mapping into
'The .request property has not been set.' for non-OpenAI providers.
Adds Anthropic and generic-chunk unit tests, an integration test across
chat, the /v1/messages bridge and the /v1/responses bridge, and moves
main's token-estimate test onto a finished stream with a sibling that
checks the cut stream still bills the estimate on its failure log.
* fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole
Gemini marks every chunk unfinished and a later usage chunk reset
intermittent_finish_reason, so a finished Gemini stream failed at EOF.
LangGraph's __aiter__ opened a new aiter_lines on every wrapper call,
which hit StreamConsumed after the first chunk and now surfaced as a
false EOF error instead of a silent truncation.
* fix(streaming): keep the synthetic stop for a stream that sent no chunks
Gemini's OpenAI-compatible endpoint answers with only [DONE] when
reasoning uses up max_tokens. Failing every openai/azure stream that
sent nothing turned that 200 into a 500, while every other provider
already kept the stop for an empty stream. Adds integration cells for
Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked
deployment and the [DONE]-only answer.
* test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream
An empty stream writes no spend row on main either, so the [DONE]-only
cell checks only what the client sees.
* fix(streaming): read the chunk's finish reason for the sticky flag
* fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message
* test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names
* fix(agentcore): say why the end-of-stream flags rebind
* test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy
* test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream
* test(agentcore): inject the async transport instead of patching the aiohttp flag
* test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does
* test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server
* test(streaming): keep each stream's ending and land every spend row once through a spend log outage
* test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst
* test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop
* test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres
* test(streaming): type the new stream EOF unit tests
* test(streaming): a merged reasoning stream cut mid reasoning now fails
Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content
---------
Co-authored-by: King Star <mcxin.y@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): expect the kyma reported-cost headers to follow the cost map rate split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): keep the kyma test file at its original formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price the response-cost header split from the served model, not the request alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses
pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw
stop_reason in provider_specific_fields. Responses reports a paused turn as
incomplete with max_output_tokens and shows the unrun server search as an
in_progress web_search_call instead of a client function_call.
A refusal's stop_details.explanation now fills message.refusal and the final
stream delta.refusal on chat, and becomes a refusal content part on Responses,
streamed or not.
* fix(anthropic): let a chat client resend a paused turn so Claude continues it
* fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message
Chat cannot say where Claude wrote each thinking block in a paused web search turn, and
Anthropic answers 400 when one moves but accepts the turn without them. A new message after
a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped.
* fix(responses): send the streamed refusal part's type and text in content_part.added
* fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text
A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does
* fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses
* fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going
* fix(responses): mark a paused turn's message item completed
* fix(responses): mark every item of a paused turn completed
* fix(responses): keep a refusal's explanation when a refused turn is sent back
A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed
* fix(responses): scan refusal parts in Responses guardrails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits
A native Responses WebSocket turn logs its raw events instead of a usage-carrying
response object, so the v3 rate limiter read no usage and never charged the key
or tag token window. Read the turn's usage from its events the way the cost
calculator does, and charge the TPM-limited tags the socket was opened under.
* fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage
A later response.create turn on an open Responses WebSocket now gets a read-only
check of every TPM window it counts toward (key, team, user, org, end user, team
member, model-scoped limits and the socket's tags). A turn with no tokens left gets
the 429 rate_limit_exceeded frame and the socket stays open. The check never
reserves or increments a counter and leaves RPM, budget, PTU and project windows
to their own paths. It runs on native and managed sockets.
Managed turns carry the socket's call id in their metadata, so they charge the
socket's TPM-limited tags. A failed native turn charges the partial usage its
failure row already carries. A socket refused at open by a rate limit gets the
same refusal frame and closes 1008, logged once without a traceback.
/v1/realtime sessions now count toward TPM through the realtime usage combiner,
and the v1 limiter reads socket usage through the same helper
* fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn
A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with.
* fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero
* fix(proxy): stop charging a realtime session's whole usage into its close minute
Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case.
* test(proxy): check each per-turn TPM window on its own
* fix(proxy): send a fail-closed per-turn socket refusal with its 503 status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags
A Decisions deployment's model_info flags for audio and video now win over
the cost map, per media kind, the same way supports_box_questions already
does. An unlisted model flagged true gets its clips sent upstream, and a
false flag refuses them (or drops them under drop_params). OpenAI-format
providers still refuse clips since their request shape has no field for
them, and the SystemOne request builder no longer re-reads the cost map.
* fix(decisions): ignore media flags on a deployment whose own mode is not decisions
A chat-mode deployment's supports_audio_input describes chat input, so it no longer
turns on audio or video clips for Decisions. The cost map decides for it, as before.
Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params
deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info.
* fix(decisions): follow the cost map's mode for a flagged deployment without its own mode
A chat deployment flagged for audio or video usually leaves mode unset, so
its flags described chat input yet still let clips through on /v1/decisions.
When the deployment sets no mode, the cost map's mode for the model now
decides whether its media flags apply, and an unlisted model still takes
them. The audio and video lookups also run only when the request carries a
clip.
* fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map
The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations
* fix(decisions): route clips in a load-balanced group to a deployment that takes them
A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media.
* fix(decisions): keep deployments that drop or cannot resolve clips in the media filter
* fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering
Running the media filter before cooldown turned the 400 a group answered
when its only clip-taking deployment was cooling down into a 429. It now
runs next to the image filter, just before order filtering, in both the
async and sync selection paths.
* fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments
* test(router): list the router default params helper in the router coverage check
* test(bedrock): stop reloading the cost map module in the GovCloud fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes
* fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes
The whole-object forward sent keys Anthropic refuses and reached other
providers, so a request that used to pass could start failing with a 400.
Read the client's user_id after hooks, drop it unless it is a string with no
email or phone number, and send only that key when the upstream URL path ends
in /v1/messages. Share the validator with the translated route, whose email
check now matches any embedded address as Anthropic's does.
* fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans
Limit the pass-through user_id to Messages create on api.anthropic.com or the
configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep
the field off any request a pre-call guardrail inspects. The translated route
goes back to its own user_id filter.
* fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases
* refactor(proxy): pass the user_id check a plain mapping instead of an optional body
* fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped
Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill Claude web search per request on deployments priced in model_info
A vertex_ai deployment priced in its model_info is registered under its hashed id, so the
web search cost check on the model key never saw "claude" and billed Claude searches with the
Gemini grounding calculator: one search per call, or Gemini's $0.035 default with no search
price. The check now also reads the model the response names.
* fix(vertex_ai): bill token-priced Claude deployments at the served model's search price
A Vertex Claude deployment priced in model_info without a search price now bills each web search at the served model's cost map price, read at cost time and never written into the deployment entry. An explicit search price, zero included, still wins.
* fix(vertex_ai): price web search on the Vertex Claude Sonnet 4.5 and Haiku 4.5 rows
Google bills Claude web search at $10 per 1,000 searches on these models, so token-priced deployments of them no longer bill searches at 0
* fix(cost): bill served-model search price on token-only Anthropic and Azure AI Claude deployments
Token-only Claude deployments on anthropic and azure_ai billed web search at 0 because the deployment entry has no search price. They now fall back to the served model's cost map search price, as vertex_ai already does. Vertex routing reads the served model first, so a Gemini deployment whose id contains claude keeps per-prompt grounding
* fix(proxy): price streamed usage frames under the deployment's model so usage.cost matches the spend row
* test(proxy): give the Vertex streamRawPredict logging doubles the request's model, as a real pass-through Logging has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover text verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity as chat verbosity on the chat completions bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(responses): type the bridged text verbosity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text verbosity validation on the codex tool search follow-up turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): send text.verbosity on the client tool search follow-up bridge turn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover text.verbosity on previous_response_id client tool_search follow-up
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp): hand out bytes buffered before a dropped connection so a cut stream bills its partial usage
test(integration): stop the CyberArk override test failing when a short config value lands inside ciphertext
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(aiohttp): cover bytes already read before a dropped connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
