* test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test
GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a
captured Linear browser login to click through a real consent page. The OAuth
flows it exercised are already integration tests against an owned
authorization-server double, except one behavior: the stored per-user token
must be served by a proxy process that never saw the consent, with no shared
Redis to fall back on. That case is now in test_mcp_oauth_flows.py
The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker
and the JUnit gate script only that workflow used are removed
* test(mcp): chat completions run an OAuth server tool with the caller's own stored token
* test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers
The behavior it checked, a chat completion running an OAuth server's tool with
the caller's stored per-user token, is now an integration test in
test_mcp_llm_endpoints.py that needs no saved Linear browser login
* test(mcp): decode the stored per-user credential as urlsafe base64
The plaintext check used b64decode, which drops the - and _ of a urlsafe value
and raised on roughly one ciphertext in five
* test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat
security/test_mcp_slots.py already sends a per-user OAuth token through
/v1/chat/completions and asserts the peer receives it. Every mutant the new test
killed was also killed by the existing suite, so it added no signal
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens
* fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests
Vertex has no count-tokens route for Mistral or codestral models, so they now
get the same 400 as other partner models without a count API instead of a 404
that points operators at IAM.
* fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens
* fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model
* fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors
* fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer
* fix(count_tokens): pass through an OpenAI provider error that omits param or code
* fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route
* test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy
* test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes
The integration conftest raised UsageError from pytest_collection_modifyitems when a
shard-selected file collected nothing. Under xdist that ran on the workers after the
unfiltered item list was already published, so the worker ended its session with tests
still queued and the controller hit xdist's crashitem assertion, naming an unrelated
queued test and hiding the real ImportError.
A selected file that collects nothing now produces a failed collection report instead.
Files whose own collection already failed keep pytest's real error. Serial runs stop with
"Interrupted: N errors during collection", xdist runs report the error and exit 1, and
both still fail the shard.
WorkerKill returned once the victim's main thread was a zombie. On Linux the
process keeps its sockets open until its last thread exits, so a pooled
keep-alive connection to the killed worker still looked idle and the next
request through it got ECONNRESET. A zombie now counts as gone only once
num_threads() is at most 1. Where psutil cannot read threads of a zombie
(macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy
These two files register deployments with their own model settings on the
gateway. They now boot a module-scoped proxy on a scratch database, so their
setup cannot affect other tests that share the worker's proxy
* test(integration): seed the isolated gateway with the same router settings as the worker gateway
* chore: Add failing test case
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Restamp nested response.model in streamed Responses events
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Add tests
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Run format and lint, and fix
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Type nested Responses restamp to satisfy basedpyright gate
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): restamp the nested model on streamed response.queued events
Background streams send response.queued as a GenericEvent, so its nested
response.model still carried the backend name. Also drop the new comments
* fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events
* test(proxy): drive typed events through the nested restamp no-op test
* fix(proxy): restamp the nested model on guardrail block frames in streamed responses
* test(proxy): assert a cache-polled background response reports the requested model
* fix(proxy): restamp nested model on dict events and on every SSE data line
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(vector-stores): reproduce team key search routed to a team model of the wrong provider
A team key's model-less vector store search picks the team's first deployment, so a langgraph team model turns a Bedrock store search into "Vector store search is not supported for langgraph". Cover map_team_model picking by provider and a registered Bedrock store search reaching Bedrock with the registry's credentials.
* fix(router): match model-less team deployment to the requested provider
map_team_model now takes an optional custom_llm_provider. With no model given, it returns the first team deployment serving that provider, or None when none does, instead of the team's first deployment. Without a provider it behaves as before, so BYOK teams calling raw vector store IDs keep their team key.
* fix(proxy): pass the vector store provider when mapping team models
Vector store search, retrieve, update and delete pass the store's custom_llm_provider to map_team_model, so a team key's model-less call lands on a team deployment of that provider, or on the direct path with the store's own credentials when the team has none. Other model-less routes keep today's behavior.
* fix(proxy): type the vector store provider passed to map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector-stores): cover team key vector store search provider routing end to end
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format route_llm_request.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type vector store routing tests and mock Bedrock transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin model-less vector store calls when a public name mixes providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep team provider routing in map_team_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries
* fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5
A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow
* refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping
Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass
* chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client
The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py
* fix(vertex): match standalone 403 status tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep text 403 fallback for wrapped 500 errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): keep BadRequestError for explicit 403 text matches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): let structured body codes override the 403 text fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(register_model): keep bedrock geo deployments priced across price data reloads
A request-time pricing registration inserting into litellm.model_cost while a
price data reload walked it could abort the reload (dictionary changed size
during iteration in add_known_models), or trip the lookup map rebuild inside
get_model_info so the deployment replay fell through to the unmapped path and
wrote a raw bedrock/au.anthropic.<model> entry with no token pricing, a $0
shadow re-rolled on every later reload. Walk snapshots of the map, bind the
case-insensitive lookup map locally, and let the unmapped path inherit the
whole built-in entry of a provider get_model_info would accept.
* fix(register_model): keep synthesized zero costs out of inherited built-in pricing
An unmapped alias that inherits its built-in entry now takes only the token
prices the catalogue really carries, so a synthesized zero never lands next to
the caller's custom price. The router regression test reports the writer's
failures instead of swallowing them, and the local names carry Final.
* fix(register_model): match inherited built-ins by the key provider prefix when the value names none
* fix(utils): resolve a tripped lookup from the key's own catalog row before its region sibling
* fix(register_model): keep custom flat prices above inherited tier rates
* fix(register_model): keep the other direction's tier rates when a custom flat price covers one
* fix(register_model): keep a catalog row's tiers when its lookup trips, and snapshot request-path cost map walks
A tripped get_model_info on a key that is itself a catalog row now inherits that row whole, so a custom flat price bills a long prompt at the row's above-threshold rate on both lookup paths instead of flipping until the next reload. The fal_ai, fireworks, ollama and gpt-5 request-path walks over litellm.model_cost iterate a snapshot so a concurrent reload cannot raise mid-request.
* fix(register_model): price route-prefixed geo keys on a tripped lookup and replace cost map entries atomically
* fix(register_model): move every alias of a cost map entry to its registered price
* perf(register_model): read alias names from the shared entry instead of scanning the cost map
Each registration scanned every cost map entry to find the names sharing it,
about 1 ms on the 4,189-entry map, and a proxy request to a custom-priced
deployment registers twice. Aliased entries now carry the names they are
served under, so a registration reads them directly, and an alias the caller
pointed at its own entry keeps that entry. The tests this PR added lose their
docstrings
* fix(utils): keep capability flags and write through on a tripped lookup in register_model
A key the lookup cannot map now seeds its registered entry from the capability
rules its unregistered lookup resolves to, so registering a region alias no
longer turns off supports_system_messages and similar flags. A lookup that
trips mid-walk on a key naming a catalog row writes into that row instead of
leaving a shadow key that goes stale. Adds regression tests for both and for
the request-path snapshot walks.
* fix(utils): retry a tripped registration lookup and rebuild a registration a reload races
A lookup a concurrent write trips is retried instead of writing into the
catalog row its key names, so a per-request price never reprices a shared
row. The inherited entry leaves the matched row's provider behind, so a
provider-less azure_ai key stays priced, and a registration whose map a
reload swapped mid-build is rebuilt from the new map.
* fix(utils): write each registration into the map it was built from and rebuild it when a reload replaced that map
* test(integration): price data reload keeps every registered deployment priced
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and
ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the
dict-gated marker write skipped them and the served hit still stamped the full
price. The marker write is now container-agnostic, which it already was for the
partial-embedding stamp, and a regression test exercises the carrier end to end.
_uncached_response_cost also restores billed_token_rates, zero_cost_warned, and
the response_cost_failure_debug_information key alongside cost_breakdown, so the
probe leaves no pricing state on the call it reads.
* test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler
* fix(logging): keep the uncached price probe's restores to the state it can leak
* fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): read cache payload fields defensively without casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel): clarify pyright suppression reason on cache payload reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11
* fix(otel): report cache hits at zero cost everywhere and mark misses
Cache hits now carry a zero cost breakdown and a zero hidden response cost,
the /v1/messages hit cost header follows chat and responses hits, every span
carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps
the provider-prefixed model name, and the V2 attribute budget counts preset keys
* fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM
Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and
/v1/messages, while a fresh call whose cost is 0 still sends no header.
The async hit path stamps the same model name as main again. Stamping the
provider-prefixed name made a hit's tokens count toward the deployment's
TPM limit, so repeated cached requests hit 429s under usage-based routing.
Drop comments that restated fields and annotate new test values as Final.
* fix(otel): keep the served pricing basis on a cache hit's cost breakdown
A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path.
* fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys
A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0.
* test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans
* test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double
* test: annotate the new cache-hit unit test locals with Final
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130)
Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main
with the unit test moved to tests/unit after the test tree relocation
* fix(presidio): label an overlapping run by its outermost detection, add integration coverage
A run of overlapping detections now takes its entity type from the highest-scoring
detection that no other detection in the run contains, matching Presidio's own
anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER.
The resolver returns typed detections instead of Any.
* fix(presidio): drop the redundant comment in the numbered token loop
* fix(presidio): bump a numbered token another part of the request already holds
Each message and content part numbers its tokens from 1 into one request-wide
pii_tokens map. Collapsing overlapping detections made an overlapping value's
token well formed, so it collided with the same type in another part and both
restored as the later value. A token that maps to a different value now moves
to the next free number, and a token for the same value is reused
* fix(presidio): give a clashing token a spare number instead of scanning for one
Scanning upward from the clashing number probed every token an earlier part
had claimed, so two parts with thousands of values of one type took seconds
of synchronous work. No token in the map is numbered above the map's size,
so numbers above both that size and the part's run count are always free
* test(presidio): make the anonymizer double reject detections the analyzer never returned
Pick the token in one conditional expression instead of reassigning it, and
have the scripted anonymizer answer 500 when the proxy forwards spans that
differ from the analyzer's, so the default-scope cells check what was sent
---------
Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost
* test(responses): keep the websocket unit test file on main's formatting
* fix(responses): type the websocket frame guard for the lint gates
* fix(responses): pass the websocket frame guard as a named parameter and mark the guardrail dict edits
* fix(responses): bill a guardrail-rejected Responses WebSocket frame as a failed request with its guardrail cost
* fix(responses): type the recorded guardrail records as Mapping for the lint gate
* fix(responses): bill a rejected websocket frame through the spend logger only so the connection keeps its rate-limit slot
* fix(responses): guard nested response.create frames, strip forged user_api_key_* frame metadata, copy guarded tools back and give each rejected frame its own spend row id
* fix(responses): mark the spend logger's bare-dict failure hook for the type gate
* fix(responses): reject response.create frames that carry input at both levels and keep billing of a rejected frame non-blocking
* fix(responses): name the non-blocking billing catch for the strict lint gate
* fix(responses): keep the admitted guardrail list on every websocket frame, a frame cannot replace it with its own guardrails list
* test(responses): wait for the key spend total before asserting it in the websocket guardrail cost test
* fix(responses): copy the admitted metadata shallowly per websocket frame, the OTEL parent span cannot be deep-copied
* fix(responses): reject a WebSocket turn a pre_call guardrail rerouted to another model
* fix(responses): bill a rerouted WS turn and carry Presidio pii_tokens from the frame guard to the socket unmasker
* fix(responses): default pii_tokens through default_factory, Python 3.13 rejects a mappingproxy dataclass default
* fix(responses): run the per-frame guardrail walk on managed Responses WebSockets too
* fix(responses): unmask managed Responses WebSocket replies with the frame guard's Presidio pii_tokens
* fix(responses): strip client guardrail cost records from WS frames so a blocked frame bills the real scan
* fix(responses): forward a WS frame without the fields a guardrail deleted
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events
When the proxy renders a complete Anthropic Messages response as an SSE
stream for a client that asked to stream, the message_delta event carried
only stop_reason and stop_sequence. A real Anthropic stream puts
safeguard_results and stop_details inside message_delta.delta, so a client
such as Claude Code auto mode lost the server-side classifier verdict and
a refusal lost its stop_details. Both stream builders now copy those keys
from the source response through one shared helper when they are present.
* fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream
A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early
* fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations
A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the
dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a
masked stream never echoes the unmasked text through those fields.
* fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams
A rewritten stream now takes stop_details and safeguard_results from the
upstream message_delta alone, so a refusal the upstream sent without
stop_details is never given one by the OpenAI round trip. Unknown safeguard
types, statuses, reasons and extra fields pass through, a verdict follows a
tool call only when its input is unchanged, and tool_permission keeps the
explanations since it rewrites no text.
* fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites
* fix(router): deliver a refused stream built from a finished agentic turn instead of falling back
Websearch, MCP and advisor streams replay a complete reply whose tools already
ran, so a refusal fallback would run every tool again. Also give the recovered
response's message_start a null stop_details as the API sends it, drop
safeguard_results a masking rewrite cannot read, and add the audit cells for
synthesized streams.
* test(router): annotate the finished-turn refusal stream test locals with Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards
* test(rust): check the dangerous-tool-use beta reaches the wire with safeguards
* fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column
DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json
keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests.
* fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts
Azure AI and Claude Platform Messages transforms now add the
dangerous-tool-use beta for a non-null safeguards field, through the
Azure AI and Anthropic beta-map columns. The safeguards wire and chaos
tests get unique file names so they do not collide with the Python
fix's tests, the Python-route test asserts the fixed behavior behind a
BUG skip, and the DeepSeek test checks that no beta is added
* fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it
Claude Platform has no beta-map column and no QA account, so it goes back to sending
safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests
before the thread pool waits on them, and mark the no-column test as rstest.
* fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks
Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR.
* fix(rust): drop a null safeguards key instead of sending it without its beta
Anthropic rejects any safeguards key, null included, when the
dangerous-tool-use beta is missing, so an explicit "safeguards": null on the
Rust gateway still answered 400. The shared Messages transform now drops a
null safeguards key on every host, as the proxy already drops None params
before calling Rust, so it answers 200 with no beta.
The scripted peer in the integration wire tests now rejects the key on
presence, matching the real API.
* test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording
The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): add the dangerous-tool-use beta when a /v1/messages request carries safeguards
* test(azure_ai): pin the outgoing Foundry request keeps the dangerous-tool-use beta next to safeguards
* test(azure_ai): inject the Foundry client and scope the beta cache reset to the test
* test(azure_ai): own and close the mock transport client, drop the docstring entry
* test(integration): audit the safeguards beta derivation across anthropic-compatible providers
* fix(github_copilot): keep safeguards from deriving the dangerous-tool-use beta
* test(integration): kill the chaos victim worker through the harness helper
* fix(anthropic): derive the dangerous-tool-use beta only where the beta map supports it and honor additional_drop_params
The safeguards beta is now added only for providers whose beta map column accepts it, so github_copilot, Eden AI and the OpenAI-compatible JSON providers keep their base wire. additional_drop_params now runs before the provider headers are built, so a dropped param no longer drives its derived beta.
* fix(anthropic): load the beta map off the event loop and pin a dropped context_management
The first /v1/messages request that needs the remote beta map now loads it in a worker thread,
so a slow fetch no longer stalls every other request on the worker. An integration cell pins that a
deployment dropping context_management sends neither the field nor its derived beta.
* fix(anthropic): load the beta map only for safeguards requests, once per worker
* fix(anthropic): gate safeguards on the bundled beta map and load the remote map on its own thread
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): report the deployment's api_base when a stream fails mid-way
* fix(responses): report the deployment's api_base when a native responses stream fails mid-way
* fix(streaming): name the deployment's configured api_base in mid-stream errors and keep prompt text out
Mid-stream errors on chat and native /v1/responses streams now prefer the router's configured api_base over the logged request URL, websocket turns keep the connection's api_base, and the mid-stream mapping no longer adds the prompt to the debug info.
* fix(streaming): keep api_base credentials out of error debug info and leave websocket turn logging unchanged
* fix(streaming): name the websocket connection's api_base on turn failures and strip scheme-less userinfo
* fix(streaming): strip api_base userinfo before cutting the query so a raw ? or # in a password stays out
Adds the audit's integration tests for the mid-stream alert API Base line and the SDK debug info.
* fix(streaming): leave the API Base line out when an @ follows the host
A raw / ? or # in a password and an @ in a query or path read the same in a malformed api_base, so either guess could show a secret or a wrong host.
* fix(streaming): keep the API Base line when an @ sits in a provider path
Vertex partner model ids like claude-opus-5-5@default put an @ in the URL path, so the fail-closed rule left out the line main printed. The line is now left out only when the text before the first / ? or # is not a host, and a :// counts as the scheme only after a valid scheme name.
* test(streaming): cover a numeric password with a slash in the API Base line
The audit found that a password like 1234/function-secret, which looks like a port, had no case. The line is left out on this branch and printed the raw secret on main
* feat(bedrock): apply bedrock_request_metadata_fields on the /bedrock pass-through routes
* fix(bedrock): move Invoke body requestMetadata into the signed header and match Bedrock's ASCII whitespace rule
* fix(bedrock): keep Invoke body requestMetadata in the body and answer bad metadata as AWS does
Invoke requests whose body already carries requestMetadata keep the resolved pairs in the body
instead of the 8,500-character signed header. Malformed caller metadata now answers 400 with the
ValidationException error type the AWS SDK expects, and drop_params drops bad pairs on both the
pass-through routes and /v1/chat/completions. The caller-pair dedupe is linear, and the chat
validator treats \s as ASCII whitespace, as Bedrock does
* fix(bedrock): match Bedrock's 400s and header limit on pass-through request metadata
Invoke metadata too long for the 8,500-character header now goes in the body,
Invoke requestMetadata null answers 400 like Bedrock, Converse type errors carry
SerializationException, violation messages name the failing key, and the reserved
user_api_key_ prefix is matched case-insensitively after trimming whitespace
* fix(bedrock): trim the Invoke metadata header to Bedrock's limit and drop bad requestMetadata under drop_params without the setting
* fix(bedrock): leave /bedrock pass-through bodies as sent when request metadata forwarding is off
Global drop_params no longer rewrites requestMetadata on the pass-through routes unless
bedrock_request_metadata_fields is set. Chat keeps dropping bad pairs under drop_params.
Adds a tests/integration wire test for the four pass-through actions and removes an unused
parameter from the Invoke header helper.
* test(bedrock): drive the pass-through metadata wire test with boto3's bearer key env var
* fix(bedrock): trim the pass-through invoke metadata header and stop Nova invoke from checking metadata it drops
* test(bedrock): cover pass-through metadata header trimming, malformed caller metadata, and chat drop_params on the wire
* fix(bedrock): match requestMetadata keys and values against the whole string so a trailing newline cannot pass the 256-character limit
* fix(azure-ai): support Entra auth for Decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure-ai): select Decisions provider by name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): prioritize deployment credentials over env key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): restore api key precedence for Decisions auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618)
Invoke streams of Claude models now go through the same frame-by-frame relay as
converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages
stream translation, a block ends the stream with an AWS exception frame, and a held
stream gets its masks written back into re-encoded chunk frames. Claude is told by
its model id or, behind an application inference profile, by the anthropic_version
its request body carries. Streams of other model families stay as they were
* test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618)
* fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618)
A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while
converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered
path, with the rewrites written back into the chunk frames that carried each text.
Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does,
so a vendor can tie the scans of one streamed reply together.
* refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618)
* fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618)
An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned
as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned.
The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream.
* test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618)
* test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618)
* test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests
* test(bedrock): type the remaining parameters of the Claude invoke stream tests
* refactor(bedrock): bind the buffered invoke rewrite in its condition
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515)
* fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515)
A post_call guardrail that scans through apply_guardrail now runs on
/bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the
caller as the upstream sends them, every fifth frame and the end of the stream
are scanned, and a block after the first frame ends the stream with an AWS
exception frame. streaming_buffer_until_moderated still holds the stream and
applies masks per content block. Guardrails that need the whole response
(own async_post_call_success_hook, mask_response_content, native hooks, policy
pipelines) keep the buffered path.
* fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515)
* fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515)
* fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515)
- The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the
upstream ends with messageStop, an exception or error frame, or a dropped connection
- A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the
failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no
failure row
- A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does
- Closing the response before its body starts closes the guardrail layers and the upstream
- Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never
reaches frames a later layer still holds
- Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser
* fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515)
* test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515)
* fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515)
Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client
disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them
from the request the way chat completions streams do
* fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515)
* fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515)
* fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515)
A held post_call guardrail with streaming_buffer_release_on_scan releases each
window as received once its scan passes, so the relay delivered the original
text instead of the guardrail's masks. Such a stream keeps the buffered path,
which applies masks per content block as on main.
* fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515)
The relay ran every guardrail in config order, so a run_in_parallel guardrail
listed before a masking guardrail scanned the unmasked text. Main's buffered
path runs parallel guardrails after the sequential ones and discards their
rewrites, which the relay cannot reproduce, so these streams stay buffered
* fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves
- Noma v2 scans on the converse-stream relay carry stream, and the last one
carries the finished Converse response with its stopReason, as chat sends
them, so Noma keeps one record per reply. Other guardrails get the request
unchanged. When the last sampled scan already covered the whole answer, the
end-of-stream scan still runs for Noma so it gets the finished answer
- A held relay stream whose client leaves before the first frame stops
reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet,
the budget reservation and the parallel-request slot are released
- A cancelled wait for the first streamed chunk now stops the upstream read
instead of leaving it running
- A Noma v2 scan cancelled before Noma answers records not_run instead of
success with an empty response
* fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks
* fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input
* test(bedrock): expect the ValidationException main now returns for a held block before the first frame
Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400.
* fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message
A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw
event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the
messageStop, skipped its write-back and released the held text without its masks. The Converse handler now
reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text
* fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame
The relay skipped releasing the slot and budget reservation whenever a spend
flush was scheduled, but a flush of a partial first frame builds no response
and logs nothing, so the slot stayed held until it expired. The flush now says
whether it logged the stream.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
kill() returns once the killed worker's main thread is a zombie, but its other threads can
still be exiting and holding its sockets open. A keep-alive connection pooled to that worker
still looks healthy, so the next request goes out on it and the closing socket answers with
a reset. The old reap wait used to hide this. A new connection can only be accepted by a
live worker, so the surviving-worker assertions now run on a fresh client
Under worker isolation the harness rewrote every proxy config into the owned proxy's directory, so modules a test placed next to its own config (custom_auth) no longer resolved and the proxy died at boot. The copy now lands next to a test-supplied config, while configs inside the checkout still go to the worker directory so nothing is written into the repo tree
* test(integration): wait for both workers to load each metered model before sending
* fix(prometheus): read remaining rate-limit headers as numbers and let the router fill unreadable ones
* fix(prometheus): keep deployment metrics on non-string header keys and cover decimal and router-filled gauges
Accept any mapping key when reading additional headers, so a non-string key no longer drops every success metric. The router fallback test now uses a real Router and covers inf, nan and unreadable requests values, the decimal test uses a fractional requests value, and the integration cell sends 12.5 and an unreadable tokens value to deployments with tpm set
* fix(prometheus): leave the router fallback unchanged and keep only the decimal header fix
* fix(prometheus): read remaining rate-limit headers with float() like every release did
* test(integration): check the remaining rate-limit gauges on /v1/responses and /v1/messages too
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it
Bedrock accepts outputConfig structured output on every reachable profile of
Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so
json_schema now goes out natively there instead of as a forced tool, which kept
Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its
keys go back to the tool path. GovCloud keys keep their flags, and the drift
tests let GovCloud lag its commercial base but never lead it.
* fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path
Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf,
additionalProperties true, and recursive or unresolved $ref, all of which the
forced tool accepted. Claude schemas on the Bedrock native paths now go through
the cleaner the Anthropic native path already uses, and a schema with a
recursive or unresolved $ref keeps the forced tool, inline, or drop path it had.
* fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key
Older installs fetch the model map from main, so flipping supports_native_structured_output
moved them onto native output with their old schema handling. The new
supports_bedrock_native_structured_output key is ignored by older code. Schemas native output
cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new
key, models on the old key keep their current behavior for those shapes, and the Anthropic
API, Vertex AI, and Azure keep their current behavior.
* chore(cost-map): regenerate the model prices schema for the Bedrock native output key
* fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool
Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since
Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters.
Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects
typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1.
* fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking
* fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model
Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1
* fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas
The Bedrock native output key now applies only when the call asks for thinking, so calls without
thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts,
or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open
objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned
them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request.
* fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke
* test(bedrock): cover native structured output and json tool routing through the proxy
Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking
* fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode
Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas.
* fix(anthropic): keep native output for dict fields unless the json tool can be forced
* fix(anthropic): keep native output for dict fields when the caller sends tools
* test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects
* test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas
Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool
* fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output
* test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects
* fix(proxy): unset DB environment variables removed from the row on every worker by row revision
* fix(proxy): order the environment variables revision migration after main's and set its lock timeout
* fix(proxy): snapshot the environment at the first DB sync even when no environment_variables row exists yet
* fix(proxy): close gaps in DB environment variable reconciliation
Grant the revision sequence to every role so a writer role with only
table grants can still save environment_variables. Unset SSO keys that
only the environment_variables row stored on the next SSO sync, while
keeping keys the SSO settings table holds. Skip a revision-less row once
a revision has been applied, so a stale cached row cannot bring a
removed key back. Keep plaintext and undecryptable values when rotating
the master key instead of dropping them from the row.
* fix(proxy): keep env var revisions monotonic across DB copies and restores
Stamp at least the old revision plus one on update so a copied row keeps
rising, re-read a stale cached row from the writer and apply it when the
database itself went back, unset dropped SSO keys when sso_settings are not
loaded from the DB, and keep env values master key rotation cannot decrypt
exactly as stored
* fix(proxy): skip an env var row read before an overlapping sync that already applied a newer one
* test(proxy): cover a read replica back on an older row and an SSO client removed from DB environment_variables
* fix(proxy): keep config reads up when the writer is down, find the revision sequence by table schema, read the writer before rewriting env vars, and add the trigger under db push
* test(proxy): model the writer database in the retry policy config update fake
* fix(proxy): recheck the environment_variables row revision right before applying it
A sync that started later could read a stale environment_variables row
before an earlier sync applied a newer one, then pass the read-order check
and apply the stale row, bringing back keys the newer row removed. The
behind-the-applied-revision check now runs at apply time, re-reading the
writer when the row is behind, and a writer row read while another sync
applied a newer revision is dropped.
The config sync tests now inject a fake database client and use the real
get_config_param and its cache instead of replacing get_config_param.
* test(proxy): keep encryption real in the SSO replica-lag test
* test(proxy): inject a fake Prisma client in the env var replica-lag test
* fix(proxy): keep the applied environment variables when the writer database is out of connections
* test(proxy): cover a writer database out of connections in the env var integration tests
#46493 reads the first stream chunk inside the router's retry and fallback
chain, so an invalid prompt rejected before any token now falls back to the
configured fallback deployment instead of answering 400. The author
confirmed this is intended
The fallback test now asserts the spare serves the stream. The old 400
without a fallback attempt still holds when fallback_on_unsupported_params
is false, so that contract moves to a second proxy with that setting
* fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice
* test(bedrock): name the malformed realtime setting test for its string cases
* fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit
Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params.
* fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats
* fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices
* test(bedrock): cover realtime replay, refusal and committed failure through a real proxy
Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again
* fix(openai_like): surface provider response headers on non-streaming chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(openai_like): mark the header test locals Final
* fix(proxy): leave out response header values HTTP cannot carry
Non-streaming OpenAI-like chat now forwards every provider header, so on the
httpx transport a provider header value outside latin-1 made the proxy answer
500 after the provider had served the call. get_custom_headers now leaves such
a value out and keeps the other headers, which also ends the same 500 on
streaming
Adds integration cells for the provider request id on non-streaming chat and
the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend
parity under an upstream cost header, the latin-1 case, and the logging
callback payload
* fix(cost): ignore an upstream response cost header that is negative or not a number
* test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint
* fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number
* test(prometheus): type the rate-limit header test and drop its clock reads
* test(integration): cover deployment metrics under an unparseable provider rate-limit header
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets
* test(integration): wait for the team spend flush before killing a worker
* test(integration): poll for the S3 attempt the batch sinks write concurrently
* test(integration): queue the terminal object and its sibling into the same s3_v2 flush
* test(integration): time the long Bedrock version against an equally long unversioned name
* test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it
* test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries
* feat(anthropic): workload identity federation and pluggable identity sources
Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one
commit on top of litellm_internal_staging without the dashboard changes.
Deployments on anthropic/ without a static api_key can exchange an OIDC
workload assertion for a short-lived sk-ant-oat01 token through a shared
RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file,
an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment,
per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation
fields are server-owned: refused inline in request bodies and on
POST /model/new, proxy-admin only on credentials, and the token exchange
is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS
adds a host. GET /credentials/{name}/jwks exports the public key set of a
LiteLLM-signed credential for the Claude Console.
The OpenAI federation trio from #39613 rides along on the backend side with
the same server-owned handling.
Fixes#28607
Resolves LIT-6107
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries
The files handler enabled workload identity on batch-result downloads but never received the
deployment's litellm_params, so a deployment authenticating through a named credential could only
mint from process-wide env vars. It now threads litellm_params through to the auth header the way
the batch retrieve path already does.
LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme,
so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the
gateway. Entries are now parsed as network locations whether or not they carry a scheme.
* fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle
* test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries
* fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly
* fix(proxy): decrypt stored litellm_params before the WIF write gate
* fix(proxy): hide WIF secret references from /health output
* fix(proxy): keep the proxy error shape on credential endpoint refusals
* fix(proxy): hide identity token file paths from /health output
* feat(openai): honor per-deployment workload identity on non-chat routes
Image generation, audio speech and transcription, moderation, text
completions, files, batches, and fine-tuning built their OpenAI client from
the env vars alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file failed with the api_key error on every route
other than chat, Responses, and embeddings. Those handlers now build their
client through the shared workload identity resolver, and the managed batch
file-access credential extraction keeps the trio
* fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys
Non-chat handlers built a fresh SDK client per call, so files, batches,
fine-tuning, text completion and sync moderation exchanged the OIDC token on
every request. The builders now reuse the cached workload identity client per
identity, base, timeout, retries and organization; static-key clients are
still built fresh.
A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no
longer silently overrides a deployment's openai_identity_* params. An explicit
distinct api_key on the deployment or call still wins.
* fix(openai): keep shared workload identity clients on the process http session
The workload identity client is cached and shared across requests, so it must
never borrow the transport a single caller passed in. Static-key clients keep
using the caller's http client; federated clients always ride the process-wide
session. Also makes the http client helpers on BaseOpenAILLM public since the
builder now needs them from outside the class.
* test(openai): pin partial deployment identity beating the process key
* test(openai): type the managed-object invoke callback
* fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working
The Bedrock Claude Platform route already reads anthropic_workspace_id from
optional_params, so banning that spelling as a server-owned federation
parameter broke a pre-existing client capability. The federation field is now
anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID),
which restores the base branch's behavior for Bedrock callers, drops the
Bedrock-specific hint from the refusal message, and deletes the unconditional
ban constant that no longer had a reader
* fix(auth): share one exchanged token across workers reading the same assertion
Anthropic accepts each identity assertion exactly once, so two uvicorn
workers reading the same token file both minting from it means the second
exchange is denied with jti_reused. Minted tokens now land in a per-user
0700 cache directory guarded by a file lock, so workers on the same host
reuse one exchange until the token expires or the assertion rotates. A 401
is only retried when the re-read assertion actually differs, and the denial
hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache
and an empty value disables it
* feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals
The httpx-based OpenAI handlers built their Authorization header from the
deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying
openai_identity_provider_id, openai_service_account_id, and
openai_identity_token_file sent "Bearer None". They now resolve the bearer
through resolve_openai_bearer_token, which exchanges the deployment's
identity token the same way the SDK-based routes do.
A deployment that carries only part of the identity trio now raises a 500
naming the missing field instead of silently falling back to the process
key, and the SDK client builders take a factory for the static-key http
client so the federated branch never touches litellm.client_session.
* fix: keep anthropic federation from being shadowed or leaked
An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated
deployment sent an empty x-api-key on every call instead of minting a token.
Blank values now read as unset, and a real static key on a federated deployment
logs once that it outranks federation and nothing is being federated.
The exchange-host allowlist matched hostnames only, so a second process on
another port of an allowed host was trusted with the workload's identity token.
An entry that names a port now trusts that port alone, while a bare host still
trusts every port.
The shared token store exists so the workers reading one projected token file do
not each spend its single-use jti. A source that mints its own assertion per
exchange shares nothing with another worker, so it no longer writes a live token
to disk for a lookup that can never hit.
* feat(openai): resolve per-deployment workload identity for the assistants routes
OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders.
* fix: unlink a staged token file a failed write leaves behind
The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads.
* refactor: move anthropic jwks derivation behind a provider-owned tagged union
* feat(openai): resolve per-deployment workload identity for image variations
Build the image variations client through the shared workload identity
builders instead of scraping locals(), and give the async path the async
http session it was missing. Let a federation trio serve the model-less
batch fallback, and hold the standard LRU size of exchanged tokens.
* fix: unlink the staged token file when its write fails at close
A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token.
* fix(anthropic): close the staging descriptor before writing the shared token file
* fix(wif): judge federation writes by what they set, not what is stored
The admin gate read the stored deployment, so a team admin lost edit, delete
and Test Connection on any deployment carrying federation params. It now
returns early unless the submitted fields touch the federation surface, and a
Test Connection probe that points the deployment at its own api_base is still
refused, with the 403 no longer wrapped into a 500
The rest of the same review pass: POST /model/new refuses only a blocking
value of `blocked`, so a client that always sends `blocked: false` is not
turned away; a request body can no longer pick which federated identity to
mint as by naming a stored credential; an advisory refresh the executor
refuses disarms the entry instead of wedging the identity until the follower
timeout; the static-key shadow warning resolves its env fallback inside the
cache instead of once per request; credential writes drop nulls before
storing them; the token exchange validates the endpoint URL before reading an
assertion and keeps refusing redirects across a client heal; /health hides
every server-owned federation field from non-admins; and the async create_file
and create_batch paths say which setting is missing when the provider resolves
no URL
* style(tests): keep two blank lines before the batch_cost_is_final block
* test(batches): type the workload identity fallback test fixtures
* fix(moderation): stop validating unrelated kwargs on the sync workload identity path
* fix(auth): refuse server-owned workload identity keys in array form and inside a client router config
* fix(openai): keep a deployment's own key ahead of workload identity on every route
Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation,
image generation, and assistants so a deployment key equal to the process key still wins. A partial
identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and
/v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments
* fix(containers): pass deployment params to container file auth headers
Container file routes built their auth headers without the deployment's
litellm_params, so a keyless deployment with an OpenAI workload identity
sent Bearer None on upload, list, retrieve, delete and content calls
* fix(openai): keep process-wide key precedence and move token exchange off the event loop
Restore the merge base order where an explicit or process-wide OpenAI key wins over
federation, so exporting OPENAI_API_KEY keeps working the way it did before.
Run the header-building validate_environment calls (image edit, videos, containers,
vector stores, vector store files) and the decisions and System One request
preparation in a worker thread, so a slow token exchange no longer stalls the loop.
Federate realtime websocket sessions, live sessions, realtime client secrets and
transcription sessions, and the OpenAI moderation guardrail when no key is set.
* fix(openai): answer 401 for a partial identity trio and federate realtime health checks
A deployment with part of the workload identity trio and no static key now
fails as an auth error naming the missing params, so the router does not retry
it as a server error. The realtime health check sends the exchanged bearer for
a keyless federated OpenAI deployment, and API token counting reads the same
static keys as every other route before minting.
* fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config
* fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting
* test(integration): refuse caller-chosen identity params in user_config and bracketed form keys
* fix(openai): send the honored secret manager key from SDK client builders
A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key
on image, assistants, files, batches, and other SDK-built routes instead of failing with
no api_key, matching what the workload identity resolver already treats as a static key
* test(openai): check the token exchange runs off the event loop instead of timing the loop
---------
Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(sso): concurrent first sign-ins land on the account the winner created, worker-kill test pins half its streams to each worker
* test(integration): a worker kill returns once the worker is dead, the reap waits at block exit
uvicorn can take a full worker healthcheck to reap a worker that died mid-ping, so kill() held
the caller that long and peers held on the surviving worker hit their client timeout
* fix(sso): a sign-in that loses the first-insert race joins the default teams
The losing sign-in could read the winner's row before new_user added the default teams, so its
CLI key was minted with no team. The race tests now run the real new_user against a fake user table
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing
A non-list tools value crashed the vector store lookup during auth and the
built-in tool cost lookup in the failure logging path, so the proxy answered
an unhandled 500 and closed the keep-alive connection. count_tokens now
answers a 400 invalid_request_error for a malformed body, drops malformed
tools when drop_params is on, and both lookups skip values of the wrong shape.
* fix(proxy): skip non-list tools in the tool allowlist check
Also validate the count_tokens body without recursion, which the recursion detector rejects.
* test(integration): restore the int tools case on count_tokens
The proxy now answers a non-list tools value with a 400, so the case is deterministic again.
* fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids
* fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids
Check tools after request setup so the Claude Code default and the logged call type apply, read the request,
global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini
generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off
* fix(proxy): keep the request's call type when logging a gate rejection after request setup
The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses
rejection as aembedding, so it now only applies when the logging object carries no known call type
* fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry
* fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket
* refactor(proxy): leave the responses websocket out of the request-level tool check
* fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections
* fix(proxy): read router default drop lists and deployment ids in the malformed tools check
* fix(proxy): leave malformed tools to the router when it can pick another deployment
Fallbacks, auto-routers, model strings, comma batches and team model names
let the router pick a deployment after the proxy checks the request, so the
proxy now forwards a malformed tools field there as main does, and only
rejects it when the deployment it checks is the one that serves. Token
counting never reroutes, so count_tokens and input_tokens keep reading the
group they count with. additional_drop_params names are read the way core
reads them, so a list with a non-string entry still names its strings.
* test(integration): cover malformed tools on reroutable models, router defaults and deployment ids
* fix(proxy): resolve malformed tools at the deployment the router picks
* fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools
Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells.
* test(integration): expect a 400 for an empty-string tools field on /v1/messages
Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test
* fix(router): keep web search routing for tools passed as a tuple
* fix(router): keep a deployment's own tools when drop_params drops a malformed request tools
* test(router): list _merged_tools as covered through the deployment tools merge tests
* fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them
* fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError
* fix(prometheus): keep deployment health metrics clean on a malformed tools rejection
* test: type the new malformed tools tests
* test: mark new test locals Final and type kwargs dicts
* docs(agents): keep directory rules in nested AGENTS.md and pin integration tests to CircleCI
* docs(agents): scope the CircleCI rule to the files it covers and ask agents to read nested AGENTS.md
* ci(coverage): reject GITHUB_FILES integration contracts run from a test-unit.yml shard
* ci(coverage): report integration ownership findings under their own header
* ci(coverage): flag GITHUB_FILES contracts run from any non-dedicated workflow
* docs(integration): say a pre-merge run is no reason to route an integration file to GitHub
* ci(coverage): tie each dedicated integration workflow to the directory it owns
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open
GuardrailToolParam required a type key, so Gemini native tools
(function_declarations, googleSearch) failed validation inside the hook
and either returned a 500 or skipped the scan when fail_on_error was
false. The type key is now optional and left out of the forwarded tool
when absent, so an echoing guardrail server cannot write type: null into
the provider request.
structured_messages was re-validated against the chat completion message
types, which silently emptied tool_result, tool_use, image and document
content blocks to content: [] and dropped undeclared keys before the
guardrail saw them. The request model now forwards rows as opaque
mappings, which also retires the write-back shim that restored the
original rows for echoed-back entries.
The fail-open critical log line now carries error_type so a timeout, a
refused connection and an upstream HTTP status can be told apart without
the traceback.
* fix(httpx): retry aiohttp server disconnects the way httpx does
aiohttp raises ServerDisconnectedError when a kept-alive connection is
closed by the peer before a response arrives, which is what a load
balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces.
The transport mapped it to httpx.ReadError, which the handler's
connection-error retry does not cover, so the request failed instead of
being resent on a fresh connection. It now maps to
httpx.RemoteProtocolError, the error httpx itself raises for the same
condition, so the existing retry fires.
* test(httpx): build the stub server reply in one bytes literal
* fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable
The connection-error retry closed its fresh client before a streamed body was read,
so an aiohttp disconnect now billed a second upstream request and still failed. It
also skipped masking on a status error from the resend, resent PATCH and DELETE as
POST, dropped multipart files and ignored the handler's ssl_verify.
The retry now resends the request the first attempt built, on a client with the
handler's TLS settings, closes that client when a streamed body is closed, and masks
a status error the same way the first attempt does.
* fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type
* fix(guardrails): resend a generic guardrail check cut before any response
A load balancer that closes an idle keepalive connection made the guardrail POST
fail with "Server disconnected", because #46336 only resends POST on ConnectError.
AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to
False, that gives a RemoteProtocolError the same single fresh-client resend PUT
and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a
guardrail check is a pure verdict over the request. Every other POST keeps the
#46336 rule
The fail-open log line now puts error_type at the end so the pre-PR text stays
contiguous for existing alert matchers
test_rows_reach_the_guardrail_verbatim now reads the body off a real local
guardrail server instead of mock call args
* fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration
A ServerDisconnectedError raised while reading a response body now stays
httpx.ReadError, as it was before this PR. Only a disconnect before any
response maps to RemoteProtocolError, which is the case the resend rule is
about. Stream readers that end on ReadError, like the SAP stream iterator,
keep their old behavior
The loopback-server disconnect, TLS and guardrail resend tests move to
tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is
in-process only. Unit keeps in-process MockTransport checks for the idempotent
POST resend, masking on the resend, streamed and buffered client closing, the
guardrail resend, and the guardrail receiving rows verbatim
* fix(guardrails): resend a reset guardrail check and keep typed echoes on texts
An idempotent POST now resends once on any network error after connect
(ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load
balancer that answers a stale keepalive with a TCP reset rides it out too.
That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes
back to ReadError. This stops PUT and DELETE from stacking litellm's resend on
top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil
Guard from replaying its analyze POST on a disconnect
Every resend now runs inside the method's own error mapping, so a timeout on
the resend is a litellm.Timeout like the first attempt and a status error is
masked once by the caller
A guardrail server that parses rows with its own message types and echoes them
back without the keys those types lack (annotations, tool_calls[].index) is
read as an echo again, so its texts redaction reaches the LLM. A row only counts
as an echo when no value changed and every key the chat completion types
declare is kept
The wire tests for the PUT/DELETE resend, its masking and its TLS settings now
run on the httpx transport, where litellm's resend is the only retry, and the
POST and guardrail tests cover both transports and both cut styles
* test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend
Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head
* fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement
* refactor(guardrails): move the dropped-connection guardrail resend out of this PR
The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line
* test(guardrails): pick the closed guardrail port from the integration port allocator
Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port
* feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent
generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Both interactive transactions in /team/delete used Prisma's 5s default.
A lock wait or a short database stall past 5s expired the transaction
after the keys were already deleted, so the call returned 503 and left
the team row, memberships and team models behind
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* chore: drop duplicate litellm import and mark the stream_ended rebind
* fix(proxy): log only router-superseded attempts as error-frame failures
A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log
* test: type the _settle helper's recorded events
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): drop dangerous beta without safeguards
* fix(bedrock): drop the dangerous-tool-use beta only next to structured outputs, keep it on Mantle
Bedrock rejects the pair with 'invalid beta flag' on Claude models before Opus 4.7, while requests without structured outputs and Bedrock Mantle accept the beta, so the drop now only applies to Invoke requests with output_config.format or a strict tool and no safeguards.
* fix(anthropic): keep the AnthropicThinkingParam import that main now uses
* fix(bedrock): drop the client's dangerous-tool-use beta whenever safeguards is absent
Bedrock rejects the beta with 'invalid beta flag' on Sonnet 4.5, Opus 4.5, and Opus 4.6 even without structured outputs, so dropping it only next to structured outputs left those beta-only requests failing. The beta does nothing without safeguards, so Invoke now drops it in every such request. Mantle keeps it. The thinking display helper goes back to main's version.
* fix(bedrock): drop the client's dangerous-tool-use beta on Invoke chat without safeguards
* test(bedrock): wire cells for the dangerous-tool-use beta on Invoke messages, chat and responses
* test(bedrock): type the new dangerous-tool-use beta tests
* test(bedrock_mantle): type the dangerous-tool-use beta Mantle test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): route gpt-transcribe to GPT audio transcription config
* test(openai): move gpt-transcribe tests under tests/unit
* fix(openai): bill gpt-transcribe from provider usage seconds when duration is unknown
Fall back to the provider-reported usage.seconds when neither a measured
file duration nor a response duration is available, so json responses
for undecodable uploads are no longer billed at zero. Move the routing
test next to the other provider config tests and replace the structure
test with a behavior test.
* fix(openai): bill provider usage seconds only for gpt transcription models
* fix(cost): decide provider-billed transcription seconds from the deployment model, not its pricing id
* test(integration): cover gpt-transcribe default request, provider-billed spend, and health probe
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
