Commit graph

22368 commits

Author SHA1 Message Date
mateo-berri
0fcd566222 Merge origin/main into litellm_aws_event_stream_empty_200
Some checks are pending
Cost map pins recheck / rerun-cost-map-guard (push) Waiting to run
LiteLLM Rust DB / db-queries (push) Waiting to run
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Waiting to run
Terraform Modules / fmt, validate, test (gcp) (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Waiting to run
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Waiting to run
2026-11-17 12:43:15 -08:00
kenneth-berri
3115254fc3
fix(dashscope): bill 1h cache writes at the 1h rate (#52331)
* fix(dashscope): bill 1h cache writes at the 1h rate

DashScope reports the 5m/1h cache-write split as a nested
prompt_tokens_details.cache_creation object. That split is now read into
cache_creation_token_details, and the DashScope calculator prices the 1h
tokens at cache_creation_input_token_cost_above_1hr, falling back to the
regular write rate when no 1h rate is set.

* Bill a DashScope tier without a 1h rate at the tier write rate
2026-11-17 12:07:01 -08:00
joshua-berri
45115cbfa2
feat(shadow-eval): add optional rubric grades (#52321)
* feat(shadow-eval): add optional rubric grades

* chore(shadow-eval): generate rubric API types

* test(shadow-eval): cover unavailable rubric result storage

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-17 12:04:31 -08:00
yuneng-jiang
c0b9496ac0
test(integration): stop requiring safeguards calls to wait for the anthropic beta map (#52329)
#52188 made a filtered /v1/messages call filter against the bundled beta table while the
remote map loads in the background, instead of waiting for it. The chaos test still asserted
that Foundry safeguards calls stay unanswered until the map loads, so it failed on every
integration run since. The other assertions keep covering what the test is for.
2026-11-17 11:40:48 -08:00
devin-ai-integration[bot]
9fde4a1b58
chore(cost-map): rolling registry update, fal_ai kling v4, azure gpt-6-luna flex, snowflake, bedrock/mantle claude 5.5 (#52299)
* chore(cost-map): add fal_ai kling v4 text-to-image, image-to-image and long-image priced per billable unit

* chore(cost-map): add 8K per-image fallback prices to the fal_ai kling v4 rows

* chore(cost-map): drop the 8K per-image fields the catalog schema rejects from the fal_ai kling v4 rows

* fix(cost-map): correct Snowflake Cortex prices and restore azure/gpt-6-luna flex prices

Absorbed from #52297 and #52289.

Co-authored-by: David Hinckley <david.hinckley5@gmail.com>
Co-authored-by: danielguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost-map): price Claude Sonnet 5.5 and Haiku 5.5 on bedrock/mantle/ like bedrock_mantle/

Absorbed from #52325.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost-map): fix snowflake cache write test expectations

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost-map): fix bedrock mantle claude 5.5 parity test setup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: David Hinckley <david.hinckley5@gmail.com>
Co-authored-by: danielguthruf <102602010+danielguthruf@users.noreply.github.com>
2026-11-17 19:36:09 +00:00
Kerry Lu
1efca39054
fix(openai): bill the audio output tokens OpenAI reports for gpt-4o-mini-tts speech (#52317)
* fix(openai): bill the audio output tokens OpenAI reports for token-priced speech models

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(openai): type the speech usage stream and expect stream_format on the wire test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(openai): flush the last speech event and ask for uncompressed events

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 11:35:30 -08:00
kenneth-berri
52412090b9
fix(proxy): leave cache-read tokens out of the priority limiter's token count (#52209)
* fix(proxy): leave cache-read tokens out of the priority limiter's token count

The priority limiter counted cache-read tokens that the TPM limiter leaves out, so mostly cached prompts were throttled early. Usage dicts in Anthropic's own shape now count uncached input plus cache writes.

Fixes #51970

Co-authored-by: yifei tang <fl2326151118@163.com>

* fix(parallel_request_limiter): count cache writes and cache reads in the prompt of an Anthropic-shaped usage dict

A usage dict in Anthropic's own shape reports input_tokens without its cache reads and
cache writes. The Rust reconciliation read input_tokens as the full prompt and subtracted
cache reads from it, so billable input dropped the cache writes and clamped to 0.

Fixes #51970

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yifei tang <fl2326151118@163.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 11:32:25 -08:00
Kerry Lu
b5e719392e
fix(cost): price forwarded-credential clones by their source deployment (#52313)
* fix(cost): price forwarded-credential clones by their source deployment

A request that forwards the client's own Authorization header goes through
`_clientside_deployment`, which builds a request-scoped deployment with a
new id derived from the credential and keeps the configured deployment's
id as `model_info.original_model_id`. Deployment prices are registered
under the configured id, so the clone's id has no cost-map entry and
`select_model_name_for_cost_calc` falls back to the public price map.

`get_router_model_id` exists only to find the deployment pricing entry.
Return a registered clone id when one exists, and otherwise fall back to
the registered `original_model_id`, so configured prices still apply to
forwarded-credential requests. Spend attribution still reads
`model_info.id`, so the per-credential model id is unchanged.

Fixes #52038

* fix(cost): type the forwarded-credential price lookup and cover it with an integration test

The original_model_id fallback added six basedpyright errors over the merge-base ceiling. The lookup now runs in a typed helper, and a proxy integration test sends /v1/messages with a forwarded client credential and checks the response cost header and spend row against the deployment's configured rates.

Co-authored-by: xDipzz <deepakmahajan.2004@gmail.com>

---------

Co-authored-by: xDipzz <deepakmahajan.2004@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-18 03:30:02 +08:00
Hoya
518a6b39ed
fix(logging): reuse the cached GenericAPILogger so config reloads stop leaking flush tasks (#50879)
* fix: safely reuse and retire generic API loggers

* test: cover generic API logger reuse and rotation

* fix(proxy): unregister the current generic API logger after its headers rotate

Removing a database-configured generic API callback after a header rotation put the retired logger back, and it kept sending logs through its replacement. The database callback sync now compares and removes callbacks by the logger that is currently active

* fix(logging): let a retired generic API logger finish its sends and move its cache tests

A header change used to give the old logger 30 seconds to finish, then cancel it and drop any logs still queued on it, even when its own timeout and retries allowed longer. The old logger now finishes its last sends under its own send settings

The cache tests move into the existing logging callback manager test file, with typed parameters and settings swapped in through monkeypatch

* fix(logging): hand a retired generic API logger's queued logs to its replacement

Logs still queued when a header change retires a logger now go out through the new logger with the new header. Also drop the flush_queue test double, add Final to the proxy test, and cover both current-logger mappings in DB callback sync

* fix(logging): send queued logs with the old config when the new logger filters their event types

* test(logging): read delivered logs and headers through one helper

* fix(logging): check the hand-off target's event types under the flush lock

* fix(logging): stop retired loggers from resending queued logs or rejoining callback lists

* fix(logging): read the cached generic API logger under the callback lock

Two threads rotating the same logger could each build a replacement, leaving a listed logger that is not the cached one. Also drop a no-op mapping of displaced database callbacks, which are only ever strings, and add a regression test for a file callback that rotates during database registration

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 19:23:31 +00:00
devin-ai-integration[bot]
6af8df3934
test(spend): compare FOCUS RequesterDetails alias exactly after dropped-connection recovery (#52198)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 19:20:09 +00:00
tavren27
4a83cbc0d6
fix(scim): answer every /scim/v2 error with the RFC 7644 SCIM Error schema (#51322)
* fix(scim): answer every /scim/v2 error with the RFC 7644 SCIM Error schema

Okta reads the detail of an RFC 7644 SCIM Error when it rejects a provisioning call, and the /scim/v2 routes answered with the proxy's OpenAI error shape or FastAPI's detail object instead, so a rejected create showed nothing after "Errors reported by remote server"

A route class on the SCIM router now turns every HTTPException, ProxyException and request validation error, auth and premium dependency failures included, into a SCIM Error served as application/scim+json with the same HTTP status. Duplicate userName and duplicate group creates carry scimType uniqueness, and a failed user create logs the userName, status and reason

* fix(scim): keep placeholder admin errors in the proxy shape and never drop an error's reason

The /scim/v2/placeholders list and merge routes are LiteLLM admin tools, so they
go back to the proxy's error body. An error detail SCIM cannot read now answers
its own text instead of the bare status phrase. The integration tests that read
SCIM error bodies now expect the SCIM Error schema.

* fix(scim): answer every SCIM route error as a SCIM Error and keep the span's error type

Every 409 on a SCIM route now answers scimType uniqueness, so a duplicate email
that only differs in case gets it too. An unexpected error answers a SCIM Error
with the status phrase and is logged like the handlers' own errors. The OTel
server span reports the ProxyException or a public RequestValidationError the
route raised instead of ScimError

* fix(scim): keep unexpected 5xx text out of SCIM detail and trace an HTTPException as itself

* fix(scim): keep deliberate 5xx reasons and log every refused SCIM create with its userName

A 5xx answers the status phrase only when its text is the raw text of an error raised without a status. The roster sync retry hint, the auth database outage 503, and other deliberate reasons keep their text, and each failed roster write is named with the same rule. The refused-create warning moves to the SCIM route, so a create refused by the key check or the body schema logs its userName too

* fix(scim): answer the status phrase for a 5xx whose reason quotes an unexpected error

A team write that wraps a database error in its own 500 put the table name in the SCIM roster sync detail. A 5xx now answers the status phrase whenever its reason carries the text of an error raised without a status, and keeps a reason that does not quote one

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 19:10:33 +00:00
tavren27
93b7ff564d
fix(prompt_management): apply a prompt manager once per request on acompletion and SDK fallbacks (#51836)
* fix(prompt_management): apply a prompt manager once per request on acompletion and SDK fallbacks

acompletion ran the prompt manager's async hook and then completion() ran its sync hook on the same request, so a template was sent twice, and once more per attempt under SDK fallbacks. acompletion now passes an internal marker to completion() and to each fallback attempt when its manager has its own async hook, and both skip their pass when it is set. The proxy strips a client-sent marker so a caller cannot skip an admin's prompt manager

* fix(prompt_management): reuse the primary's template on fallbacks and apply it once on Responses-bridged chat calls

Fallback attempts reuse the rendered messages and the settings the template returned, an attempt whose model names a prompt integration prefix still runs it, the chat-to-Responses bridge marks its call so the Responses side skips the template, and the applied marker is an enum value a JSON request cannot carry

* fix(prompt_management): template each fallback attempt once from the caller's own messages

With SDK fallbacks the outer call no longer applies a registered prompt manager, so each
attempt renders its own template once, keeping a fallback's own prompt_id and the
template's precedence over the fallback's settings. A prefix-routed model gets no marker,
so a registered manager still applies after litellm_agent/ resolves the model, and
completion(fallbacks=...) skips its discarded outer pass

* fix(prompt management): skip the template on a chat call the Responses API hands back to chat, and keep a missing model a 400

* fix(prompt management): apply the template once on the Responses-to-chat bridge, keep cache points, and detect an applied async hook by its result

* fix(responses): build the empty prompt marker default with a factory so litellm imports on Python 3.11

* fix(fallbacks): keep the caller's credentials on a fallback behind the same prompt prefix

The SDK fallback runner compared providers on the prompt prefix (litellm_agent/, dotprompt/), so a
litellm_agent/openai fallback of an OpenAI primary counted as a provider switch and dropped the
caller's api_key and client

* fix(fallbacks): keep a fallback's own prompt template and drop credentials when its template can switch provider

* fix(fallbacks): resolve a prompt-prefixed fallback's provider from its own model name

* fix(responses): treat a prompt hook that only changed parameters as applied

A Responses call handed to chat now marks the template as applied when the
hook changed only parameters, so chat skips re-running it and still places
the cache points

* fix(responses): pass a template's chat-only settings to the bridged chat call

A Responses call handed to chat now forwards the template settings the
Responses request cannot carry, such as max_tokens, the way the chat side
applies a hook's settings

* fix(responses): keep the Responses call's own parameters out of a template's bridged chat settings

A template that set timeout or extra_headers made the bridged chat call get that keyword twice and fail. Those stay the caller's, as on base and on the native Responses path

* fix(prompt management): keep a template from sending the deployment's key to another host or provider

A Responses call handed to chat took a template's api_base and sent the deployment's key there, and a litellm_agent/ fallback that names its own prompt kept the caller's key when that prompt picked another provider. The bridged chat call now takes only a template's chat inference settings, and that fallback drops the caller's credentials, both as on main

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 19:04:49 +00:00
Agung Besti
d7ab7255e2
feat(guardrails): scan an image on /apply_guardrail with a Bedrock guardrail (#51016)
* fix(guardrails): record the guardrail verdict on a request dict that starts empty

The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.

* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail

A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.

The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.

* feat(guardrails): scan an image on /apply_guardrail with a Bedrock guardrail

/apply_guardrail only took text. It now takes `images`, one base64 data URI per request, and `text` becomes optional. The image is validated with the attachment reader the Bedrock guardrail already uses for chat completions and sent to ApplyGuardrail through the same call, with source OUTPUT for a response-direction scan. The image is scanned first, and the text scan is skipped only when the request has neither text nor messages.

A request the guardrail cannot judge answers 422 instead of 200 or 400. That covers an unreadable image, a guardrail that is not a Bedrock ApplyGuardrail guardrail, and an answer in which Bedrock does not report the image as guarded.

* test(guardrails): run the image tests through the endpoint's real request setup

The endpoint test helper on main now goes through the real request setup and guardrail lookup, and expects each request to leave one log entry. A body the request model refuses never reaches the endpoint, so those two cases are checked on the model. The guardrail that cannot scan images now carries the name the request asks for.

* fix(guardrails): log an image Bedrock did not evaluate as not run

An ApplyGuardrail image call that Bedrock answered without guarding the image
was logged as a guardrail success, so the request log read as a pass for an
image nobody checked. It now logs as not_run. A Bedrock error inside a 200
body on /apply_guardrail answers 502 naming the AWS error type instead of the
Image modality message

* fix(guardrails): refuse message text a Bedrock guardrail would not scan

* fix(guardrails): answer 422 when Bedrock cannot evaluate an image on /apply_guardrail

* refactor(guardrails): return image scan failures as values and type the test override

* fix(guardrails): keep Bedrock API errors reporting as HTTPException and log image-only prompts

* fix(guardrails): keep the scanned image in the apply_guardrail spend log request

* fix(guardrails): redact the apply_guardrail image when message logging is off and keep Bedrock API errors a plain HTTPException

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 18:13:28 +00:00
paveln83
c38b3c3747
fix(cohere)!: map tool_choice, reasoning_effort and json_schema response_format on the v2 chat route (#50834)
* fix(cohere): map tool_choice, reasoning_effort and json_schema response_format on the v2 chat route

get_optional_params pinned cohere and cohere_chat to the v1 param mapper, so the v2 mapper never ran. The v2 mapper now maps tool_choice none/required, reasoning_effort to thinking, and json_schema response_format to Cohere's json_object with the schema. Values Cohere cannot take drop under drop_params and answer 400 otherwise. The cost map no longer claims command-a-plus-05-2026 supports tool_choice.

* fix(cohere): keep the schema on Cohere's native json_object response_format

A {"type": "json_object", "json_schema": ...} request is Cohere's own structured output shape, so the mapping now forwards its json_schema instead of keeping only the type

* fix(cohere): drop response_format beside tools under drop_params and mark 08-2024 Command R without tool_choice

Cohere documents response_format with tools or documents as unsupported, and command-r-08-2024 and command-r-plus-08-2024 answer 422 to it, so drop_params now drops it there. Those two models also accept tool_choice but ignore it, so the cost map marks them supports_tool_choice false

* fix(cohere): gate JSON output per model, cap thinking at max_tokens, and list every served Cohere chat model

* fix(cohere): honor allowed_openai_params and deployment model_info, and force a named sole tool

The drop-or-400 checks for tool_choice, reasoning_effort and response_format now run when the request is built, so a param in allowed_openai_params goes to Cohere as sent and a deployment's model_info or base_model unlocks it on both cohere_chat/<model> and cohere_chat/v2/<model>. A named function that is the only tool maps to REQUIRED. response_format beside tools or documents is checked per model, only the two 08-2024 Command R models rejecting it

* fix(cohere): warn when drop_params drops a param and send a mapped reasoning_effort as thinking only

Each tool_choice, reasoning_effort or response_format value that drop_params drops now logs a warning naming the param, model and reason. A reasoning_effort LiteLLM translates to Cohere thinking is no longer also sent raw when allowed_openai_params lists it, since Cohere rejects that field

* fix(cohere): list response_format only for Cohere models with JSON mode

The v2 config listed response_format for every Cohere model, so router
pre-call checks kept Aya, tiny-aya and North deployments as candidates for
JSON requests, and /model_group/info said they took it. It is now listed only
where the cost map marks supports_response_schema, the same way
reasoning_effort is gated, and model_info or base_model can still turn it on.

* fix(cohere): keep response_format contents out of the drop warning and the 400

A dropped response_format now logs only its type, and the 400 for an unknown type or a model without JSON mode no longer repeats the value, since a schema can hold the caller's data

* fix(cohere): leave out tool_choice without tools, check json_schema keywords Cohere rejects, use Cohere's context lengths

A tool_choice sent with no tools is left out again, as it was before this PR, instead of answering 400

A json_schema using a keyword or pattern Cohere rejects (minimum, allOf, minItems, an anchored pattern, and others) drops under drop_params, else answers 400 naming it

tool_choice support now follows a model registered after the first request, and error messages shorten long values

The new and touched Cohere cost-map rows take max_input_tokens from Cohere's models API context_length, so pre-call checks no longer refuse prompts Cohere accepts

An unreachable response_format branch is removed, since get_supported_openai_params already leaves response_format out for those models

* fix(cohere): read model_info on cohere/ deployments and stop the schema keyword check 64 levels deep

A cohere/ deployment of a model the cost map does not list ignored its model_info supports_tool_choice,
since the capability lookup only asked for the cohere_chat provider. A json_schema nested about 300 levels
deep hit the recursion limit and answered 500. Deeper levels now go to Cohere unchecked.

* fix(cohere): drop or reject a json_schema nested past the 64-level check

A schema deeper than the keyword check now drops under drop_params and
otherwise answers 400, as every schema did before this PR, so a keyword
past level 64 no longer reaches Cohere and fails under drop_params

* fix(cohere): read a string allowed_openai_params with the shared helper

* test(cohere): compare the whole request body Cohere gets in the v2 param tests

* fix(cohere): check only the schema parts Cohere validates and refuse list-form items

Cohere validates properties, anyOf and a single items schema, but not prefixItems or additionalProperties, so a keyword there no longer turns into a 400. A list in items, which Cohere refuses, now drops under drop_params or answers 400

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 18:10:37 +00:00
devin-ai-integration[bot]
20b2f02297
test(gemini): use valid base64 thought signature fixtures (#52311)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 18:03:38 +00:00
joshua-berri
bcb1f486a6
fix(interactions): retain async stream success logging task so it survives GC and shutdown drain (#52300)
Route the async Interactions API stream success logging through spawn_logging_task
instead of a bare asyncio.create_task, so the pending task is strongly referenced,
failures are logged, and proxy shutdown drains it.

Fixes #50894

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-17 09:43:45 -08:00
Matthew Lapointe
6a8d259f27
fix(router)!: let an unmatched content_policy_fallbacks list fall through like context_window_fallbacks (#52132)
A content policy error whose group has no content_policy_fallbacks entry
now walks the order tiers and the generic fallbacks, as an unmatched
context window error has since #49779. Before, an entry for any group
made the error raise for every other group.

A matching content policy chain now wins over the order tiers and over a
flat request fallbacks list, and a failed order chain keeps the original
content policy error. Refusals returned as a 200 are unchanged.
2026-11-17 09:28:49 -08:00
kenneth-berri
69438c1e3f
fix(anthropic, bedrock): split concatenated tool call arguments and answer every split tool use (#52216)
* fix(anthropic, bedrock): split concatenated tool call arguments and answer every split tool use

* fix(anthropic, bedrock): scope split tool results to their turn, reserve existing ids, cap the fan-out

* fix(anthropic, bedrock): share one split id map per turn, reuse ids for repeated calls, reject splits past the cap

* perf(anthropic): skip the split id scan for messages without tool calls
2026-11-17 09:28:19 -08:00
jesus-berri
7b88d94d61
fix(together_ai): declare reasoning_effort_levels on Kimi-K3.1-Pro so max is not sent as high (#52294) 2026-11-17 08:20:41 -08:00
Gabriel Creti
60d05972ae
feat(tts)!: stream audio from the provider as it arrives on /v1/audio/speech (#46222)
* feat(tts): stream audio from the provider as it arrives on /v1/audio/speech

The proxy read the whole audio body from the provider before sending the
first byte to the client, so time-to-first-byte was the full synthesis
time. Pass stream=True to get the bytes forwarded as they arrive.

The response type does not change: it is still HttpxBinaryResponseContent,
only unread, so every existing caller, logging path and spend hook works
the same. Providers opt in with supports_streaming on their config, the
others ignore the flag and buffer as before.

* fix(tts): keep streamed speech inside its concurrency and cleanup guarantees

Streaming the audio out moved two things that used to be covered by the
buffered response, and the review on #46222 caught both.

A deployment's max_parallel_requests counted syntheses that had started
rather than syntheses in flight, because aspeech now returns at the
response headers with the audio still arriving. The slot is held until
the response is closed, the way the chat streaming path already does it.

A client that walks away mid-clip leaves the body unread, and httpx only
releases the connection when it is read to the end. The body iterator
closes it on the way out, and the route arms the same close as a
background task for a response whose body is never read at all. These
are two different paths on purpose: Starlette raises ClientDisconnect
out of the response before background tasks run, so neither one covers
both cases.

Also pulls the speech body cast into one helper beside
speech_request_body, which drops a banned typing.cast import.

* test(tts): cover the slot-holding helper directly

router_code_coverage.py wants every function in router.py called by name
from a unit test, and _hold_slot_until_closed was only reached through
router.aspeech(). Test it on its own instead of adding it to the ignore
list: which responses keep their slot is the part worth pinning, so the
cases are buffered body, non-binary response and unread body.

* fix(tts): stream ElevenLabs speech from its /stream endpoint

ElevenLabs' convert endpoint synthesizes the whole clip before it sends the
first byte, so stream: true still waited for full synthesis. Requests with
stream: true now call /v1/text-to-speech/{voice_id}/stream.

* fix(tts): log streamed speech like buffered speech

A deployment-hook sync copied stream=True onto the logging object for
speech, so the success handler waited for an assembled chat stream that
never comes and streamed speech never reached spend logs or callbacks.

* fix(tts): forward streamed speech audio as it arrives

* fix(tts): log streamed speech when the audio ends

Streamed speech now logs success when the caller reads the audio to the end or closes it, and logs a failure when the upstream breaks mid-clip. Key max_parallel_requests slots stay held for the whole stream, the logged duration and bytes match what was sent, and the router slot is no longer pinned by a pending logging task.

* fix(tts): bill and map streamed speech failures, defer logging only when streaming

A streamed clip that breaks mid-way is now logged as a failure at the clip's cost, since speech is priced per input character and the provider bills the whole request. Transport errors map to litellm.Timeout or litellm.APIConnectionError, and the proxy runs post_call_failure_hook with the clip's cost. Success logging waits for the stream end only when the caller asked for stream=True.

* fix(tts): stream only on a literal true, mark hang-ups, and cut off without a fake DB outage

* fix(tts): put the 499 hang-up marker on the spend row

* fix(tts): keep the caller's metadata and positional base_url intact, add speech stream integration cells

* fix(speech): keep stream keyword-only and buffer the speech health check

Passing stream positionally skipped the logging wrapper, and a deployment
with stream: true left the health check's provider stream open. Also gives
each scripted chat reply its own id so the non-boolean stream cases do not
collide on the spend log key, and adds the audit's round-2 cells

* test(speech): check the health check calls the buffered path

* test(integration): read the speech chaos spend rows from the worker database

Main now gives each owned proxy a private database unless the test file opts
into the worker's state. The chaos tests read billed rows from the worker
database, so they opt in with the shares_worker_state marker like the other
providers chaos files

* fix(tts): release the router slot when a streamed read fails

The deployment slot was only released through Response.aclose, which httpx
does not call when the body read raises. A provider cutting a clip off
mid-read left the slot held, so later Router.aspeech calls got 429.

Release the slot on any stream end, and close the upstream stream when a
read fails.

Co-Authored-By: Claude <noreply@anthropic.com>

* test(tts): drive speech stream tests with events instead of sleeps

Co-Authored-By: Claude <noreply@anthropic.com>

* test(utils): capture streamed sync speech usage when the audio ends, not at the headers

* fix(proxy): close streamed speech when the route fails before responding

A response headers hook, or any other step between aspeech returning an unread
audio stream and the route returning its StreamingResponse, could raise and leave
that stream open. The upstream connection and the deployment's
max_parallel_requests slot then stayed held, so later clips got 429s. The route
now closes the audio on any error before it returns

* fix(proxy): end streamed speech as failed when the route errors before responding

Closing the unread audio on a pre-response error went through aclose, which always
ends as StreamClosedEarly: the clip was success-logged with a 499 ClientDisconnected
marker while the route returned 500. The route now ends the audio with the error,
through every OnEndAsyncByteStream wrapper, so it is logged as a failure. A
cancellation still closes it as a hang-up.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(proxy): close unread speech audio when a deployment guardrail rejects it

A guardrail's post-call deployment hook runs before the audio stream's end logger is attached, so a rejection left the unread provider response open and held its connection. The failure path of the async wrapper now closes it, and a close that fails never replaces the rejection

Also annotates the new test locals in test_routes_audio.py with Final

* fix(proxy): free the key's parallel slot before a streamed speech clip ends

A streamed clip released its key's max_parallel_requests slot only from the
success logging task, which defers the Redis write until after the caller has
seen the end of the audio. A next clip sent to another instance at that moment
could still be counted against the key and get 429. The speech route now
releases the slot when the audio finishes, before the response ends, the same
way chat streams do.

* fix(speech): log a failed streamed clip once per failure hook

A logger with both failure hooks got both for one failed aspeech stream. The
stream's failure path now marks its sync run as part of an async request, as the
non-stream path does. Also type the route test helpers and check that the speech
health check leaves no provider connection open.

* test(speech): expect one async failure event from a cut-off aspeech stream

* fix(proxy): shield the speech upstream close and tag cut-off rows as aspeech

A hang-up while the provider is silent no longer lets the anyio cancel abort the upstream close
on servers that cancel the stream on disconnect. Mid-clip cut-off failure rows now store the
aspeech call type instead of an empty one.

* refactor(logging): record a speech client disconnect through the logging object

* test(speech): hold the provider mid-clip in the guardrail-rejection wire test

The test sent the whole short clip at once on a keep-alive connection, so when the clip reached the HTTP client before the guardrail rejected it, closing the response handed the finished connection back to the pool and the test timed out waiting for a close. The peer now holds after the first audio chunk until the rejection is raised, as a real provider still sending a long clip would

* fix(speech): close the unread audio when a sync speech call fails after its headers

The async wrapper already closed it. The sync wrapper left the provider
connection open until garbage collection when post-call processing raised.

* fix(router): keep hung-up speech clips out of latency samples

A streamed clip the caller hangs up on ends early, so its short duration made
the deployment look fast to latency-based routing. The clip still counts as a
request.

* test(proxy): type the speech request helper in the audio route tests

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-17 15:03:12 +00:00
Vansh
0cc072c999
feat(litellm): add Vispark provider with Vision Small/Medium/Large (#41192)
Some checks are pending
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / mcp-elicitation (push) Blocked by required conditions
unit / proxy-auth (push) Blocked by required conditions
unit / mcp-server (push) Blocked by required conditions
unit / proxy-extras (push) Blocked by required conditions
unit / proxy-hooks-client (push) Blocked by required conditions
unit / proxy-infra-db-spend (push) Blocked by required conditions
unit / proxy-infra-root-a-p (push) Blocked by required conditions
unit / proxy-infra-root-q-z (push) Blocked by required conditions
unit / docs (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / helm (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(litellm): add Vispark provider with Vision Small/Medium/Large

Rebased onto latest main: re-applied vispark registrations (model sets,
provider enum, JSON provider registry, pricing catalogs, dashboard
fields, endpoints matrix) and moved provider test to
tests/unit/llms/openai_like/ per current layout.

* fix(litellm): keep basedpyright reportUnknownArgumentType budget flat for vispark

* fix(litellm): satisfy type-discipline budget for vispark cast

* chore: retrigger CI (prior MCP dep-resolution leg cancelled by infra)

* test(vispark): cover routing, text completion and cost; sync endpoint matrix backup

* test(vispark): tolerate cost-map entries without a provider when listing models

* chore(vispark): regenerate the provider endpoint matrix with main's generator

* fix(vispark): bill the 600 input token minimum and bridge JSON schemas and PDF file parts

Vispark charges every request for at least 600 input tokens, so the cost map
gains min_billed_input_tokens and the generic cost path bills up to it.
Vispark answers 500 to a json_schema that sets additionalProperties, which the
OpenAI SDK always sends for a Pydantic model, so the provider strips it.
Vispark reads PDFs only as image_url parts, so file parts are sent that way.

* fix(vispark): bill the input token minimum under custom pricing and strip every additionalProperties value

* refactor(router): inline the billing minimum lookup into the deployment cost payload

* fix(vispark): scope the input minimum per deployment and reject map-field schemas

Keep a deployment's min_billed_input_tokens on its own cost-map entry so it
no longer changes billing for other deployments of the same backend model.
Strip boolean additionalProperties for Vispark, answer 400 for schema-valued
ones unless drop_params is set, and do it without recursion.

* fix(budget): reserve at least a deployment's billed input minimum

* test(budget): patch the renamed deployment info helper

* fix(vispark): reject file_id parts, keep properties named additionalProperties, bill the minimum on dashscope and perplexity

* fix(vispark): send inline file_data when a file part also carries a file_id

* fix: keep provider-stated cost for minimum-only deployments and skip the minimum without a counted prompt

A deployment that sets only min_billed_input_tokens no longer drops Perplexity or Kyma stated cost, streamed cost headers no longer bill the minimum before usage arrives, and tiered deployments now carry the minimum

* fix: bill provider-stated cost for minimum-only deployments and charge a disconnect only the group's shared minimum

* style: drop the restating comment on the vispark provider entry

* fix(cost): keep the provider-reported video cost for a deployment that sets only a billing minimum

* fix(cost): read the billing minimum the way the cost map guard accepts it

A cost map row can write min_billed_input_tokens as "600" or 600.0, which the
catalog's count parser accepts. The chat cost path subtracted the raw value and
raised TypeError on a string, and the custom-rate path billed no minimum unless
the value was an int. Both now read it through one non-negative int adapter

Also loosen the Scaleway file part test, since main now adds a default filename

* fix(vispark): keep schema literals and reserve at the short prompt's tier

Removing additionalProperties now skips the literal data in const, enum,
default and examples, so a value holding that key reaches Vispark unchanged

Budget reservation picks the pricing tier from the counted prompt and only then
applies the billed input minimum, the way billing does, so a minimum that
crosses into a cheaper tier no longer reserves less than the request bills

* fix(utils): list JSON providers whose API key env var differs from their name

* fix(vispark): answer 400 for a file part with no file instead of crashing

* fix(cost): use a deployment's own input minimum with per-token rates

/spend/calculate and /cost/estimate priced custom-rate deployments with the
cost map's min_billed_input_tokens, so a deployment that sets its own
minimum (0 or 1000, say) got a quote that differs from what its live
requests bill. Both endpoints now pass the deployment's minimum with the
rates, and the per-token floor also reads it from the requested model info
before falling back to the cost map

* chore(vispark): keep catalog and endpoint formatting aligned

* fix(vispark): mark provider registry reference final

* fix(vispark): reject missing provider credentials before fallback

* fix(vispark): preserve authenticated text completion clients

* fix(vispark): preserve authenticated chat completion clients

* fix(vispark): reject unused SDK credentials on HTTP transport

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-17 00:48:52 -08:00
devin-ai-integration[bot]
42f901ad65
feat(cost-map): price Claude on Bedrock Mantle under bedrock_mantle/ and bedrock/mantle/ (#52231)
* feat(cost-map): price Claude on Bedrock Mantle under bedrock_mantle/ and bedrock/mantle/

* fix(cost-map): set Mantle claude-haiku-4-5 batch prices to half the In-Region rate

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 07:12:16 +00:00
Kerry Lu
fd12cba0b2
test(cost): mark the cost tracking cases the reconciliation automation bills against the real provider (#52226)
* test(cost): mark cost tracking cases that the reconciliation automation bills against the real provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): add real-provider GPT-6, image, audio, embedding and realtime cost cases for the reconciliation automation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): make the GPT-6 prompt cache cases cacheable by the real provider and add the 30 minute Responses cache write

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: drop handle_jwt.py that was re-added by mistake

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 07:01:32 +00:00
Nate Armstrong
f5e911afd0
feat(callbacks-parallel-request-limiter): port parallel request limiter v3 to a rust callback (#51537) 2026-11-16 22:56:38 -08:00
Mateo Wang
4b12a4051e
feat(proxy): opt-in prefix block hashes in spend logs for workload replay (#52120)
* feat(observability): add opt-in prefix block hashes to spend metadata

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(observability): address prefix block hash callback review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(observability): cover prefix hash callback registration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(observability): bound prefix hashing and honor logging privacy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(observability): inject prefix hash worker admission bound

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 22:52:42 -08:00
ryan-crabbe-berri
da296ee4bf
fix(gemini): keep thought signatures for proxy model names without gemini in them (#52212)
* fix(gemini): keep thought signatures for proxy aliases without gemini in the name

The proxy ran the non-Gemini signature strip during request setup, before
routing, so it judged the alias (g38flash) instead of the deployment
(vertex_ai/gemini-3.8-flash) and replaced every replayed signature with
Google's placeholder. The strip now runs in the SDK call wrapper, where the
model is the deployment's, and skips messages that carry no signature.

* fix(gemini): type the tool call id signature strip and match outbound snake_case parts in the test

* test(gemini): check signature stripping where the deployment model is known, and that proxy setup keeps it for aliases

* test(gemini): cover streamed alias replay and an OpenAI to Vertex alias fallback in one request
2026-11-17 06:20:51 +00:00
fangkang
98e83feeb7
fix: keep LiteLLM-internal params out of provider request bodies (#46083)
* fix(embeddings): keep prompt-management params out of embedding provider requests

A deployment whose litellm_params carry cache_control_injection_points,
knowledge_bases or vector_store_ids fails every embedding call against a
strict provider, because embedding() forwarded those keys as provider
params. Bedrock Titan rejects them with "extraneous key ... is not
permitted" (#30314). Embeddings never run prompt-management hooks, so
nothing consumes these params there. They are now excluded where
embedding() builds provider params, and an explicit extra_body is still
passed through

Refs #30301

Co-authored-by: tsushanth <78000697+tsushanth@users.noreply.github.com>

* fix(mcp): own the MCP handler keys and read fake_stream as a control

completion() and embedding() already keep owned kwargs out of provider
params, but skip_mcp_handler, mcp_handler_context and fake_stream were
in no registry, so a caller that passed them sent them to the provider.
Some OpenAI-compatible providers crashed with a TypeError before
sending, others forwarded the field and failed on strict APIs

The MCP keys now live in McpHandlerState. filter_internal_params builds
its set from the same declaration instead of a separate hand-written
list, and stays MCP-only because fallback callers pass the remaining
kwargs back into acompletion

fake_stream is a ControlOptions field next to stream_chunk_size. A value
from a caller or a deployment is validated, carried in litellm_params and
read by the HTTP and Converse handlers alongside the flag provider configs
set internally, so it turns on fake streaming and never reaches the
provider body

Refs #30301 #30371

Co-authored-by: Chenglun Hu <chenglunhu@gmail.com>

* test(providers): walk every chat and embedding route with internal params seeded

The wire test covered six providers. This walks every chat and embedding
route that can reach a local wire server, including each Bedrock route,
Invoke family and Bedrock embedding family. Each call carries every
LiteLLM-owned name a plain call can take, and the test fails if one of
them, a prompt-management param on embeddings, any other owned name at
the top level, or any top-level underscore key reaches the request body

The seeds follow the registry. Every owned name is seeded or listed with
the reason it can't be, so a new field fails until someone decides

A provider without a route must either have a stated reason it can't
reach the wire and send nothing to it, or raise LiteLLMUnknownProvider
without sending. A provider that gains a handler fails until it is walked

Routes the walk shows sending or rejecting max_retries are pytest.skip
with a BUG reason, per tests/integration/AGENTS.md

Closes #30301

* fix(bedrock): filter LiteLLM-internal params from invoke request bodies

Carried over from #30398 by kimnamu and applied to current main, where
the test file now lives under tests/unit

Fixes #30371

* refactor(bedrock): check invoke params against the owned MCP names in one pass

The invoke filter now reads MCP_HANDLER_KWARG_NAMES from the registry
inside the existing comprehension, so it adds no rebind and no import of
the untyped helper. Behavior and the regression test from #30398 are
unchanged

* test(providers): walk the providers and owned names that landed on main

Main added the opencode_go, coralbricks, openinfer, databricks_agent and inferenceapis chat providers and the
aws_bedrock_control_plane_endpoint, kubernetes_pod_discovery and _encrypted_content_affinity_pinned owned
names, so the wire walk now covers them too

* test(providers): walk the providers and owned names that landed on main

Main added the bigmodel, aiand, abliteration, empiriolabs, kyma, refuseless, hostinger_router, clf_ai_gateway,
zerogpu, streamlake and wally chat providers, the empiriolabs and hostinger_router embedding routes, and the rpd,
tpd, daily_reset_time, switchyard_router_config and thought_signature_in_tool_call_id owned names. chatgpt now
sends with a plain api_key, so the walk covers it as a chat route instead of keeping it off the wire

* style(tests): match main's formatting in the Bedrock Converse handler tests

The file now differs from main only by the caller fake_stream test and its imports

* fix(main): bound a caller or deployment fake_stream by the request timeout

completion() picks the faked-stream timeout by asking _completion_fakes_stream, which only looked at
optional_params. A caller's or deployment's fake_stream lives in the stored control options instead, so a
faked stream was bounded by the short stream_timeout alone and a slow non-streamed answer timed out. The
check now also reads control_options.fake_stream, which covers the first attempt and the Bedrock native
output retry

* fix: drop only the invalid control option under drop_params

A bad stream_chunk_size under drop_params reset every control option, so a
valid fake_stream was lost and the stream went upstream for real. Keep each
option that validates on its own instead.

Also annotate the new Bedrock locals and tests with Final and return types.

* fix: replay fake_stream as a stream on every chat route

completion() now makes one plain provider call when a caller or deployment
sets fake_stream and replays the answer as a stream, so OpenAI, Anthropic and
every other chat handler honor it instead of streaming live with the faked
stream timeout. The replayed chunks keep the answer's own model name

fake_stream accepts only true or false, as a bool or a string, so values like
"yes", 1 or 1.0 get the invalid control option 400 instead of turning it on

filter_internal_params takes a typed Mapping and the Converse handler reads
stream as a bool

* fix: keep token logprobs on a replayed fake stream

The replayed chunk now carries each choice's logprobs from the plain answer,
so a fake_stream request with logprobs gets the same token probabilities a
live stream returned

* fix: keep provider headers, service tier and fingerprint on a replayed fake stream

The replayed stream now carries the plain answer's provider response headers, so
the proxy still returns llm_provider-* and x-ratelimit-* headers with fake_stream

Replayed chunks keep the answer's service_tier and system_fingerprint

Image generation and image edit drop deployment prompt-management params like
cache_control_injection_points, which Bedrock rejected as an invalid field

* fix: replay a text completion answer when fake_stream is set

text_completion, atext_completion and the Codestral completion route returned a raw TextCompletionResponse from the plain call, so the replay crashed with "'coroutine' object is not iterable" or an async-for error after the provider answered. The replay now turns that answer into a chat answer first and streams it like the other routes

* fix: keep every answer when a text completion stream holds several choices

A text completion stream converted only the first choice of each chunk, so a
fake_stream replay with n=2 lost the second answer. Each chunk now converts
every choice it carries

---------

Co-authored-by: tsushanth <78000697+tsushanth@users.noreply.github.com>
Co-authored-by: Chenglun Hu <chenglunhu@gmail.com>
Co-authored-by: kimnamu <jihoon.lucas.kim@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 22:20:05 -08:00
quietjules
0134008628
fix(responses): write one failure row per Responses WebSocket turn the deployment refuses (#50731)
* fix(responses): write one failure row per refused Responses WebSocket turn

A turn the deployment refuses on an open native or managed Responses WebSocket now writes its own spend
log failure row when it is refused, with its own request id, the turn's start time and request, the
socket's resolved model group and the websocket call type. Before, refused turns wrote nothing until the
socket closed, and then only one row for the whole socket, carrying the socket's open time and first frame.

Guardrail-blocked turns go through the same row builder, so their rows now carry the model group and call
type too. Turns refused or crashed in the turn check before a session-limit cancel also keep their rows.

The socket's close-time failure hooks still run once to settle the admission budget reservation, but when
the close failure is the first refused turn the spend logger skips the duplicate row.

* fix(responses): keep forged logging state out of refused-turn rows and wait for each row

A refused turn's failure row now leaves out the top-level fields the HTTP path strips from a client body, plus standard_callback_dynamic_params and litellm_params, so a frame cannot forge its own logging state or turn off message redaction. The relay now waits for each refused turn's row before it reads the next frame, and the write still finishes if the socket is cancelled meanwhile

* ci: start the unit workflow GitHub never created for the previous push

* fix(responses): keep the client's request tags on refused-turn rows

A refused turn's row now carries the key's tags plus the client's User-Agent tags, computed from the socket's request headers like its success rows, while the headers themselves stay out of the row

* fix(responses): keep a refused turn's frame from forging its spend row and store no blocked prompt

A refused turn's failure row now takes only the turn's model and call id from the frame, ignores frame metadata tags and model_group, and logs non-object metadata as empty instead of dropping the row. Guardrail-blocked frames no longer carry a stored request.

* fix(proxy): keep the socket's session id, guardrail records, and router-key strip on refused WebSocket turn rows

* fix(proxy): use the public router-reserved metadata field list, as main renamed it

* fix(responses): give refused WebSocket turn rows the socket's session id

When a socket's request carries no trace id, the router minted its own for the relayed turns, so success rows and refused-turn rows had different session ids. The endpoint now hands the router the socket's trace id

The refused-turn row test's session-limit case jumps the loop clock past the limit once the relay is parked, instead of racing a 50 ms timer

* test(responses): expect the parallel-slot refusal at the WebSocket handshake

Main now refuses a cached key that is over its limits before the request body is read, so a second socket on a key whose only parallel slot an open socket holds is refused at the upgrade instead of with a 429 frame. The test still checks that the refused turn's failure row keeps the slot held until the first socket closes

* fix(responses): take a refused WebSocket turn's agent and session fields from the socket

A frame's metadata.agent_id, billing_agent_id or session_id no longer lands on the refused turn's spend row. The row copies those fields from the socket, like its success rows

The integration test no longer pins the reopened socket's 1011 close, which is incidental to the held slot it checks

* test(responses): expect a 403 when a held parallel slot refuses a second WebSocket handshake

* test(responses): expect a held parallel slot to refuse a second socket's first turn with a 429 frame

Main's #51675 skips the headers-only limit check for a bodyless GET, and every WebSocket upgrade is one, so a second socket on a key whose only parallel slot is held now opens and its first turn gets a 429 max_parallel_requests frame and close 1008 instead of a 403 at the handshake

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 22:08:06 -08:00
joshua-berri
2314e5e7d1
fix(router): keep both errors when a streamed cross-group fallback also fails (#52217)
* fix(router): keep both errors when a streamed cross-group fallback also fails

On /v1/messages and /v1/responses streams, a recovery walk appended the
fallback outcome to the MidStreamFallbackError envelope and the stream then
raised the inner provider error without it. A fallback hop whose stream opened
and then failed before any content never reached the walk, so the primary's
error was dropped. The outcome now lands on the surfaced error, and a fallback
hop's pre-content failure reads as the walk's own failure would.

* fix(proxy): keep the router fallback outcome in Responses response.failed events

The response.failed formatter preferred the upstream body message, so the
primary error and fallback outcome the router put on the exception never
reached the client. Use the exception message when it carries that outcome.

* fix(router): only add the primary error to a fallback stream's failure before any content

Count any chunk past message_start and pings, split or not, as content on
/v1/messages, track forwarded output events on Responses, and switch the
response.failed message only on a fallback-also-failed outcome.

* fix(router): scope the streamed fallback outcome to pre-content messages and Responses failures

Leave the router core and chat streams as they were. Carry the outcome the
walk put on the stream's envelope onto the provider error only before any
content, and read a fallback hop's /v1/messages stream as whole SSE frames so
split or merged chunks are judged by event, not by chunk.

* fix(router): keep a Responses stream's own error once it has sent output

A primary Responses stream that already sent output no longer lets its
fallback stream's failure read as the primary's. Keep CRLF line endings in a
rewritten error frame, and read a bridge chunk's type through a typed mapping.

* test(router): cover partial frames on a fallback stream's end and raise

* fix(router): carry the primary error through a fallback stream's retry and redact the combined frame

A fallback group's same-group stream retry keeps the fallback's error context,
every fallback-level /v1/messages stream is read as whole frames, and the
rewritten error frame goes through the client message redactor.

* fix(router): treat a cut-off message_start on a fallback stream as before content

* fix(router): redact the primary error and the fallback outcome separately before joining them

A traceback in the primary's error made the client redactor drop the fallback
outcome that followed it. Make the fallback-retry tests deterministic with a
single-deployment fallback group.

* test(router): cover a split error frame on a fallback group's retried stream

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 22:04:16 -08:00
Vansh Singh
c2c021c8ca
fix(copilot): accept empty or whitespace stored key with key ref (#52014) (#52032)
* fix(copilot): accept empty or whitespace stored key with key ref (#52014)

* fix(copilot): treat blank GitHub App fields as unset and keep other values fail-closed

Booleans and other non-string App field values stay configured so a
non-App request carrying them still fails closed, and a non-string key
ref no longer passes as a blank one

* fix(copilot): never store blank GitHub App fields on a credential

POST /credentials drops a blank or whitespace-only App field instead of
storing it, and PATCH treats one as a deletion. A stored blank key came
back masked as ***** and the dashboard then edited it as an uploaded key

* fix(copilot): keep non-admin credential creates with blank App fields forbidden

* test(copilot): pin that a null App key on update keeps the stored key

* test(copilot): pin that a false or 0 key ref next to an uploaded key fails closed

* fix(copilot): keep signing a whitespace-only key that has no key ref

A named credential with a whitespace-only App key and no key ref answered
400 at the head, which the router never retries. The merge base reached JWT
signing and answered 401, so a model group with a working sibling kept
serving. Whitespace still counts as unset next to a key ref.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 22:03:15 -08:00
joshua-berri
21746eb287
fix(router): honor fallback opt-out for 413 and 422 bad requests (#52195)
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:51:23 -08:00
moe-berri
1efd8b38d3
feat(proxy): support Cursor CLI subscriptions (#52205)
* feat(proxy): relay Cursor CLI subscription sessions

Adapt dm3ch's native Cursor relay to current main, preserving separate gateway authentication, opaque streams, and policy admission checks.

* fix(proxy): enforce Cursor relay policies and preserve stream accounting

* fix(proxy): allow logging callbacks on Cursor relay

---------

Co-authored-by: Dmitry Chepurovskiy <me@dm3ch.net>
2026-11-16 21:49:12 -08:00
ryan-crabbe-berri
9a61ebd087
fix(gemini): stop replaying a client-mangled thought signature from a tool call id (#52207)
* test(gemini): cover client-mangled thought signatures in tool call ids

* fix(gemini): ignore a client-mangled thought signature in a tool call id

Only replay the signature embedded in a tool call id when it is still valid base64. Clients that strip or replace +, / and = in ids now get the existing placeholder fallback instead of a Vertex 400 on every later turn. Fixes #52001.
2026-11-16 21:39:16 -08:00
joshua-berri
9e81aa3f40
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served

A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.

Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.

The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.

* test(shadow-eval): drive same-model skip tests off explicit completion signals

Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.

Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.

* style(tests): remove extra shadow evaluation test spacing

* fix(ui): send skip_same_model false from the shadow eval start form

The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.

* fix(migrations): bound shadow evaluation schema lock waits

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:38:00 -08:00
joshua-berri
87c94e58b6
feat(router): select conversation headers for session affinity (#52193)
* feat(router): select conversation headers for session affinity

* test(router): assert legacy session routing results

* fix(router): validate conversation request fields at typed boundaries

* fix(router): synchronize conversation header API types

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:37:48 -08:00
tin-berri
adcaa4d374
fix(skills): preserve legacy Skills Hub capabilities (#51977) 2026-11-16 21:21:52 -08:00
Kerry Lu
e46fa61681
fix(cost-map): declare mid-conversation system support on Databricks Claude Opus 5.5 (#52214)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 05:09:07 +00:00
yujonglee
c25c669167
feat(auth): decide key and JWT authentication in a Rust core (#51374)
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.

The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.

Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.

A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.

The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
2026-11-17 14:07:47 +09:00
paveln83
ea96ea3db5
fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output (#52116)
* fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output

* fix(anthropic): close rebuilt-stream thinking blocks with a signature_delta

* fix(anthropic): drop unsigned thinking before Claude, stream raw reasoning_text, close rebuilt thinking blocks with signature_delta

A Claude target on /v1/messages now gets unsigned thinking blocks left out of the request, so a switch from a reasoning_content model makes one call and keeps thinking, except when that would leave the final tool turn without its opening thinking block

Raw response.reasoning_text deltas from the Responses bridge stream as thinking, and a stream rebuilt from a full answer or rewritten by a guardrail closes each thinking block with a signature_delta, the real signature or an empty string

* fix(anthropic): stream a whole reasoning item as a closed thinking block

A Responses stream can send a reasoning item whole in output_item.added and output_item.done with no text deltas, as litellm's own stream does for gpt-oss on OpenRouter. The /v1/messages stream now builds that item's thinking block with the same helper the non-streamed response uses, so streamed and non-streamed answers carry the same reasoning

* test(anthropic): mark the new streaming test locals Final

* fix(anthropic): drop unsigned thinking before every Claude call, a pending tool turn included

Anthropic answers 400 to a thinking block with an empty signature but accepts a
tool turn with no thinking, so the strip no longer keeps the block that opens the
pending tool turn. The check now looks for "claude" in the model name, which covers
Claude behind every messages route and the chat adapter, not only four providers

* fix(anthropic): strip unsigned thinking when a deployment's base_model names Claude

* fix(anthropic): drop unsigned thinking for an alias whose base_model names Claude on the native messages path

The chat adapter already read base_model before leaving unsigned thinking out of a Claude request. The native messages transform and the Rust request transform now read it too, so an aliased Claude deployment gets one call instead of a 400 and a retry

* test(anthropic): build the pending tool turn without mutating it in the Rust strip test

The Rust test now builds its expected messages from the same helper instead of editing them afterward, and a doc comment that only restated a helper's name is gone

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 20:44:27 -08:00
ryan-crabbe-berri
fea939843d
fix(gemini): keep Google's reason on non-streamed content_filter choices (#52208)
* test(gemini): cover content_filter reason on non-streamed chat completions

* fix(gemini): keep Google's reason on non-streamed content_filter choices

* refactor(gemini): take content_filter reason as a read-only mapping
2026-11-16 20:38:14 -08:00
maciej-berri
adaa460566
fix(messages): load the anthropic beta map only when a filter has a beta to filter (#52188)
* fix(messages): load the anthropic beta map only when a filter has a beta to filter

Since #52042 the Rust messages route asked Python for the beta map on every
azure_ai, Bedrock Invoke and Foundry call while decoding the request, so the
first such call on each worker downloaded the remote map even when it sent no
anthropic-beta, and the download ran synchronously on the request path.

The bridge now installs the map only once Python has loaded it. When a filter
first meets a beta and no map is installed, it asks the host to load it on the
loader thread and uses the compiled table for that call, as before #52042.

* fix(messages): start at most one background beta map load at a time

A burst of beta-carrying calls while the first download was still in flight
queued one load task per call on the loader thread. A non-blocking lock now
lets only one load wait in the queue, and it is released when that load ends.
2026-11-16 20:03:08 -08:00
Kerry Lu
8b0bb6ae42
fix(bedrock): route bedrock/<region>/<model> to that region on /v1/messages (#52136)
* fix(bedrock): route bedrock/<region>/<model> to that region on the Invoke path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): keep LIT010 flat and update the invoke region-path integration test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(bedrock): drop the comment that repeats the helper name

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 19:58:53 -08:00
ryan-crabbe-berri
c37bd7532f
fix(spend-logs)!: match key alias filter exactly through the api_key index (#47795)
* fix(spend-logs): match key aliases exactly

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(spend-logs): drop unused key alias ORM lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate exact key alias API descriptions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend-logs): keep key alias filter matching CLI session logs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 19:56:16 -08:00
devin-ai-integration[bot]
f7902877ee
test(security): run the owned security proxy in offline mode so the ECB rate lookup never leaves the rig (#52202)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 03:37:16 +00:00
Stanley Zhou
326702cc94
fix(gemini): parse tool call arguments with the shared helper (#50020)
Gemini tool call arguments now go through parse_tool_call_arguments, the shared helper the other providers use, so malformed or empty argument strings are repaired or rejected the same way instead of raising a raw JSONDecodeError
2026-11-16 19:31:25 -08:00
devin-ai-integration[bot]
5008a9a11d
test(providers): expect batch alias rows rewritten to the deployment model on model-routed uploads (#52199)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 03:28:04 +00:00
moe-berri
dc9bbb762f
fix(complexity_router): classify short asks in long conversations (#52183)
* fix(complexity_router): classify short asks in long conversations

* fix(complexity_router): validate active context limits without recursion

* test(rag): wait for keepalive frames before releasing retrieval

* test(ui): control availability debounce in context-limit regressions
2026-11-16 19:16:10 -08:00
ryan-crabbe-berri
d2664e6b64
fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers (#52179)
* test(providers): cover upstream 403 from OpenAI-compatible providers as permission denied and not retried

* fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers
2026-11-17 03:14:09 +00:00
zoveki
11cd3c3f7f
fix(bedrock): send guarded_text parts in system and assistant messages as Converse guardContent (#51776)
* fix(bedrock): send guarded_text parts in system and assistant messages as Converse guardContent

* fix(bedrock): drop guarded_text parts on the Nova invoke route under drop_params

* fix(bedrock): send guarded_text parts as text under drop_params when the request cannot carry guardContent

* fix(bedrock): trim a final guarded assistant turn and honor batch drop_params for guarded parts

Claude on Converse rejects a final assistant turn that ends in whitespace,
and the trim covered only text blocks, so a guarded_text prefill ending in
whitespace now answered 400. The trim now covers guardContent text too,
which also covers the drop_params text fallback that runs after it.

Nova batch records built their request without the record's drop_params,
so a guarded system part stayed guardContent, which Nova InvokeModel
rejects. The record's drop_params now reaches the transform.

The new Nova tests split a nested generator and gain type annotations.

* fix(bedrock): keep the full guardrail input check and reject malformed content parts

A guarded assistant or mid-conversation system part goes as text under guardrailConfig when no user turn is guarded, since guardContent there would narrow Bedrock's input check to that part. A system or guarded_text part whose text is not a string now gets a 400 instead of being dropped.

* fix(bedrock): keep the full guardrail input check when a guarded system message ends the request

A guarded system message after the last assistant turn or tool result became a trailing user turn that the guardrail auto-wrap then guarded alone, so Bedrock skipped the user's text. The rule now reads the trailing turns after the system conversion and leaves them unguarded when nothing else would be. A user guarded_text part whose text is not a string now gets litellm's 400 that names it.

* fix(bedrock): keep guarded parts guarded when tool history without tools gets guarded

With tool history and no tools param, litellm rewrites each tool result into a guarded user turn after the input check decision ran, so a guarded system part sent as text escaped the check. The decision now keeps the part guarded when that rewrite will guard a tool result.

* fix(bedrock): keep the full guardrail input check after an unanswered tool call without tools

* fix(bedrock): guard trailing turns only when a user turn is among them

A plain system message that ends a request after an assistant turn or a tool result
became a user turn, and the trailing guardrail wrap guarded it alone, so Bedrock
checked only the system text and a blocked word in an earlier user turn got through

* test(bedrock): cover a function result without tools keeping a guarded part guarded

* test(bedrock): expect the full input check for a guarded assistant part before a trailing system turn

A trailing system turn alone no longer gets the trailing guard wrap, so a guarded assistant part before it goes as
text whatever the system text holds, and Bedrock checks every message

* fix(bedrock): keep the full guardrail input check when tool history comes without tools

* fix(bedrock): keep guarded assistant text guarded when its tool calls turn into text

Without a tools param, an assistant turn with tool calls is flattened to text. When a guarded user turn makes the guardrail check only guarded blocks, that flattening also turned the assistant's guarded_text parts into plain text, so the guardrail skipped them. In that case keep those parts guarded and append the tool-call text as its own text part. Everywhere else the turn is flattened as before

* fix(bedrock): keep a user question before rewritten tool results in the guard wrap

With tool history and no tools param, litellm rewrites each tool result as a user turn. The trailing guard wrap stopped at the tool result, so a plain user question right before it went as plain text beside guarded blocks and its blocked word skipped the guardrail. Rewritten tool results now count in the trailing run, which matches the merge base

* fix(bedrock): keep guarding a trailing system message next to a caller-guarded user turn

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 19:11:35 -08:00
joshua-berri
43d6b45cb7
fix(bedrock)!: split the OpenAI breakpoint and Bedrock cachePoint model flags (#52180)
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags

supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint
format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a
Bedrock Claude deployment silently turned off its prompt caching.

Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching
as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old
key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a
Bedrock deployment still on the old key logs a one-time migration warning.

Fixes #52140

* fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models

* test(bedrock): mark the system prompt directly in the routed cache point regression

* fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides

* fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog

* test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 18:50:32 -08:00