Commit graph

2377 commits

Author SHA1 Message Date
mateo-berri
f9010a1c45 fix(auth): type the 403 by the key's grant and check auto-router previews as their team 2026-11-17 03:41:42 -08:00
mateo-berri
671ebeac68 fix(auth): keep empty access groups so a fully foreign group grants nothing
A group whose members all belong to other teams was dropped from the
team-filtered map, so its name leaked into /v1/models as a plain model.
Keep the group with no members, and treat a grant that expands to
nothing as no-default-models instead of unrestricted. All-team-models
team keys get team_model_access_denied again, as on main.
2026-11-16 23:52:29 -08:00
mateo-berri
305fe443b5 Merge remote-tracking branch 'origin/main' into litellm_access_group_team_scope
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	litellm/proxy/auth/model_checks.py
#	litellm/proxy/auth/user_api_key_auth.py
#	litellm/proxy/proxy_server.py
#	litellm/proxy/utils.py
#	tests/unit/llms/anthropic/messages/test_safeguards.py
#	tests/unit/proxy/auth/test_model_checks.py
#	tests/unit/proxy/proxy_server/test_team_model_name_translation.py
#	tests/unit/proxy/utils/helpers/test_model_access.py
2026-11-16 22:41:06 -08:00
joshua-berri
21746eb287
fix(router): honor fallback opt-out for 413 and 422 bad requests (#52195)
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:51:23 -08:00
moe-berri
1efd8b38d3
feat(proxy): support Cursor CLI subscriptions (#52205)
* feat(proxy): relay Cursor CLI subscription sessions

Adapt dm3ch's native Cursor relay to current main, preserving separate gateway authentication, opaque streams, and policy admission checks.

* fix(proxy): enforce Cursor relay policies and preserve stream accounting

* fix(proxy): allow logging callbacks on Cursor relay

---------

Co-authored-by: Dmitry Chepurovskiy <me@dm3ch.net>
2026-11-16 21:49:12 -08:00
ryan-crabbe-berri
9a61ebd087
fix(gemini): stop replaying a client-mangled thought signature from a tool call id (#52207)
* test(gemini): cover client-mangled thought signatures in tool call ids

* fix(gemini): ignore a client-mangled thought signature in a tool call id

Only replay the signature embedded in a tool call id when it is still valid base64. Clients that strip or replace +, / and = in ids now get the existing placeholder fallback instead of a Vertex 400 on every later turn. Fixes #52001.
2026-11-16 21:39:16 -08:00
joshua-berri
9e81aa3f40
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served

A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.

Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.

The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.

* test(shadow-eval): drive same-model skip tests off explicit completion signals

Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.

Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.

* style(tests): remove extra shadow evaluation test spacing

* fix(ui): send skip_same_model false from the shadow eval start form

The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.

* fix(migrations): bound shadow evaluation schema lock waits

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:38:00 -08:00
joshua-berri
87c94e58b6
feat(router): select conversation headers for session affinity (#52193)
* feat(router): select conversation headers for session affinity

* test(router): assert legacy session routing results

* fix(router): validate conversation request fields at typed boundaries

* fix(router): synchronize conversation header API types

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:37:48 -08:00
tin-berri
adcaa4d374
fix(skills): preserve legacy Skills Hub capabilities (#51977) 2026-11-16 21:21:52 -08:00
Kerry Lu
e46fa61681
fix(cost-map): declare mid-conversation system support on Databricks Claude Opus 5.5 (#52214)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 05:09:07 +00:00
yujonglee
c25c669167
feat(auth): decide key and JWT authentication in a Rust core (#51374)
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.

The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.

Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.

A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.

The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
2026-11-17 14:07:47 +09:00
paveln83
ea96ea3db5
fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output (#52116)
* fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output

* fix(anthropic): close rebuilt-stream thinking blocks with a signature_delta

* fix(anthropic): drop unsigned thinking before Claude, stream raw reasoning_text, close rebuilt thinking blocks with signature_delta

A Claude target on /v1/messages now gets unsigned thinking blocks left out of the request, so a switch from a reasoning_content model makes one call and keeps thinking, except when that would leave the final tool turn without its opening thinking block

Raw response.reasoning_text deltas from the Responses bridge stream as thinking, and a stream rebuilt from a full answer or rewritten by a guardrail closes each thinking block with a signature_delta, the real signature or an empty string

* fix(anthropic): stream a whole reasoning item as a closed thinking block

A Responses stream can send a reasoning item whole in output_item.added and output_item.done with no text deltas, as litellm's own stream does for gpt-oss on OpenRouter. The /v1/messages stream now builds that item's thinking block with the same helper the non-streamed response uses, so streamed and non-streamed answers carry the same reasoning

* test(anthropic): mark the new streaming test locals Final

* fix(anthropic): drop unsigned thinking before every Claude call, a pending tool turn included

Anthropic answers 400 to a thinking block with an empty signature but accepts a
tool turn with no thinking, so the strip no longer keeps the block that opens the
pending tool turn. The check now looks for "claude" in the model name, which covers
Claude behind every messages route and the chat adapter, not only four providers

* fix(anthropic): strip unsigned thinking when a deployment's base_model names Claude

* fix(anthropic): drop unsigned thinking for an alias whose base_model names Claude on the native messages path

The chat adapter already read base_model before leaving unsigned thinking out of a Claude request. The native messages transform and the Rust request transform now read it too, so an aliased Claude deployment gets one call instead of a 400 and a retry

* test(anthropic): build the pending tool turn without mutating it in the Rust strip test

The Rust test now builds its expected messages from the same helper instead of editing them afterward, and a doc comment that only restated a helper's name is gone

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 20:44:27 -08:00
ryan-crabbe-berri
fea939843d
fix(gemini): keep Google's reason on non-streamed content_filter choices (#52208)
* test(gemini): cover content_filter reason on non-streamed chat completions

* fix(gemini): keep Google's reason on non-streamed content_filter choices

* refactor(gemini): take content_filter reason as a read-only mapping
2026-11-16 20:38:14 -08:00
maciej-berri
adaa460566
fix(messages): load the anthropic beta map only when a filter has a beta to filter (#52188)
* fix(messages): load the anthropic beta map only when a filter has a beta to filter

Since #52042 the Rust messages route asked Python for the beta map on every
azure_ai, Bedrock Invoke and Foundry call while decoding the request, so the
first such call on each worker downloaded the remote map even when it sent no
anthropic-beta, and the download ran synchronously on the request path.

The bridge now installs the map only once Python has loaded it. When a filter
first meets a beta and no map is installed, it asks the host to load it on the
loader thread and uses the compiled table for that call, as before #52042.

* fix(messages): start at most one background beta map load at a time

A burst of beta-carrying calls while the first download was still in flight
queued one load task per call on the loader thread. A non-blocking lock now
lets only one load wait in the queue, and it is released when that load ends.
2026-11-16 20:03:08 -08:00
Kerry Lu
8b0bb6ae42
fix(bedrock): route bedrock/<region>/<model> to that region on /v1/messages (#52136)
* fix(bedrock): route bedrock/<region>/<model> to that region on the Invoke path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): keep LIT010 flat and update the invoke region-path integration test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(bedrock): drop the comment that repeats the helper name

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 19:58:53 -08:00
ryan-crabbe-berri
c37bd7532f
fix(spend-logs)!: match key alias filter exactly through the api_key index (#47795)
* fix(spend-logs): match key aliases exactly

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(spend-logs): drop unused key alias ORM lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate exact key alias API descriptions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend-logs): keep key alias filter matching CLI session logs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 19:56:16 -08:00
Stanley Zhou
326702cc94
fix(gemini): parse tool call arguments with the shared helper (#50020)
Gemini tool call arguments now go through parse_tool_call_arguments, the shared helper the other providers use, so malformed or empty argument strings are repaired or rejected the same way instead of raising a raw JSONDecodeError
2026-11-16 19:31:25 -08:00
moe-berri
dc9bbb762f
fix(complexity_router): classify short asks in long conversations (#52183)
* fix(complexity_router): classify short asks in long conversations

* fix(complexity_router): validate active context limits without recursion

* test(rag): wait for keepalive frames before releasing retrieval

* test(ui): control availability debounce in context-limit regressions
2026-11-16 19:16:10 -08:00
ryan-crabbe-berri
d2664e6b64
fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers (#52179)
* test(providers): cover upstream 403 from OpenAI-compatible providers as permission denied and not retried

* fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers
2026-11-17 03:14:09 +00:00
zoveki
11cd3c3f7f
fix(bedrock): send guarded_text parts in system and assistant messages as Converse guardContent (#51776)
* fix(bedrock): send guarded_text parts in system and assistant messages as Converse guardContent

* fix(bedrock): drop guarded_text parts on the Nova invoke route under drop_params

* fix(bedrock): send guarded_text parts as text under drop_params when the request cannot carry guardContent

* fix(bedrock): trim a final guarded assistant turn and honor batch drop_params for guarded parts

Claude on Converse rejects a final assistant turn that ends in whitespace,
and the trim covered only text blocks, so a guarded_text prefill ending in
whitespace now answered 400. The trim now covers guardContent text too,
which also covers the drop_params text fallback that runs after it.

Nova batch records built their request without the record's drop_params,
so a guarded system part stayed guardContent, which Nova InvokeModel
rejects. The record's drop_params now reaches the transform.

The new Nova tests split a nested generator and gain type annotations.

* fix(bedrock): keep the full guardrail input check and reject malformed content parts

A guarded assistant or mid-conversation system part goes as text under guardrailConfig when no user turn is guarded, since guardContent there would narrow Bedrock's input check to that part. A system or guarded_text part whose text is not a string now gets a 400 instead of being dropped.

* fix(bedrock): keep the full guardrail input check when a guarded system message ends the request

A guarded system message after the last assistant turn or tool result became a trailing user turn that the guardrail auto-wrap then guarded alone, so Bedrock skipped the user's text. The rule now reads the trailing turns after the system conversion and leaves them unguarded when nothing else would be. A user guarded_text part whose text is not a string now gets litellm's 400 that names it.

* fix(bedrock): keep guarded parts guarded when tool history without tools gets guarded

With tool history and no tools param, litellm rewrites each tool result into a guarded user turn after the input check decision ran, so a guarded system part sent as text escaped the check. The decision now keeps the part guarded when that rewrite will guard a tool result.

* fix(bedrock): keep the full guardrail input check after an unanswered tool call without tools

* fix(bedrock): guard trailing turns only when a user turn is among them

A plain system message that ends a request after an assistant turn or a tool result
became a user turn, and the trailing guardrail wrap guarded it alone, so Bedrock
checked only the system text and a blocked word in an earlier user turn got through

* test(bedrock): cover a function result without tools keeping a guarded part guarded

* test(bedrock): expect the full input check for a guarded assistant part before a trailing system turn

A trailing system turn alone no longer gets the trailing guard wrap, so a guarded assistant part before it goes as
text whatever the system text holds, and Bedrock checks every message

* fix(bedrock): keep the full guardrail input check when tool history comes without tools

* fix(bedrock): keep guarded assistant text guarded when its tool calls turn into text

Without a tools param, an assistant turn with tool calls is flattened to text. When a guarded user turn makes the guardrail check only guarded blocks, that flattening also turned the assistant's guarded_text parts into plain text, so the guardrail skipped them. In that case keep those parts guarded and append the tool-call text as its own text part. Everywhere else the turn is flattened as before

* fix(bedrock): keep a user question before rewritten tool results in the guard wrap

With tool history and no tools param, litellm rewrites each tool result as a user turn. The trailing guard wrap stopped at the tool result, so a plain user question right before it went as plain text beside guarded blocks and its blocked word skipped the guardrail. Rewritten tool results now count in the trailing run, which matches the merge base

* fix(bedrock): keep guarding a trailing system message next to a caller-guarded user turn

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 19:11:35 -08:00
joshua-berri
43d6b45cb7
fix(bedrock)!: split the OpenAI breakpoint and Bedrock cachePoint model flags (#52180)
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags

supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint
format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a
Bedrock Claude deployment silently turned off its prompt caching.

Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching
as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old
key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a
Bedrock deployment still on the old key logs a one-time migration warning.

Fixes #52140

* fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models

* test(bedrock): mark the system prompt directly in the routed cache point regression

* fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides

* fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog

* test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 18:50:32 -08:00
ishaan-j-berri
b46fc82e5e
feat(harness): add typed agent evaluations (#52143) (#52174) 2026-11-17 02:35:52 +00:00
Mateo Wang
a41f124bdc
refactor: share Replicate HTTP client factories and declare PagerDuty type alias (#51594)
* refactor: deduplicate Replicate clients and declare PagerDuty type alias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover shared Replicate HTTP client factories

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: exercise real Replicate HTTP client factories

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-17 02:32:54 +00:00
yuneng-jiang
7f7bb13707
fix(auth): apply current team limits in the pre-body rate limit check (#52169)
* fix(auth): apply current team limits in the pre-body rate limit check

The check that refuses an exhausted key before reading the request body
used the team limits copied onto the cached key. Raising a team's rpm or
tpm limit with /team/update did not reach that copy, so a throttled key
stayed throttled until the key cache expired, and the full auth path that
does refresh team fields never ran.

The check now overlays the cached team's newer fields the same way full
auth does, through one shared helper. When the team is not in the cache,
as on a peer right after an invalidation, it defers to full auth instead
of judging the key on stale limits.

* fix(auth): keep key limits in the early check when the team is not cached

On a team cache miss the early check skipped every pre-body limit, so an
exhausted key read its body before being refused. Apply the key's own
limits and defer to full auth only when the cached team limit refuses.
2026-11-16 18:24:43 -08:00
tin-berri
f33dcfd1dd
feat(ui): add bulk actions for virtual keys (#52138) 2026-11-16 17:48:05 -08:00
jesus-berri
43fc7d5cd3
fix(mcp): keep an MCP server's own Authorization key when the JWT signer is on (#51867)
* fix(mcp): keep an MCP server's own Authorization key when the JWT signer is on

With mcp_jwt_signer enabled, a server whose configured authentication_token is sent in
Authorization (bearer_token, token, authorization, basic) had that key replaced by the
signer's JWT on tools/list and tools/call, so the upstream returned 401. Both signer guards
now also treat the server's own key as occupying Authorization, using the same
to_server_spec mapping that sends the key upstream.

Resolves LIT-10782
Fixes #47988

Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>

* fix(mcp): keep a deferred oauth2-delegate server's own token when the JWT signer is on

Servers that defer to the v1 client, such as oauth2 with delegate_auth_to_upstream,
send their authentication_token in Authorization (or upstream_token_header), but the
signer guard only read the v2 spec and let the gateway JWT overwrite it. The guard
now falls back to v1's own header rule when to_server_spec returns None. BYOK keeps
its existing behavior.

* test(mcp): cover a SigV4 server's token leaving Authorization to the JWT signer

* fix(mcp): only let upstream_token_header claim Authorization for auth types v1 writes a key for

A SigV4 server with a stray authentication_token and upstream_token_header
set to Authorization lost the signer JWT, though v1 never sends that token.
The v1 rule now resolves the header v1 would write the key to, or none, before
applying the override.

* fix(mcp): keep the signer JWT on tools/call when the caller's auth header displaces the server key

A per-request MCP auth header replaces the server's authentication_token, so
the configured key is not what reaches Authorization. The tools/call guard now
only consults the server's configured key when the caller sent none, matching
the tools/list guard, so the upstream no longer ends up with no Authorization.

---------

Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>
2026-11-17 01:47:56 +00:00
Kerry Lu
fe3e6bbb01
fix(mcp): let zero-budget users call MCP tools that cost nothing (#52130)
* fix(mcp): let zero-budget users call MCP tools that cost nothing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): price a tool call made by its display name as the tool it runs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep budget checks for bare display names a paid server can run and for cut non-tool-call bodies

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): budget check reserved tool names and display names that resolve on to a paid tool

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mcp): follow chained display names without recursion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): price a scoped tool call on the servers it is scoped to and follow display names as execution does

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep the budget check for MCP servers that use display names

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 17:44:44 -08:00
kenneth-berri
49b2058c1a
fix(bedrock): recognize the ca. cross-region inference profile prefix (#52152)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 16:53:51 -08:00
zoveki
b45fae77cd
fix(azure_ai): send reasoning_effort to Foundry reasoning models (#51775)
* fix(azure_ai): send reasoning_effort to Foundry reasoning models

Foundry models outside the GPT-5 family that the cost map marks as reasoning
models, like DeepSeek V4, now accept reasoning_effort. A level the model row
declares is sent as is. A level it does not declare is dropped under
drop_params and gets a 400 otherwise. Declares the levels DeepSeek V4 Flash
and Pro accept on Foundry

* fix(azure_ai): send reasoning_effort only to Foundry rows that declare levels

Gate reasoning_effort on the cost map's reasoning_effort_levels instead of
supports_reasoning, so undeclared reasoning rows keep the drop_params behavior
they had before. Look the levels up case-insensitively, keep a dropped effort
out of the body when allowed_openai_params lists it, model the level check as
values, and name the proxy drop_params setting in the 400

* fix(azure_ai): read Foundry reasoning levels from the loaded catalog row and honor allowed_openai_params

Levels now come only from the model's own azure_ai catalog row, checked for the deployment's base_model first, so a sibling deployment's model_info or a bare openai row no longer decides them. A request that allows reasoning_effort sends any level, as before this PR

* fix(azure_ai): honor register_model reasoning levels on Foundry rows

The level check read only the loaded cost map, so a model registered with
litellm.register_model lost its declared reasoning_effort_levels. Read the
loaded map with register_model registrations on top, still without the
router's per-deployment registrations, so sibling deployments stay separate

* fix(azure_ai): leave default and auto to the model and keep rows without levels unchanged

default is stripped even when allowed_openai_params skips the level check, since Foundry refuses it on DeepSeek V4
auto leaves the level to the model, the same way the Fireworks config treats it, so Kimi K3 auto keeps working
A model whose catalog row lists no levels gets reasoning_effort exactly as it was sent
Updated the Kimi K3 level tests to the seven-level Foundry row, and the catalog-only test to the level check

* fix(router): drop a classifier reasoning_effort level the deployment refuses

The complexity router's classifier kept reasoning_effort once azure_ai listed it, so a level Foundry's row refuses failed the classifier call and sent every request to the capable tier. The classifier now drops a level the deployment's provider config refuses, unless the deployment declares its own levels or allows reasoning_effort

* fix(azure_ai): keep the catalog row's reasoning levels under a registration spelled in another case

A price-only register_model call under azure_ai/fw-kimi-k3 hid the azure_ai/FW-Kimi-K3 row, so its levels were lost and reasoning_effort answered 400 or was dropped
The level lookup now merges the catalog row under the registration, the way register_model itself merges
The Foundry Kimi K3 level test cites the live check behind its seven levels

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 16:40:16 -08:00
moe-berri
dd7d77fea1
fix(opencode_go): forward chat reasoning effort (#52156)
* test(opencode_go): cover chat reasoning effort forwarding

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode_go): forward chat reasoning effort for reasoning models

Preserve reasoning_effort through chat parameter validation. Keep models without declared reasoning support unchanged. Related contributor change in #51879.

Co-authored-by: PCTEJA <39472754+PCTEJA@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(opencode_go): tighten capability and translation assertions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: PCTEJA <39472754+PCTEJA@users.noreply.github.com>
2026-11-16 16:31:56 -08:00
yucheng-berri
0e61e99120
feat(mcp): add Parallel to the MCP discovery registry (#52141)
* feat(mcp): add Parallel Search to discovery registry

* fix(mcp): point the Parallel catalog entry at mcp.parallel.ai

Parallel's docs call search.parallel.ai/mcp legacy and say new installs should use mcp.parallel.ai/mcp, which keeps web_search and web_fetch and stays free without a key

* test(mcp): cite the Parallel docs beside the pinned catalog URL

---------

Co-authored-by: georgeatparallel <george@parallel.ai>
2026-11-16 16:30:58 -08:00
kenneth-berri
4e9c728d38
fix(bedrock): log the prompt, settings and stream flag of passthrough requests (#52146)
* fix(bedrock): log the prompt, settings and stream flag of passthrough requests

* refactor: keep passthrough request logging within the lint gates

* fix: keep the logging object deep-copyable

* fix(bedrock): log Anthropic tool exchanges and skip guarded blocks without text
2026-11-16 16:10:52 -08:00
kenneth-berri
a14d17822b
fix(caching): log a cache hit with the deployment's vertex_location (#52150)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 16:10:30 -08:00
joshua-berri
f94c79290c
feat(ui): show an optional currency estimate on the Usage spend card (#51979)
* feat(ui): show the usage spend total in another currency at ECB reference rates

Adds GET /get/reference_exchange_rates, which returns the current ECB
reference rates as USD per unit for every feed currency. It refreshes at
most once a day under a lock, keeps fresh rates when a refresh fails and
answers 503 instead of serving rates older than seven days.

The usage overview's spend card gets a Spend display selector. It asks for
rates only once opened and then shows an estimated amount with the rate date
under the unchanged USD total. Every other figure stays in USD.

* fix(ui): fall back to USD when spend display rates stop being usable

The selector now shows USD whenever no estimate can be shown, so a failed
refetch never leaves EUR selected beside a USD-only notice, and an opened
selector refreshes its rates hourly. The endpoint gets its HTTP client
inside the async fetch so the client belongs to the serving event loop.

* fix(proxy): import the ECB rates endpoint HTTP client helper at module level

Binds get_async_httpx_client at import so its type is known. The client is
still built inside the awaited fetch on the serving event loop.

* fix(proxy): fetch display rates with a short-lived handler closed after each refresh

Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, and drops the descriptive comment on the spend display component.

* chore(ui): regenerate API types for the reference exchange rates endpoint

* fix(proxy): use the cached async client factory for display rate refreshes

Direct AsyncHTTPHandler construction fails the async client check, so the
refresh fetch goes back to get_async_httpx_client, called inside the awaited
fetch, with a named type suppression for the factory's untyped params.

* fix(ui): tidy spend display helper, hook options and fallback test

Groups the rate snapshot into one estimatedSpend argument, names the rates
query options, names the missing-estimate condition, and waits for the
unavailable notice before asserting the USD fallback.

* chore(ui): drop a restating comment on the spend estimate helper

* test(proxy): prove concurrent first rate requests share one ECB fetch

* fix(proxy): wait a minute before retrying a failed display rate refresh

A failed ECB refresh left rates due, so every request after it waited on
the lock and fetched again, and N queued callers paid N timeouts during an
outage. The endpoint now skips refreshing for 60 seconds after a failed
attempt, serving held rates or a 503, and retries on the first request
after that. Concurrent waiters share the failed attempt. Also drops the
usd_rates docstring.

* fix(ui): retry unavailable currency estimates on reopen

* refactor(ui): name the currency selector open handler

* style(ui): format currency selector props

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 16:00:20 -08:00
mika-lt
530a5094be
fix(router)!: stop a client's metadata flag from turning off silent_model mirroring (#52083)
* fix(router)!: stop a client's metadata flag from turning off silent_model mirroring

A request that sent metadata.is_silent_experiment true skipped every mirror the
deployment's silent_model configured. The check was a recursion guard from when
mirroring first shipped. The router now stops a mirror from mirroring again on
its own, so the metadata check only ever acted on client input

BREAKING CHANGE: clients can no longer opt out of an admin's traffic mirroring by
sending is_silent_experiment in request metadata

* test(router): assert a caller-sent silent experiment flag still mirrors in the router core tests

The two core tests pinned the old opt-out, where a request that sent
metadata.is_silent_experiment skipped the deployment's silent_model

* test(router): check a mirrored call is never mirrored again on the sync path

* test(router): name why the sync mirror test suppresses the endpoint attribute check

* ci: start the zizmor check GitHub skipped at the last push

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 15:45:56 -08:00
Mateo Wang
8fadc09434
feat(focus): upgrade FOCUS export to the 1.5 Cost and Usage schema (#52105)
* feat(focus): upgrade cost and usage export to FOCUS 1.5

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(focus): identify the combined daily request SKU meter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 15:43:15 -08:00
joshua-berri
5792794691
feat(alerts): report observed Anthropic cache misses as advisories (#51973)
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache

Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.

Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.

Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.

* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration

Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.

After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.

Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.

* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading

Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.

A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.

* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset

* fix(alerts): describe prompt cache alerts as advisory and type their boundaries

Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.

* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof

A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.

* fix(alerts): serialize each message's fields once in the prompt cache prefix

Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.

* test(proxy): keep a leaked worker config out of the prompt cache lifespan test

An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.

* test(alerts): clean each prompt cache deployment tag in its own comprehension

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 15:36:07 -08:00
moe-berri
ea448a8951
feat(alerts): add email labels, bulk actions and threshold editing (#52139)
* feat(alerts): add bulk actions and enterprise thresholds

* fix(alerts): accept fractional multipliers above one
2026-11-16 15:32:19 -08:00
Agung Besti
c482013a8a
fix(guardrails): log requests that a guardrail blocks on /apply_guardrail (#51014)
* fix(guardrails): record the guardrail verdict on a request dict that starts empty

The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.

* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail

A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.

The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.

* fix(guardrails): keep Presidio and Gray Swan verdicts on empty request data

Presidio's text-only path and Gray Swan's API-failure path replaced an empty request dict with a throwaway one, so the verdict never reached the logged request. The apply_guardrail logging tests now run the real request setup and read what a logging callback receives.

* test(guardrails): cover the endpoint failure row, the sync decorator, and an unreachable Gray Swan without real sockets

* fix(logging): redact prompt text in guardrail records when message logging is off

* fix(logging): redact the failure payload when message logging is off

* fix(logging): keep guardrail details for other callbacks when one opts out of message logging

* fix(logging): redact failure events per callback and keep guardrail compression stats

A callback that turns off message logging now gets the same redacted copy on failure events as on success events, so a blocked /apply_guardrail call no longer hands it the raw prompt and verdict

Guardrail redaction keeps the numeric compression stats (tokens saved, compression ratio) that spend tracking reads, through one helper shared with the spend-log sanitizer

* fix(policy): test each batch input on its own copy of the request data

The guardrail logging decorator now records a verdict on an empty request dict instead of dropping it. The policy test endpoint shared one request dict across every input and merged it into the agent call, so each later input and agent request carried the earlier inputs' verdicts. Each input now gets its own copy, which keeps the caller's request data unchanged

* test(policy): look up the counting guardrail through the real registry

The batch isolation test now registers its guardrail as a litellm callback, so the endpoint finds it through the real guardrail registry instead of a mocked one, and its apply_guardrail override carries the same concrete types as the base class

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 15:18:08 -08:00
joshua-berri
1540acd7e4
feat(proxy): accept request token prices in a declared currency (#51971)
* feat(proxy): convert per-request token prices from pricing_currency to USD

A trusted chat completion request can now send pricing_currency with
input_cost_per_token and/or output_cost_per_token. The proxy converts the
supplied prices to USD once at ECB reference rates in pre-call, right after
the existing pricing strip, and removes the field before routing. Requests
without the field, and USD, behave as before and make no network call.

The field sits behind the same two gates as request prices. Other routes,
invalid codes or prices, other pricing fields alongside it, and rates that
are unavailable, stale or out of range are rejected instead of being priced
as USD. Cold lookups share one bounded refresh and fresh rates never wait.

* test(proxy): type pricing currency tests and cover failed rate refreshes

Types the new pre-call tests, waits for the shared refresh to finish even
when the due-refresh test fails, and covers stale rates with a failed
refresh (503), a failed background refresh keeping fresh rates, and a
refresh that raises.

* test(proxy): restore an unrelated import line reformatted by mistake

* test(proxy): use the current chat model in request pricing fixtures

* fix(proxy): reject pricing_currency on generic pass-through requests

An authorized key keeps request prices on a generic pass-through, which
bills them as USD, while pricing_currency was forwarded upstream untouched.
The pass-through body boundary now rejects a non-null pricing_currency with
a 400 and drops a null one, matching the chat-only conversion.

* fix(proxy): type the request pricing conversion call and ECB client import

Validates the request body to a typed mapping only when pricing_currency is
present, converts the supplied prices from that view, and writes them back
onto the original request. Imports the async HTTP client helper at module
level so its binding is typed. The client is still built inside the fetch.

* test(proxy): cover a null pricing_currency through the pre-call pipeline

* fix(proxy): fetch ECB rates with a short-lived handler closed after each refresh

Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, instead of binding the untyped cached client factory.

* fix(proxy): wait a minute before retrying a failed ECB rate refresh

A failed refresh left rates due, so every later non-USD request started a new
ECB fetch during an outage. The shared lookup now reuses a failed refresh
for 60 seconds from its start, so requests in that window get the held
rates or a 503 without fetching again, and the next request after it
retries. It also goes back to the cached async client factory, with a named
type suppression for the factory's untyped params argument.

* fix(proxy): convert pricing_currency on every chat completions alias

The /engines/{model}/chat/completions and /openai/deployments/{model}/chat/completions
aliases reach the same chat handler but were rejected, because the check
compared the literal request path. It now compares the registered route
template, falling back to the path when no template is available. Also
covers a cancelled refresh not blocking the next lookup.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 15:16:24 -08:00
mika-lt
f0548f9a19
fix(proxy): rewrite batch JSONL alias rows to the deployment model on model-routed uploads (#51843)
* fix(proxy): rewrite batch JSONL rows to the deployment model on model-routed uploads

* fix(proxy): rewrite only alias rows and authorize them against the alias at batch create

* fix(proxy): keep batch create working when the credentials lookup names no model

* fix(proxy): spool the batch alias rewrite to disk and close it after upload

* fix(proxy): rewrite only alias rows in model-routed batch uploads and charge both project quotas

Rows that do not name the alias now reach the provider exactly as uploaded, and a file with no alias row is sent unchanged. Rewritten rows are compact UTF-8 JSON. Rows routed through the alias count against the project quota of both the alias and the model the row names

* fix(proxy): keep every byte of non-alias batch rows and skip files with nothing to rewrite

* fix(proxy): rewind batch uploads instead of calling seekable() so Python 3.10 works

* fix(vertex_ai): honor deployment and row drop_params for sampling params in batch file rows

* fix(batches): rewrite escaped alias rows and charge shared quota counters once

Parse each row's model to find rows naming the alias, so an alias written
with JSON escapes is rewritten too. Merge a row's alias and real model into
one project quota descriptor per counter, so a model_group_alias that puts
both in one group charges the row once against the lower limit.

* test(batches): prove the shared quota counter is charged once on the Redis store

* test(files): capture model-routed batch uploads at the provider HTTP boundary

* fix(batches): charge a shared project counter once per batch and keep rows JSON cannot re-encode

A batch file that mixes rows naming the proxy alias and rows naming the
deployment model built one project ITPM/OTPM descriptor per row group, so
the in-memory limiter checked each group against the quota on its own and
let the batch through. Descriptors now merge by counter with summed tokens,
and the 429 detail reports the merged total.

A row whose numbers JSON cannot write back, such as 1e400, was rewritten as
Infinity. It now stays byte-identical, the same as rows the rewrite skips.

* fix(batch_rate_limiter): hold a shared project counter to its strictest limit

When a batch file mixes rows naming the alias and rows naming a model in the
same group, both row groups charge one project ITPM/OTPM counter. The merged
descriptor kept the first group's limit, so a stricter limit set on the other
name was skipped. Merge them with the limiter's coalesce_rate_limit_descriptors,
which keeps the lowest limit, the same rule a single request already follows.

* chore(rust): restore generate_stdlib_fixtures.py as it is on main

An earlier merge commit reformatted this ruff-excluded fixture script. Restoring main's bytes keeps the PR to the batch alias fix

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 15:12:52 -08:00
norkelbyte
83486ddfbd
fix(proxy): preserve team aliases in realtime websocket routing (#52124)
* fix(proxy): preserve team aliases in realtime websocket routing

* test(proxy): cover translations and live sessions team alias routing

* test(proxy): cover a team alias that differs from the provider model on realtime routes

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 15:00:07 -08:00
kenneth-berri
113766d4b9
fix(gemini): return n images for image generations and edits when n > 1 (#52103)
* fix(gemini): serve image n with n single-image requests

Gemini image models reject a candidate count above one on the images routes,
so n > 1 failed with a provider 400 on AI Studio and Vertex AI. The image
generation and edit handlers now ask the provider config how many identical
requests to send, and merge the returned images and token usage into one
response. Vertex AI Gemini image edits also accept n now.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): cap image fan-out and bill served images when one request fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(images): price each Gemini fan-out request on its own usage and give the merged response its own hidden params

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Bill completed Gemini image calls when a generations fan-out request fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Expect the shared logging object in the image generation router call test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 14:50:13 -08:00
joshua-berri
e42e0db034
feat(alerts): display prompt cache advisories in existing alerts (#51943)
* feat(alerts): read and show prompt cache alerts alongside the existing alert types

Saved alerts can now carry a prompt_cache type on a deployment resource,
with evidence made of opaque fingerprints and counters only. One incident
exists per deployment and cache configuration, whatever window or prefix
found it, so a dismissed or resolved one stays the only one.

The performance list, the daily summary, the free-alert allowance and the
history lookup now name the four metric types instead of excluding model
deprecations, so these rows never reach metric charts, digests or the quota.

The All view lists prompt cache alerts from their own query, leaving the
performance query and its pagination unchanged. An alert's page shows the
deployment, model, misses in a row, the proving write, the cache lifetime
and when the misses happened, in place of a metric chart, and keeps the
existing dismiss, resolve and reopen controls.

* fix(alerts): pin prompt cache evidence to what is observed and prove the readers against the database

Prompt cache evidence now only accepts the anthropic provider and the
five-minute TTL. The title says cache usage was not observed rather than
that caching stopped, and the scope copy names the checked requests:
non-streaming Anthropic Chat Completions with one five-minute breakpoint,
after a confirmed write, with never-proven prefixes left unassessed.

Alerts are presented by their evidence shape. Metric labels come from the
evidence unit and the activity link label skips deployments, so the
unrecognized-alert fallback is gone.

The real-database alert tests now round-trip a prompt cache alert, keep it
out of the performance list and the daily summary, keep a dismissed one
as the only incident, and leave it out of the free allowance. They replace
the unit tests that only checked SQL arguments. The UI tests reopen an
incident, page both lists in All from their own cursors, and drop loaded
prompt cache alerts when Performance is chosen.

* chore(ui): regenerate API types for prompt cache alerts

Alerts can now carry the prompt_cache type, the deployment resource and prompt cache evidence, so the generated dashboard types include them.

* fix(alerts): satisfy the dashboard lint and the bounded IN check for prompt cache alerts

The prompt cache list query reuses useAlerts with the prompt_cache type, so
no five-parameter helper is needed. The new prompt cache mock handling in
the Alerts integration test lives in its own function, a presentation test
names its expected details, and AlertDetail is formatted. The performance
list's IN filter over the four fixed metric types records that bound.

* fix(alerts): reject metric alerts on a deployment resource

Only prompt cache alerts use the new deployment resource. Metric alerts have no deployment history source, so a metric alert on a deployment is now invalid. Every combination accepted before this change is still accepted.

* test(alerts): match the plain token count the prompt cache evidence renders

* fix(ui): describe prompt cache alerts as advisory observed evidence

The alert reports requests the proxy observed. Delayed, missing or unlogged requests can fall between them, so the list and detail say they are not necessarily consecutive and do not prove an outage. The serialized consecutive_misses field is unchanged.

* fix(ui): title prompt cache alerts as advisories and attribute the limit to observations

The title no longer reads as a deployment-wide finding, and the limitation says delayed, missing or unlogged observations mean the requests are not necessarily consecutive and do not prove a caching outage.

* docs(alerts): describe consecutive_misses as observed requests that are not necessarily consecutive

* chore(ui): regenerate API types for the advisory prompt cache evidence description

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 14:49:52 -08:00
joshua-berri
e7c89b0dd4
fix(proxy): preserve terminal evidence in disconnect billing (#52088)
* fix(proxy): preserve terminal evidence in disconnect billing

* fix(proxy): retain held finish reasons through stream wrappers

* refactor(proxy): use immutable finish evidence results

* fix(proxy): type disconnect evidence boundaries and cover wrappers

* fix(proxy): preserve object type before chunk narrowing

* fix(proxy): retain disconnect finish evidence per choice

* fix(proxy): preserve genuine Gemini finish evidence

* fix(proxy): use typed finish evidence storage

* test(proxy): exercise real router stream attempts

* fix(proxy): keep unspecified Gemini evidence unknown

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 14:43:03 -08:00
joshua-berri
4af6fb70d8
feat(cost): add automatic ECB reference-rate lookup (#51934)
* feat(cost): add ECB reference rate cache for USD conversion

Adds a typed adapter for the ECB daily euro reference rates feed and an
in-process cache with a synchronous, no-network USD conversion lookup.
Refresh is an explicit async call with a bounded timeout, a daily cadence,
and last-good retention. Rates older than seven calendar days, future-dated
or malformed feeds, and date rollbacks are reported as values. USD is an
identity rate available offline.

Helper only. Nothing calls it yet, so no proxy or SDK behavior changes.

* fix(cost): reject unusable ECB cross rates, encoded DTDs and stale feeds

Feeds whose USD cross rate overflows or underflows are rejected, the feed
must be UTF-8 without NUL bytes and is parsed with a forced UTF-8 decoder so
a declared UTF-16 or UTF-7 encoding cannot hide a DOCTYPE, and a feed dated
more than seven days before the refresh no longer counts as the daily check.
Fetch failures carry only the error class or HTTP status.

Tests cover each case and a delayed older response racing a newer refresh.

* fix(cost): apply same-date ECB rate corrections on daily refresh

A valid feed with the held rate date now replaces the held snapshot instead
of being discarded while reporting success. Strictly older dates are still
rejected, and rates already returned to callers are unchanged.

* fix(cost): type the ECB feed client as a narrow protocol

fetch_ecb_daily_feed now takes a protocol for the one async get call it
makes, so strict type checking no longer sees AsyncHTTPHandler's untyped
params, and compares the status against HTTPStatus.OK. No behavior change.

* test(cost): prove a stalled ECB fetch is cancelled at the overall deadline

A get that never returns is driven past the fetch deadline on an event loop
whose clock the test skips forward, so the test fails if the overall
timeout is removed or not enforced, and checks the stalled call was cancelled.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 14:26:43 -08:00
Kerry Lu
de20250735
fix(proxy): check pre-call fallback access with the saved model_group_alias, never the one the client sent (#52077)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 22:02:42 +00:00
Kerry Lu
0ba3d312d4
fix(dynamic_rate_limiter): keep counting and reporting priority requests after a priority pool's tokens run out (#52081)
* fix(dynamic_rate_limiter): keep counting and reporting priority requests after a priority pool's tokens run out

* fix(dynamic_rate_limiter): return the tracked priority statuses as a tuple

* fix(dynamic_rate_limiter): read optional rate limit fields without assuming they are set

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 22:00:36 +00:00
Kerry Lu
c7e5f78724
fix(proxy): register new customer org aliases one at a time per worker so a burst of first requests is served (#52080)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 14:00:04 -08:00
Nate Armstrong
1c45174926
fix(messages): close high and medium rust messages parity bugs (#52042)
* fix(messages): sign bedrock invoke with sigv4 when api_key is empty

With api_key set to an empty string and AWS_BEARER_TOKEN_BEDROCK in the env, the handler dropped the empty key before it reached the Bedrock Invoke config, so Rust read the env token and sent it as a bearer. Python's bedrock_bearer_token treats an empty key as no bearer without reading the env and signs the call with SigV4. The empty key is now dropped only when the azure_ai module api_key fallback is in play. Fixes AUTH-084, AUTH-049 and BI-021.

* fix(messages): guard the anthropic env base like a caller api_base

With no caller api_base, an anthropic call took its base from ANTHROPIC_API_BASE or ANTHROPIC_BASE_URL in Rust, and the host's malformed-base guard only looked at the caller api_base. So ANTHROPIC_API_BASE=http:///d/anthropic was parsed by the native client as host d, and a user:pass@h base went to h, both with the operator x-api-key. Rust also skipped a whitespace-only ANTHROPIC_API_BASE and fell through to ANTHROPIC_BASE_URL or api.anthropic.com. Python resolves api_base or get_secret_str("ANTHROPIC_API_BASE") or get_secret_str("ANTHROPIC_BASE_URL") and hands the result to its HTTP client, which refuses those URLs, and keeps a whitespace-only value so the call fails locally. The host now resolves the anthropic env base the same way, runs it through the guard so these bases post through Python's client, and seeds the values it read into the native route. Rust now only skips an empty env base, as Python does. Catalog ids ANT-012, ANT-009, ANT-010 (ruling R1)

* fix(messages): bound connect and response head per phase and retry a failed connect once

A native Messages call whose TCP connect never completes was bounded by a fixed 10 s client connect cap and by one tokio timeout over connect plus response head, so it failed as a read timeout that the proxy billed at the prompt estimate, and a connect error was never retried. Python bounds connect by the call timeout (5 s when none is set) and the head by the read timeout separately, raises ConnectTimeout labelled Connection that is not billed, and its async client retries a ConnectError once. Rust now applies a per-call connect bound through a connector layer, times the head from when the connection is ready, and retries a failed connect once on async calls. Catalog ids ERR-082, ERR-083, ERR-084, ERR-088, COST-073, RTR-066.

* fix(messages): give the agentic follow-up the caller's system, not the hoisted wire system

With system "S" and a leading role system turn "R" on azure_ai or bedrock_invoke, the Rust host built the agentic follow-up params from the upstream wire body, whose system was already hoisted to [S, R], while replaying the un-hoisted messages, so the follow-up hoisted R again and sent [S, R, R]. Python hands its follow-up the caller's request params, whose system is still S, since the hoist only rewrites the transformed request, so the follow-up sends [S, R]. The host now takes system from the caller's requested params whenever the wire body carries one. AZ-091. ANT-085 is left as is: Python's transform pops max_tokens from the same params dict it later hands the agentic hooks, so Python's follow-up also falls back to 1024 or budget plus 1024, which the host already matches.

* fix(messages): keep explicit nulls in typed request fields on the wire

A request carrying cache_control null or cache_control.ttl null, tool_result.content or tool_use.input null, a block type null, output_config.effort or format null, thinking.display null, or context_management.edits null lost those keys in the Rust typed round-trip, and with output_config.effort null plus reasoning_effort Rust filled in the mapped effort. Python forwards these dicts as sent and its setdefault keeps the null effort. The fields now deserialize an explicit null into a kept value through the recognized helpers, as budget_tokens already does, while beta selection and effort decisions still read null as unset like Python's get checks. Catalog ids SHAPE-078, SHAPE-112, SHAPE-132, SHAPE-133.

* fix(messages): add the anthropic oauth beta when the oauth key comes from ANTHROPIC_API_KEY

With no caller api_key and ANTHROPIC_API_KEY=sk-ant-oat01-..., or with an api_key that only matches
the OAuth prefix after trimming, Rust sent the token as a bearer without anthropic-beta:
oauth-2025-04-20. Python's _static_auth_header builds the bearer through _oauth_bearer_header, which
adds that beta (merged with the caller's betas, no direct browser access header), so Rust now builds
that case like the minted OAuth environment. A Bearer-prefixed key keeps the F10 ruling and is sent
once without the beta. Catalog id ANT-018

* fix(messages): assume the irsa role explicitly before a cross-account bedrock role

With IRSA (AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN set), no keys, and an aws_role_name naming a different role, Rust built an aws-config AssumeRoleProvider whose source credentials came from the default chain, so the web identity hop used an SDK session name and no GetCallerIdentity was sent. Python reads the token file and calls AssumeRoleWithWebIdentity on the IRSA role with the caller's session name, then GetCallerIdentity and AssumeRole on the target with those credentials, all against the same STS endpoint and region. Rust now makes the same three calls with the same parameters. Catalog id AUTH-033

* fix(http): drop tls 1.2 chacha20 suites from the rust client under fips_mode

With general_settings.fips_mode on, Python's outbound SSL contexts drop RSA key exchange and TLS 1.2 ChaCha20 (set_ciphers with :!kRSA:!CHACHA20, falling back to TLS 1.3 only when nothing is left), but the Rust client built its rustls config from the ring provider and kept every TLS 1.2 suite, so Rust-served Messages traffic and its auth calls still offered ECDHE ChaCha20. The host now passes the exclusion flag in its http settings and the Rust client removes the TLS 1.2 ChaCha20 suites from the configured or default list, offering TLS 1.3 only when none remain. rustls has no RSA key exchange, so nothing else changes. Catalog DISP-074, ruling R15.

* fix(messages): filter anthropic-beta through the beta config python loaded

SHAPE-020. The Rust messages route filtered anthropic-beta values for azure_ai, bedrock, bedrock_mantle and Foundry against its compiled copy of the bundled table, while Python filters against whatever _load_beta_headers_config loaded: the remote JSON by default, or the bundled file under LITELLM_LOCAL_ANTHROPIC_BETA_HEADERS, offline mode or a failed fetch. A remote config that differs from the bundled one therefore sent different betas on each path

The host now hands Rust that loaded map for every call whose betas are filtered, Rust installs it once per distinct config object, and the azure_ai and bedrock policy, the Foundry and bedrock_mantle allowlist and Bedrock Invoke's own filter read it with Python's lookup order (provider aliases, the header-only column, then the provider column) before falling back to the compiled table. Checks Python makes against the bundled config alone keep the compiled table
2026-11-16 21:08:25 +00:00