* test(gemini): cover client-mangled thought signatures in tool call ids
* fix(gemini): ignore a client-mangled thought signature in a tool call id
Only replay the signature embedded in a tool call id when it is still valid base64. Clients that strip or replace +, / and = in ids now get the existing placeholder fallback instead of a Vertex 400 on every later turn. Fixes#52001.
* feat(shadow-eval): let reverse jobs skip samples the baseline group served
A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.
Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.
The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.
* test(shadow-eval): drive same-model skip tests off explicit completion signals
Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.
Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.
* style(tests): remove extra shadow evaluation test spacing
* fix(ui): send skip_same_model false from the shadow eval start form
The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.
* fix(migrations): bound shadow evaluation schema lock waits
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
gateway-auth-core gains plan/ and credential/, which plan and decide key and
JWT authentication on top of its route rules. The Python proxy now runs that
core through python-bridge with a PythonAuthHost, which replaces
handle_jwt.py and the key and JWT paths of user_api_key_auth.py.
The core defines Token, MasterKey and KeyHash, and config gains the JWT auth
settings. Route groups named in allowed_routes match as starlette's
compile_path compiles them, like _allowed_routes_check, and a team's
allowed and denied pass-through routes read as RouteLists. allow_user_auth,
allow_requests_on_db_unavailable and enforce_rbac read through python-compat's
IsTrue, StrBool and Truthy adapters, and StrBool reads text with
flags::str_to_bool.
Fixes that landed on main in the code this moves carry over: every user read
the core makes from a JWT hands the raw email_verified claim to the host,
which reads it with read_email_verified, the impersonation header applies
after a master key, cached admin, virtual key or mapped JWT key is accepted,
and the team member budget step uses the native budget decision.
A JWT email claim that is not a string reads as no email, the same as a
missing claim, instead of being stored as the number's text.
The native extension size cap rises from 55 MB to 65 MB, since the auth core
brings the Linux build to 56.85 MB.
* fix(ui): use full page navigation for SSO login
* style(ui): format SSO navigation regression test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output
* fix(anthropic): close rebuilt-stream thinking blocks with a signature_delta
* fix(anthropic): drop unsigned thinking before Claude, stream raw reasoning_text, close rebuilt thinking blocks with signature_delta
A Claude target on /v1/messages now gets unsigned thinking blocks left out of the request, so a switch from a reasoning_content model makes one call and keeps thinking, except when that would leave the final tool turn without its opening thinking block
Raw response.reasoning_text deltas from the Responses bridge stream as thinking, and a stream rebuilt from a full answer or rewritten by a guardrail closes each thinking block with a signature_delta, the real signature or an empty string
* fix(anthropic): stream a whole reasoning item as a closed thinking block
A Responses stream can send a reasoning item whole in output_item.added and output_item.done with no text deltas, as litellm's own stream does for gpt-oss on OpenRouter. The /v1/messages stream now builds that item's thinking block with the same helper the non-streamed response uses, so streamed and non-streamed answers carry the same reasoning
* test(anthropic): mark the new streaming test locals Final
* fix(anthropic): drop unsigned thinking before every Claude call, a pending tool turn included
Anthropic answers 400 to a thinking block with an empty signature but accepts a
tool turn with no thinking, so the strip no longer keeps the block that opens the
pending tool turn. The check now looks for "claude" in the model name, which covers
Claude behind every messages route and the chat adapter, not only four providers
* fix(anthropic): strip unsigned thinking when a deployment's base_model names Claude
* fix(anthropic): drop unsigned thinking for an alias whose base_model names Claude on the native messages path
The chat adapter already read base_model before leaving unsigned thinking out of a Claude request. The native messages transform and the Rust request transform now read it too, so an aliased Claude deployment gets one call instead of a 400 and a retry
* test(anthropic): build the pending tool turn without mutating it in the Rust strip test
The Rust test now builds its expected messages from the same helper instead of editing them afterward, and a doc comment that only restated a helper's name is gone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(messages): load the anthropic beta map only when a filter has a beta to filter
Since #52042 the Rust messages route asked Python for the beta map on every
azure_ai, Bedrock Invoke and Foundry call while decoding the request, so the
first such call on each worker downloaded the remote map even when it sent no
anthropic-beta, and the download ran synchronously on the request path.
The bridge now installs the map only once Python has loaded it. When a filter
first meets a beta and no map is installed, it asks the host to load it on the
loader thread and uses the compiled table for that call, as before #52042.
* fix(messages): start at most one background beta map load at a time
A burst of beta-carrying calls while the first download was still in flight
queued one load task per call on the loader thread. A non-blocking lock now
lets only one load wait in the queue, and it is released when that load ends.
* fix(bedrock): route bedrock/<region>/<model> to that region on the Invoke path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep LIT010 flat and update the invoke region-path integration test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(bedrock): drop the comment that repeats the helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): match key aliases exactly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(spend-logs): drop unused key alias ORM lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate exact key alias API descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): keep key alias filter matching CLI session logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Gemini tool call arguments now go through parse_tool_call_arguments, the shared helper the other providers use, so malformed or empty argument strings are repaired or rejected the same way instead of raising a raw JSONDecodeError
* fix(complexity_router): classify short asks in long conversations
* fix(complexity_router): validate active context limits without recursion
* test(rag): wait for keepalive frames before releasing retrieval
* test(ui): control availability debounce in context-limit regressions
* test(providers): cover upstream 403 from OpenAI-compatible providers as permission denied and not retried
* fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers
* fix(bedrock): send guarded_text parts in system and assistant messages as Converse guardContent
* fix(bedrock): drop guarded_text parts on the Nova invoke route under drop_params
* fix(bedrock): send guarded_text parts as text under drop_params when the request cannot carry guardContent
* fix(bedrock): trim a final guarded assistant turn and honor batch drop_params for guarded parts
Claude on Converse rejects a final assistant turn that ends in whitespace,
and the trim covered only text blocks, so a guarded_text prefill ending in
whitespace now answered 400. The trim now covers guardContent text too,
which also covers the drop_params text fallback that runs after it.
Nova batch records built their request without the record's drop_params,
so a guarded system part stayed guardContent, which Nova InvokeModel
rejects. The record's drop_params now reaches the transform.
The new Nova tests split a nested generator and gain type annotations.
* fix(bedrock): keep the full guardrail input check and reject malformed content parts
A guarded assistant or mid-conversation system part goes as text under guardrailConfig when no user turn is guarded, since guardContent there would narrow Bedrock's input check to that part. A system or guarded_text part whose text is not a string now gets a 400 instead of being dropped.
* fix(bedrock): keep the full guardrail input check when a guarded system message ends the request
A guarded system message after the last assistant turn or tool result became a trailing user turn that the guardrail auto-wrap then guarded alone, so Bedrock skipped the user's text. The rule now reads the trailing turns after the system conversion and leaves them unguarded when nothing else would be. A user guarded_text part whose text is not a string now gets litellm's 400 that names it.
* fix(bedrock): keep guarded parts guarded when tool history without tools gets guarded
With tool history and no tools param, litellm rewrites each tool result into a guarded user turn after the input check decision ran, so a guarded system part sent as text escaped the check. The decision now keeps the part guarded when that rewrite will guard a tool result.
* fix(bedrock): keep the full guardrail input check after an unanswered tool call without tools
* fix(bedrock): guard trailing turns only when a user turn is among them
A plain system message that ends a request after an assistant turn or a tool result
became a user turn, and the trailing guardrail wrap guarded it alone, so Bedrock
checked only the system text and a blocked word in an earlier user turn got through
* test(bedrock): cover a function result without tools keeping a guarded part guarded
* test(bedrock): expect the full input check for a guarded assistant part before a trailing system turn
A trailing system turn alone no longer gets the trailing guard wrap, so a guarded assistant part before it goes as
text whatever the system text holds, and Bedrock checks every message
* fix(bedrock): keep the full guardrail input check when tool history comes without tools
* fix(bedrock): keep guarded assistant text guarded when its tool calls turn into text
Without a tools param, an assistant turn with tool calls is flattened to text. When a guarded user turn makes the guardrail check only guarded blocks, that flattening also turned the assistant's guarded_text parts into plain text, so the guardrail skipped them. In that case keep those parts guarded and append the tool-call text as its own text part. Everywhere else the turn is flattened as before
* fix(bedrock): keep a user question before rewritten tool results in the guard wrap
With tool history and no tools param, litellm rewrites each tool result as a user turn. The trailing guard wrap stopped at the tool result, so a plain user question right before it went as plain text beside guarded blocks and its blocked word skipped the guardrail. Rewritten tool results now count in the trailing run, which matches the merge base
* fix(bedrock): keep guarding a trailing system message next to a caller-guarded user turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(ui): cover Aria tool date schema compatibility
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): use the SDK tool argument parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): drop format:date from Aria Agent tool schemas
Co-authored-by: tomasWade <4695743+tomasWade@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): name the Aria operation context fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cite the observed DeepSeek format restriction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: tomasWade <4695743+tomasWade@users.noreply.github.com>
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags
supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint
format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a
Bedrock Claude deployment silently turned off its prompt caching.
Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching
as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old
key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a
Bedrock deployment still on the old key logs a one-time migration warning.
Fixes#52140
* fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models
* test(bedrock): mark the system prompt directly in the routed cache point regression
* fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides
* fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog
* test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(auth): apply current team limits in the pre-body rate limit check
The check that refuses an exhausted key before reading the request body
used the team limits copied onto the cached key. Raising a team's rpm or
tpm limit with /team/update did not reach that copy, so a throttled key
stayed throttled until the key cache expired, and the full auth path that
does refresh team fields never ran.
The check now overlays the cached team's newer fields the same way full
auth does, through one shared helper. When the team is not in the cache,
as on a peer right after an invalidation, it defers to full auth instead
of judging the key on stale limits.
* fix(auth): keep key limits in the early check when the team is not cached
On a team cache miss the early check skipped every pre-body limit, so an
exhausted key read its body before being refused. Apply the key's own
limits and defer to full auth only when the cached team limit refuses.
The customer daily usage migration gave LiteLLM_DailyCustomerMemberSpend.updated_at a
DEFAULT CURRENT_TIMESTAMP that schema.prisma never declared, and the customer hierarchy
cleanup renamed the default budget template foreign key to a 65 character name that
Postgres cut to 63 characters, which is not the name Prisma derives. prisma migrate diff
reported both, so test_schema_migration_in_sync failed on every main integration run.
The new migration drops the updated_at default, matching every other daily spend table.
The only writer, the daily spend bulk upsert, always sets updated_at itself. It also
renames the truncated foreign key to the name Prisma expects, guarded so databases that
never got the truncated name are left alone.
* fix(mcp): keep an MCP server's own Authorization key when the JWT signer is on
With mcp_jwt_signer enabled, a server whose configured authentication_token is sent in
Authorization (bearer_token, token, authorization, basic) had that key replaced by the
signer's JWT on tools/list and tools/call, so the upstream returned 401. Both signer guards
now also treat the server's own key as occupying Authorization, using the same
to_server_spec mapping that sends the key upstream.
Resolves LIT-10782
Fixes#47988
Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>
* fix(mcp): keep a deferred oauth2-delegate server's own token when the JWT signer is on
Servers that defer to the v1 client, such as oauth2 with delegate_auth_to_upstream,
send their authentication_token in Authorization (or upstream_token_header), but the
signer guard only read the v2 spec and let the gateway JWT overwrite it. The guard
now falls back to v1's own header rule when to_server_spec returns None. BYOK keeps
its existing behavior.
* test(mcp): cover a SigV4 server's token leaving Authorization to the JWT signer
* fix(mcp): only let upstream_token_header claim Authorization for auth types v1 writes a key for
A SigV4 server with a stray authentication_token and upstream_token_header
set to Authorization lost the signer JWT, though v1 never sends that token.
The v1 rule now resolves the header v1 would write the key to, or none, before
applying the override.
* fix(mcp): keep the signer JWT on tools/call when the caller's auth header displaces the server key
A per-request MCP auth header replaces the server's authentication_token, so
the configured key is not what reaches Authorization. The tools/call guard now
only consults the server's configured key when the caller sent none, matching
the tools/list guard, so the upstream no longer ends up with no Authorization.
---------
Co-authored-by: Arvind Kumar <arvindkumar.acad@gmail.com>
* fix(mcp): let zero-budget users call MCP tools that cost nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a tool call made by its display name as the tool it runs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep budget checks for bare display names a paid server can run and for cut non-tool-call bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): budget check reserved tool names and display names that resolve on to a paid tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): follow chained display names without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): price a scoped tool call on the servers it is scoped to and follow display names as execution does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the budget check for MCP servers that use display names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): send reasoning_effort to Foundry reasoning models
Foundry models outside the GPT-5 family that the cost map marks as reasoning
models, like DeepSeek V4, now accept reasoning_effort. A level the model row
declares is sent as is. A level it does not declare is dropped under
drop_params and gets a 400 otherwise. Declares the levels DeepSeek V4 Flash
and Pro accept on Foundry
* fix(azure_ai): send reasoning_effort only to Foundry rows that declare levels
Gate reasoning_effort on the cost map's reasoning_effort_levels instead of
supports_reasoning, so undeclared reasoning rows keep the drop_params behavior
they had before. Look the levels up case-insensitively, keep a dropped effort
out of the body when allowed_openai_params lists it, model the level check as
values, and name the proxy drop_params setting in the 400
* fix(azure_ai): read Foundry reasoning levels from the loaded catalog row and honor allowed_openai_params
Levels now come only from the model's own azure_ai catalog row, checked for the deployment's base_model first, so a sibling deployment's model_info or a bare openai row no longer decides them. A request that allows reasoning_effort sends any level, as before this PR
* fix(azure_ai): honor register_model reasoning levels on Foundry rows
The level check read only the loaded cost map, so a model registered with
litellm.register_model lost its declared reasoning_effort_levels. Read the
loaded map with register_model registrations on top, still without the
router's per-deployment registrations, so sibling deployments stay separate
* fix(azure_ai): leave default and auto to the model and keep rows without levels unchanged
default is stripped even when allowed_openai_params skips the level check, since Foundry refuses it on DeepSeek V4
auto leaves the level to the model, the same way the Fireworks config treats it, so Kimi K3 auto keeps working
A model whose catalog row lists no levels gets reasoning_effort exactly as it was sent
Updated the Kimi K3 level tests to the seven-level Foundry row, and the catalog-only test to the level check
* fix(router): drop a classifier reasoning_effort level the deployment refuses
The complexity router's classifier kept reasoning_effort once azure_ai listed it, so a level Foundry's row refuses failed the classifier call and sent every request to the capable tier. The classifier now drops a level the deployment's provider config refuses, unless the deployment declares its own levels or allows reasoning_effort
* fix(azure_ai): keep the catalog row's reasoning levels under a registration spelled in another case
A price-only register_model call under azure_ai/fw-kimi-k3 hid the azure_ai/FW-Kimi-K3 row, so its levels were lost and reasoning_effort answered 400 or was dropped
The level lookup now merges the catalog row under the registration, the way register_model itself merges
The Foundry Kimi K3 level test cites the live check behind its seven levels
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(mcp): add Parallel Search to discovery registry
* fix(mcp): point the Parallel catalog entry at mcp.parallel.ai
Parallel's docs call search.parallel.ai/mcp legacy and say new installs should use mcp.parallel.ai/mcp, which keeps web_search and web_fetch and stays free without a key
* test(mcp): cite the Parallel docs beside the pinned catalog URL
---------
Co-authored-by: georgeatparallel <george@parallel.ai>
* fix(bedrock): log the prompt, settings and stream flag of passthrough requests
* refactor: keep passthrough request logging within the lint gates
* fix: keep the logging object deep-copyable
* fix(bedrock): log Anthropic tool exchanges and skip guarded blocks without text
* feat(ui): show the usage spend total in another currency at ECB reference rates
Adds GET /get/reference_exchange_rates, which returns the current ECB
reference rates as USD per unit for every feed currency. It refreshes at
most once a day under a lock, keeps fresh rates when a refresh fails and
answers 503 instead of serving rates older than seven days.
The usage overview's spend card gets a Spend display selector. It asks for
rates only once opened and then shows an estimated amount with the rate date
under the unchanged USD total. Every other figure stays in USD.
* fix(ui): fall back to USD when spend display rates stop being usable
The selector now shows USD whenever no estimate can be shown, so a failed
refetch never leaves EUR selected beside a USD-only notice, and an opened
selector refreshes its rates hourly. The endpoint gets its HTTP client
inside the async fetch so the client belongs to the serving event loop.
* fix(proxy): import the ECB rates endpoint HTTP client helper at module level
Binds get_async_httpx_client at import so its type is known. The client is
still built inside the awaited fetch on the serving event loop.
* fix(proxy): fetch display rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, and drops the descriptive comment on the spend display component.
* chore(ui): regenerate API types for the reference exchange rates endpoint
* fix(proxy): use the cached async client factory for display rate refreshes
Direct AsyncHTTPHandler construction fails the async client check, so the
refresh fetch goes back to get_async_httpx_client, called inside the awaited
fetch, with a named type suppression for the factory's untyped params.
* fix(ui): tidy spend display helper, hook options and fallback test
Groups the rate snapshot into one estimatedSpend argument, names the rates
query options, names the missing-estimate condition, and waits for the
unavailable notice before asserting the USD fallback.
* chore(ui): drop a restating comment on the spend estimate helper
* test(proxy): prove concurrent first rate requests share one ECB fetch
* fix(proxy): wait a minute before retrying a failed display rate refresh
A failed ECB refresh left rates due, so every request after it waited on
the lock and fetched again, and N queued callers paid N timeouts during an
outage. The endpoint now skips refreshing for 60 seconds after a failed
attempt, serving held rates or a 503, and retries on the first request
after that. Concurrent waiters share the failed attempt. Also drops the
usd_rates docstring.
* fix(ui): retry unavailable currency estimates on reopen
* refactor(ui): name the currency selector open handler
* style(ui): format currency selector props
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router)!: stop a client's metadata flag from turning off silent_model mirroring
A request that sent metadata.is_silent_experiment true skipped every mirror the
deployment's silent_model configured. The check was a recursion guard from when
mirroring first shipped. The router now stops a mirror from mirroring again on
its own, so the metadata check only ever acted on client input
BREAKING CHANGE: clients can no longer opt out of an admin's traffic mirroring by
sending is_silent_experiment in request metadata
* test(router): assert a caller-sent silent experiment flag still mirrors in the router core tests
The two core tests pinned the old opt-out, where a request that sent
metadata.is_silent_experiment skipped the deployment's silent_model
* test(router): check a mirrored call is never mirrored again on the sync path
* test(router): name why the sync mirror test suppresses the endpoint attribute check
* ci: start the zizmor check GitHub skipped at the last push
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(focus): upgrade cost and usage export to FOCUS 1.5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(focus): identify the combined daily request SKU meter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache
Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.
Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.
Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.
* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration
Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.
After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.
Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.
* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading
Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.
A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.
* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset
* fix(alerts): describe prompt cache alerts as advisory and type their boundaries
Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.
* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof
A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.
* fix(alerts): serialize each message's fields once in the prompt cache prefix
Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.
* test(proxy): keep a leaked worker config out of the prompt cache lifespan test
An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.
* test(alerts): clean each prompt cache deployment tag in its own comprehension
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(model_prices): registry audit 2026-11-16, add dashscope qwen4-max, max reasoning effort for GPT-5.6 and Azure Astra
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(model_prices): credit absorbed max reasoning effort flags from #48098
Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): keep max effort off chat-route GPT-5.6 and Azure Astra rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
* fix(guardrails): record the guardrail verdict on a request dict that starts empty
The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.
* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail
A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.
The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.
* fix(guardrails): keep Presidio and Gray Swan verdicts on empty request data
Presidio's text-only path and Gray Swan's API-failure path replaced an empty request dict with a throwaway one, so the verdict never reached the logged request. The apply_guardrail logging tests now run the real request setup and read what a logging callback receives.
* test(guardrails): cover the endpoint failure row, the sync decorator, and an unreachable Gray Swan without real sockets
* fix(logging): redact prompt text in guardrail records when message logging is off
* fix(logging): redact the failure payload when message logging is off
* fix(logging): keep guardrail details for other callbacks when one opts out of message logging
* fix(logging): redact failure events per callback and keep guardrail compression stats
A callback that turns off message logging now gets the same redacted copy on failure events as on success events, so a blocked /apply_guardrail call no longer hands it the raw prompt and verdict
Guardrail redaction keeps the numeric compression stats (tokens saved, compression ratio) that spend tracking reads, through one helper shared with the spend-log sanitizer
* fix(policy): test each batch input on its own copy of the request data
The guardrail logging decorator now records a verdict on an empty request dict instead of dropping it. The policy test endpoint shared one request dict across every input and merged it into the agent call, so each later input and agent request carried the earlier inputs' verdicts. Each input now gets its own copy, which keeps the caller's request data unchanged
* test(policy): look up the counting guardrail through the real registry
The batch isolation test now registers its guardrail as a litellm callback, so the endpoint finds it through the real guardrail registry instead of a mocked one, and its apply_guardrail override carries the same concrete types as the base class
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): convert per-request token prices from pricing_currency to USD
A trusted chat completion request can now send pricing_currency with
input_cost_per_token and/or output_cost_per_token. The proxy converts the
supplied prices to USD once at ECB reference rates in pre-call, right after
the existing pricing strip, and removes the field before routing. Requests
without the field, and USD, behave as before and make no network call.
The field sits behind the same two gates as request prices. Other routes,
invalid codes or prices, other pricing fields alongside it, and rates that
are unavailable, stale or out of range are rejected instead of being priced
as USD. Cold lookups share one bounded refresh and fresh rates never wait.
* test(proxy): type pricing currency tests and cover failed rate refreshes
Types the new pre-call tests, waits for the shared refresh to finish even
when the due-refresh test fails, and covers stale rates with a failed
refresh (503), a failed background refresh keeping fresh rates, and a
refresh that raises.
* test(proxy): restore an unrelated import line reformatted by mistake
* test(proxy): use the current chat model in request pricing fixtures
* fix(proxy): reject pricing_currency on generic pass-through requests
An authorized key keeps request prices on a generic pass-through, which
bills them as USD, while pricing_currency was forwarded upstream untouched.
The pass-through body boundary now rejects a non-null pricing_currency with
a 400 and drops a null one, matching the chat-only conversion.
* fix(proxy): type the request pricing conversion call and ECB client import
Validates the request body to a typed mapping only when pricing_currency is
present, converts the supplied prices from that view, and writes them back
onto the original request. Imports the async HTTP client helper at module
level so its binding is typed. The client is still built inside the fetch.
* test(proxy): cover a null pricing_currency through the pre-call pipeline
* fix(proxy): fetch ECB rates with a short-lived handler closed after each refresh
Builds an AsyncHTTPHandler inside the refresh fetch and closes it when the
fetch ends, instead of binding the untyped cached client factory.
* fix(proxy): wait a minute before retrying a failed ECB rate refresh
A failed refresh left rates due, so every later non-USD request started a new
ECB fetch during an outage. The shared lookup now reuses a failed refresh
for 60 seconds from its start, so requests in that window get the held
rates or a 503 without fetching again, and the next request after it
retries. It also goes back to the cached async client factory, with a named
type suppression for the factory's untyped params argument.
* fix(proxy): convert pricing_currency on every chat completions alias
The /engines/{model}/chat/completions and /openai/deployments/{model}/chat/completions
aliases reach the same chat handler but were rejected, because the check
compared the literal request path. It now compares the registered route
template, falling back to the path when no template is available. Also
covers a cancelled refresh not blocking the next lookup.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): rewrite batch JSONL rows to the deployment model on model-routed uploads
* fix(proxy): rewrite only alias rows and authorize them against the alias at batch create
* fix(proxy): keep batch create working when the credentials lookup names no model
* fix(proxy): spool the batch alias rewrite to disk and close it after upload
* fix(proxy): rewrite only alias rows in model-routed batch uploads and charge both project quotas
Rows that do not name the alias now reach the provider exactly as uploaded, and a file with no alias row is sent unchanged. Rewritten rows are compact UTF-8 JSON. Rows routed through the alias count against the project quota of both the alias and the model the row names
* fix(proxy): keep every byte of non-alias batch rows and skip files with nothing to rewrite
* fix(proxy): rewind batch uploads instead of calling seekable() so Python 3.10 works
* fix(vertex_ai): honor deployment and row drop_params for sampling params in batch file rows
* fix(batches): rewrite escaped alias rows and charge shared quota counters once
Parse each row's model to find rows naming the alias, so an alias written
with JSON escapes is rewritten too. Merge a row's alias and real model into
one project quota descriptor per counter, so a model_group_alias that puts
both in one group charges the row once against the lower limit.
* test(batches): prove the shared quota counter is charged once on the Redis store
* test(files): capture model-routed batch uploads at the provider HTTP boundary
* fix(batches): charge a shared project counter once per batch and keep rows JSON cannot re-encode
A batch file that mixes rows naming the proxy alias and rows naming the
deployment model built one project ITPM/OTPM descriptor per row group, so
the in-memory limiter checked each group against the quota on its own and
let the batch through. Descriptors now merge by counter with summed tokens,
and the 429 detail reports the merged total.
A row whose numbers JSON cannot write back, such as 1e400, was rewritten as
Infinity. It now stays byte-identical, the same as rows the rewrite skips.
* fix(batch_rate_limiter): hold a shared project counter to its strictest limit
When a batch file mixes rows naming the alias and rows naming a model in the
same group, both row groups charge one project ITPM/OTPM counter. The merged
descriptor kept the first group's limit, so a stricter limit set on the other
name was skipped. Merge them with the limiter's coalesce_rate_limit_descriptors,
which keeps the lowest limit, the same rule a single request already follows.
* chore(rust): restore generate_stdlib_fixtures.py as it is on main
An earlier merge commit reformatted this ruff-excluded fixture script. Restoring main's bytes keeps the PR to the batch alias fix
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>