4096 digests held about 20 long agent sessions per worker, so interleaved
sessions evicted each other and every resend rescanned its history;
65536 digests (about 10 MB) hold a few hundred. Tool payload secrets are
now replaced longest first after JSON unescaping, so a shorter overlapping
match cannot leave part of a secret behind. The key-context test checks
the work it bounds instead of wall-clock time, which CI could not meet
Agents resend their whole conversation on every turn, so each turn
rescanned every earlier tool result. Each guardrail now keeps a bounded
set of digests of scan inputs that held no secret and skips those; text
with a secret is always rescanned and no secret material is kept
Responses carries the same tool call arguments and system prompt as chat
and messages under function_call.arguments, custom_tool_call.input and
instructions, so secrets there reached the provider unscanned
List items keep their parent key in the scanned JSON line, capped to its
last 64 characters, so a credential key over a list is still detected
while a long key over many items no longer multiplies the scan text
Treat tool arguments that decode to a lone surrogate as raw text, decode
JSON-escaped detections back to the leaf value, quote unkeyed leaves
without repeating a parent key per list item, drop cycle tracking that
parsed JSON cannot reach, and use the CI license instead of a fake
license server
Scan each tool value beside its key instead of an indented dump, read
tool_result content one level deep, keep non-finite tool arguments as
raw text, and give read-only guardrails one fragment per tool payload
* fix(azure-ai): support Entra auth for Decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure-ai): select Decisions provider by name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): prioritize deployment credentials over env key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): restore api key precedence for Decisions auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618)
Invoke streams of Claude models now go through the same frame-by-frame relay as
converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages
stream translation, a block ends the stream with an AWS exception frame, and a held
stream gets its masks written back into re-encoded chunk frames. Claude is told by
its model id or, behind an application inference profile, by the anthropic_version
its request body carries. Streams of other model families stay as they were
* test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618)
* fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618)
A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while
converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered
path, with the rewrites written back into the chunk frames that carried each text.
Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does,
so a vendor can tie the scans of one streamed reply together.
* refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618)
* fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618)
An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned
as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned.
The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream.
* test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618)
* test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618)
* test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests
* test(bedrock): type the remaining parameters of the Claude invoke stream tests
* refactor(bedrock): bind the buffered invoke rewrite in its condition
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515)
* fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515)
A post_call guardrail that scans through apply_guardrail now runs on
/bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the
caller as the upstream sends them, every fifth frame and the end of the stream
are scanned, and a block after the first frame ends the stream with an AWS
exception frame. streaming_buffer_until_moderated still holds the stream and
applies masks per content block. Guardrails that need the whole response
(own async_post_call_success_hook, mask_response_content, native hooks, policy
pipelines) keep the buffered path.
* fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515)
* fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515)
* fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515)
- The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the
upstream ends with messageStop, an exception or error frame, or a dropped connection
- A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the
failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no
failure row
- A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does
- Closing the response before its body starts closes the guardrail layers and the upstream
- Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never
reaches frames a later layer still holds
- Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser
* fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515)
* test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515)
* fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515)
Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client
disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them
from the request the way chat completions streams do
* fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515)
* fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515)
* fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515)
A held post_call guardrail with streaming_buffer_release_on_scan releases each
window as received once its scan passes, so the relay delivered the original
text instead of the guardrail's masks. Such a stream keeps the buffered path,
which applies masks per content block as on main.
* fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515)
The relay ran every guardrail in config order, so a run_in_parallel guardrail
listed before a masking guardrail scanned the unmasked text. Main's buffered
path runs parallel guardrails after the sequential ones and discards their
rewrites, which the relay cannot reproduce, so these streams stay buffered
* fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves
- Noma v2 scans on the converse-stream relay carry stream, and the last one
carries the finished Converse response with its stopReason, as chat sends
them, so Noma keeps one record per reply. Other guardrails get the request
unchanged. When the last sampled scan already covered the whole answer, the
end-of-stream scan still runs for Noma so it gets the finished answer
- A held relay stream whose client leaves before the first frame stops
reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet,
the budget reservation and the parallel-request slot are released
- A cancelled wait for the first streamed chunk now stops the upstream read
instead of leaving it running
- A Noma v2 scan cancelled before Noma answers records not_run instead of
success with an empty response
* fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks
* fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input
* test(bedrock): expect the ValidationException main now returns for a held block before the first frame
Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400.
* fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message
A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw
event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the
messageStop, skipped its write-back and released the held text without its masks. The Converse handler now
reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text
* fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame
The relay skipped releasing the slot and budget reservation whenever a spend
flush was scheduled, but a flush of a partial first frame builds no response
and logs nothing, so the slot stayed held until it expired. The flush now says
whether it logged the stream.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(e2e): warm the memory test up with two batches before the measured one
On a cold worker the second batch of failing requests can still raise the
allocator's high-water mark by about 50 MB while the live Python heap stays
flat, so the 48 MB budget failed about 1 in 10 Buildkite runs without a leak.
The test now runs two warmup batches, measures the third against the same
budget, and records every batch's settled growth per worker as an
rss_batch_growth JUnit property for later calibration
* test(e2e): record each memory batch's growth as soon as it settles
A failure in a later batch no longer drops the growth already measured, and
the worker keys are no longer sorted, which raised on a mix of named and
unnamed hosts
* test(e2e): silence the xunit2 record_property warning on the memory test
* test(e2e): event_loop_stalls perf case reads the gateway's asyncio debug log and fails on a loop hold over 100 ms that is not on its allowlist
The gateway runs the stdlib loop in debug mode (new LITELLM_EVENT_LOOP=asyncio knob next to
PYTHONASYNCIODEBUG=1), the case replays the mixed body mix at 1 rps and groups asyncio's
'Executing <Task ...> took X seconds' lines by coroutine and location. Known holds are listed with
the count they show today; anything else, or a known one past its count, fails the case. Opt-in
through E2E_PERF_EVENT_LOOP=1 so the nightly build skips it
* test(e2e): event_loop_stalls reads the pod list and pod logs through e2e_http instead of a raw httpx client
* test(e2e): event_loop_stalls charges a hold to garbage collection when the gateway's debug-mode gc log accounts for it
The proxy registers a gc.callbacks hook only when the event loop runs in debug mode and logs every
collection over the loop's slow-callback threshold (generation, collected count, seconds). The case
parser pairs such a pass with the hold line that follows it and reports the hold under a bounded
garbage collection bucket instead of the task asyncio happened to blame, which made 7 of 10 local
runs fail on a different task name each time. Holds the collector does not account for keep the
strict allowlist, which gains the prisma first-query warm-up and the second pub/sub subscriber
* test(e2e): event_loop_stalls allowlist caps set at twice the ten-run maximum
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(logging): keep the async request's logging payload when sync callbacks run late
Treat the "cache" success callback as internal when deciding whether to start
the sync success thread for async requests. It has no branch in the sync
handler, so with `cache: true` that thread did nothing except rebuild and
overwrite model_call_details["standard_logging_object"].
When the sync handler does run for an async request (a real sync-only logger
is configured), it now builds its own payload and hands it only to its own
callbacks. The async handler keeps sole ownership of the shared payload, so
edits made by async logging hooks (for example the v3 rate limiter's mirrored
x-ratelimit headers that Prometheus reads) are no longer discarded. Sync
requests keep their existing behavior.
* fix(logging): keep sync cache hits logged and isolate the async-request payload build
The sync completion cache-hit path reused handle_sync_success_callbacks_for_async_calls,
so treating "cache" as internal silently dropped sync callbacks on sync cache hits. It now
submits the sync handler through its own method, as it effectively always did.
For async requests the sync handler builds its payload from a copy of the call details, so
message redaction cannot rewrite the shared standard_logging_object, and when it builds no
payload its callbacks keep the shared one instead of getting None.
* test(caching): lock sync cache-hit logging for completion, embedding and text_completion
* test(logging): keep sync hook edits off a caller-supplied payload for async requests
* fix(logging): give sync hooks a copy of the async payload when the thread builds none
* test(logging): check every legacy sync logger gets the thread's own payload
* test(logging): cover traceloop and supabase sync logging for async requests
* fix(caching): keep prompt cache usage on streamed cache hits
A stream=true request served from the response cache rebuilt its usage from
the prompt, completion and total counts only, so prompt_tokens_details and the
cache read and creation counts were dropped on the replayed stream. /v1/messages
clients routed to Bedrock Converse saw input_tokens equal to the whole prompt
and no cache tokens on a hit.
Both replays now build the usage through one helper that also restores
prompt_tokens_details and the integer cache read and creation counts, skipping
cached values that do not validate.
* fix(caching): coerce replayed token counts the way the live usage does
The replay accepted only exact ints for the token counts, so an integral float
or numeric string in a cached entry became 0 where the previous replay kept it.
Counts now go through pydantic int coercion and fall back only when they do not
validate.
kill() returns once the killed worker's main thread is a zombie, but its other threads can
still be exiting and holding its sockets open. A keep-alive connection pooled to that worker
still looks healthy, so the next request goes out on it and the closing socket answers with
a reset. The old reap wait used to hide this. A new connection can only be accepted by a
live worker, so the surviving-worker assertions now run on a fresh client
With E2E_PROVIDER_CACHE=1 the test's Anthropic deployment went through the
shared response cache. Each call's recording expires on its own, so a run
could replay the cache-writing call while a follow-up went live against a
cold provider cache, recording creation instead of a read. That recording
then replayed as a failure for 24 hours. Mark the test provider_live, as
PROVIDER_CACHE.md requires for prompt-caching tests
Under worker isolation the harness rewrote every proxy config into the owned proxy's directory, so modules a test placed next to its own config (custom_auth) no longer resolved and the proxy died at boot. The copy now lands next to a test-supplied config, while configs inside the checkout still go to the worker directory so nothing is written into the repo tree
* ci(cost map): remove the generated model prices JSON schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost map): keep schema-independent cost map tests after dropping the schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(cost map): keep the schema file until main's guard stops requiring it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(cost map): let bot sync PRs keep touching the schema file until it is deleted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for both workers to load each metered model before sending
* fix(prometheus): read remaining rate-limit headers as numbers and let the router fill unreadable ones
* fix(prometheus): keep deployment metrics on non-string header keys and cover decimal and router-filled gauges
Accept any mapping key when reading additional headers, so a non-string key no longer drops every success metric. The router fallback test now uses a real Router and covers inf, nan and unreadable requests values, the decimal test uses a fractional requests value, and the integration cell sends 12.5 and an unreadable tokens value to deployments with tpm set
* fix(prometheus): leave the router fallback unchanged and keep only the decimal header fix
* fix(prometheus): read remaining rate-limit headers with float() like every release did
* test(integration): check the remaining rate-limit gauges on /v1/responses and /v1/messages too
* perf(cli): start lite without initializing the litellm SDK
lite and litellm-proxy now enter through a small litellm_cli package that imports the CLI against an uninitialized litellm package and runs the SDK __init__ in place only when something first uses it, so commands start in a fraction of a second instead of loading about 3,000 modules first.
* perf(cli): import only the lite subcommand being run
Each lite subcommand now loads its module when Click looks it up, so
lite logout no longer imports the autoroute wizard, agents, or the other
command groups. importlib.import_module and attribute reads on the SDK
also initialize it on first use, and an already imported litellm is
kept as is.
* fix(cli): initialize the SDK on a star import and test command bodies in thin mode
A from litellm import * after lite loaded the CLI used to see only the names the thin package had, so it now runs the full SDK init first. The thin-vs-full parity test now also runs logout and whoami, not just --help
* fix(cli): keep the import hooks until the lazy SDK init finishes
A thread that imported SDK code after the init restored the interpreter
imports could hold that module's import lock while waiting for the init,
and the init then waited on the same lock, so both threads hung. Other
threads now wait for the init before they load any new SDK code
* fix(cli): retry a failed lazy SDK init on the next use instead of leaving it half-run
* test(cli): bind the patched credentials client and autoroute dir in an order-independent way
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it
Bedrock accepts outputConfig structured output on every reachable profile of
Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so
json_schema now goes out natively there instead of as a forced tool, which kept
Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its
keys go back to the tool path. GovCloud keys keep their flags, and the drift
tests let GovCloud lag its commercial base but never lead it.
* fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path
Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf,
additionalProperties true, and recursive or unresolved $ref, all of which the
forced tool accepted. Claude schemas on the Bedrock native paths now go through
the cleaner the Anthropic native path already uses, and a schema with a
recursive or unresolved $ref keeps the forced tool, inline, or drop path it had.
* fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key
Older installs fetch the model map from main, so flipping supports_native_structured_output
moved them onto native output with their old schema handling. The new
supports_bedrock_native_structured_output key is ignored by older code. Schemas native output
cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new
key, models on the old key keep their current behavior for those shapes, and the Anthropic
API, Vertex AI, and Azure keep their current behavior.
* chore(cost-map): regenerate the model prices schema for the Bedrock native output key
* fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool
Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since
Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters.
Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects
typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1.
* fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking
* fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model
Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1
* fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas
The Bedrock native output key now applies only when the call asks for thinking, so calls without
thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts,
or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open
objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned
them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request.
* fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke
* test(bedrock): cover native structured output and json tool routing through the proxy
Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking
* fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode
Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas.
* fix(anthropic): keep native output for dict fields unless the json tool can be forced
* fix(anthropic): keep native output for dict fields when the caller sends tools
* test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects
* test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas
Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool
* fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output
* test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects
* ci(cost map): classify token_count_model as an object key in the schema generator
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(proxy): unset DB environment variables removed from the row on every worker by row revision
* fix(proxy): order the environment variables revision migration after main's and set its lock timeout
* fix(proxy): snapshot the environment at the first DB sync even when no environment_variables row exists yet
* fix(proxy): close gaps in DB environment variable reconciliation
Grant the revision sequence to every role so a writer role with only
table grants can still save environment_variables. Unset SSO keys that
only the environment_variables row stored on the next SSO sync, while
keeping keys the SSO settings table holds. Skip a revision-less row once
a revision has been applied, so a stale cached row cannot bring a
removed key back. Keep plaintext and undecryptable values when rotating
the master key instead of dropping them from the row.
* fix(proxy): keep env var revisions monotonic across DB copies and restores
Stamp at least the old revision plus one on update so a copied row keeps
rising, re-read a stale cached row from the writer and apply it when the
database itself went back, unset dropped SSO keys when sso_settings are not
loaded from the DB, and keep env values master key rotation cannot decrypt
exactly as stored
* fix(proxy): skip an env var row read before an overlapping sync that already applied a newer one
* test(proxy): cover a read replica back on an older row and an SSO client removed from DB environment_variables
* fix(proxy): keep config reads up when the writer is down, find the revision sequence by table schema, read the writer before rewriting env vars, and add the trigger under db push
* test(proxy): model the writer database in the retry policy config update fake
* fix(proxy): recheck the environment_variables row revision right before applying it
A sync that started later could read a stale environment_variables row
before an earlier sync applied a newer one, then pass the read-order check
and apply the stale row, bringing back keys the newer row removed. The
behind-the-applied-revision check now runs at apply time, re-reading the
writer when the row is behind, and a writer row read while another sync
applied a newer revision is dropped.
The config sync tests now inject a fake database client and use the real
get_config_param and its cache instead of replacing get_config_param.
* test(proxy): keep encryption real in the SSO replica-lag test
* test(proxy): inject a fake Prisma client in the env var replica-lag test
* fix(proxy): keep the applied environment variables when the writer database is out of connections
* test(proxy): cover a writer database out of connections in the env var integration tests
* build(docker): cache the Rust extension build in its own stage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): shorten the rust-native stage comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): copy .cargo config into the rust-native stage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): drop comment restating the wheel install
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cache-creating call now reports cache_creation_input_tokens on top of
Vertex's own prompt count and bills the write at the input rate, as Google
does. The old assertions assumed neither, so the test went red on every run.
Check the cache write, the discounted read, and that a second identical call
reuses the cache instead.
* test(e2e): repair fallback round trip for config-owned fallbacks
The e2e stack config sets router_settings.fallbacks, and since #45860 a
/fallback write to a key the config file owns is refused with a 400. The
round trip now runs on context_window fallbacks, which the stack config
leaves to the database, and also deletes the rule and checks it is gone.
A second test locks the new contract: a /fallback write to the
config-owned fallbacks key returns 400 naming router_settings.fallbacks,
and the config rule keeps being served unchanged.
* test(e2e): assert the refused fallback rule is never served
#46493 reads the first stream chunk inside the router's retry and fallback
chain, so an invalid prompt rejected before any token now falls back to the
configured fallback deployment instead of answering 400. The author
confirmed this is intended
The fallback test now asserts the spare serves the stream. The old 400
without a fallback attempt still holds when fallback_on_unsupported_params
is false, so that contract moves to a second proxy with that setting
* fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop
create() walks every message, content part and tool of the request through
openai._utils._transform before posting, 91 ms at 300k tokens and ~300 ms at
1M tokens on the event loop, and returns the same dict for the plain dicts
litellm builds. client.post is what create() calls after that walk, same
request, headers, retries, raw response and stream objects, so the chat
helpers now call it directly through the helper embeddings already used
* fix(openai): type the chat client.post result as LegacyAPIResponse so the type gate sees .headers and .parse
* fix(openai): assert the chat client.post result is a LegacyAPIResponse instead of casting it
* test(openai): stub client.post with a LegacyAPIResponse spec in the empty-response and no-leak tests
* test(core-utils): stub client.post with a LegacyAPIResponse spec in the agentic loop leak test
* test: every client.post stub on the OpenAI chat path returns a LegacyAPIResponse-spec'd mock
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The old docstring called completion_response required, said total_time was in
seconds (it is milliseconds), and only covered a handful of the parameters.
Replace it with a short accurate summary and fix the "calculat" typo in the
unmapped-model error.
Co-authored-by: Kerry Lu <kerry@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice
* test(bedrock): name the malformed realtime setting test for its string cases
* fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit
Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params.
* fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats
* fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices
* test(bedrock): cover realtime replay, refusal and committed failure through a real proxy
Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again
* fix(openai_like): surface provider response headers on non-streaming chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(openai_like): mark the header test locals Final
* fix(proxy): leave out response header values HTTP cannot carry
Non-streaming OpenAI-like chat now forwards every provider header, so on the
httpx transport a provider header value outside latin-1 made the proxy answer
500 after the provider had served the call. get_custom_headers now leaves such
a value out and keeps the other headers, which also ends the same 500 on
streaming
Adds integration cells for the provider request id on non-streaming chat and
the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend
parity under an upstream cost header, the latin-1 case, and the logging
callback payload
* fix(cost): ignore an upstream response cost header that is negative or not a number
* test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint
* fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number
* test(prometheus): type the rate-limit header test and drop its clock reads
* test(integration): cover deployment metrics under an unparseable provider rate-limit header
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): preserve key settings drafts during background refreshes
* fix(ui): refresh untouched key settings while preserving edits
* fix(ui): keep team-change MCP clears across key refreshes
clearMcpGrants now marks the MCP fields dirty, so a background refresh with
keepDirtyValues no longer restores the grants a team change just cleared.
Also adapts the draft tests to the onSubmit(values, dirtyFields) signature
on main and asserts a refresh does not mark untouched fields dirty, since the
team admin member-key path rejects saves with non-budget dirty fields.
KeyEditView becomes a one-line arrow to stay under the max-lines limit.
* fix(ui): keep MCP tool defaults out of a key's edited fields
MCPToolPermissions writes non-delete tool defaults for unrestricted servers
on mount. Marking that write dirty made a team admin's budget-only save on a
member key get rejected for editing mcp_tool_permissions. onChange now says
whether a write is a default or a user edit, and the key editor dirties only
edits.
The edit session is now keyed by token or token_id, matching the form's own
fallback, so switching between keys that only carry token_id starts fresh.
* fix(ui): keep a new MCP server's default tool limits across refreshes
Default tool writes for a server the user just added in this edit session
are part of that edit, so they are marked dirty and survive a background
refresh. Defaults written for an already saved server stay clean, so a team
admin's budget-only save on a member key is still accepted.
* fix(azure_ai): count Grok reasoning tokens in completion_tokens
Grok on Azure AI Foundry reports reasoning_tokens outside completion_tokens,
like xAI does, so azure_ai/grok-* calls billed only max(visible, reasoning)
output tokens. Fold them with XAIChatConfig.fold_reasoning_tokens_into_completion
in transform_response and in a new streaming iterator, matching the xai provider.
Split out of #46377.
* fix(azure_ai): fold reasoning tokens for every deployment name, streaming or not
Non-streaming checked the deployment name for grok while streaming checked the
chunk model field, so a Grok deployment named without grok was only folded when
streaming. The fold already only fires when total_tokens equals prompt plus
completion plus reasoning, so it now runs on every azure_ai response.
* feat(ui): shareable URL state for usage views and users-to-usage deeplinks
The Usage page now keeps the view, global view tab, date range and entity filters in the
URL, so links can be bookmarked and shared. Relative presets are stored as range=7d and
keep rolling, custom ranges as UTC from/to. Invalid params are dropped, and a link to a
view or user the viewer cannot access shows a toast and falls back to the default view.
Internal Users rows get a "View usage" action that opens usage filtered to that user, and
the per-user usage views link back to the user's record. A "Copy Share Link" button sits
next to "Ask AI"
* fix(ui): clean usage links for sessions without the organizations query
The organizations query only runs for premium sessions, and a disabled query stays
pending forever, so the URL cleanup never ran for other sessions. Gate on isLoading
instead, and give the access denied toast an id so a re-run effect shows it once
* fix(ui): type-check usage entity filters and address review findings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep team usage available to internal users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format users table usage action
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync date picker preset with the current usage range
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): restore dashboard AGENTS.md from main and drop unused usage state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): share custom usage date ranges as calendar days
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep per-user agent usage page across rerenders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
