Both restart tests boot two 2-worker proxies but capped the case at a fixed
180s, below even one calibrated 2-worker readiness budget. On a loaded runner
each boot took 80-90s, so pytest-timeout fired mid-boot or mid-shutdown
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): surface mid-stream upstream errors that arrive without an event type
Bedrock's native Responses route can end a stream with a bare {"error": {...}}
chunk. It parsed as a typeless GenericEvent and crashed the iterator with
AttributeError, so clients saw that instead of the upstream error. Such chunks
are now read as error events, which raise the mapped upstream error and reach
the client as response.failed.
* refactor(responses): inline the untyped error chunk check to keep LIT001 at the base count
* fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF
With OpenSSL enforcing FIPS, hashlib.scrypt raises, so every MCP session
token and bridge envelope request failed with a 500. Under FIPS the keys
now come from LITELLM_SALT_KEY through SP 800-108r1 KBKDF run by the
provider's EVP_KDF. Without FIPS the scrypt keys are unchanged
* fix(mcp): keep master key rotation invalidating MCP tokens under FIPS
Pass the master key as the KBKDF context, so rotating it changes the
three FIPS-mode keys as it changes the scrypt keys
* test(mcp): annotate new FIPS key test locals with Final
* fix(mcp): derive the FIPS MCP keys only when general_settings.fips_mode is on
On a FIPS OpenSSL 1.1.1 host (RHEL 8) scrypt still works, so keying the switch on
openssl_enforces_fips() sent those hosts to the OpenSSL 3 only KBKDF and broke MCP
sessions and bridge envelopes that worked on main. fips_mode_active() is the same
reader B1 adds, set by enforce_fips_mode at startup.
* refactor(fips): move the salt key KBKDF into encrypt_decrypt_utils
The KBKDF helper derives keys from LITELLM_SALT_KEY, so it now sits next to the other salt key helpers, and fips.py goes back to main's version.
* Pin the MCP key KBKDF fetch to the FIPS provider
Fetch KBKDF with the property query fips=yes, so a change to the process's default
properties after boot can't move the derivation to a non-FIPS provider. The shared
fips_openssl test fixture drops the pin, since the test OpenSSL has no FIPS provider.
* feat(ui): set deployment weights in key and team router settings
The proxy already applies router_settings.weights per key and team, but no
dashboard screen could set them, and saving a team in the UI dropped weights
set through the API
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): only offer deployments the key or team may weight
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show weight load errors, inherited strategy and team deployments on key create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): keep reliability fields on create forms and label weight inputs by deployment id
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(ui): name the accordion value identity object
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(ui): expect weights in the key create router settings payload
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): keep weight drafts, loading state and team switches correct
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): only accept weight keystrokes that save as a number
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(guardrails): add prebuilt credential patterns for JWT, and OpenAI, Anthropic, Gemini API keys
* fix(guardrails): prevent quadratic backtracking in jwt_token pattern
* test(guardrails): move credential pattern tests into test_patterns.py
* feat(guardrails): remove jwt_token to keep credential patterns strictly bounded
* fix(guardrails): mask whole provider keys and keep openai_api_key to OpenAI's key shapes
End each pattern so a key ending in "-" is masked in full, require OpenAI's
T3BlbkFJ marker on bare sk- keys so other vendors' sk- keys and LiteLLM
virtual keys are left alone, add sk-None- user keys, and replace the
registration-only tests with whole-match, length-boundary, and negative cases
* fix(guardrails): require the T3BlbkFJ marker on every OpenAI key form and match keys after escaped separators
openai_api_key now needs the marker on the sk-proj-, sk-svcacct-, sk-admin- and sk-None- forms too, as gitleaks and TruffleHog do. anthropic_api_key and gemini_api_key also start a match right after a literal \n or \t or a percent-encoded byte, so keys in pasted JSON strings and URL-encoded logs are masked.
* fix(guardrails): bound the OpenAI key regex, match Gemini prefixes case-sensitively, and hold back long keys in streamed replies
* fix(guardrails): size the streaming holdback per pattern, stop OpenAI key tails before the next key, and allow non-ASCII text after a key
* fix(guardrails): mask content filter pattern matches found in the original text so overlapping patterns cannot split a key
* fix(guardrails): let a block pattern see the text after earlier masking patterns, as before
Masking patterns still match the original text and merge overlapping matches into one tag, but a BLOCK pattern checks the text with the masking patterns listed before it applied, so a request main masked and passed is not blocked now. Detections name only the patterns whose tag was written.
* fix(guardrails): keep content filter masks in order and hold back the longest OpenAI key
Masking patterns run in configured order on the masked text as on main, so a contextual
pattern still matches next to an earlier tag, and each also matches the original text so an
earlier tag cannot split a key. Detections follow the configured pattern, not its name, and
the OpenAI streaming holdback covers the pattern's 419-character longest match.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cost-map): move -latest model aliases into the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the finalized local cost map in the moonshot alias test and assert alias pricing by behavior
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: compare alias cost map rows to their targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): give each -latest alias its own alias_of row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 91e6cf2c27)
* fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit a54c42218a)
* feat(cost-map): add -latest alias rows for the tier 1 and 2 hosting providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-map): derive hosted alias targets from the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): skip alias rows when checking cache-write rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): round-trip encrypted reasoning through /v1/messages
The chat bridge stored encrypted reasoning on reasoning_items, the Anthropic
adapter never read that field, stream chunks that only carried thinking
blocks were dropped, and the messages adapter stripped the signed blocks
before a chatgpt request could turn them back into reasoning items.
Co-authored-by: DDT <1786035110@qq.com>
* fix(anthropic): keep Responses reasoning on tool turns and retry once when a backend cannot decrypt it
* fix(anthropic): stop logging rejected encrypted reasoning
Quote the retry helper return type so importing litellm no longer evaluates a string alias with |. The warning keeps provider, model, status, and error code, and leaves the ciphertext out.
* fix(anthropic): keep encrypted reasoning on OpenAI thinking and xAI web search turns, and stream signatures only to Messages clients
The keep check now knows when the handler moves an OpenAI thinking turn to Responses, and when web_search_options sends an xAI turn there.
The Responses-to-chat stream iterator adds the signed reasoning chunk only when the caller is /v1/messages, so /v1/chat/completions streams stay unchanged.
The reasoning retry fires only when the translated request still carries encrypted reasoning.
* fix(anthropic): keep each streamed reasoning summary with its own item, and read the Messages marker before the bridge call
* fix(anthropic): send openai/ Messages turns again once without encrypted reasoning a backend rejects
The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge.
* fix(anthropic): keep every non-streamed reasoning item for Messages clients and read a deployment's reasoning_summary in the keep check
* fix(anthropic): log only the outcome of a turn sent again without rejected reasoning
A /v1/messages turn whose replayed reasoning the backend rejects is sent once more without it. The rejected first attempt ran the failure callbacks on the shared logging object, which logged a failure for a turn that then succeeded, kept the retry's own failure from being logged, and freed the parallel-request slot while the retry was still in flight.
The first attempt now runs inside resent_on, and the client wrapper skips failure logging and the budget unbind for a failure that matches it. The retry runs outside that context, so its own outcome logs as usual.
* test(anthropic): type the parameters of the failure-recording test loggers
* fix(anthropic): skip the SDK num_retries retry for an attempt that is sent again
* fix(anthropic): stop SDK retries from resending a rejected block or clearing the global num_retries
A transient 429 or 500 on a /v1/messages turn that carries a replayed encrypted
block was retried by an SDK call's num_retries loop, which sent the block again
until every retry was rejected. On async calls the loop then raised the first
transient error, so the turn failed instead of being sent again without the
block. The retry loop now stops at that rejection and passes it on, so the turn
is sent again once without encrypted blocks.
A rejected first try on a sync call also cleared the process-wide
litellm.num_retries, so later calls in the process were no longer retried. That
setting now stays as configured.
* fix(anthropic): stop a cancelled SDK retry from starting another call
The resent-failure retry predicate returned true for BaseException
subclasses such as asyncio.CancelledError, so a cancelled
acompletion_with_retries or aresponses_with_retries call could start
another provider call. Retry only Exception subclasses again, as
tenacity's default did, while still skipping a resent failure.
* test(anthropic): type the retry helper parameter in the cancellation test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cost-map): read alias_of pointer rows in the catalog and the loader
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): preserve streamed tool argument fragments verbatim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): send the schema-less stream test the null properties form providers accept
* fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece
* fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams
* fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON
A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches.
* fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook
* fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON
* fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers
Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks.
* fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers
* refactor(anthropic): encode JSON mode answers without a recursive helper
* fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation
* fix(anthropic): type the decoded JSON mode answer as object and test non-object answers
json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores
RedisCache and the native Redis client each kept their own breaker, so once
the cache learned the coordination Redis was wedged, a spend-counter write
through the native store still paid a full socket timeout. Every Redis target
now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected
into both the RedisCache guard and the native client. The Python FSM is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(rust): cargo fmt the shared breaker
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(redis): close the breaker when the recovery probe gets a Redis reply error
A command error such as WRONGTYPE means Redis answered, but the probe was left
unsettled and its drop reopened the breaker for another recovery timeout. Both
the RedisCache guard and the native client now settle such a probe as a success.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test
The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw.
* fix(redis): key the shared breaker by server and close the three audit gaps
The breaker registry now keys on the Redis server (host and port, unix socket,
cluster node set or sentinel service) instead of the full connection target, so
caches that differ only in db, credentials or url spelling share one breaker.
A READONLY reply from a replica mid-failover no longer counts as Redis answering,
so it can't reset a run of failures. A new RedisCache refreshes the breaker state
gauge, which the deleted Python breaker did at construction. A shared native client
is only reused by a store that passes the same breaker.
* refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations
lib.rs now only declares modules and re-exports, with resolution, target, tls,
auth and endpoint each in their own file. Derived FromPyObject already reads a
struct field as the attribute of the same name, so #[pyo3(attribute)] on the
breaker and store settings did nothing.
* refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot
Derived FromPyObject already reads each field as the attribute of the same name.
The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in
the same order.
* refactor(redis): return a HashMap from the three memory-tier snapshots
The snapshots read cache-memory's HashMap, so there is no order to keep.
PyO3 builds the same dict from a std HashMap, which replaces the hand-written
PyDict loops in the spend counter and router usage stores and the IndexMap in
the cooldown store. The rate-limit slot registry keeps IndexMap because it
hands back a Python dict in the order Python wrote it.
* refactor(redis): share Redis breakers and clients through one RedisHosts per host
redis-connect's RedisHosts now owns the per-server breaker and the shared clients
that python-bridge kept in two process-wide statics. A client takes breaker
settings and admits through its server's breaker, so no caller can pair a client
with a breaker of its own, and clients are keyed by Resolution alone. This drops
RedisClient::guarded_by and Breaker::same.
python-bridge keeps one RedisHosts per process and calls it with the GIL released.
The gateway builds one per connect, so its coordination and response cache on the
same server now share a breaker. RedisStoreSettings.breaker is back to breaker
settings, as on main.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(vertex_ai): run auth once per Vertex pass-through request
Drive /vertex_ai, /vertex-ai and /vertex_ai/discovery requests through the real
proxy app with a counting custom_auth, and assert each request authenticates
once and is served as the caller that auth returned. The generate-content
routes fail today because the route authenticates a second time.
* fix(vertex_ai): authenticate Vertex pass-through requests once
vertex_proxy_route already resolves the caller through Depends(user_api_key_auth),
but _base_vertex_proxy_route ignored it and authenticated the request again. A
custom_auth function ran twice per request, and a key with max_budget reserved
its estimated cost twice, so the unreleased first reservation exhausted the
budget after one or two calls.
The base route now uses the caller it is given and authenticates once itself
only when none is passed, which is the discovery route's case.
* fix(vertex): read the caller's key only from x-litellm-api-key or Authorization on Vertex routes
The single auth check on /vertex_ai and /vertex-ai now reads the same two headers main's effective check did, so a key sent only in another key header gets 401 again instead of authenticating.
* fix(vertex): authenticate the discovery route through the same Vertex key check
Both Vertex pass-through routes now resolve the caller once through
_authenticate_vertex_caller and hand it to _base_vertex_proxy_route, which
no longer re-runs auth. x-litellm-api-key accepts the same scheme prefixes
on /vertex_ai/discovery as on /vertex_ai and every other proxy route.
* fix(proxy): size a Gemini request's budget hold from its prompt and maxOutputTokens
A Gemini generateContent body's budget hold now counts its contents and
systemInstruction and reads generationConfig.maxOutputTokens. Before, it held
the model's whole context window, about $0.85 on gemini-3.8-flash, so a key
with a small budget got 422s on back-to-back calls it could afford
Also removes get_litellm_virtual_key, which nothing calls since the Vertex
routes stopped using it
* fix(proxy): count every Gemini system part and candidateCount in the budget hold
A Gemini generateContent body with several systemInstruction text parts lost
every part after the first when LiteLLM translated it to a chat request. That
dropped the later system text for non-Gemini models behind
/v1beta/models/<model>:generateContent, and made the budget hold count only the
first part. The adapter now keeps every text part, as a plain string when there
is one part and as text blocks when there are several.
The output hold now also multiplies by generationConfig.candidateCount, the
same way chat bodies use n, so a request asking for several candidates holds
budget for all of them.
* fix(proxy): keep the context-window hold for Gemini bodies the prompt count cannot see
A Gemini generateContent body with media, cachedContent, other roles, or
non-function tools keeps the full context-window hold, since the chat
translation drops or undercounts those parts. The hold also reads the
snake_case generation_config output fields, and several systemInstruction
text parts reach non-Gemini models joined into one system string
* fix(proxy): read only camelCase Gemini output fields in the budget hold
/v1beta forwards only generationConfig.maxOutputTokens to non-Gemini models, so a snake_case cap the model never receives must not shrink the hold. Snake_case bodies now get the default output hold.
* fix(proxy): count snake_case candidate_count in the Gemini budget hold
* fix(vertex_ai): skip malformed system parts and read Gemini output fields only on Gemini bodies
A systemInstruction part whose text is not a string is skipped, as main did, instead of failing the request. The budget hold reads generationConfig output and candidate fields, and counts contents, only when the body has no messages, prompt, input, query or documents field. An output cap of 0 or below counts as no cap. Test recorders keep tuples instead of mutating lists.
* fix(proxy): take the budget hold's request shape from the route
On generateContent routes the hold counts only contents and reads its output
cap and multiplier from generationConfig, so stray fields of other APIs can no
longer hide the prompt. A max_tokens of 0 holds no output again, while negative
values and a Gemini maxOutputTokens of 0 or below hold the default output
* fix(proxy): hold one candidate for a count too large to price and read system parts like content parts
* fix(proxy): keep the spend already counted when a budget hold is far over the budget
* fix(proxy): count candidateCount in the budget hold only for models that generate the candidates
A non-Gemini model reached through /v1beta generateContent never receives candidateCount, so a huge count held the whole budget for the length of the call while the model produced one answer
* fix(proxy): hold a candidate count only when every deployment generates it, and drop n when every deployment drops it
* fix(proxy): count candidateCount when any deployment generates it and always count n
The stricter rules from the previous commit let a key spend past max_budget through a Gemini name no deployment serves, an allowed_openai_params n, and a router fallback. Restore the earlier rules and add regression tests for those three shapes.
* fix(proxy): cap the candidateCount hold at main's default output hold
A candidateCount like 1e300 sent to a mixed Gemini and Claude group, or through a key alias from a Gemini name to Claude, held a whole key while Claude served the call and answered a strict key 422. On generateContent routes the counted candidates now raise the output hold at most to the default output hold main used for the model, so no count makes a call hold more than on main. Chat n is unchanged.
* fix(proxy): hold every Gemini candidate up to the merge base's whole hold
The previous cap stopped the candidateCount output hold at the default
16384 tokens. Because the hold now counts the Gemini prompt instead of the
whole input window, a strict key admitted a call like 4 candidates of 8000
tokens and then spent past max_budget, where the merge base refused it.
Every candidate is now held at its output cap, like chat n, and the whole
hold is clamped at what the merge base held for the same call (the model's
input window plus the default output), never below one candidate's hold.
A huge count still holds no more than the merge base did. Chat n is
unchanged
* fix(proxy): let a Gemini output cap that /v1beta drops only raise the hold
The Vertex and Gemini pass-throughs forward every spelling of the output
cap, but /v1beta passes only generationConfig.maxOutputTokens to other
models. The hold read only that spelling, so a snake_case cap above the
default output held less than the pass-through could spend. A cap in any
other spelling now raises the hold to it and never lowers it
* fix(proxy): hold Gemini candidates of a model with no context window up to the default output
A Gemini model the cost map has no context window for, priced in its deployment, held one
candidate for any candidateCount, so parallel calls could pass their budget holds. Its
candidates are held again up to its counted prompt plus the default output hold
* fix(proxy): keep main's Gemini hold unless the request caps its output
An uncapped generateContent call, or one capped only by a snake_case
spelling, now holds what main holds (the context window plus the default
output). A call with generationConfig.maxOutputTokens holds its counted
prompt plus cap times candidateCount, never clamped to the deployment's
max_output_tokens and never more than main's hold. A model with no context
window keeps main's no-hold, and candidateCount always counts on
generateContent routes. Also keeps get_litellm_virtual_key in
pass_through_endpoints/common_utils.py, so an out-of-tree importer
does not break
* fix(proxy): keep main's budget hold on generateContent routes
The PR now leaves the generateContent budget estimate as main has it.
A live risk check showed the counted Gemini hold under-reserves (the
prompt counter undercounts Gemini bodies, tiered and image output rates
are not read, and request fields the handler merges later are not
counted), so that estimate moves to a follow-up. The single auth pass
on the Vertex pass-through and the systemInstruction adapter fix stay
* fix(vertex_ai): keep main's Gemini adapter and pass a missing Authorization as no key
The Gemini adapter change for later systemInstruction parts goes back to
main, since it is not the double-auth fix this PR is for. The Vertex auth
dependency now hands a missing Authorization header to the auth check as
no key, like main's first auth pass, so custom auth and the 401 text see
what they see on main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(caching): don't cache an Anthropic Messages stream whose content blocks are all empty
A non-Anthropic model routed through the Messages adapter turns an empty reply into a single empty text block. The no-output check counted any content_block_start as output, so the async stream cache stored that empty reply and replayed it on every repeat. Judge each content_block_start and content_block_delta by its payload instead, like chat streams judge a choice: a block carries output when a field other than type holds something besides "", [] or null. Empty replies are no longer stored and stored ones become misses. A bare {} is not treated as empty, since the only block fields that hold one (tool_use input) come with an id and a name
Fixes#49931
* fix(caching): treat thought-signature-only and non-streamed empty replies as no output
* fix(caching): treat whitespace-only replies as no output
* fix(caching): keep whitespace replies and text completion fills cached
* refactor(caching): drop the unused streamed-choice branch of the output check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Add per-key and team prompt storage restrictions for spend logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply spend-log prompt restrictions to passthrough requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply team prompt storage restrictions to queued requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings
The spend-event collector decided whether to ship request and response
bodies from the global setting alone, so an opted-out key or team still
sent prompts through the collector transport. It now uses the same
key/team check as the spend log writer.
A key or team setting of the string "false" was ignored and prompts kept
being stored. It now counts as false, the same way the global setting
parses strings.
Splits the 42-case tenant storage test into one test for the policy and
one for the pass-through logging contexts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows
Failure rows, executed batch rows, semantic index embeddings, websearch
interception calls and errors rejected before request setup built their
logging metadata without the key and team metadata, so a restricted key
or team still had prompts stored on those rows. They now stamp it through
one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata.
The dashboard toggle now shows a saved string "false" as off, matching
the server, and its label and help text come from the translation file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(xai): use billed image cost and resolution prices
* test(xai): mark image generation tests as a package
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): forward Codex agent messages that carry encrypted content intact
Codex multi-agent v2 sends worker messages as agent_message items whose
encrypted_content block only the model server can decrypt. Bedrock Mantle and
Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into
assistant text, pasting the ciphertext into the prompt and dropping the sender
and recipient. Bedrock now decrypts these items itself, so the rewrite left the
recipient model reading an opaque token instead of its task.
Agent messages with an encrypted_content block now go to Bedrock untouched on
both endpoints. The Bedrock configs declare supports_encrypted_agent_messages,
and the shared normalizer keys off that flag so the two cannot drift. Mantle's
/v1/responses route for open-weight models still rejects agent_message in any
form, so it keeps the old rewrite. Plaintext-only agent messages keep the
rewrite on every route.
* fix(bedrock): only keep worker messages locked for OpenAI GPT models
Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so
Codex puts plain text in the locked slot and Bedrock rejected it with
invalid_encrypted_content once the route alone decided to forward it intact.
Decide from the model instead, and restore the full-value rewrite assertion.
* feat(videos): add first-class OpenRouter video request fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): remove recursive image parameter fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve EdenAI native parameter passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): deduplicate frame roles in linear time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve image selection and dropped parameter semantics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve provider scalar parameters and raw extra body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): retain accepted scalar inputs for Vertex and DashScope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): add Kimi K3.1 rows inherited from Kimi K3
* fix(bedrock): sync published Kimi K3.1 Highspeed prices
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): format dashboard values through i18next formatters
Register named formats on i18next's formatter (usd, compact, duration,
relativetime, number and five date presets) with a cache keyed on the
language and options, add format.* keys to en.json, and move the usage
stat strip onto them.
* fix(ui): keep number formats in USD and the format cache bounded
Force USD on the number and compact formats, clean underscore language
codes like i18next does, fall back when options cannot be serialized,
keep formatted placeholders alone in format keys, and pin the remaining
stat strip values in the German test.
* refactor(ui): format durations with Intl.DurationFormat and load its polyfill on demand
Browsers without Intl.DurationFormat (Chrome and Edge before 129, Firefox before 136) load the pinned FormatJS polyfill through a dynamic import and re-render translations once it is installed. The milliseconds-to-units split and the 0s fallback stay, so the output matches the previous hand-built formatter.
* test(ui): cover duration edge cases on the polyfill path
* refactor(ui): drop the unused duration format and its polyfill
Nothing outside tests called format.duration, and its 1m 15s style doesn't match the decimal-second latencies the dashboard shows today. A duration format will come with the pages that need it, in the style they already use.
#50009 moved Aria out of the dashboard header into the sidebar with its own
page, and #50195 renamed the Skills nav link and AI Hub tab to Skills Hub and
relabeled the skills search box. The UI e2e specs still looked for the old
header button, dialog and labels, so the scheduled litellm-e2e run has been
red since build 815.
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations
A team newrelic (or other preset) destination exported spans without the
operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback
was listed, because only the OTel logger's async_pre_call_hook seeded
requester_metadata into Baggage and that hook runs only for loggers in
litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the
request data once requester_metadata exists.
Resolves LIT-9783
* fix(otel): seed only request metadata from the proxy pre-call hook
The pre-call re-seed also passed the request model, which by then can be an
alias target, mcp-tool-call or an internal embedding model, so it rewrote
gen_ai.request.model on the server span for V2 deployments without otel listed.
Seed only the request metadata and keep the model auth seeded. Adds a unit test
for the pre-call seed and the guardrails_only skip, drops the explanatory
comments, and requires one server span plus the model span in the integration
test.
* test(otel): annotate new test variables Final and build test config without mutation
Addresses review: the new unit and integration tests now follow AGENTS.md
(LIT010 Final on every variable, no in-place mutation, read-only parameter
types, one for-clause per comprehension).
* test(otel): cover key-level callbacks and assert tenant spans carry no credentials
Parametrizes the tenant metadata test over team and key callbacks, and adds a
test that sends credential headers and a planted New Relic key with an
allowlist naming credential headers, then asserts no exported span carries the
master key, the virtual key, the team's New Relic key or any credential header
value, and that every export used only the team's own New Relic key.
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy
checked the same response a second time (LIT-10481). The deployment hook now records guardrail +
response hash on the request and on the response, and the proxy skips a guardrail only on a
matching record. The hash covers every field except ids, usage, model and other metadata set
between layers, so a response rewritten between the layers (OCR included) is still checked.
Guardrails with their own post_call hook keep their pre-#40154 behavior
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(usage): tolerate entity metadata database failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(usage): keep spend when metadata lookups hit database failures
Treat transport errors and the query engine crash on a closed connection as database
failures in the key, user and guardrail lookups of the aggregated, keys, users,
model_top_keys and cache_leakage_keys routes, and keep prisma out of module-level imports
so a proxy install without prisma still imports the usage endpoints.
* fix(usage): fall back only on database outages, on every daily activity route
Data-layer Prisma errors such as a rejected query or a missing table return 500 again, a key whose details failed to load reports key_exists null instead of false, and the non-aggregated daily activity routes keep spend when the key or entity metadata lookup hits an outage
* fix(proxy): skip the key metadata lookup when the daily rows carry no keys
* fix(usage): keep the guardrail breakdown fallback for any Prisma error
A missing guardrail usage table or column made the Usage call return 500 after the outage-only narrowing. The guardrail lookup now falls back on outages and on any Prisma error, as it did before this PR
* fix(usage): keep spend when the team, organization, customer, or agent name lookup drops
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): jais-30b-chat price was per 1K tokens, store it per token
* fix(watsonx): jais-13b-chat price was per 1K tokens, store it per token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(proxy): serve Codex its own release's model catalog, fetched from GitHub
When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that
release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the
fetch fails, times out after 2 seconds, does not validate, the version has no tag, or
LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight
most recent versions are kept. The integration tests ask as the bundled release so their expectations
hold whether or not the proxy can reach GitHub
* fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions
Failed versions now live in their own cache so they never push out a fetched catalog, requests for a
version already being fetched share that fetch, each worker starts at most six fetches a minute and
serves the bundled catalog past that, and the version check only accepts ASCII digits
* fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap
Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine
failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the
codebase
* refactor(typing): clear basedpyright Any errors (wave 1)
* refactor(typing): clear basedpyright Any errors (wave 2)
* refactor(typing): clear basedpyright Any errors (wave 3)
* refactor(typing): clear basedpyright Any errors (wave 4)
* refactor(typing): clear basedpyright Any errors (wave 5)
* refactor(typing): clear basedpyright Any errors (wave 6)
* fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves
* fix(logging): keep callbacks running when litellm_params is a params model
should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request
Adds behavior tests for the changed lines codecov listed as not run
* fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter
The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call.
* fix: bill regional uplift multipliers float() accepts, bytes included
* fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type
* fix(bedrock): run realtime guardrails on Nova Sonic sessions
* fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output
* fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames
* fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes
* fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs
Text parts of one message are joined before screening again, so a phrase split
across parts is still blocked, and a mask collapses them into one part. The
spoken notice is realtime_violation_message or the block reason, quoted only on
Nova Sonic. Client-written output_text and text parts are screened. Key and team
guardrail opt-outs reach regular and translation realtime sessions. A non-string
upstream transcript skips screening as before.
* fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages
* fix(realtime): keep every per-part mask when a joined text masks differently
* fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails
Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a
guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the
session request data with its metadata, so key and team policies apply, and a ModifyResponseException
block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock.
* fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items
* fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened
* fix(realtime): screen text parts in a loop so large messages keep their masks
* fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata
* fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): pin /v1/responses requests for prompt caching affinity
* fix(router): pin previous_response_id turns and keep native tools in the Responses pin key
A Responses turn that continues a stored response now goes to the deployment
that answered it, unless that deployment already failed this request. Native
Responses tools are hashed into the pin key as sent instead of converted, so
two requests with different native tools no longer share a pin and a native
request no longer logs a false dropped-tool warning. The pin keeps its
cacheable deployment ids in a sorted tuple so loggers can serialize request
metadata.
* fix(router): leave a Responses request with a malformed tool to the provider
* fix(router): keep the Responses prompt cache pin out of logged metadata
The pre-call check now keeps the pin in a router-owned in-flight store and
puts only a random token in litellm_metadata, so loggers that ship metadata
never see the routing key or deployment ids. The store is shared across check
instances, so turning prompt_caching off and on at runtime keeps pinning, and
a retry drops its previous attempt's token. The skipped-input debug log now
names only the error type.
* fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries
The prompt caching check no longer narrows the candidates inside the callback
filters. It names the deployment it prefers, and the router core applies that
preference at the pinned deployment step, after tag, region, pre-call and plugin
filters, only when the deployment is still a candidate. A pin can no longer
send a request to a deployment its tags do not allow.
A retry skips the pin when the pinned deployment was already tried in this
request, read from the retry breadcrumbs in both metadata buckets and the
excluded and retry-skipped ids. When the retry count outruns the breadcrumbs
the check stands down, so a pinned deployment that keeps failing is not
retried on every attempt.
* test(router): list the prompt caching check as an in-memory cache caller
The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers
* fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records
* test(router): lift test cooldowns through the cooldown store's evict_local
* fix(router): key Responses tools like their chat twin and drop the pin token after success
A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict.
* test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): stop parsing formatted dates and spend back into values
Regenerating a key stored its new created_at as locale text and parsed it
back for display, which shows "Invalid Date" on browsers whose date format
JavaScript cannot parse. It now stores an ISO timestamp.
The model health table stored check times as locale text and sorted by
parsing that text, which misorders or drops rows outside US English. It now
keeps the raw timestamp and formats it in the table cell, so the shown text
is unchanged.
getSpendString decided whether spend rounds to zero by stripping commas from
its own formatted output. It now compares the number against half of the
smallest shown unit, which gives the same output for every value and no
longer depends on how the digits are rendered.
* fix(ui): keep getSpendString exact at high precision
0.5 / 10 ** decimals drifts by one ulp once 10 ** decimals is no longer exact
(past 22 decimals), so a value right at the threshold could flip from "< $..."
to a row of zeros. Parsing 5e-(decimals + 1) gives the correctly rounded
threshold for every precision.
