Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential
* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential
* fix(proxy): read credentials from the writer db before the router update and look a credential up once
* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
* fix(proxy): kill the whole prisma process group when a boot migration command times out
* fix(proxy): keep boot alive when the installed proxy extras has no prisma runner
The no-tools skip path in async_pre_call_hook now logs at DEBUG and the
denied-by-rule messages in async_pre_call_hook and _evaluate_tool_calls
log at INFO. The malformed tool arguments warning is unchanged. Adds
regression tests pinning each level
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rank streaming latency routing by raw TTFT, not TTFT per token
Latency-based routing divided time-to-first-token by completion_tokens
before storing it, so a deployment that streamed a long answer looked
faster to first token than one that answered briefly. TTFT is now stored
as plain seconds (first token time minus request start) in both the sync
and async success handlers, which is what the routing decision compares.
Non-streaming latency normalization per output token is unchanged.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* fix(router): store streaming TTFT under a seconds-only cache key
Workers on the previous release keep writing seconds-per-token samples
under "time_to_first_token" in the shared router cache during a rolling
deploy, so mixing the new raw-seconds samples into the same list averaged
incompatible units. Raw TTFT now lives under "time_to_first_token_seconds"
and routing reads only that key.
Also fix the regression test's token counts: with 50 tokens on the fast
deployment and 500 on the slow one the old per-token formula picks the
slow deployment, so the routing assertion now catches the bug.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* test(router): cover the TTFT sliding window from the unit-test shard
Move the TTFT list trimming checks from the CircleCI-only suite into the
mapped unit test file as one sync/async parametrized test, so the changed
lines in lowest_latency.py are exercised by the GitHub unit-test shard
that reports patch coverage.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
Review response. The previous commit tested both cache buckets for absence, which
was right for reads and wrong for writes, and it was wrong for reads on a model
that does not cache at all.
Reads now require an explicit rate AND supports_prompt_caching. Six cost-map
entries pair cache_read_input_token_cost of 0 with supports_prompt_caching false,
including gemini-robotics-er-1.5-preview and openrouter/z-ai/glm-4.7. On those the
zero is a placeholder for a model that has no cache, not a free one, so honouring
it priced 20,000 baseline tokens at $0.00 instead of $0.006.
Writes go back to truthiness. A zero cache-creation price does not mean free, it
means writes bill at the plain input rate, and 36 entries rely on that inheritance
including deepseek/deepseek-chat. Reading it as a real price dropped a 10,000 token
first-turn baseline from $0.0028 to $0.00.
The helper-only test is replaced with cases that go through _baseline_usage on real
cost-map entries, covering all three shapes: supported free reads stay in the read
bucket, unsupported reads move to text_tokens, and a zero write rate moves to
text_tokens. Both new tests fail if either leg is reverted.
_baseline_cache_rate_keys tested the rate with bool(), so an explicit
cache_read_input_token_cost of 0.0 read as absent. The cache-read tokens then
fell through to ordinary text input and were charged the full input rate, which
overstates savings for deployments where reads are genuinely free.
Tests for absence instead. A rate that is present and zero is now honoured, a
rate that is missing still falls back to plain input, which is what the existing
docstring describes and what OpenAI, Azure and Gemini entries rely on for cache
writes.
explicit free (0.0) before (False, False) after (True, True)
absent before (False, False) after (False, False)
Final batch of the fifth basedpyright Any reduction round. Every change is
typing-only and leaves runtime behavior identical.
These are the densest remaining files, so the yield per file is small and most
of the batch was left alone deliberately. The Riva transcription handler
describes the SDK module attributes it reads with Protocols instead of a bare
ModuleType, the AWS secret manager stops hiding a botocore header object behind
Any, and the sensitive data masker, MCP SSO assertion store and Ovalix guardrail
move payload and option annotations to object and Mapping[str, object].
Third batch of the fifth basedpyright Any reduction round. Every change is
typing-only and leaves runtime behavior identical.
Provider transformation configs, video and rerank base classes, OTel metadata
and the guardrail and realtime type modules move their payload, header and
optional-parameter annotations from Any to object, Mapping[str, object] or the
concrete model the call site already produces. Repositories and endpoints that
reached Prisma through an untyped handle now name the actions they call with the
repo's own TableActions protocol. The pydantic field retypes were checked
against pydantic to confirm object and Any validate, serialize and generate JSON
schema identically.
Second batch of the fifth basedpyright Any reduction round. Every change is
typing-only and leaves runtime behavior identical.
Guardrail hooks and file, vector store and usage endpoints move their payload,
header and response annotations from Any to object, Mapping[str, object] or the
concrete response model the call site already produces. Two private aggregation
helpers in the guardrail usage endpoints take a key accessor function instead of
an attribute name string, so the key they read is checked against the row type.
The verification token repository reaches its two tables through Protocols that
name the handles it calls, rather than reading them off an untyped prisma
client, and the Azure AD credential wrapper describes the azure-identity
credential it wraps the same way.
Fifth round of basedpyright Any reduction. Every change is typing-only and
leaves runtime behavior identical.
Prisma table access now goes through the PrismaTableRepository and TableActions
protocols the repo already has, instead of reading untyped attributes off
prisma_client.db. Payload and parameter annotations move from dict[str, Any] to
dict[str, object] or Mapping[str, object]. Guardrail constructors that took
**kwargs: Any now take Unpack of a PEP 728 TypedDict, the same
_CustomGuardrailOptions shape three other guardrails already use. Calls into the
OpenAI and Azure assistants SDKs pass explicit keywords rather than splatting a
dict, so the arguments are checked against the real SDK signatures.
An aggregate gateway DCR authorize whose RFC 8707 resource resolves to exactly one
gateway-managed oauth2 server sealed that server into the flow and then sent the browser to
the generic connect grid anyway, so the user had to find the server the client had already
named and click Connect.
The connect URL now carries only the flow handle. GET /authorize/flow classifies the sealed
flow as unscoped, interactive, M2M, or stale, and returns the matching state to the page.
Interactive flows require a live per-user vendor credential before minting and do not burn the
flow on an early submit. M2M flows use the gateway's configured service credential and finish
without an interactive OAuth trip. Stale flows fail closed instead of becoming unscoped.
The existing explicit Finish action and a new Cancel path preserve deliberate user intent.
Preserve deployment identity through savings calculation, with canonical model fallback only when either ID is absent. Cover negotiated rates, unchanged deployments, alias/base-model cache accounting and missing IDs.
Fixes#38811. Based on the deployment-identity approach proposed by @QuantumBreakz in #38834.
Co-authored-by: Claude Code <noreply@anthropic.com>
post was the only method the mock-transport tests reached, leaving the anchor
in async delete and in sync patch/put/delete uncovered in the tree Codecov
measures. Parametrizing also means a method added later is covered here rather
than being the one that forgets to anchor.
A streamed Responses API relay handed the success handler a bare ResponsesAPIResponse, which the streaming assembly step drops, so the relay never reached the spend callbacks. Hand it the terminal response.completed event instead, which the assembly step already converts, and cover the whole flush path with a regression test that fails on the previous tip.
tests/local_testing runs on CircleCI, which does not feed the patch-coverage
check, so the anchor read as half-uncovered. These three drive it over a mock
transport, which is enough for the lifetime contract even though it cannot
reproduce the teardown the loopback tests exist for.
litellm's azure_ai config rejects reasoning_effort for gpt-chat-latest and Azure documents a fixed reasoning level for it, so the entry no longer advertises reasoning_effort_levels. The catalog metadata tests compare cost_per_token and the whisper transcription cost with the entry the calculator read instead of with list-price literals, the pattern #40195 removed
The merge base read the variable by truthiness, so any non-empty value
turned the global flag on. Parsing it as a flag made a value such as
temperature or enabled silently turn it off, and the only docs for the
variable describe it as a list of parameter names, so keep those values
on and log a warning that asks for true or false. A blank value stays off
without a warning
A key allowed to call a Bedrock model could delete any object under the
deployment's buckets through DELETE /bedrock/v1/files/{s3 id}?model=...
because the managed-file ownership check only runs for unified ids. Raw
cloud storage ids now answer 403 on every delete route unless the caller
is a proxy admin; managed ids and require_managed_files are unchanged
_handler_may_close_client withholds the finalizer's close from a client that
someone else references. A streaming response is not one of those referrers: it
holds the connection it reads from, never the client, so the refcount says
"sole referrer" for exactly the client that is busiest, and __del__ tears the
pool down mid-body. Both handlers, both transports, and it is the shape #24929
reports: a provider's streaming call returns the response and drops the
handler, and get_async_httpx_client lets a cached handler go after an hour.
A streaming send now anchors the handler to the response it returns, so the
handler is finalized once the caller is done with the body and the ordinary
close applies with nothing in flight. The anchor is a weakref.finalize, which
holds the handler in its own registry rather than on the response. That keeps
the handler out of the response's reference cycle, so it is finalized by
refcount and can still schedule an async close, rather than inside a cyclic
collection that reaps its aiohttp session in the same pass; and a handler
serving several streams is released only once all of them are done, because
each anchor holds it separately.
Only a streaming send anchors. A non-streaming response has been read in full
by the time the method returns, so pinning the handler to it would delay every
client close behind whatever the caller does with the response.
The alternative was to ask the connection pool whether a request was in flight,
reusing EvictedClientCloser's check. It reads client._transport, so it answers
"idle" for any client with a proxy configured, where httpx routes through
client._mounts; and having found the client busy it can only poll, which never
terminates for a response the caller abandons unread, since httpx leaves that
connection checked out. The response's own lifetime is the condition both were
approximating.
Nothing calls it since the streaming pipeline detects rewrites at run time
through the stream observer, so the base method and the content filter's
override were dead code with dead tests
