Key objects share the 200-entry UserApiKeyCache in-memory store with teams,
end users, tags and memberships, so churn in those objects evicts hot keys
and forces a LiteLLM_VerificationToken lookup on the next request. Route
bare hashed-token keys to a dedicated InMemoryCache inside UserApiKeyCache
while keeping Redis, TTL, serialization and invalidation shared
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A LiteLLM virtual key in the Authorization header admits the caller but is not an Entra assertion the guardrail can exchange, so the connect-time RFC 9728 challenge now fires unless the bearer is a compact JWS. The guardrail parses the inbound bearer with the same predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Reading LITELLM_DISABLE_LOGIN_RATE_LIMIT through get_secret_bool on every
unauthenticated sign-in attempt meant a hosted secret manager in read mode
was queried once per password guess, before any counter was checked
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Building the guardrail's tool metadata from the local registry entry that
dispatch resolved, instead of re-deriving it from the tool name, keeps an
OpenAPI operation whose name starts with its own server prefix from being
reported with the shorter operation's description and schema. The registry
branch in get_listed_tool is gone with it, and the test doubles for the
local registry now carry a string description and dict schema like the real
entries do
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Use RedisCache.async_increment_with_floor (a single Lua INCRBY + EXPIRE) for the
shared login counters instead of the two-step INCRBYFLOAT then EXPIRE, so a
counter can never be committed to Redis without its expiry. The repair in
_remaining_window now only covers expiries stripped out of band (PERSIST, a
restore) and uses the same atomic call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
OpenAPI-generated and legacy local-registry tools dispatch through execute_mcp_tool's
local branch, which called pre_call_tool_check without the cached MCPTool. Agent 365
therefore received bare {"name"} payloads for those tools while managed-server tools
carried description and inputSchema. Both local call sites now pass get_listed_tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The recursion detector flags any unignored recursive function, so the
timeout classification now walks the explicit cause chain with a
bounded generator instead of calling itself
Scopeless Agent 365 gated servers now advertise api://<client_id>/access_as_user instead of
staying silent, so a client can still sign in. Entra rejecting the gateway's own credentials
(invalid_client, unauthorized_client, invalid_scope, invalid_resource) follows unreachable_fallback
rather than telling the caller to sign in again with a 401
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
