Pad the locust payload to tens of KB so per-request bookkeeping cost scales with
body size instead of hiding behind a 40-byte prompt. Turn on
use_redis_transaction_buffer in the chaos config and JSON_LOGS in the workflow so
the spend buffer, pod lock, and JSON-encoded breaker tracebacks are all part of
the measured chaos cost. Add a log-bytes-per-request budget alongside latency,
RSS, and CPU, reading the proxy's log file size at each phase split; its ceiling
is uncalibrated since no chaos run has measured it yet.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The Anthropic Messages route reaches the same Redis touchpoints and cost-tracking
callback through its own request path, so a failure-path regression there would not
surface from chat completions alone. Each simulated user now picks one endpoint round
robin and stays on it, and the per-endpoint split is asserted and reported so a run
that silently drove only one route fails instead of passing.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(proxy): keep call_type and request start time on failed-request spend logs
post_call_failure_hook pops litellm_logging_obj before the failure callbacks
run, so the spend row built from request_data had a blank call_type and used
datetime.now() as the start time. A guardrail-blocked MCP tool call therefore
showed up in the Logs page as an LLM row with no call type and a 0s duration.
Lift call_type and start_time off the logging object alongside the fields
already lifted, and have the DB failure hook prefer the lifted start time.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the spend writer into _ProxyDBLogger instead of patching a module global
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): fall back from unhealthy auto-router tier
Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit 00c7fd8376)
* fix(router): treat budget and tag exhaustion as a no-capacity verdict
The eligibility probe only read typed router errors as "nothing here can
serve this". Provider and deployment budget exhaustion, and tag routing
with no matching deployment, report it as a bare ValueError carrying a
RouterErrors marker, so the probe read a spent tier as live, skipped the
peer and default recovery, and failed the request.
---------
Co-authored-by: Tin Chi Lo <tin@berri.ai>
Co-authored-by: Claude Code <noreply@anthropic.com>
* fix(proxy): authorize every Responses API id, not only the ones the proxy issued
The ownership check on the Responses API only ran when the id arrived in the
proxy's own encrypted format. An id in any other shape skipped the check and
was forwarded upstream, so a key that did not own the response could retrieve,
cancel, delete, or chain off it.
Every addressed id now goes through one authorization step shared by retrieve,
cancel, delete, list-input-items, and create's previous_response_id. An id the
proxy did not issue is refused with 403 unless the deployment opts in with
general_settings.allow_unmanaged_response_ids, has responses id security
disabled, has no signing key configured, or the caller is a proxy admin.
* fix(proxy): re-authorize the retained responses id instead of trusting it
CLIENT PAUSE ALL for the length of the chaos phase instead of CLIENT PAUSE WRITE, so every Redis touchpoint on the request path times out rather than just the writes. The pause is sized to the phase because it freezes the control connection too; teardown's CLIENT UNPAUSE is a safety net for a phase that overran
Latency, RSS and CPU are now budgeted as chaos-over-baseline ratios (p50/p90/p99 for latency and RSS, CPU seconds per request once) through a small phase_budget module, replacing the machine-shaped absolutes. The Redis timeout rate is reported but no longer asserted
The final /metrics scrape waits for litellm_deployment_failure_responses_total to stop moving, since that counter is bumped from the async logging queue and lagged the load generator by thousands of increments. The model group carries a unique marker so a deployment left behind by an aborted run cannot absorb this run's retries
Co-Authored-By: Claude Code <noreply@anthropic.com>
The docstring cited session_affinity_ttl_seconds as the keepalive bound,
but the Router-level knob feeding ttl_seconds is
deployment_affinity_ttl_seconds; session_affinity_ttl_seconds is the
separate per-request PreRoutingHookResponse override. Anyone grepping
the docstring's name to shrink the Router default finds only the
override. Name both, scoped correctly.
Starlette scans the route table in registration order, so a request pays one
regex match per route registered ahead of its own. The proxy registers several
hundred routes and left the liveness probe near position 280 and the lazy
loaded /v1/messages at the very end. Move /health/liveliness, /health/liveness,
/v1/chat/completions, /chat/completions and /v1/messages to the front of the
route table after startup registration and again after a lazy router loads.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Both reasoning_effort mappers ended their if/elif chain in a bare ValueError.
exception_type() has no branch for ValueError, so it fell through to the shared
APIConnectionError fallback and the proxy answered a malformed client request
with a retryable HTTP 500 carrying no hint of the accepted values.
Raise UnsupportedParamsError (400) instead, listing the supported set, matching
what the Anthropic and Bedrock transforms already do and what this same file
already does at its five other param-validation sites.
This also covers 'xhigh' and 'max', which are members of litellm's own
REASONING_EFFORT literal but have no Gemini mapping, so callers bridging from
OpenAI-shaped code were hitting the 500 without typing anything wrong.
Fixes#40474
Claude-Session: https://claude.ai/code/session_01XT1qsbjLwnhiN5sQ2hNUxr
* feat(proxy): make the in-memory management cache capacity configurable
Add general_settings.user_api_key_cache_max_size (positive int, default 200) to resize the
in-memory tier of the shared user_api_key_cache at startup and on DB config reloads, expose it
in the Admin UI general settings, and cover it with behavioral tests. Prior art: #34726
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): resize the in-memory tier from DualCache so any cache instance honours the cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): wrap the cache capacity field description to the 120 col limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response
Bedrock pass-through streaming kept every relayed chunk in memory until EOF and
then decoded, parsed and translated the whole stream again for spend logging.
Large or concurrent streams could exhaust proxy worker memory.
Sync and async passthrough wrappers now hand each chunk to a provider stream
collector as it is relayed. Bedrock decodes event-stream frames incrementally,
folds consecutive text deltas, and keeps only what stream_chunk_builder needs
for usage, tool calls and metadata. Text deltas are no longer retained in the
Bedrock and Anthropic stream decoders either. Providers without a collector
keep the previous raw-bytes behavior. Collector failures are isolated so spend
tracking can never interrupt the customer stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): assert the spend payload the collector builds instead of mock internals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Key objects share the 200-entry UserApiKeyCache in-memory store with teams,
end users, tags and memberships, so churn in those objects evicts hot keys
and forces a LiteLLM_VerificationToken lookup on the next request. Route
bare hashed-token keys to a dedicated InMemoryCache inside UserApiKeyCache
while keeping Redis, TTL, serialization and invalidation shared
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
