* fix(proxy): keep team member budget enforced at the cap and across Redis counter expiry
The cached-key auth path admitted a request when the team member counter sat exactly at max_budget, and a Redis counter that expired during a long stream was reconciled against this pod's stale in-memory copy, driving the shared counter negative and reopening the budget. Reject at >= like every other budget check, read Redis before the per-pod copy when judging the reconcile delta, and add the settled request cost after a DB reseed since reserved keys skip the normal increment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): reseeded reservation counter also carries the settled request cost
The reseed test asserted counter == DB floor. The floor is read before the async spend flush writes this request, so the counter now lands at floor plus settled cost, matching the after leg in the PR proof
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The per-chunk streaming hook skipped every guardrail stepped by a post_call pipeline, even when the pipeline is left out of the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that gate the stream, the same way the iterator hook already computes it
* fix(router): keep deployment tags out of retry and fallback tag routing
Deployment-level tags merged into request metadata for spend attribution were also read as caller tag constraints on later attempts, so a tag-filtered group re-narrowed to the deployment that just failed. Snapshot the caller's routing tags before the merge and have tag routing read that snapshot.
Resolves LIT-7113
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): drop redundant comment in tag routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): make tag retry regression deterministic and cover routing snapshot helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): give cost-based routing its own cache key so it stops overwriting latency samples
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): prefix the cost routing cache key so it cannot alias another group's latency key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
simple_shuffle only looked at healthy_deployments[0] to decide whether a
metric was configured, so a weight, rpm, or tpm on a later deployment was
ignored and the pick fell back to uniform random. Decide the metric from
all healthy deployments and keep the total_weight <= 0 fall-through.
Resolves LIT-7112
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep per-request routing_strategy override selectors out of global callbacks
Override selectors built by _get_override_strategy_selector were registered in litellm.callbacks / litellm.input_callback and cached for the router's lifetime, so one request with router_settings_override made every later request on the worker run that selector's pre-call check (rpm enforcement for usage-based-routing-v2). Build them with register_callbacks=False and run the selector's own pre_call_check / async_pre_call_check only for the request that carried the override.
Resolves LIT-7064
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): run override selector pre-call check on specific-deployment early returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover pass-through and affinity-pinned override pre-call paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add traces_endpoint (env OTEL_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, per-exporter key traces_endpoint, Admin UI field) as a complete OTLP/HTTP trace URL used verbatim, so collectors that do not serve /v1/traces can receive traces. endpoint keeps its existing base-URL + signal-path normalization.
Resolves LIT-7218
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A client behind an auto-router sends one max_tokens for every tier, so a value
sized for the smallest tier starves a bigger tier's thinking budget and a value
sized for the biggest is rejected by the smallest. After the complexity router
picks a tier, its per-tier litellm_params now carry max_tokens set to the
smallest max_output_tokens across that tier model's deployments (model_info,
then the cost map), applied the same way a per-tier reasoning_effort already
is, on every routing exit including plan mode, the empty-ask default and the
classifier fallback. The router seam collapses whichever ceiling alias a tier
carries onto the surface's own name, so one tier max_tokens reaches chat,
/v1/messages and /v1/responses alike, drops the caller's other carriers of the
same setting before the merge, and stamps the caller's original once so a
fallback into a group no tier owns gets it back instead of a ceiling sized for
the tier that failed.
Proxy-level reservations were sized from the caller's cap before routing, so a
raised cap left them short. Both owners now re-validate at the deployment hook:
the v3 limiter tops up its combined-TPM and project-OTPM reservations to the
final cap or writes the admitted cap back, and the budget limiter re-estimates
on the chosen deployment and grows the reservation or writes the admitted cap
back. An auto-router alias also reserves budget at its priciest tier model now
instead of pricing to zero.
An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens
still wins, and max_tokens_from_tier_model: false forwards the caller's value
unchanged.
* feat(deploy): expose SSE keepalive, pre-call checks and a metrics sidecar in Helm and Terraform
Typed reliability values on both Helm charts and the AWS/GCP Terraform
modules, a dedicated ClusterIP Service for the separate Prometheus port,
a /health route on the metrics server and dead-worker pruning so the
aggregate does not keep stale multiprocess samples.
Resolves LIT-7142
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(deploy): drop reliability config from Helm and Terraform, keep only the metrics sidecar
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover startup pruning of dead workers' live gauges and unsignalable pids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
* fix(responses): record spend for native Responses API WebSocket sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): bill usage from response.incomplete WebSocket turns
A turn cut short by max_output_tokens ends in response.incomplete, which
OpenAI bills but the processor only read response.completed, so those
sessions still logged zero spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): hoist websocket usage test imports to module scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): price websocket sessions through the standard cost path
The realtime completion_cost branch skips cost_discount_config and cost_margin_config, so a native
Responses WebSocket session was priced differently from the same usage over HTTP /v1/responses.
Drop the explicit widening so the LiteLLMRealtimeStreamLoggingObject built by
normalize_logging_result flows through the generic usage path, and pin WS == HTTP cost under a
50% provider discount in the regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun proxy-infra after flaky test_check_migration process tree test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(proxy): eagerly initialize string callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): materialize string callbacks after load_config so later litellm_settings keys are applied
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop casts when snapshotting string callbacks so LIT006 stays at base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(guardrails): keep guardrail telemetry when a policy pipeline blocks or modifies the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): count every raw-snapshot guardrail evaluation and type the telemetry carry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(policy_engine): type the recording guardrail hooks and telemetry test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Only the leading run of system messages (before the first non-system
message) is joined into the Responses `instructions` field. A system
message that arrives after a user, assistant, or tool turn now becomes a
system input item at its position, whether its content is a string or a
list, so a client that re-sends the same reminder as a string on the
next request produces byte-identical input and `instructions` stays
stable. Claude Code >= 2.1.237 appends such reminders after every user
turn, and folding them into `instructions` made Azure treat every
request as a cold prompt (cached_tokens 0 on every request of a
session).
Fixes#40198
Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential
* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential
* fix(proxy): read credentials from the writer db before the router update and look a credential up once
* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
* fix(proxy): kill the whole prisma process group when a boot migration command times out
* fix(proxy): keep boot alive when the installed proxy extras has no prisma runner
The no-tools skip path in async_pre_call_hook now logs at DEBUG and the
denied-by-rule messages in async_pre_call_hook and _evaluate_tool_calls
log at INFO. The malformed tool arguments warning is unchanged. Adds
regression tests pinning each level
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rank streaming latency routing by raw TTFT, not TTFT per token
Latency-based routing divided time-to-first-token by completion_tokens
before storing it, so a deployment that streamed a long answer looked
faster to first token than one that answered briefly. TTFT is now stored
as plain seconds (first token time minus request start) in both the sync
and async success handlers, which is what the routing decision compares.
Non-streaming latency normalization per output token is unchanged.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* fix(router): store streaming TTFT under a seconds-only cache key
Workers on the previous release keep writing seconds-per-token samples
under "time_to_first_token" in the shared router cache during a rolling
deploy, so mixing the new raw-seconds samples into the same list averaged
incompatible units. Raw TTFT now lives under "time_to_first_token_seconds"
and routing reads only that key.
Also fix the regression test's token counts: with 50 tokens on the fast
deployment and 500 on the slow one the old per-token formula picks the
slow deployment, so the routing assertion now catches the bug.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* test(router): cover the TTFT sliding window from the unit-test shard
Move the TTFT list trimming checks from the CircleCI-only suite into the
mapped unit test file as one sync/async parametrized test, so the changed
lines in lowest_latency.py are exercised by the GitHub unit-test shard
that reports patch coverage.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
An aggregate gateway DCR authorize whose RFC 8707 resource resolves to exactly one
gateway-managed oauth2 server sealed that server into the flow and then sent the browser to
the generic connect grid anyway, so the user had to find the server the client had already
named and click Connect.
The connect URL now carries only the flow handle. GET /authorize/flow classifies the sealed
flow as unscoped, interactive, M2M, or stale, and returns the matching state to the page.
Interactive flows require a live per-user vendor credential before minting and do not burn the
flow on an early submit. M2M flows use the gateway's configured service credential and finish
without an interactive OAuth trip. Stale flows fail closed instead of becoming unscoped.
The existing explicit Finish action and a new Cancel path preserve deliberate user intent.
Preserve deployment identity through savings calculation, with canonical model fallback only when either ID is absent. Cover negotiated rates, unchanged deployments, alias/base-model cache accounting and missing IDs.
Fixes#38811. Based on the deployment-identity approach proposed by @QuantumBreakz in #38834.
Co-authored-by: Claude Code <noreply@anthropic.com>
The merge base read the variable by truthiness, so any non-empty value
turned the global flag on. Parsing it as a flag made a value such as
temperature or enabled silently turn it off, and the only docs for the
variable describe it as a list of parameter names, so keep those values
on and log a warning that asks for true or false. A blank value stays off
without a warning
Nothing calls it since the streaming pipeline detects rewrites at run time
through the stream observer, so the base method and the content filter's
override were dead code with dead tests
A policy that both adds a guardrail and steps it in a post_call pipeline
used to drop the guardrail from the request's guardrail list outright, so
its pre_call stage never ran. The per-hook loops already skip guardrails
by pipeline mode, so the mode-agnostic subtraction only lost coverage
CustomGuardrail.__init_subclass__ wrapped _StreamRewriteObserver.apply_guardrail in log_guardrail_information, so every streaming step recorded a second standard_logging_guardrail_information entry and span next to the inner guardrail's own. The observer's method now carries the marker that skips the wrapper. The step also adds the guardrail to the applied guardrails header the way the non-streaming unified path does, so streamed spend rows name the guardrail that scanned them
The streaming loop skipped every guardrail stepped by a post_call pipeline, even when the pipeline was dropped from the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_iterator_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that will gate the stream
A deployment drop_params string that is not a flag value (a typo like ture) stayed silently off. The router now logs one warning per deployment. LITELLM_DROP_PARAMS and litellm_settings.drop_params share the same helper, so a non-flag value there warns as well instead of flipping silently from on to off
