Upstream grew ClassificationMethodConfig.tsx to 783 lines, so the 14 lines this
PR added there pushed the merge result past the 800-line max-lines cap. The
helper is standalone logic with its own unit tests, so it moves out rather than
the cap moving up.
* fix(a2a): forward caller identity headers on message/send and message/stream
_forwarding_headers() stamped X-LiteLLM-User-Id/-Team-Id from the authenticated
caller, but was only wired into the tasks/* and tasks/resubscribe branches. The
primary message/send and message/stream conversational path forwarded
agent_extra_headers unchanged, so a downstream agent never learned which end
user was calling it except on secondary task-management calls. This broke
per-user MCP scoping and per-customer FinOps budget enforcement for any agent
invoked through the normal conversational flow.
Resolves LIT-7342
* fix(a2a): snapshot key-bound caller identity before pre-call processing
user_header_mappings lets add_litellm_data_to_request rewrite
user_api_key_dict.user_id from a client header, so the identity stamped
onto X-LiteLLM-User-Id is now captured before that step runs. Header
tests updated to expect the caller identity on message/send.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): drop redundant docstrings from message identity tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): type the message method test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): make message method test helpers immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): default max_idle_connection_lifetime on componentized DB URLs
DatabaseURLSettings.apply_to_env() now appends max_idle_connection_lifetime=60
(or DATABASE_MAX_IDLE_CONNECTION_LIFETIME) to DATABASE_URL and DIRECT_URL before
the reader inherits the writer's connection params, so the gateway, backend and
migrations entrypoints get the same idle-connection reaping as the classic CLI.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep DB URL connection params across IAM/Entra token refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger proxy-infra after process-tree test flake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Direct Anthropic http image inlining, Vertex AI Anthropic forced base64 conversion, Ollama completion image download and the watsonx GPT-OSS Hugging Face chat template lookup all ran synchronous HTTP inside the async request path. Each provider config now transforms through async_inline_remote_media on the async path, the Anthropic handler awaits the config's async_transform_request before dispatch and in the Rust fallback, and watsonx text exposes async_transform_request and always awaits ahf_chat_template
Resolves LIT-7028
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): add http/json export protocol for OTel v2 traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/json was accepted but routed to the protobuf
OTLP/HTTP exporter, so collectors that only decode JSON rejected every batch.
Route it to an OTLP/JSON span exporter that reuses the SDK HTTP transport and
expose the protocol as a select field on the OpenTelemetry callback in the
admin UI.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): walk the fixed OTLP shape instead of recursing when hex-encoding ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): map stored callback variables onto their form fields when editing a callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep team member budget enforced at the cap and across Redis counter expiry
The cached-key auth path admitted a request when the team member counter sat exactly at max_budget, and a Redis counter that expired during a long stream was reconciled against this pod's stale in-memory copy, driving the shared counter negative and reopening the budget. Reject at >= like every other budget check, read Redis before the per-pod copy when judging the reconcile delta, and add the settled request cost after a DB reseed since reserved keys skip the normal increment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): reseeded reservation counter also carries the settled request cost
The reseed test asserted counter == DB floor. The floor is read before the async spend flush writes this request, so the counter now lands at floor plus settled cost, matching the after leg in the PR proof
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Concurrent usage reads on one worker now share a single spend-log query
instead of each scanning the same window, and a digest that comes back
nameless a second time is remembered for the full ten minutes rather than
thirty seconds, so a key that never resolves costs at most two scans per
worker per window per ten minutes. The first miss still expires after
thirty seconds so a read that lands between the daily spend flush and the
spend-log flush recovers on the next read
The per-chunk streaming hook skipped every guardrail stepped by a post_call pipeline, even when the pipeline is left out of the stream for lacking the unified apply_guardrail interface, so a default_on guardrail that only implements async_post_call_streaming_hook stopped governing streams it governed on the merge base. The skip set now comes from the pipelines that gate the stream, the same way the iterator hook already computes it
* fix(router): keep deployment tags out of retry and fallback tag routing
Deployment-level tags merged into request metadata for spend attribution were also read as caller tag constraints on later attempts, so a tag-filtered group re-narrowed to the deployment that just failed. Snapshot the caller's routing tags before the merge and have tag routing read that snapshot.
Resolves LIT-7113
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): drop redundant comment in tag routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): make tag retry regression deterministic and cover routing snapshot helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): give cost-based routing its own cache key so it stops overwriting latency samples
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): prefix the cost routing cache key so it cannot alias another group's latency key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
simple_shuffle only looked at healthy_deployments[0] to decide whether a
metric was configured, so a weight, rpm, or tpm on a later deployment was
ignored and the pick fell back to uniform random. Decide the metric from
all healthy deployments and keep the total_weight <= 0 fall-through.
Resolves LIT-7112
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep per-request routing_strategy override selectors out of global callbacks
Override selectors built by _get_override_strategy_selector were registered in litellm.callbacks / litellm.input_callback and cached for the router's lifetime, so one request with router_settings_override made every later request on the worker run that selector's pre-call check (rpm enforcement for usage-based-routing-v2). Build them with register_callbacks=False and run the selector's own pre_call_check / async_pre_call_check only for the request that carried the override.
Resolves LIT-7064
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): run override selector pre-call check on specific-deployment early returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover pass-through and affinity-pinned override pre-call paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add traces_endpoint (env OTEL_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, per-exporter key traces_endpoint, Admin UI field) as a complete OTLP/HTTP trace URL used verbatim, so collectors that do not serve /v1/traces can receive traces. endpoint keeps its existing base-URL + signal-path normalization.
Resolves LIT-7218
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A client behind an auto-router sends one max_tokens for every tier, so a value
sized for the smallest tier starves a bigger tier's thinking budget and a value
sized for the biggest is rejected by the smallest. After the complexity router
picks a tier, its per-tier litellm_params now carry max_tokens set to the
smallest max_output_tokens across that tier model's deployments (model_info,
then the cost map), applied the same way a per-tier reasoning_effort already
is, on every routing exit including plan mode, the empty-ask default and the
classifier fallback. The router seam collapses whichever ceiling alias a tier
carries onto the surface's own name, so one tier max_tokens reaches chat,
/v1/messages and /v1/responses alike, drops the caller's other carriers of the
same setting before the merge, and stamps the caller's original once so a
fallback into a group no tier owns gets it back instead of a ceiling sized for
the tier that failed.
Proxy-level reservations were sized from the caller's cap before routing, so a
raised cap left them short. Both owners now re-validate at the deployment hook:
the v3 limiter tops up its combined-TPM and project-OTPM reservations to the
final cap or writes the admitted cap back, and the budget limiter re-estimates
on the chosen deployment and grows the reservation or writes the admitted cap
back. An auto-router alias also reserves budget at its priciest tier model now
instead of pricing to zero.
An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens
still wins, and max_tokens_from_tier_model: false forwards the caller's value
unchanged.
* feat(deploy): expose SSE keepalive, pre-call checks and a metrics sidecar in Helm and Terraform
Typed reliability values on both Helm charts and the AWS/GCP Terraform
modules, a dedicated ClusterIP Service for the separate Prometheus port,
a /health route on the metrics server and dead-worker pruning so the
aggregate does not keep stale multiprocess samples.
Resolves LIT-7142
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(deploy): drop reliability config from Helm and Terraform, keep only the metrics sidecar
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover startup pruning of dead workers' live gauges and unsignalable pids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
The daily spend rows and the spend logs of one batch are written a moment apart, so a usage read landing between them used to remember the session as nameless for ten minutes on that worker. Found identities keep the ten minute entry
The ten minute TTL and the 10000 item ceiling need no env override, and the documentation env-key check flags any os.getenv read that the docs do not list
Cuts the explanatory comments and docstrings added here down to one line each, or
removes them where the code already says it. Restores the four pre-existing
docstrings this PR had reworded to their original text; the one remaining edit to
existing text is TierDefinition.description, whose hardcoded tier list would
otherwise misstate that a tier named NON_REASONING may also omit its description.
The spend-log lookup for permanently unresolvable digests is back to a single DISTINCT ON scan over the requested window, keeping only rows that carry an alias, user, or team so a newer nameless row cannot hide an older named one. Results and misses are cached per worker for ten minutes keyed by digest and window, failed queries are not cached, and JWT rows keyed hashed-jwt-<sha256> now pass the digest gate. Tests cover the JWT gate, cache reuse and partial misses, window changes, error handling, the with-window guard, and the daily activity wiring
* fix(responses): record spend for native Responses API WebSocket sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): bill usage from response.incomplete WebSocket turns
A turn cut short by max_output_tokens ends in response.incomplete, which
OpenAI bills but the processor only read response.completed, so those
sessions still logged zero spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): hoist websocket usage test imports to module scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): price websocket sessions through the standard cost path
The realtime completion_cost branch skips cost_discount_config and cost_margin_config, so a native
Responses WebSocket session was priced differently from the same usage over HTTP /v1/responses.
Drop the explicit widening so the LiteLLMRealtimeStreamLoggingObject built by
normalize_logging_result flows through the generic usage path, and pin WS == HTTP cost under a
50% provider discount in the regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun proxy-infra after flaky test_check_migration process tree test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(proxy): eagerly initialize string callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): materialize string callbacks after load_config so later litellm_settings keys are applied
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop casts when snapshotting string callbacks so LIT006 stays at base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Review flagged the added comments as over-explaining. Cut the call-site comment
that restated the helper's own docstring, and shortened the rest to the fact the
code cannot state itself: why the constant excludes the tier, why the flag is
cleared on a classifier change, and why the edit modal reads both keys back.
The switch adds a row at the top of the tier list, so sitting below Reasoning
put the control and the thing it changes at opposite ends of the card. It now
heads the list, with a separator between it and the first row.
Three review findings, all in the dashboard.
Switching off the LLM classifier left the toggle checked but disabled, so the
flag could not be cleared and every save was refused by the backend. The
classifier-change handler now drops the flag and the tier's pool the same way it
already drops the other classifier-specific keys.
builtInTierInfo resolved rows against the four-tier order, so the new row
rendered with no description, no examples and no rename field. It now resolves
against every built-in tier, and the duplicate BUILT_IN_TIER_ORDER constant is
gone in favour of the one in tier_rows.
The preset schema widening is reverted. It was speculative, no published catalog
carries the tier, and prefill would have discarded it while the route-wide null
exclusion changed the endpoint's passthrough contract for every other field.
Also splits NonReasoningTierToggle and TierConfigIntro into their own files to
get ComplexityRouterConfig.tsx back under the max-lines limit, and applies ruff
format to the two backend files CI flagged.
error_status_code only read status_code, so a ProxyException raised
before routing (which stores its status as the string code) answered
500 with its 4xx type through the rerank, images, realtime, files, and
pass-through tails. It now falls back to a decimal code. A 408 maps to
timeout_error instead of invalid_request_error.
Tail regressions for rerank, images, realtime calls, and the chat
pass-through fail at the merge base with ('None', 'None'); the new
files-test helpers are fully typed.
Fireworks renders a Responses request through a chat template that only
accepts a system message at the very beginning, so a request carrying
`instructions`, a developer item, and a replayed reasoning item (the shape
Codex CLI sends from its second prompt on) came back 400 with "System
message must be at the beginning".
The leading system or developer items, and any developer item later in
the conversation, now fold their text into top-level `instructions`,
joined with blank lines, and leave `input`. A developer item that closes
the conversation right after an assistant turn stays where it is as a
system item, as does any system or developer item with an image or file
part, so those parts still reach Fireworks. Mid-conversation system items
stay untouched. Non-string `instructions` pass through unchanged.
Folding into `instructions` rather than a leading system item keeps
`previous_response_id` chaining working, since Fireworks prepends the
stored history to `input` and a leading system item would land after it.
This supersedes the leading system item approach from deaadc21d3 and
4807630c2a on this branch. The leading and closing block rules match the
chat path change in #39852.
