* fix(prometheus): count provider-side 401s in LLM failure and deployment metrics
The invalid virtual key skip no longer applies to an error the LLM provider
returned while the deployment was called with its own credential, so a 401
for a bad deployment credential increments litellm_llm_api_failed_requests_metric,
litellm_deployment_failure_responses and litellm_deployment_total_requests and
marks litellm_deployment_state as a partial outage again. Proxy-gate rejections
and a failing credential that belongs to the caller (a rejected per-user
credential, an api_key sent with the request, a per-user GitHub Copilot
session) keep the skip
* fix(prometheus): keep a caller's own Anthropic OAuth token failure out of the deployment metrics
A provider 401 on an Anthropic OAuth token the caller forwarded through the proxy is the caller's credential
failing, so it stays out of the deployment failure and outage series like a caller-supplied api_key. Adds
integration cells for an Anthropic deployment's own key, the forwarded caller OAuth token and a direct SDK
call that brings its own key
* test(prometheus): pin how a caller's own key or Authorization header on a deployment is counted
An SDK Router call that passes its own api_key is counted against the one-off deployment the router builds for that key, so the operator deployment keeps only its own failure. A provider 401 on an Authorization header the caller sends through the proxy counts against the deployment, as on v1.80.12
* fix(prometheus): keep caller-sent provider credentials and token-exchange 401s out of deployment metrics
A caller can send its own provider credential in the request body, such as
azure_ad_token, azure_scope, the Azure Entra app fields, watsonx token,
gigachat_access_token, OCI keys, AWS keys or Vertex credentials, at the top
level or inside extra_body. A provider 401 on that credential is the caller's
failure, so it stays out of the shared deployment's failure, request,
LLM-failed and state series, as on main. A token-exchange deployment trades the
caller's identity token on every call, so its 401s stay out as well, matching
the router cooldown.
* test(prometheus): Final locals, precise helper types and one for per comprehension in the new test helpers
* test(prometheus): pin that a provider 401 at a caller-sent api_base counts against its one-off deployment
* fix(prometheus): keep a provider 401 on caller-sent headers out of deployment metrics
A caller can override the deployment's credential with headers it sends in the
request body: headers or provider_specific_header, or an Authorization, api-key,
x-api-key, x-goog-api-key or Ocp-Apim-Subscription-Key entry in extra_headers.
A provider 401 on such a request is the caller's failure, so it stays out of the
shared deployment's failure, request, LLM-failed and state series, as on main.
* fix(prometheus): skip a provider 401 only when the caller's headers carry a credential
The proxy now records headers, extra_headers or provider_specific_header in
credential_fields only when they name a provider credential header
(Authorization, api-key, x-api-key, x-goog-api-key, Ocp-Apim-Subscription-Key),
the same way api_key is recorded. A caller that sends a plain header such as a
trace id still has a provider 401 on the deployment's own key counted against
the deployment.
* fix(prometheus): stamp credential_fields after the proxy forwards the caller's headers
A provider credential header the proxy forwards from the caller's HTTP request
(forward_llm_provider_auth_headers + forward_client_headers_to_llm_api) lands in
data["headers"] after proxy_server_request is built, so the stamp missed it and a
401 on that caller credential was counted against the deployment. Compute the
stamp after the header forwarding steps and pin both forwarded cases.
* fix(prometheus): skip a provider 401 when the caller sends any header or an OpenAI organization
The proxy now records headers, extra_headers or provider_specific_header in
credential_fields whenever the caller sends or the proxy forwards any header
there, and records organization when the caller sends one in the body or as a
forwarded OpenAI-Organization header. A forwarded header can be a gateway key
such as x-portkey-api-key or an OpenAI-Project id, and a wrong organization
gets a 401 from OpenAI, so these 401s stay out of the deployment metrics as on
main instead of marking a healthy deployment as a partial outage.
* fix(prometheus): ignore the user information the proxy adds when stamping caller headers
With add_user_information_to_llm_headers on and header forwarding enabled, the
proxy adds x-litellm-user_api_key_* headers to the forwarded headers. Those
carry the proxy's own key and user details, not the caller's, so they no longer
make a request look caller-scoped, and a provider 401 on the deployment's own
key on such a request is counted.
* fix(prometheus): skip a 401 raised before the provider is called or at the caller's own endpoint
A deployment with no key of its own fails with LiteLLM's local "Missing
Anthropic API Key" 401 before any provider call, and a caller who sends
aws_bedrock_runtime_endpoint (or another provider endpoint field) can get a
401 from their own gateway. Both were counted against the shared deployment
and marked it as a partial outage, where main skipped every 401.
Count a 401 only once the provider was called (api_call_start_time is set),
and treat caller-sent provider endpoint fields like caller credential fields.
* fix(prometheus): count the deployment's own 401 on requests that carry only anthropic-version or anthropic-beta
The Anthropic SDK sends anthropic-version on every request, and the proxy copies
anthropic-version and anthropic-beta into provider_specific_header for Anthropic,
Bedrock and Vertex. Both were treated as caller headers, so a 401 on an Anthropic
deployment's own key from any Anthropic SDK client stayed out of the deployment
metrics.
These two headers only pick the API version and beta features. Anthropic answers
a bad anthropic-version or an unknown anthropic-beta with a 400, never a 401, so
they cannot turn the deployment's own key into a 401 and no longer mark the
request as carrying a caller header.
* test(prometheus): read and write the owned proxy's own database in the expired-key and MCP cells
Since #47213 an owned proxy boots on a private database, so the expired-key update and the MCP guardrail row read now go through the gateway's database_url instead of the worker's.
* fix(prometheus): keep credential_fields as main stamps it and give Prometheus its own caller_credential_fields
credential_fields tells the Rust OCR bridge which stripped body fields the
caller sent. Repurposing it for the Prometheus caller-credential check changed
its value (organization and extra_headers added, anthropic-version-only
headers dropped). Restore main's stamp unchanged and stamp the Prometheus
classification under caller_credential_fields, which only Prometheus reads.
* fix(prometheus): skip a provider 401 on a stored credential the caller picks by litellm_credential_name
A request body litellm_credential_name swaps the deployment's stored
credential for one the caller chose, so a 401 on it is the caller's, not the
deployment's. Treat it like the other caller credential fields so it no longer
marks a healthy shared deployment as a partial outage.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(team): keep the team reset schedule when a temp budget increase is set on a member inheriting the team default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the temp-only member row to carry the team default cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): serve a team's wildcard deployment before a proxy-wide wildcard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep the wildcard precedence helpers free of partially typed locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): call each pattern router once in the wildcard precedence helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ptu): a team's wildcard reservation serves beside a proxy-wide catch-all
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): budget period tooltip on key, team, user and org spend tiles
Adds a shared BudgetPeriodTooltip to the four detail-view spend tiles showing the current budget period, max budget, reset interval and spend so far, and makes organization create schedule budget_reset_at and organization info return it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): give the budget period tooltip trigger its own aria label
The default SimpleTooltip trigger shares the question-circle label with InheritedBudgetHint, which broke the key budget display tests that count that icon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): match the proxy's budget window rules in the period tooltip
30d and Nmo now span calendar months like get_budget_window_start, the arithmetic runs in UTC so a viewer's DST change cannot shift the start, the hourly/daily/weekly/monthly aliases the proxy accepts are normalized before parsing, and spend without a duration reads as lifetime spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): gate key budget_duration and budget_id updates like max_budget
A key owner or team member could shorten budget_duration or swap budget_id on
/key/update without the budget authorization max_budget gets, so the reset job
cleared spend on a capped key or a bigger tier lifted its cap
Fixes#46916
* test(proxy): cover tier keys and self-serve budget policies for key budget updates
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
OAuthScopeResponse holds a tools/call response's http.response.start so a managed OAuth scope challenge can replace it with a 403. Once the call was allowed, the held start still waited for the next body message, which for a slow tool is the 15s SSE keep-alive ping. Send it as soon as the call is allowed instead.
Fixes#46659
* fix(router): stop concurrent deployment deletes from crashing deployment lookups
delete_deployment and upsert_deployment shrink model_list before they repair
the index maps, so a lookup on another thread in that window read a stale
index: IndexError, or the neighbouring deployment and its credentials. An add
racing a delete could corrupt the index for good.
One RLock now covers model_list and the three index maps. Delete, upsert, add
and the set_model_list reset change them under it, and every lookup that turns
an index into a deployment reads under it.
Fixes#45242
* test(router): release the held delete when the read reaches the index lock
The race tests released the held write after a fixed 0.2 s, so a reader the runner
stalled past it read the repaired list and passed on unfixed code. The read now runs
on its own thread and the write stays held until the read returns or reaches the
router's index lock; the 10 s bound only fails a broken synchronization.
The integration test gains the audit cells: chat, streaming, Responses and
Anthropic messages calls (sync and async, by model name and team public name)
during a held delete, the same for upsert, add and set_model_list, a request for
the deployment being deleted, a provider error, a double delete, and thread and
async bursts during upsert churn.
* test(router): cover the early return of a delete and the team filter of get_model_ids
delete_deployment now returns None from inside the index lock, and get_model_ids by
name reads its deployments through _deployments_named; pin both: deleting an unknown
id returns None and leaves the list alone, and exclude_team_models drops only the
team deployment.
* fix(router): read deployments by name under the index lock outside the Router
The response polling provider check, the complexity router's adaptive pool and the
savings baseline turned the name index into list positions themselves, so a concurrent
delete could still hand them a stale slot. They now read through Router.deployments_named,
which takes the deployment index lock.
* fix(router): type the locked deployment lookups as DeploymentTypedDict
deployments_named and _team_deployments_named returned tuple[Any, ...]. They now return
DeploymentTypedDict rows, which is what model_list holds. The adaptive pool setup in the
Router and the complexity router drop their branches for pydantic Deployment rows, which
these lookups never return, and should_include_deployment accepts the typed row.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text
The pass-through guardrail translation json-dumps the request body as the
scan text when no request_fields are configured. The unified guardrail hook
injects litellm_metadata into that body first, so the dump carried the
caller's key metadata. Exclude it alongside metadata and litellm_logging_obj;
vendors still receive it through request_data.
* test(guardrails): type the pass-through scan-text tests per AGENTS.md
Preserve oversized JSON integers in proxy requests and model rewrites
Parse large bodies in a worker thread to avoid blocking request handling
Fixes#47947
Co-authored-by: Jeevan Babu Gotru <114808955+jeevan1098@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
* fix(proxy): reconcile team member total_spend with current cycle spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read /team/list as a list and drain the scripted provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(repositories): compose the membership reset writer and drop SQL text asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the cascade repair test and flatten memberships without a stacked comprehension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep the reset job test file's existing typing untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(teams): show caller member spend and budget on Your Teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): use member-linked reset dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep type-discipline suppressions on cast lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): preserve empty page list response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep the member reset date inside the spend column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): match email-only roster members for caller membership
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show team member budget reset date in UTC
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(teams): address review on caller membership enrichment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): label the caller's team spend line My Spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill OpenCode Go requests from the reported list_cost
OpenCode returns usage.opencode.list_cost on every response. Read it into the provider-reported cost the cost calculator checks before the price map, for chat and Responses, streamed and not. Keep the static rows as the fallback, and stop the generic usage.cost path from billing Go requests at 0 (usage.cost is what hit the prepaid balance, 0 inside the plan).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(core): resolve reported cost breakdowns without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(opencode): cover reported list_cost spend through the proxy
Replaces the recursive _reported_cost_value with a flat _cost_number helper
so the code-quality gate passes, and adds an integration test that runs the
proxy against a scripted OpenCode upstream for a non-stream and a stream
chat completion, asserting the response cost header and the spend log row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill native /v1/messages from the reported list_cost
The native Anthropic Messages route priced OpenCode Go bodies from the price
rows twice over: the non-stream body is a TypedDict the response-cost header
path priced as-is, and the stream assembled its ModelResponse with the rows
cost already stamped. Both now read usage.opencode.list_cost, the non-stream
body through billable_result and the stream from the final message_delta.
The integration test now covers chat, responses and messages, stream and
non-stream, through the proxy against a scripted OpenCode upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): keep the stream cost stamp inside the lint gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): read list_cost off dict chunks and trust it only from OpenCode on /v1/messages
Stream builders receive chunks both as mappings and as ModelResponseStream
objects, so the list_cost scan now reads usage through chunk_usage. The
native Messages route only trusts a reported cost from opencode_go, every
other upstream keeps the price map, which also keeps a zero usage.cost from
being billed as free. Served bodies without stop_sequence still price, and
the new code stays inside the basedpyright gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add disable_pre_routing_error_logs to skip spend rows for pre-routing rejections
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): decide pre-routing rejections by pipeline stage, not exception type
Guardrails such as AIM raise ProxyException(400), the same shape as an unparsable
body, so the status code list skipped their spend rows. The pipeline stamps
litellm_call_id only after auth, body parsing and model resolution, so a failure
reaching the hook without one died before any provider was considered.
_handle_logging_proxy_only_error flags those requests when it mints the call id,
and is_pre_routing_rejection keys off that flag plus ProxyModelNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark pre-routing rejections with a server-owned sentinel
request_data is the client body, so a boolean marker or a litellm_call_id in it
can be sent by the client to hide a guardrail block or force a spend row. The
failure handler now stores an object() sentinel whenever it finds no logging
object, and is_pre_routing_rejection checks identity, which JSON cannot forge.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): skip declared text-only deployments for image requests without enable_pre_call_checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): run the vision filter after the cooldown filter so a cooled vision leg falls back to the text leg
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): apply the vision filter after pre-call checks and tidy the new tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): mark the remaining locals in the vision filter tests Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add native Reco guardrail provider (pre_call)
Registers guardrail: reco as a first-class vendor integration, built on
the existing generic guardrail API wire contract rather than reimplementing
the HTTP client and payload mapping. Maps reco_tenant_id to the
X-Reco-Tenant-Id header and hardcodes fail_open/fail_on_error since those
should not be customer-configurable. v1 only supports mode: pre_call;
post_call and streaming redaction already work against the existing
generic pilot and are deferred.
Also overrides apply_guardrail to re-raise blocked-request errors with the
guardrail's own configured name: the base class hardcodes a module-level
name in that exception, which would otherwise mislabel every blocked Reco
request in logs.
* feat(guardrails): configurable Reco fail modes
Expose unreachable_fallback and fail_on_error on the Reco guardrail
and keep fail-open as the default. Values under optional_params win.
Top-level litellm_params values apply only when set explicitly, since
their defaults are fail-closed and would otherwise flip the default
Also type RecoGuardrail's forwarded kwargs with an Unpack'd TypedDict
instead of silencing ANN003
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(guardrails): align Reco with main's CI gates
Upstream moved unit tests from tests/test_litellm to tests/unit, so
test_reco.py was invoked by no CI job. Move it next to the other
guardrail tests
Replace RecoGuardrail's forwarded kwargs with explicit keyword
params, read optional_params through a pydantic adapter instead of
untyped dict/getattr access, freeze RecoOptionalParams, and drop
mutable-ok markers that suppress nothing, to stay within main's
type-discipline and basedpyright budgets
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(guardrails): fail closed by default in the Reco guardrail
Matches GenericGuardrailAPI and every other guardrail here: a Reco error or outage now blocks the request unless the admin sets fail_on_error: false. Tests cover the default, explicit fail-open, and top-level vs optional_params precedence
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): honor explicit fail-open on Reco when optional_params is a model
model_dump(exclude_unset=True) so pydantic defaults in optional_params no longer override an explicit top-level fail_on_error. Also drop the duplicate log_guardrail_information decorator, call the parent by keyword, freeze RecoConfigModel, and type the tests
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep Reco config model assignable and the hook decorator the gate requires
frozen=True on RecoConfigModel froze the shared LitellmParams mixin, which the guardrail registry assigns resolved secrets on, so every guardrail failed to load. Dropped it and added a test that LitellmParams stays assignable. Restored log_guardrail_information on the Reco override since tests/code_coverage_tests/check_guardrail_apply_decorator.py requires it on every hook. Test params typed as Mapping for LIT001
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): format the Reco test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Yaron Bratspiess <yaronb@reco.ai>
Co-authored-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): buffer tool discoveries in redis and let the leader write LiteLLM_ToolTable in one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the redis family on the tool discovery buffer and drop the blind except and cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy LIT010 and LIT014 in the tool discovery buffer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the aggregation fold Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): requeue drained tool rows to redis when the write provably did not apply
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drain tool discoveries on their own lock and scheduler job, not the spend commit
* chore(proxy): suppress the untyped apscheduler add_job on the tool discovery job
* refactor(proxy): move the LiteLLM_ToolTable bulk upsert into its own module next to the other bulk upserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): name the LiteLLM_ToolTable bulk write span so the prisma span check resolves it
* refactor(proxy): move the tool discovery drain out of DBSpendUpdateWriter into tool_registry_writer
* refactor(proxy): share db_write_failure_is_requeue_safe from exception_handler so the spend writer stays untouched
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): configure the default Usage time range
* fix(ui): retain the selected calendar preset during input sync
* fix(ui): preserve fallback dates when applying across DST
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)
* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)
* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts
* fix: restore pass-through behavior on paths the typing pass tightened
MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites
* fix(runwayml): keep a malformed 2xx task body on the 500 path
* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report
EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.
`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.
The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.
* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed
* test: annotate the new regression tests' locals with Final
* test: annotate the remaining new test locals with Final
* fix(spend): await the per-request rollup enqueues and report their failures
The daily and entity spend rollups ran in an unreferenced asyncio task that
nothing awaited or drained, so a shutdown after the spend log was enqueued
could drop every rollup for that request, and a malformed metadata payload
dropped the request from all daily tables at debug level. The rollups now run
inline in update_database, each helper failure is logged through
spend_log_error with the request id, and unreadable metadata only costs the
row its metadata-derived fields
Resolves LIT-6027
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): type the rollup failure helpers for the lint gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): track the rollup enqueue task so shutdown drains it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): shutdown drain waits for an in-flight rollup enqueue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): register the rollup task without blocking the cost callback
Awaiting the rollups inline made a blocked queue put stall increment_spend_counters, so budget checks could run on stale counters. The task is now only registered via spawn_logging_task; drain_logging_tasks still waits for it on shutdown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own
* fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced
* test: isolate the offline-mode tokenizer tests from the local Hugging Face cache
* test: cover the offline policy templates and the cached Kimi K3 tokenizer
* test(integration): count Kimi K3 tokens in the offline-mode egress test
* refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(chat): port Moyai UI with Model Hub and images
* feat(chat): add Models sidebar with embedded Model Hub
* feat(chat): polish Model Hub with provider cards and logos
* refactor(chat): use a provider dropdown in Model Hub
* feat(chat): keep workspace switching visible with a Chat identity
* Add minimal opt-in Chat usage metrics and prioritize Chat in workspace selector
* fix(chat): enable usage metrics by default only when chat is used
* fix(chat): count requesting users with frontend-only metrics
* feat(chat): send default-on request metrics to litellm-tel
The router plants its deployment fields in litellm_metadata for Interactions
calls, while the proxy put the request's metadata (spend_logs_metadata, tags,
key fields) under metadata. The spend log resolves litellm_metadata and only
carries user_api_key* fields over, so spend_logs_metadata never reached
LiteLLM_SpendLogs. Route /interactions like /responses so both land in
litellm_metadata.
Fixes#47841
Co-authored-by: Igor Tavares <2163472+igorborgest@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): sort every teams table data column
* fix(ui): order unlimited team limits as infinity and verify org scope
* fix(ui): validate team sort filters without recursion
* feat(ui): sort every users table data column
* fix(ui): order unlimited user budgets as infinity and verify org scope
* fix(ui): cover user sort queries and document pagination bound
* fix(ui): preserve Aria usage token breakdown and viewer model access
* fix(ui): make Aria conversations easier to read and reply to
* fix(ui): wrap long headings in assistant tables
* feat(ui): simplify Aria and add usage charts with PDF answer export
* fix(ui): retain full chart values in Aria PDF exports
* feat(ui): consolidate gateway and usage assistants as Aria Agent
* refactor(ui): give Aria an original blue sprite mascot
* feat(ui): prompt shared Aria from usage and retire duplicate usage agent
* fix(ui): keep Aria naming consistent and reachable across dashboard
* feat(ui): open Aria in a floating popup with suggested questions
* fix(ui): simplify the shared Aria Agent title
* feat(proxy): add LITELLM_DISABLE_LATEST_RELEASE_CHECK to stop the release check
The dashboard asks the proxy for the latest LiteLLM release, and the proxy fetches it from api.github.com on every cache miss. Deployments that block or audit outbound traffic had no way to turn that off. With LITELLM_DISABLE_LATEST_RELEASE_CHECK=true the endpoint returns null without making a request, which the dashboard already treats as no upgrade banner.
* fix(proxy): regenerate dashboard types for the release check flag
The endpoint docstring feeds the OpenAPI description, so the ui-api-types
check needs schema.d.ts regenerated. The docstring now says the flag skips
every outbound request, which stays true once the changelog feed is read
too
---------
Co-authored-by: Sujit Nirmal <nirmalsujit891@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The Admin UI upgrade notice read GitHub's latest release, which the release
pipeline sets on its own, so admins were offered releases the changelog had
not announced yet and sent to the auto-generated GitHub release page. The
notice now reads the newest stable release from the docs changelog feed and
links to its changelog page. GitHub release notes are still used for the
feature and fix counts of that version's minor line.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(responses): round-trip Anthropic compaction blocks as compaction items
/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.
Fixes#41456
* fix(responses): replay compaction blocks natively only when the request compacts
* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails
Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.
* fix(responses): count, route, guard, and redact compaction items consistently
A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.
* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it
* fix(responses): keep a stored compaction summary whole in spend logs
* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning
A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.
* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order
Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.
* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes
A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.
* fix(spend_logs): lift only the redaction opt-in header into failure rows
The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main
* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes
* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary
A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.
* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id
* fix(responses): count only message items with a string role as compaction instructions
* test(responses): flatten the last two-for comprehension in the compaction wire test
* fix(responses): replay from a compaction in stored Responses output and SDK compaction items
A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.
* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history
* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary
A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.
* fix(responses): replay a compacted session only through the referenced response
A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item
* fix(spend-logs): keep main's stored value for a prompt length limit under 2
At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value
* fix(spend-logs): keep main's stored tail for a negative prompt length limit
At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start
* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages
* test(responses): run the compaction wire tests on the worker database under isolated workers
* fix(router): keep the encrypted content affinity pin out of provider requests
* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction
* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message
* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses
* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call
* test(proxy): run the dispatched-call-type failure test through the real request setup
* fix(proxy): count a failed Cursor stream's input from the newest compaction item
The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.
* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item
* fix(responses): keep a tool call a later output answers when trimming before a compaction item
* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes
* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time
* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests
* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item
* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item
* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content
* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks
* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once
* fix(responses): count input_tokens after a compaction item the way inference sends it
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage
A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object
across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception,
and a second drop never reached the failure handlers because the first had already marked the object as
logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the
router discards the stashed usage right before it opens the next attempt, on same-group retries and on
every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage.
* fix(router): dispatch chat stream failures through the attempt-scoped logging copy
The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types.
* test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy
* test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader
The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead.
* test(router): use real Logging in fallback cooldown test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): preserve superseded retry usage for billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill retry input after prior stream usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): track retry dispatch for terminal billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): use deterministic retry timestamp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): avoid real clock in fallback tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): remove duplicate streaming logger import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): retain error-frame usage for exhausted retries
* test(proxy): fix retry billing expectations and helper annotations
* fix(proxy): preserve failure estimate without carried usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): audit retry helpers and fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fix retry fixture timestamps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe retry stash at attempt boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): keep terminal failure attribution when dropped attempts log on copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish a dropped attempt's payload only while it is still the current attempt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover spend-log encoding error propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): cool down the final fallback attempt when its failure callbacks were blocked
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): import Final in adapter regression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop spend-log encoding expectation superseded by main's poison isolation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): finish the wrapped stream's failure log before surfacing the error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin pass-through stream fixture start times
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload before awaiting failure callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): publish failed attempt payload from the failure handler after baseline invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging): drop unused discard_partial_usage_for_failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
