* fix(ui): keep the request log model cell to one model
A session row rendered every model the conversation ever used, comma
joined. A long conversation produced a huge string that the auto-layout
table turned into a very wide Model column, pushing the rest of the
columns off screen.
Render only the model of the row's own call (the newest call in the
session) and show an overflow count. The full model list stays in the
hover popup, including a note when the server capped the list.
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(ui): use the shared Badge and an i18n key in the request log model cell
The "+N" overflow marker now uses the shadcn Badge instead of a hand-styled span, and the
"More models not shown" note reads from en.json so the dashboard's i18next/no-literal-string
lint rule passes.
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: add customer spend breakdown and filtering by current team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: recover legacy customer team spend and tolerate lookup outages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf: resolve customer team keys once per daily activity query
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: paginate customer team activity in SQL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): keep Claude Code subagent and side-request traffic off other agents' tier pins
Claude Code sends one x-claude-code-session-id for the main agent, every
subagent and their side requests. In user_turn mode a subagent's new ask
rewrote the parent's pin (#50099), and the agent progress summaries Claude
Code fires during a subagent run rewrote the subagent's pin to the cheapest
tier (#50444).
Subagent requests now get their own tier pin keyed on session id plus
x-claude-code-agent-id. Requests marked x-claude-code-request-class:
auxiliary route as housekeeping, and only main and subagent turns may write
a pin. Without the hint header, the summary and prompt-suggestion prompts
are matched as housekeeping sentinels.
* chore(ui): regenerate schema.d.ts for the housekeeping_patterns description
* fix(complexity_router): let the request-class hint only mark housekeeping, not suppress pin writes
A client-sent request class must not decide whether a decision is saved.
Tagging a real ask as workflow or auxiliary dropped its routing decision,
so later continuations replayed an older pin with the new history. The
auxiliary hint now only routes the request through the housekeeping path,
which is already never pinned and defers to custom classifiers and
keyword rules.
* test(complexity_router): cover the auxiliary hint on a prompt no sentinel matches
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(router): let deployments declare a shared encrypted content boundary
Deployments that set the same model_info.encrypted_content_boundary are
treated as encryption boundary peers by encrypted_content_affinity, so
regional endpoints of one provider organization with different api_base
values keep encrypted reasoning when a follow-up moves between them.
Deployments without the key keep the api_base and api_key match.
* refactor(router): type the encryption boundary peer lookup's deployments
* test(router): cover order fallback hops onto a declared encryption boundary peer
* chore(ui): regenerate API types for encrypted_content_boundary
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(ui): fail lint on hardcoded English in dashboard JSX
Adds eslint-plugin-i18next no-literal-string in jsx-only mode with type info for dashboard source files, skipping tests and the two untranslated files, plus a no-restricted-syntax selector that rejects an English literal as the first argument to toast.success, error, warning or info.
* test(ui): lint the fetch rule fixture at a real source path
Type-aware linting needs the file to be in the TypeScript project, so a virtual path under src now fails to parse and no rule runs. The fixture now uses src/app/layout.tsx with the same code and assertions.
* feat(proxy): add OpenSearch index pass-through with per-key read/write grants
Adds /opensearch/<index>/<api> for LiteLLM vector store indexes on OpenSearch
stores. A key or team needs read or write on the index in
allowed_vector_store_indexes, and the proxy signs each call with the vector
store's AWS identity, so callers never hold AWS credentials
The route maps the LiteLLM index name to the store's provider index, refuses
any path, query parameter, or body field that could reach another index, and
sends the exact signed bytes upstream with only the signed headers
* fix(proxy): let OpenSearch _bulk and _msearch NDJSON bodies through auth
opensearch-py sends _bulk and _msearch NDJSON with content-type application/json, and the
auth-time body read parsed it as one JSON document and answered 400 before the route ran.
Auth now skips parsing OpenSearch NDJSON bodies the same way it skips Azure Speech audio.
* feat(proxy): accept a _bulk _index naming the path index and allow named search pipelines
A _bulk action line may name the index in the URL path, and LiteLLM rewrites it to the
real index, which is what opensearch-py's helpers.bulk sends. The search_pipeline query
param now goes through so hybrid search works, while an inline pipeline stays refused.
Also types the type-checking Router stub with _router_init's parameters again, which
make lint's tests/e2e check needs since the Rust core change.
* fix(opensearch): answer 401 when the AWS credential source cannot read its token file
* refactor(proxy): check OpenSearch request bodies without recursive helpers
* fix(opensearch): answer 401 when AWS credentials fail to refresh at signing time
* fix(opensearch): keep auth off OpenSearch bodies and widen the read list to index checks
Auth no longer parses any OpenSearch request body, so a document field named model, api_base or base_url is written
instead of answering 401 or 403. GET and HEAD on the index and HEAD on _doc and _source are reads, each operation
checks the exact grant the list names, an index whose store name two stores share answers 400, and an _msearch
header that names the path index is sent to the mapped index
* fix(opensearch): need read for writes that return documents and keep document fields out of LiteLLM params
* fix(opensearch): keep documents out of LiteLLM hooks, keep main's credential statuses, make _msearch admin only, and bill only searches
* fix(proxy): bill only an exact /opensearch/<index>/_search path
A document read such as /opensearch/docs/_doc/_search ended in /_search and was counted as a billable search. The check now matches the index segment and _search exactly
* fix(proxy): only proxy admins set vector store index grants, and bill no OpenSearch document call
An index grant now opens document writes through /opensearch, so key generate, update, bulk update,
regenerate, team new and update, /user/new, and bulk user creation answer 403 when a non-admin
changes allowed_vector_store_indexes, top-level or under metadata. Resending the stored grants is not a change
An /opensearch path is matched before the LLM route suffixes, so a document id such as search or
embeddings never makes a document call count as a billable request
* fix(proxy): treat an empty index grant list from a non-admin as a change
An explicit empty top-level allowed_vector_store_indexes passed the admin-only check as unchanged and then
replaced the stored grants. It now counts as a change, so a non-admin gets 403. The new grant tests also
carry parameter types
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): label every budget reset time in the Admin UI with UTC
Budget reset times rendered in the browser's timezone with no label, while the proxy resets budgets on UTC boundaries by default. Every budget reset time and budget period now renders in UTC with a UTC label, and both keys tables' Budget Reset column widens to fit it.
* fix(ui): keep the key page's 12-hour clock for the UTC budget reset time
* fix(ui): show clock times in a budget period that does not start at midnight UTC
* fix(ui): show the UTC clock time in the Teams list reset line when it is not midnight
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(ui): clear hardcoded English from dashboard JSX
Move the 283 strings the planned no-literal-string guard flags so main
starts at zero:
- missed UI text (Switchyard router form, Create User custom ID field,
advanced settings placeholders) and our own fallbacks ("unknown",
seconds suffixes) onto t() with new en.json keys
- product names into src/i18n/untranslated/productNames.ts and code
samples, sample values and JSON placeholders into
src/i18n/untranslated/codeSamples.ts
- values that never display (filter ids, enum fallbacks, query keys,
CSS, ids) onto literal-union types or module constants
The rendered English is byte-identical.
* refactor(ui): put Bedrock policy chips and test panel error on t()
The six Bedrock policy chips and the compliance test panel's error word are our own display labels, not server values, so they go through t() instead of module constants. Rendered English is unchanged.
* feat(ui): set deployment weights in key and team router settings
The proxy already applies router_settings.weights per key and team, but no
dashboard screen could set them, and saving a team in the UI dropped weights
set through the API
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): only offer deployments the key or team may weight
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show weight load errors, inherited strategy and team deployments on key create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): keep reliability fields on create forms and label weight inputs by deployment id
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(ui): name the accordion value identity object
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(ui): expect weights in the key create router settings payload
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): keep weight drafts, loading state and team switches correct
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): only accept weight keystrokes that save as a number
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add per-key and team prompt storage restrictions for spend logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply spend-log prompt restrictions to passthrough requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Apply team prompt storage restrictions to queued requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings
The spend-event collector decided whether to ship request and response
bodies from the global setting alone, so an opted-out key or team still
sent prompts through the collector transport. It now uses the same
key/team check as the spend log writer.
A key or team setting of the string "false" was ignored and prompts kept
being stored. It now counts as false, the same way the global setting
parses strings.
Splits the 42-case tenant storage test into one test for the policy and
one for the pass-through logging contexts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows
Failure rows, executed batch rows, semantic index embeddings, websearch
interception calls and errors rejected before request setup built their
logging metadata without the key and team metadata, so a restricted key
or team still had prompts stored on those rows. They now stamp it through
one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata.
The dashboard toggle now shows a saved string "false" as off, matching
the server, and its label and help text come from the translation file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(xai): use billed image cost and resolution prices
* test(xai): mark image generation tests as a package
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): format dashboard values through i18next formatters
Register named formats on i18next's formatter (usd, compact, duration,
relativetime, number and five date presets) with a cache keyed on the
language and options, add format.* keys to en.json, and move the usage
stat strip onto them.
* fix(ui): keep number formats in USD and the format cache bounded
Force USD on the number and compact formats, clean underscore language
codes like i18next does, fall back when options cannot be serialized,
keep formatted placeholders alone in format keys, and pin the remaining
stat strip values in the German test.
* refactor(ui): format durations with Intl.DurationFormat and load its polyfill on demand
Browsers without Intl.DurationFormat (Chrome and Edge before 129, Firefox before 136) load the pinned FormatJS polyfill through a dynamic import and re-render translations once it is installed. The milliseconds-to-units split and the 0s fallback stay, so the output matches the previous hand-built formatter.
* test(ui): cover duration edge cases on the polyfill path
* refactor(ui): drop the unused duration format and its polyfill
Nothing outside tests called format.duration, and its 1m 15s style doesn't match the decimal-second latencies the dashboard shows today. A duration format will come with the pages that need it, in the style they already use.
* fix(bedrock): run realtime guardrails on Nova Sonic sessions
* fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output
* fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames
* fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes
* fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs
Text parts of one message are joined before screening again, so a phrase split
across parts is still blocked, and a mask collapses them into one part. The
spoken notice is realtime_violation_message or the block reason, quoted only on
Nova Sonic. Client-written output_text and text parts are screened. Key and team
guardrail opt-outs reach regular and translation realtime sessions. A non-string
upstream transcript skips screening as before.
* fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages
* fix(realtime): keep every per-part mask when a joined text masks differently
* fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails
Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a
guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the
session request data with its metadata, so key and team policies apply, and a ModifyResponseException
block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock.
* fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items
* fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened
* fix(realtime): screen text parts in a loop so large messages keep their masks
* fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata
* fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): stop parsing formatted dates and spend back into values
Regenerating a key stored its new created_at as locale text and parsed it
back for display, which shows "Invalid Date" on browsers whose date format
JavaScript cannot parse. It now stores an ISO timestamp.
The model health table stored check times as locale text and sorted by
parsing that text, which misorders or drops rows outside US English. It now
keeps the raw timestamp and formats it in the table cell, so the shown text
is unchanged.
getSpendString decided whether spend rounds to zero by stripping commas from
its own formatted output. It now compares the number against half of the
smallest shown unit, which gives the same output for every value and no
longer depends on how the digits are rendered.
* fix(ui): keep getSpendString exact at high precision
0.5 / 10 ** decimals drifts by one ulp once 10 ** decimals is no longer exact
(past 22 decimals), so a value right at the threshold could flip from "< $..."
to a row of zeros. Parsing 5e-(decimals + 1) gives the correctly rounded
threshold for every precision.
* feat(router): add Switchyard Stage Router strategy
* feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router
Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes
ModelCall.respond and fail async. The classifier is now a tagged union of an LLM
judge and a decision model, and decision calls go through the router's adecisions
so any decisions provider works. Algorithm misbehavior and library errors route to
the capable model instead of failing the request, and the session id plus client
session headers are always passed so capable holds apply without a classifier
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): keep the Switchyard router inside the type-check gate
The gate's environment did not install the optional switchyard extra, so every
libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other
optional runtime packages, and imports go through switchyard_rust.libsy, which
ships py.typed. Step handling moves into helpers so no Final is rebound in the
loop, and the dashboard API types are regenerated for the new routing fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): route Switchyard on every endpoint and reject unsafe router saves
The router now passes each request's call type to routing strategies, and
Switchyard converts the request to chat messages with the existing converter for
that call type before translating it. /v1/messages turns with tool or thinking
blocks used to fall back to the capable model, and now route on their signals.
The Anthropic converter can keep tool_result.is_error on an analysis copy, so
Switchyard sees tool failures there, while requests sent to models never carry it
Saves of a strategy router that lists itself now fail with a 400 instead of a
500 with a dead row, the capable and efficient models must differ, judge and
decision calls are not retried before failing open, the Switchyard routing
fields are classified for redaction, and the edit dialog returns the saved
router instead of the stale one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): catch self-routing on patch and rename, and trim Switchyard tests
The self-routing check now reads the stored params through the decrypting
helper and runs when a patch carries no litellm_params, so a PATCH that points
a router at itself or renames it onto one of its targets returns 400 instead of
a 500 with a dead row. The Switchyard tests share one transcript builder and
router fixture, and reformatting noise in unrelated test files is reverted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(ui): make Switchyard a strategy in the existing auto router form
Switchyard is now a card in the existing strategy picker. Picking it swaps the
tier and classifier fields for the Switchyard fields, and the shared name, team
and save button create it through one submit branch, the same flow as the other
strategies. The separate add form, the extracted picker and the hidden-form
swap are gone, and the edit dialog keeps its own Switchyard form
The judge's session_affinity and message_hash_fallback settings are removed,
since Switchyard rejects both for the stage router
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): accept partial tool semantics and show server errors on model edit
The Switchyard tool semantics field rejected mappings that list only some
categories, although the server accepts them. Model PATCH errors were
reduced to a generic message, so it now goes through the shared API client
like model create and surfaces the server text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): show the routing strategy picker up front in the auto router form
Every strategy was hidden behind an Advanced configuration link, so the
quick setup read as the only option and never said it creates a heuristic
router. The picker now always shows with Heuristics as the labelled
default, the quick setup sits under it, and picking another strategy opens
that strategy prefilled with the chosen models
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show the server's reason when a model settings or API key update fails
Both screens replaced the rejection with a fixed message, so a 400 such as
a router routing to itself read only as a generic failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(router): route Switchyard on the Rust router core
main replaced the Python router core with the Rust one, which left
Switchyard routers registered nowhere. Register them with the other
pre-routing strategies, hand each strategy the endpoint the request came
through, and evict Switchyard routers on reload and delete like the rest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form
Switchyard session state is keyed by a hash of the caller's key hash and session id, and client
session headers no longer reach Switchyard unscoped. The add form carries the quick setup models
into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the
schema reads API-saved configs with nulls and structured decision instructions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): make the Switchyard test folder a package
* fix(router): read the Switchyard session key hash only from the proxy's own metadata
* fix(proxy): keep nested nulls as null when masking deployment params
* fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies
* fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model-hub): show each deployment's full prices in the model hub
The model hub only showed one input and one output price per model, the highest across its
deployments. Each model group now lists every deployment's prices, read from the same entry
billing charges, including cached tokens, long-context tiers, flex, priority, batch and
tiered ranges. Both the logged-in and public hub detail views render them in one table per
deployment, labeled by provider only
* refactor(ui): move model hub pricing breakdown text onto i18n
Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged
* fix(model-hub): show Vertex regional prices and explicit cached reads as billed
A Vertex deployment outside the global endpoint now shows its token prices with the regional
uplift billing applies at the deployment's location. The explicit cached read price shows per
1M tokens on its own row instead of a raw per-token value. The breakdown also says that
adjustments made per request, like off-peak discounts and regional or data residency
surcharges, aren't included
* fix(model-hub): show prices written as text the way billing reads them
A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but
the hub dropped it, so a deployment showed no price or the cost map's price instead of its own.
The hub now reads numeric text for regular prices and size-based price ranges, and skips text
that isn't a number
* style(ui): format public model hub test
* test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget
* fix(anthropic): do not surface server_tool_use as client tool_calls
Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.
Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.
* fix(responses): return Claude web_fetch calls as web_search_call open_page items
* fix(responses): keep Claude server web calls without a result as hosted items
A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call
* fix(anthropic): keep pending server calls native and hide server calls from client tool paths
Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.
* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls
Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again
* fix(anthropic): let guardrails and loggers read Claude's server tool calls again
Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.
* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing
A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.
* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge
Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.
* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses
Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.
* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions
* test(guardrails): type the fake Pillar client like its Protocol
* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool
Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result
* fix(guardrails): scan replayed Claude server calls and hosted web calls
Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.
* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks
* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows
* test: annotate the new server tool call test locals with Final and type their parameters
* test: type the new guardrail test helpers with litellm's tool call types instead of Any
* test: narrow the remaining new test helper parameters from bare dict and object
* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs
Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls
* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera
* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): discover upstream LiteLLM Gateway wildcard models
* fix(proxy): use the typed shared client for gateway discovery
* fix(ui): show the selected LiteLLM Gateway logo
* feat(ui): let admins set tpm_limit and rpm_limit on tags
The tag create modal and edit form said tag TPM/RPM limits were unsupported and never sent them, while the proxy has enforced tag-level rpm_limit and tpm_limit since #41807. Replace the notice with TPM Limit and RPM Limit inputs wired to /tag/new and /tag/update, validated as nonnegative integers, with a cleared limit sent as null so the proxy drops it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): prefill tag budget and limit fields from the Edit Tag button
The edit form only seeded max_budget and budget_duration when opened from the row's Edit action, so the Edit Tag button inside the tag view showed empty budget and limit inputs and a cleared field could not be sent as null. Seed from the loaded tag on both paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): accept null budget fields when editing a tag
A tag whose budget row has max_budget or a limit set to null failed the edit form's zod schema with 'expected string, received null' once the fields were prefilled. Treat null like an empty input and type Tag's budget fields as nullable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): accept a null description when editing a tag
A tag created without a description comes back from /tag/info with description null, which the edit form's zod schema rejected with 'expected string, received null' on save. Allow null and type Tag.description as nullable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): ignore invalid limits left in a collapsed tag budget section
An invalid value typed in Budget & Rate Limits and then hidden by collapsing the section blocked Create Tag and Save Changes with the error out of sight. When the section is collapsed and the only errors are on its fields, drop them and submit the visible values, which already discards the hidden budget fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): name the expected payload so the inline-object lint budget holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move remaining outlier text onto i18n
Moves 146 English messages that the earlier i18n batches missed into en.json.
They live in helpers, zod schemas, label fallbacks and shortcut descriptions
that the batch scope script did not count, so no batch picked them up. The
English on screen is unchanged.
* refactor(ui): keep exported compliance error text in English
The compliance results' "Error: ..." text is also written to the CSV export,
so it stays a literal. The policy table's unnamed fallback now reads i18n.t
inside its helper instead of taking t as a new parameter.
* feat(team): let default_team_params attach access_group_ids to every new team
* fix(types): keep default access_group_ids as Sequence for the type discipline gate
* fix(settings): chunk the default access group existence lookup
* test(team): prove default access groups reach the saved team row and type the settings fixtures
* fix(team): check default access groups on the writer so a fresh group is not rejected behind a read replica
* feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests
* fix(spend-logs): keep bridged Responses rows and classifier audit in latest_turn mode, reject unknown values at boot
* fix(spend-logs): drop the recursive call in the latest_turn trim and type the stateless test cases
* test(spend-logs): format the stateless latest_turn cases
* test(spend-logs): wrap the classifier audit fixture
* fix(spend-logs): type the latest_turn trim inputs
* fix(spend-logs): leave the raw request body unvalidated before the latest_turn trim
* fix(spend-logs): trim classifier rows and ignore continuation keys off /v1/responses in latest_turn mode
* fix(spend-logs): take the classifier's originating request from the logging payload and redact a copy of vector store results
* fix(spend-logs): redact the body selected for storage, after the latest_turn trim
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise
Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the
value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way.
s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls
* refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map
* fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request
* feat(ui): add a beta language setting to the account menu
The account menu gets a Language field, marked Beta, that lists each language by its own name with the English name beside it. The choice is saved per browser in localStorage and applied with a page reload, so memoized and imperative translations are read fresh.
English stays the default. An unset value, English, an unknown or unsupported code, a locale that fails to load, and any string a locale has not translated all show English. The field stays hidden until a second locale is registered in LOCALES, so this change is invisible until the first translation lands.
* fix(ui): keep the help translate link text inside the link
<link> is a void HTML tag, so the Trans parser dropped its children and rendered an empty anchor. The note now uses <githubLink>, and the language list is at least as wide as its trigger.
The banner inherits the Alert primitive's icon layout, which spans two grid rows and nudges the icon down 2px so it lines up with a title plus description. The banner only has a one-line title, so the empty second row and the nudge left the icon 3px below the text. Keep the icon on the title's row with no offset
The /chat request errors and deep research notice, and Aria's step limit
and no answer replies, are written into assistant messages that are sent
to the model as history on the next turn. Pin them to English with
i18n.getFixedT("en") so a future UI language never changes the model
payload. English output is unchanged.
* fix(types): define the responses compact call types so during_call guardrails and deployment hooks see them
* test: regenerate dashboard api types and skip compact in the azure sdk client sweep
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Moves the English left in the Aria assistant, the MCP tool permission
groups, the shared chat composer, the Beta badge, the user banner and
the metadata key fields onto react-i18next. English output is unchanged.
Text that Aria sends to the model (system prompt, tool descriptions,
tool results and usage chart payloads) stays English byte for byte.
Where the same string is also shown on screen, a display map keyed by
the operation name or by the chart's English value translates it at
render time, and the model copy is left alone.
Compliance framework and category names, the Compare endpoint labels, the
Decisions presets and schema messages, chat metrics, MCP events, reasoning
toggle, conversation date groups, model filters and the MCP boolean argument
labels now resolve through en.json. Ids, stored values and text sent to a
model are unchanged
Constants and option lists on the keys, teams, agents, budgets, cost
tracking, router settings and vector store pages now hold message keys,
and the code that renders them translates. Values, ids, sort keys and
payloads are unchanged, and lookups keyed by server values check
Object.hasOwn first
Five tests now resolve the expected English through i18n.t instead of
reading the constant directly
Moves the leftover English in the sidebar, page descriptions, account
menu, usage card and a handful of shared widgets into en.json. Option
lists and label maps now hold message keys and are translated where they
render, while every option value, id and stored setting stays the same
string. Lookups keyed by a server value check Object.hasOwn first
* refactor(ui): move remaining models text onto i18n
Moves the leftover English on the models pages, mostly labels and hints
held in constants, onto react-i18next. Constants now hold message keys and
the code that renders them translates. Where a string is both a value and
a label (tier names, health check sentinels), the English value stays for
comparisons, sorting and payloads, and a separate label key drives display.
Lookups keyed by server values check Object.hasOwn first.
* refactor(ui): translate the tier name in display name inputs
Logs and usage option lists, filter chip labels and label maps now hold
message keys that their readers translate at render time. Tile titles, the
"Claude (unspecified)" agent label and the endpoint chart categories stay
English ids and get display maps keyed by them. Lookups keyed by URL or
server values check Object.hasOwn first.
* fix(accounting): keep one undecodable delivery from blocking the ledger
A request whose metadata carried a 401-digit tokens_saved stored that number in
its delivery payload. The query engine could not decode it when the worker
claimed the batch, so every later delivery on the proxy stayed pending
The worker now claims the payload as text and decodes each row in Python. A row
that still cannot be decoded is parked alone while the rest of the batch
applies. The payload builder also clamps integers to the JSON-safe range, so
caller-supplied counts can no longer reach the engine at that size
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): take spend locks in one order so the ledger cannot deadlock
A ledger delivery updates many spend tables in one transaction, so it can
deadlock with any writer that locks the same rows in another order. It took the
team advisory lock after the user and team rows, while team roster writers take
it first
The delivery now takes every team advisory lock before any row, then writes
tables in one canonical order with ids sorted. It also runs under the rollup
lock_timeout, applies 500 deliveries per batch by default, and spreads retries
with jitter
The writers that could still collide with it now follow the same order. The
budget reset cascade, model allowlist rename, global spend reset, bulk user
update, org delete and the delete-by-list paths lock their rows in key order
first, and access group endpoints update teams and keys in one sorted pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): prove a ledger delivery waits instead of deadlocking
Real Postgres tests hold the first lock of a team roster change, an access group
update and a budget reset cascade, start a delivery, and let the other writer
finish. The delivery must wait and then charge once. Before the lock order fix
each case ends in a Postgres deadlock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(accounting)!: make the SpendLogs ledger the default accounting path
use_spend_logs_ledger now defaults to true, so every request is charged through
LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and
Redis spend queues, which stay in place. Tests that cover those legacy writers
pin the setting off, and the rest check the queued charge the ledger stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): let the mocked client open a transaction for key and team deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): take team locks first when a model rename rewrites allowlists
A rename locked team rows before user rows while a roster change holds the team
advisory lock and a user row, so the two could deadlock. The rename now takes the
advisory lock for every team that lists the model, sorted, before any row lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): drain a backlog in one worker run, one batch per transaction
One run applied a single fresh batch, so at 500 per batch charges fell behind any
load above that rate. The worker now keeps claiming fresh batches while they come
back full, for up to five seconds, and each batch still commits on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): queue only the charge when a request's spend log is suppressed
With logging off for a request, the ledger still queued the full row, so a failed
flush or a shutdown could leave its prompt and response in Redis. The queue now
holds only the fields the charge needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(proxy): format the rename team listing query
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): remove expired delivery receipts as fast as the worker applies them
The worker now applies several batches per run but still removed one cleanup batch
of applied receipts, so the table could outgrow its retention window. Cleanup now
repeats while each pass removes a full batch, for up to five seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): retry a model rename when a team starts listing the model mid-lock
The team advisory locks come from a snapshot, so a team that gained the old name
before its row lock was locked without its advisory lock. That attempt now commits
without writing and the next one locks the fresh set of teams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
