When DATABASE_URL_READ_REPLICA is set, NativeDatabase opens a reader pool
resolved from that URL, with its own token rotation. Pools exposes writer(),
read() and read_after_write(), matching RoutingPrismaWrapper and
WriterPinnedClient. read() prefers the replica and read_after_write() the
writer. Each falls back to the other side when its pool cannot connect and
skips the failed side for 30 seconds, so reads keep working on the replica
while the writer is down
The reader pool takes its share out of the reader's connection_limit the same
way the writer does, with the same hard-coded RUST_RESERVED_CONNECTIONS. While
that is 0 the replica URL is built exactly as before and no native reader pool
opens. A replica connection_limit that is not a positive integer is left for
Prisma to judge, as it always was
NativeDatabase.connect opens a lazy SQLx writer pool from an Endpoint that
Python resolves from the final DATABASE_URL, read the way Prisma reads it:
Unix socket host, sslmode / sslaccept / sslcert with Prisma's verification
rules (prefer + strict asks the server once whether it offers TLS),
sslidentity / sslpassword handed over as PEM, schema, options, pgbouncer mode,
pool_timeout (0 waits forever) and socket_timeout. Options SQLx cannot send
are logged instead of failing boot
ConnectionBudget splits each worker's connection_limit between Prisma and the
native pool, so the budget an operator sized never grows. The Rust share is the
hard-coded RUST_RESERVED_CONNECTIONS and no setting names Rust, SQLx or Prisma.
It is 0 until a ported query needs the pool, so Prisma's URL is exactly what it
was, no native pool opens and nothing connects through SQLx. A connection_limit
that is not a positive integer is never split and goes to Prisma as before.
Once the share is above 0 Prisma keeps at least one connection, and pools are
tagged litellm-prisma and litellm-native in pg_stat_activity. Python keeps
minting RDS IAM and Entra tokens and hands each one to the pool
statement_cache_size sets the native statement cache (Prisma's default 100) and
pgbouncer=true turns it off. sqlx-postgres 0.9.0 still names every statement when
the cache is off, so Statements runs queries unnamed in that case, like Prisma,
until a release carries transact-rs/sqlx#4337. An application_name the operator
set is used on both pools
The SQL query files are pinned to LF so Windows checkouts hash them like .sqlx,
and the crate lists .sqlx in its package include so the core sdist builds offline
crates/db/AGENTS.md tracks parity with the Python Prisma stack
The bridge now links litellm-db, so the editable install compiles its queries.
The Rust DB workflow builds that install from the checked-in .sqlx metadata,
since it runs before the migrations that the online check needs
* test(providers): keep registration lookups in the model catalog
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(providers): format catalog registration lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences
Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call
fallback walk reads router settings Key > Team > Global instead of the key's
column alone, and skips a fallback the caller may not call.
A key or team with router_settings.fallback_on_saturation: true walks that same
fallback list as soon as the requested model reaches saturation_threshold in
dynamic_rate_limiter_v3, and goes back to the requested model when no fallback
admits the request. Responses carry x-litellm-fallback-reason:
rate_limit:model_saturation_fallback.
Removes the team_metadata.model_preferences steering and the
x-litellm-model-steered-from header from #51350, which did the same job through
a second knob.
* fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests
fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call.
* fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does
A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access
groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a
fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client
body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team
router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it
cannot pass before saturation.
* refactor(proxy): one helper decides which refused fallbacks end the pre-call walk
_pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller.
* fix(proxy): keep the caller's request object through the pre-call fallback walk
Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place.
* fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes
A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it).
* refactor(proxy): read fallback prices through the router registry entry point
router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info.
* fix(proxy): a fallback's group budget check skips the groups the request already holds
The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read.
* test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering
A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle.
* fix(proxy): the saturation handoff only fires under the fallback walk
_pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429.
* style(proxy): keep the cast-ok reason within the line length
* fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only load the end user for fallback checks when a database is connected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model
The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share
reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request
after admission, and _release_pre_call_hold calls it next to the registered limiter.
An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the
request on to a further alias. It now replays with the model argument the first attempt saw.
The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(files): match every spelling of a provider file id in the ownership check
enforce_provider_file_access matched the stored provider file id by exact string, and a Gemini output file
can be addressed as files/<id>, files%2F<id>, <id>/ or its full URL, all of which the Gemini file transformation
sends to the same file. Those spellings found no ownership row and fell through the no-row pass-through path.
canonical_provider_file_id collapses the spellings to the bare id; the ownership check looks up both the
spelling the caller sent and the canonical one, and store_batch_output_file_ownership stores the canonical id.
* fix(files): unwrap model-encoded ids and index canonical provider ids in ownership rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(files): assert the ownership lookup outcome instead of its query shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(files): map provider ids back to their unified file by canonical spelling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(files): record the bound on the ownership lookup's in filter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(files): reduce any scheme or case wrapping files/<id> the way gemini does before the ownership lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(files): bind the reverse lookup match with a walrus instead of a Final inside the loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(files): assert the provider id lookup covers the requested ids instead of its exact list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit e8fc243a09)
* fix(files): keep non-Gemini storage URLs opaque so an upload name cannot index another key's provider id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(files): judge the provider id resolver by the mapping it returns only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(nace): add drex-max to the catalog, image support, and the complexity router
Nace released drex-max, the Drex decision model that also reads images,
and the existing nace provider could not use it: the catalog had no row,
so cost tracking and the dashboard did not know the model, and image
input was refused before the request left the gateway.
Add the nace/drex-max catalog row to both cost map files with
supports_vision and the per-question token limit, let the nace decisions
provider forward images as the top-level images array Nace reads
(inline images as data URLs, remote URLs fetched and inlined, at most 8
per request), and accept provider nace in the Auto Router classifier
config so a tier can be chosen by drex-max with NACE_API_KEY.
Also: resolve os.environ/ references for the classifier's nace api_key
and api_base once, and refuse an unset one instead of falling back to
NACE_API_KEY; price drex-max at $0.03 per 1M input tokens and add
drex-max-v1.1; mark the retired drex-v1.0, now served by drex-max, as
vision-capable; and accept provider nace in the dashboard's classifier
config (default drex-max), the generated API types and the member
auto-router permission check.
* feat(nace): send X-Nace-Client attribution header
Nace requests, including the Auto Router classifier, identify litellm and its version in X-Nace-Client for Nace's usage attribution. Other providers do not receive it
* fix(nace): list Nace in the classifier UI and follow the test conventions
The dashboard's classifier config widened its provider type to nace without a
description entry, so the type check failed, and the Auto Router's inline
provider picker did not offer Nace. Both now include it, with tests
The header test reads the request respx captured instead of appending to a
list, and the drex-max catalog tests move into the existing catalog test file
* fix(nace): bound image downloads and refresh Drex aliases
* test(images): exercise streaming transport in provider regressions
* fix(nace): preserve text-only Drex v2 image handling
* refactor(nace): scope Drex Max to decisions integration
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(proxy): name the customer organization by alias with x-litellm-customer-org-alias
A request may carry x-litellm-customer-org-alias next to, or instead of,
x-litellm-customer-member-id. Alone, the alias charges and enforces the
organization level only. Next to the member header it must name the
member's organization, else 403. Unknown alias is 401, an alias shared
by several organizations is 400. Organization alias changes evict the
cached lookup.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for optional customer team and member ids
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): register unknown customer members in the aliased organization's default team
A request carrying x-litellm-customer-org-alias and an
x-litellm-customer-member-id the proxy has never seen creates the member
under the organization's default team, on the organization's default
budget, and serves the request. Without an alias, or when the
organization has no default budget, the member stays unknown and the
request is still refused.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare the member registrar callable as a TypeAlias
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): enforce customer org, team and member budgets and limits on the request path
Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): customer hierarchy survives virtual key auth without a legacy end user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): keys pinned to a customer organization may only act for that organization's members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): keep the customer hierarchy spend maps within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): check the key's customer org scope after the key is resolved
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the hierarchy cache key family and keep type counts flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): resolve the customer hierarchy in the shared authorization gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop a member's cached hierarchy through the declared cache target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(auth): keep the hierarchy reset within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): type the requested models as a sequence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend): add daily customer usage and budget resets (#51493)
* feat(spend): add daily customer usage and budget resets
* chore(spend): document customer reset query bounds
* fix(spend): chart customer queue metrics and update recovery test
* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit ccac47608f.
* fix(proxy): evict cached member hierarchies when team, org or template budgets change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the auth cache dependency alias as a TypeAlias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit 7542471e17.
* fix(spend): reconcile the daily customer spend table with main
Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): tolerate failure metadata without customer hierarchy ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): key daily customer spend rows on main's hashed identity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): read customer ids from failure metadata through a typed lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
Deployment affinity and the complexity router's session and tier pins move
off the router's DualCache onto RouterSessionPinStore, backed by the native
redis-store. The read and claim Lua scripts run byte-identical to main, the
pin read rides the request's pre-call batch, and keys and values keep main's
format.
* refactor(router): count the whole router_usage family on the native store
usage-based-routing-v2, the model rate limit check (tpm, rpm, tpd, rpd) and the
io-token check (itpm, otpm) count through the native router_usage store, next
to the router core's global_router tpm and rpm. The pick's usage read rides the
native pre-call batch and settles on the batch reply, keeping DualCache's
per-key read throttle, and the success tpm increment rides the native
post-call batch after counting in memory. io-token reservations stash their
counter's fields instead of a Redis key. Redis keys and values are unchanged.
A Redis service event is reported in a context named for whose it is, either
the calling store op's, the request whose post-call batch its deadline sends,
or nobody's for breaker transitions and unscoped calls. The deadline's tag is
built when a post-call declaration arms it, so that flush parents under the
request's span even though auth sets the span after the batch scope opens. An
SDK call anchors the span its caller passed, so its usage reads and io-token
refunds parent under it without a span argument threaded through the family.
* fix(redis): keep an int counter past i64 above the limit instead of wrapping
Number::add summed two ints with plain i64 addition. With Redis off and the OTPM counter at 1, a request with max_tokens=2**63-1 passed the MAX_INCREMENT check, and the memory count wrapped negative in a release build, so it passed admission and let other requests past the cap. Python's int never overflows, so main sees a count above the limit, rolls it back and answers 429
An int sum past i64 now becomes the nearest float, which compares above any limit the way Python's int does. The rollback after such a refusal can leave the count one token low, since the float cannot hold 2**63 exactly
* fix(logging): run every callback under the span an SDK caller passed
Native Redis service spans take their parent from the request span contextvar. The proxy sets it at auth and the SDK router scopes it around the routed call, but a streamed response is iterated after that scope closes, so its callbacks saw no span. Three callbacks reopened the scope from kwargs by hand, and the sync stream path had no wrap at all because its success handler runs on an executor thread
The four Logging handlers now anchor the span from model_call_details around their dispatch, so every CustomLogger callback sees it, streamed or not, sync or async, with or without a Router. The per-callback wraps in lowest_tpm_rpm_v2 and model_rate_limit_check are gone. A span already anchored, as on the proxy, is kept
* test(logging): cover a mid-stream failure, and drop the span wrap in Router._release
The callback span test now also breaks a stream after its first chunk, sync and async, through a custom provider, so the failure handlers the stream wrapper starts on a bare thread are covered too
A probe through the Router showed Router._release only runs inside the routed attempt, where the router has already anchored the span, and never after a stream returns. Its own wrap was a no-op, so it is gone
* fix(router): refuse an io-token reservation past its limit before counting it
The overflow fix let a reservation near i64::MAX become a float, so the count still read above the limit and the request got its 429. But the rollback subtracted the same amount from that float, and with OTPM usage at 40 the count came back as 0.0, so later requests reused capacity already spent
A reservation above its limit now gets the 429 before the store is touched, in the async and sync paths. On main the count went up and straight back down for such a reservation, so the answer and the count after it are the same as main's. Only the usage named in the error message differs, which now shows the reservation instead of the count plus the reservation
* test(redis): send the failed SDK call's refund through a real completion
The refund span test called ModelRateLimitingCheck.async_log_failure_event directly, so it relied on the span wrap that callback used to carry. The Logging failure handlers now anchor the span for every callback, so the test sends a failing litellm.acompletion with the reservation in its metadata and checks the refund's Redis span still lands under the caller's span
* feat(redis): serve the health checks' deployment states from a native store
DeploymentHealthCache reads and writes its one Redis entry through a new
deployment_health store family instead of the router's DualCache, like the
other router families.
* refactor(redis): type the deployment health states in the store
The health store kept a raw Value from Redis to Python and back, so Python re-checked the shape by hand and router-core only got a typed record after a Python pass. The family now has a types.rs with HealthState and HealthStates, the bridge converts through PyData, and the Python store returns DeploymentHealthStateValue mappings
Reads keep main's leniency for records another writer left: extra fields are ignored, a falsy flag is unhealthy and a missing one healthy, a timestamp that is no number reads as 0, and a record that is no dict is dropped alone. A rewrite writes only is_healthy, timestamp and reason
router-core's RouterCache now names the routing state it touches (cooldowns,
allowed fails, per-minute usage, health states) instead of spelling Redis keys.
The stores own every key, and the Python host answers each typed cache step
from its store without parsing keys back into ids.
* fix(dynamic_rate_limiter_v3): read a model's saturation from Redis so every worker agrees on it
A worker's in-memory copy of the saturation counters carried only the usage it served
itself, and steady traffic kept that copy alive past saturation_check_cache_ttl, so two
workers behind one Redis disagreed on whether a model was saturated. The saturation read
now goes to Redis and is written back to memory for the TTL; without Redis memory is all
there is.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): read the saturation count from Redis itself, not DualCache's memory-first get
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): back the Redis saturation count into memory only when memory holds nothing, and split the counter-pair comprehensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): fall back to the worker's own saturation count when the Redis read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename customer member budget_id to budget_override_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ci): let the schema gate accept the unreleased customer member rename
* fix(proxy): refuse prisma db push while the customer member budget_id column is still unrenamed
* refactor(proxy): rename customer member budget_template_id to budget_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse to baseline a database that still has the old customer member budget columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): suppress unknown member type on the optional proxy-extras budget rename guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drop the flaky current-download count refresh case in the Skills Hub panel
* test: drop four unit tests that flaked on main this week
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): show customer organization usage by date range
* feat(customers): align usage reads with daily spend contract
* fix(customers): combine spend display and resolve CI failures
* test(vector-stores): isolate search warnings from SDK verbosity
* fix(ui): cap customer usage date ranges at 400 days
* perf(rust): store the sent wire bytes as the Messages request instead of re-serializing
The Rust /v1/messages path handed record_response the body as a Python dict, and
Python ran json.dumps on the whole body again to build httpx_response.request.
Rust now keeps the bytes it serialized for the final attempt and passes them
through, so the stored request is exactly what the provider received and Python
does no extra full-body serialization. The hosted transport path sends the same
Rust bytes instead of a second Python json.dumps.
This also fixes the stored request after a retry with a wide int, NaN or
Infinity, which previously carried the internal marked string instead of the
real value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx
* perf(rust): serialize the Messages logged input in Rust instead of json.dumps
messages_input ran json.dumps on the whole pre_call body in Python for every
Rust /v1/messages request. Rust now serializes it with the same raw_json writer
it uses for the wire, at the same point (after caller substitution, before
Logging.pre_call), so callback edits still reach the wire but not the logged
input. Python only returns the top-level key order, with the handler signature
read once, and top-level scalars keep the caller's own spelling.
The logged input string becomes compact JSON with raw UTF-8. Top-level key order
and values are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx
* fix(rust): keep body keys missing from the order list in the Messages logged input
The logged input only wrote keys named by Python's order tuple, so a key left
out of that tuple would vanish from the log without an error. Keys the order
does not name now follow in their sent order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dynamic_rate_limiter_v3): give a capped borrower its own cap counter so it also gets a fresh share at saturation
A team with team_model_limit_max_borrow_multiple enforced its cap on the same counter as its share, so its
idle-time usage kept counting and it got no fresh share once the model saturated. The cap now runs on its own
counter (model_per_team_max_borrow_multiple) that counts all of the team's usage this window, while the share
counter only counts usage once the model is saturated. Also repairs the six share-metric unit tests that main
broke when #51415 and #51417 landed together, by saturating the model from a separate priority class before
the measured calls.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): size a capped team's token reservation by its share, idle or saturated
The reservation for a prompt without max_tokens was bounded by whichever descriptor was being held, the cap
while idle and the share once saturated, so the same request was refused at one moment and admitted the next
with the same counters. Reserve against the share, the tighter of the two, in both states.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): track the cap's reservation apart from the share's
The cap and the share shared one reserved-token figure and the fallback carry keyed on the share's window, so an
idle admission's hold on the cap was refunded from a share that never carried it once a WebSocket turn saturated,
stacked idle fallback holds could carry across cap windows, and an idle fallback served for a capped team
published no share gauges. Each counter now carries its own reservation and window, and a fallback hold keeps
the target's share so the gauges read the cap counter against it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep a WebSocket's share hold until it settles and read capped idle borrowing against the team share
A frame checked after the model went idle again replaced the connection's share descriptor with None while the
admission's hold on that share was still outstanding, so settlement never gave the hold back. The share is now
kept while its reservation is open. With the class share split on, the share gauges picked the wider class share
over a capped idle team's own share, understating its borrowing, so the tighter of the two is published.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The --target builder image copies the repo root to /app, so after the
schema moved under litellm/ it lives at /app/litellm/schema.prisma there.
Only the runtime stage adds the /app/schema.prisma alias.
Claude-Session: https://claude.ai/code/session_01Bne5zS8tnYUDfKJYENWcYA
Co-authored-by: Claude <noreply@anthropic.com>
* fix(cost): bill Bedrock cross-region profiles at the profile rate on /v1/messages
Bedrock reports the region-stripped model id for a cross-region inference profile,
and the cost selector preferred that served id, so streaming and non-streaming
Messages spend (input, output, cache write and cache read) used the global row.
A served id carrying a different cross-region prefix keeps its own price.
Fixes#46286
* test(langfuse): derive the Bedrock profile export cost from the profile's model info
The regional us. deployment is now billed at its profile rate, so the expected
Langfuse cost comes from that profile's rates and the mock usage instead of a
hardcoded global price.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(bedrock): send max_tokens and priority in Cohere embedding requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sagemaker): mark Cohere transform result Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rerun after unrelated checkout timeout and langsmith flake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): unwrap extra_body for providers with their own embedding handlers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): unwrap extra_body in get_optional_params_embeddings and keep lint gates flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): let mapped params win over extra_body and drop the unused merge helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): merge extra_body into the provider body after each native embedding transform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): let the transformed body win over extra_body so input and mapped params cannot be replaced
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): sign the Bedrock body that is sent and align the shared handler so extra_body never replaces the input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(embeddings): keep the shared handler's extra_body precedence, the router batch limit relies on extra_body.input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): set per-threshold budget alert recipients on virtual keys
Create Key and Edit Key get the threshold and recipients rows the team
member budget alerts already use, writing metadata.max_budget_alert_emails.
The helpers and the row editor move to key_team_helpers so the team form
and both key forms share them.
* test(ui): cover the empty metadata box path and drop a redundant payload case
* fix(ui): keep comma-separated string alert recipients stored through the API
The backend accepts a string of recipients per threshold, so Edit Key and the
team form now load them as rows instead of dropping them and deleting the
alert on the next save
* fix(ui): keep the stored key alert map on saves that do not edit its rows
Edit Key rebuilt max_budget_alert_emails from the rows on every save, so
API-written entries the rows cannot show (a null recipient list, a list with
a null, thresholds outside 1 to 100) were deleted by unrelated edits. The
map is now rebuilt only when the rows were edited and otherwise sent back
as stored
* fix(ui): load null and mixed alert recipient lists as rows and drop a typed map on Create Key
A null recipient list loads as an owner-only row and a mixed list keeps its
strings, matching what the backend sends. Create Key now ignores a
max_budget_alert_emails typed into the Metadata box, as Edit Key does, and
the hint says thresholds replace the default 80% alert
* fix(ui): keep Create Key alert rows when the Metadata box holds null
* fix(ui): move typed alert maps into rows, keep hidden alert entries, hide alert rows from team admins
A max_budget_alert_emails map typed into the Metadata box on Edit Key or Create Key now moves
into the threshold rows on blur with a toast, the same way a tags array moves to the Tags field.
Entries the rows cannot show stay in the JSON box.
When the rows are edited, the saved map keeps the stored entries the rows cannot show (thresholds
outside 1 to 100, non-numeric keys, unreadable values). Create Key keeps such typed entries too.
Team admins editing a member's key no longer see the alert rows, since that save only accepts
budget fields
* fix(ui): let the Metadata box own key alert entries the threshold rows cannot show
Edit Key now shows max_budget_alert_emails entries outside whole numbers 1 to 100 in the
Metadata JSON box and saves them from the box as the admin left it, so they can be seen,
edited and removed. Rows still own every entry they can show, and an untouched save sends
the stored map unchanged
* fix(ui): leave zero-padded alert thresholds to the Metadata box instead of duplicating a row
* fix(ui): keep loading zero-padded team member alert thresholds as rows
The canonical-key rule applies only to key alert maps, where the Metadata box
holds entries the rows cannot show. The team form has no such fallback, so it
keeps loading "080" as row 80 as it did before
* fix(ui): let the Metadata box own a non-object max_budget_alert_emails value
A string, list, number or null stored under max_budget_alert_emails stays in the
Edit Key Metadata box, is saved back as the box holds it, and is stored as typed
on Create Key. Threshold rows only own an object map, so rows the admin adds
replace a non-object value
* fix(ui): move typed alert thresholds into the rows when Edit Key saves without a blur
Submit now moves a max_budget_alert_emails map typed into the Metadata box
into the rows, as it already does for tags, and reads the edited fields at
submit time so the moved rows are saved
* refactor(ui): name the Edit Key submit handler
fal's luma/agent/ray/v3.2 endpoints default resolution to 540p, but the fal video transformation had no
Luma profile and fell back to the Seedance profile, whose default is 720p. A request that left
resolution unset was stamped video_resolution 720p and priced from output_cost_per_second_720p, which is
twice the 540p rate on text-to-video, image-to-video and reframe and 1.5x on video-to-video.
A Luma Ray profile (540p, 720p, 1080p, default 540p) stamps the real default, so those requests price
from output_cost_per_second, the 540p rate.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates
The per-team share counters only count requests admitted while the model is
saturated, so what a team used while the model was idle is not charged
against its share once contention starts. A team below its share is still
admitted ahead of a borrower, and the model's own limits bound the total.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): idle fallback targets and WebSocket turns follow the same share accounting
A router fallback target attempted while idle no longer holds the team's
share, and a Responses WebSocket connection refreshes its share descriptor
on every turn, so idle turns are forgiven and saturated turns count.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): an idle fallback target still records its hold
Settlement finds the hold, gives the admitting model's reservation back and
charges nothing for the idle target, instead of charging the target's
tokens to the admitting model's share. The integration fixture builds its
configuration without mutating the loaded one.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label
Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits
the request, a router fallback's settlement publishes the fallback target's own share under its model group, the
gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window
replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests
restore the process registry, and the integration test polls for the settled totals.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): type the gauge setter and the label factory so the type gate holds
_set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory
declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the
label test tolerates the now visible None values.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(prometheus): keep the label test diff to the two None-tolerant assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): load LITELLM_MODEL_COST_MAP_URL from gs:// with Google application default credentials
* fix(cost-map): bound the gs:// token wait by the fetch timeout and accept a bare STORAGE_EMULATOR_HOST
* fix(cost-map): retry a gs:// token that is not ready and send no credentials to a storage emulator
* fix(cost-map): spend one timeout across the gs:// token wait and the object request
* fix(fal_ai): support flat per-video output pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): handle flat-priced video catalog rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(rust): format flat video pricing parser test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse blocked and over-limit keys before reading the request body
A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path.
* fix(proxy): pick the effective key header for the early refusal and keep its failure log
The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook.
* fix(proxy): honor litellm_key_header_name in the early refusal
The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies.
* fix(proxy): keep the early-rejection path inside the type gate ceilings
Read the configured key header through general_settings_view() instead of importing the untyped
general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as
HTTPException.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): distinguish unknown timeout usage from prompt estimates
* fix(otel): handle expected tokenizer failures and cover parent spans
* fix(otel): validate failure usage inputs at typed boundaries
* fix(otel): type Logging.messages for the failed prompt token estimate
Declare Logging.messages as Sequence[object] | None at its single __init__
assignment and drop the list TypeAdapter, so the failure estimate reads a typed
value without new unknown-type diagnostics.
* fix(otel): skip the failure prompt estimate for prompts with attachments
Only estimate the failed prompt when every message is plain text, so files,
documents and images never reach PDF parsing or media decoding on the failure
path. Those requests still report unknown provider usage without an estimate.
* fix(otel): count the failed prompt off the event loop for async requests
Async failure logging now awaits the existing offload_token_count limiter for the
optional prompt estimate, and the sync half of an async request no longer counts
inline, so a slow tokenizer load cannot stall other requests.
* fix(otel): only count the failed prompt for loggers that receive failures
The estimate check now reads general callbacks, the global failure lists through
a typed get_failure_callbacks accessor, and the dynamic failure lists, so a logger
registered only for success never triggers failure token counting.
* fix(otel): return an immutable failure callback snapshot
* fix(otel): keep malformed content blocks from breaking failure logging
Only compare string block types against the text block table, and run the
text-only check inside the estimate's narrow try, so a malformed prompt leaves
usage unknown without an estimate instead of interrupting failure callbacks.
* fix(otel): stop the token counter from debug-logging raw prompts
The failure estimate counts the original prompt, so the counter's debug line
could write prompt text to application logs while message logging was off.
Drop that line and assert the prompt stays out of debug logs on both routes.
* test(otel): pin text-only eligibility and failure callback categories
Cover the public has_only_text_content contract for malformed and non-text
messages, and check get_callbacks_by_type puts sync and async failure
registrations in the failure categories through the public manager APIs.
* fix(otel): keep failure logging when the prompt estimate cannot run
Return no estimate when the offloaded count hits a worker or tokenizer
RuntimeError or OSError, so failure callbacks still run, while cancellation
still propagates. Treat SDK Message objects like their dict form when checking
whether a prompt is text only.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(anthropic_messages): key the first web search round's spend row by its own upstream reply
A non-streaming /v1/messages request answered through websearch_interception
logged its first model call with that call's usage but the follow-up reply's
id, so both spend rows shared one request_id and the first was dropped by
skip_duplicates. The logged row now takes its id and provider-reported cost
from the same upstream body its usage is parsed from, falling back to the
served id when that body has none.
Refs #49651
* refactor(anthropic_messages): read the upstream reply id after reparsing it
Drop the defensive body helper. transform_response already requires a read
JSON body, so the upstream id and reported cost are now read after it
succeeds, which leaves every existing failure path unchanged. Tests gain
return annotations and use the current Sonnet model name.
Refs #49651
* fix(anthropic_messages): key by the upstream reply only when a follow-up was served
Keying every reparsed /v1/messages row by its upstream reply overrode an id a
post-call deployment hook deliberately replaced. The anthropic agentic
executor now records on the outer logging object that it served a follow-up's
reply, and only then does that call's spend row take its id and reported cost
from its own upstream reply.
Refs #49651
* test(anthropic_messages): pin the agentic follow-up row test to a fixed start time
Refs #49651
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
A guardrail such as Pillar aliases the request's metadata to litellm_metadata,
which holds the caller's UserAPIKeyAuth. Since #51119 the Rust host converts
the whole metadata field to JSON before its own user_id filter runs, so the call
failed with "unsupported type UserAPIKeyAuth". The host now passes only
user_id, the one key the Rust validator and Python's AnthropicMetadata keep.
