mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
105 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kerry Lu
|
172e16acde
|
feat(proxy): router_settings.fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team model_preferences (#51439)
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call fallback walk reads router settings Key > Team > Global instead of the key's column alone, and skips a fallback the caller may not call. A key or team with router_settings.fallback_on_saturation: true walks that same fallback list as soon as the requested model reaches saturation_threshold in dynamic_rate_limiter_v3, and goes back to the requested model when no fallback admits the request. Responses carry x-litellm-fallback-reason: rate_limit:model_saturation_fallback. Removes the team_metadata.model_preferences steering and the x-litellm-model-steered-from header from #51350, which did the same job through a second knob. * fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call. * fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it cannot pass before saturation. * refactor(proxy): one helper decides which refused fallbacks end the pre-call walk _pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller. * fix(proxy): keep the caller's request object through the pre-call fallback walk Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place. * fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it). * refactor(proxy): read fallback prices through the router registry entry point router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info. * fix(proxy): a fallback's group budget check skips the groups the request already holds The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read. * test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle. * fix(proxy): the saturation handoff only fires under the fallback walk _pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429. * style(proxy): keep the cast-ok reason within the line length * fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only load the end user for fallback checks when a database is connected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request after admission, and _release_pre_call_hold calls it next to the registered limiter. An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the request on to a further alias. It now replays with the model argument the first attempt saw. The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
d30b2b9b26
|
feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489)
* feat(proxy): enforce customer org, team and member budgets and limits on the request path Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget, allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy allowlists in alternate-model steering, and settle spend to the member, team and org rows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): customer hierarchy survives virtual key auth without a legacy end user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(auth): keys pinned to a customer organization may only act for that organization's members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): keep the customer hierarchy spend maps within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): check the key's customer org scope after the key is resolved Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): declare the hierarchy cache key family and keep type counts flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): resolve the customer hierarchy in the shared authorization gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drop a member's cached hierarchy through the declared cache target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(auth): keep the hierarchy reset within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): type the requested models as a sequence Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(spend): add daily customer usage and budget resets (#51493) * feat(spend): add daily customer usage and budget resets * chore(spend): document customer reset query bounds * fix(spend): chart customer queue metrics and update recovery test * fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "feat(spend): add daily customer usage and budget resets (#51493)" This reverts commit |
||
|
Kerry Lu
|
eb811913f1
|
fix(dynamic_rate_limiter_v3): read a model's saturation from Redis so every worker agrees on it (#51519)
* fix(dynamic_rate_limiter_v3): read a model's saturation from Redis so every worker agrees on it A worker's in-memory copy of the saturation counters carried only the usage it served itself, and steady traffic kept that copy alive past saturation_check_cache_ttl, so two workers behind one Redis disagreed on whether a model was saturated. The saturation read now goes to Redis and is written back to memory for the TTL; without Redis memory is all there is. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): read the saturation count from Redis itself, not DualCache's memory-first get Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): back the Redis saturation count into memory only when memory holds nothing, and split the counter-pair comprehensions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): fall back to the worker's own saturation count when the Redis read fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
24f753c589
|
test: drop the unit tests that flaked on main this week (#51488)
* test(ui): drop the flaky current-download count refresh case in the Skills Hub panel * test: drop four unit tests that flaked on main this week --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
0324b9d915
|
fix(dynamic_rate_limiter_v3): a capped borrower keeps its cap on its own counter and gets a fresh share at saturation (#51494)
* fix(dynamic_rate_limiter_v3): give a capped borrower its own cap counter so it also gets a fresh share at saturation A team with team_model_limit_max_borrow_multiple enforced its cap on the same counter as its share, so its idle-time usage kept counting and it got no fresh share once the model saturated. The cap now runs on its own counter (model_per_team_max_borrow_multiple) that counts all of the team's usage this window, while the share counter only counts usage once the model is saturated. Also repairs the six share-metric unit tests that main broke when #51415 and #51417 landed together, by saturating the model from a separate priority class before the measured calls. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): size a capped team's token reservation by its share, idle or saturated The reservation for a prompt without max_tokens was bounded by whichever descriptor was being held, the cap while idle and the share once saturated, so the same request was refused at one moment and admitted the next with the same counters. Reserve against the share, the tighter of the two, in both states. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): track the cap's reservation apart from the share's The cap and the share shared one reserved-token figure and the fallback carry keyed on the share's window, so an idle admission's hold on the cap was refunded from a share that never carried it once a WebSocket turn saturated, stacked idle fallback holds could carry across cap windows, and an idle fallback served for a capped team published no share gauges. Each counter now carries its own reservation and window, and a fallback hold keeps the target's share so the gauges read the cap counter against it. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keep a WebSocket's share hold until it settles and read capped idle borrowing against the team share A frame checked after the model went idle again replaced the connection's share descriptor with None while the admission's hold on that share was still outstanding, so settlement never gave the hold back. The share is now kept while its reservation is open. With the class share split on, the share gauges picked the wider class share over a capped idle team's own share, understating its borrowing, so the tighter of the two is published. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ishaan-j-berri
|
df8b3b498e
|
feat(proxy): add customer hierarchy usage analytics (#51495) | ||
|
Kerry Lu
|
de3f0690e7
|
fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates (#51415)
* fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates The per-team share counters only count requests admitted while the model is saturated, so what a team used while the model was idle is not charged against its share once contention starts. A team below its share is still admitted ahead of a borrower, and the model's own limits bound the total. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): idle fallback targets and WebSocket turns follow the same share accounting A router fallback target attempted while idle no longer holds the team's share, and a Responses WebSocket connection refreshes its share descriptor on every turn, so idle turns are forgiven and saturated turns count. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): an idle fallback target still records its hold Settlement finds the hold, gives the admitting model's reservation back and charges nothing for the idle target, instead of charging the target's tokens to the admitting model's share. The integration fixture builds its configuration without mutating the loaded one. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
7130a5b4fd
|
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits the request, a router fallback's settlement publishes the fallback target's own share under its model group, the gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests restore the process registry, and the integration test polls for the settled totals. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prometheus): type the gauge setter and the label factory so the type gate holds _set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the label test tolerates the now visible None values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(prometheus): keep the label test diff to the two None-tolerant assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
98dbe80ada
|
fix(proxy): refuse blocked and over-limit keys before reading the request body (#51428)
* fix(proxy): refuse blocked and over-limit keys before reading the request body A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path. * fix(proxy): pick the effective key header for the early refusal and keep its failure log The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook. * fix(proxy): honor litellm_key_header_name in the early refusal The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies. * fix(proxy): keep the early-rejection path inside the type gate ceilings Read the configured key header through general_settings_view() instead of importing the untyped general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as HTTPException. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
2c7a36abe9
|
fix(rate-limit): hold router fallbacks, nested calls, WebSocket turns and oversized prompts to a borrowing team's model share (#51367) | ||
|
Kerry Lu
|
c0114e55aa
|
feat(dynamic_rate_limiter_v3): priority_class_share_splits_equally_among_active_teams splits a priority class share among the teams active on a model (#51366)
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false). When on, the class weights are renormalized over the classes with a team active on the model in the current window, each class share is divided equally among its active teams, and once the model saturates each team is held to its own share with a 429 whose rate_limited_by header reads priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL. * fix(proxy): keep basedpyright counts flat for the priority class share setting PriorityReservationSettings is built with model_validate instead of **value, so a new field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers type the handler by its implementation class. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way A registry failure falls back to the class pool instead of skipping the model limits, a team's counter is keyed by model and team so keys of different priorities spend one share, Responses WebSocket turns are held to the class share like the team model limit, a share is floored at one request and one token, and the registry declares the rate limits key family. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
52d1c4cd80
|
feat(dynamic_rate_limiter_v3): steer flexible teams to a permitted alternate when their preferred model is saturated (#51350)
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated
Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.
* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates
* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names
* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails
* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests
* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver
An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.
* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take
The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.
* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates
While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.
* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget
While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Kerry Lu
|
ece1484bc7
|
feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity (#51359)
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could use everything up to the model's own tpm and rpm while the model was idle. Setting team_model_limit_max_borrow_multiple in the team's metadata caps the team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429 whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset keeps the previous behavior. Values below 1 or non-numeric are logged and ignored. * fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check * fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
19ffe3a5f2
|
feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams lets a team's model TPM and RPM limits borrow idle capacity in the dynamic rate limiter (#51328)
* feat(rate-limit): opt-in team_model_limit_is_floor turns a team's model_tpm_limit into a guaranteed share that borrows idle capacity With priority_reservation_settings.team_model_limit_is_floor: true, a team's inherited model_tpm_limit on a model the dynamic rate limiter covers stops being a hard cap in the regular limiter. The dynamic limiter tracks it as a team_model_floor descriptor, lets the team past it while the model is below saturation_threshold, and holds the team to it once the model saturates, with rate_limited_by: team_model_floor on the 429. Default false keeps today's behavior. Key-owned model_tpm_limit, model_rpm_limit, the model-wide tpm and customer limits stay hard. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rate-limit): name the setting team_model_limit_mode with hard and soft values soft replaces the team_model_limit_is_floor flag. The 429 header reads rate_limited_by: team_model_soft_limit and the message says the team soft limit was reached. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams covers team model TPM and RPM Replaces team_model_limit_mode. A team's inherited model_tpm_limit and model_rpm_limit on a model the dynamic rate limiter covers are enforced by that limiter once the model saturates, so the team borrows idle capacity below the threshold. The 429 names the limit and reads rate_limited_by: model_per_team_saturated. Folds in the bot findings: key ownership is checked per requested model, team limits resolve through the model alias like the regular limiter, the team counter is charged only for requests the pre-call hook tracked, every response type that carries usage is settled, and tracking below saturation no longer adds a token per request. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the type gate flat in the dynamic limiter's team limit paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): settle responses usage and null auth metadata, resolve aliases for the borrowing team limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): key the borrowing team counter by resolved model group and read usage through get_usage_object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): hold Responses WebSocket turns to the borrowing team limit once the model saturates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
db6dfb6bb6
|
chore: move workspace packages under litellm (#51000)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
072290e78f
|
test(proxy): await background spend counters before asserting (#51249)
* test(proxy): wait for child readiness before supervisor cancellation * test(proxy): sync supervisor cancellation test on a socketpair handshake Replace the readiness marker poll and timed sleeps with one-byte socketpair messages. The child reports its SIGTERM handler is installed, reports the stop, and holds its exit until the parent releases it, so the assertion only passes when the cancelled supervisor waits for the child. A pytest-timeout marker bounds a deadlock. * test(proxy): drive supervisor cancellation cleanup with a controlled child Replace the real child and socket handshake with a scripted process returned by asyncio.create_subprocess_exec and a stop observed through os.killpg. The child only exits after the stop and a checkpoint, so returncode is 0 only when the cancelled supervisor stops the child and waits for it without being cancelled. * test(proxy): start the supervisor cancellation only once the reporter waits The scripted child signals when a wait begins, and the test cancels only after the exit reporter is blocked on the stop, so cancellation reaches it without relying on task start order. * test(proxy): await background failure spend counters before asserting Failure spend counters finish in a background task when the counter store is slower than the wait budget, so the tests read stale values on a slow runner. Hold the counter store on a release event with a zero wait budget, capture the background task, and await it before asserting the counters. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
59538ff1d9
|
feat(messages): serve anthropic messages on rust only (#51119)
* feat(messages): apply modify_params rewrites on rust * feat(messages): mint anthropic workload identity tokens on rust * feat(messages): apply prompt caching injection on rust * feat(messages): serve the easy python-only detectors on rust * feat(messages): retry rejected thinking signatures on rust * feat(messages): serve response cache reads on rust * feat(messages): retry rejected strict tools on rust * refactor(messages): delete the python path for anthropic messages Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a detector that raises) gets UnsupportedParamsError naming the feature, raised from one place in litellm/messages/dispatch.py. Other providers keep the Python fallback. A call whose arguments the handler signature rejects still gets Python's TypeError. Deleted Python code that only served anthropic Messages: - workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses were never eligible; the Rust host mints the token itself) - the anthropic Foundry beta-row and affinity-base branch in async_anthropic_messages_handler, and the "forward" mode of the beta filter that only that branch used The proxy MCP tools detector now uses the light tool precheck so a function tool never imports the gateway handler, since a raising detector now refuses. * feat(messages): forward message threads on rust The Rust path for anthropic /v1/messages dropped the thread body field, so a create or continue turn reached Anthropic without its thread or the message-threads-2026-08-12 beta. It now matches the Python handler from #50976: - the host forwards thread with the other body params - a non-null thread adds the message-threads beta - a well-formed continue turn keeps advisor blocks in history and derives no compaction, per-turn-control or tool-change beta from its messages - the recorded httpx_response carries the request that was sent, for JSON and streamed calls, so the proxy's thread ownership hook resends the create turn's betas on a continue as it does for Python Upstream error.details already reached the exception message unchanged, so thread_not_found relays and keeps the deployment out of cooldown on both paths. Parity cases cover create, continue, streaming, sync, retries, thread_not_found and the ownership hook round trip. * feat(messages): serve mock responses on rust * feat(messages): send caller clients and unsupported tls through python on rust * feat(messages): run pre-request hooks on rust * feat(messages): run agentic loop hooks on rust * feat(messages): run the safeguards classifier on rust * feat(messages): run registered interceptors on rust * fix(messages): match python stream cache gate, logging and interceptor order on rust Store a streamed Messages response in the cache only when Python would: a message_stop event line, no provider error frame and at least one frame with output. Leave dict results untouched by mark_rust_response so hidden params never reach result dicts, logging objects or hook digests. Ask interceptors whether they handle the call after the cache lookup, not at decode time, so a cache hit never calls can_handle. Emit the track_llm_api_timing service event for the provider exchange. Bill and log a stream superseded by a router fallback as the failure of its error frame, with the partial usage warning. Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the autorouter baseline cache sees retried requests. * fix(messages): forward python-only json values to anthropic as python does Accept any max_tokens, sampling scalar and messages collection that Python main accepts, and carry values serde_json cannot hold (ints beyond 64 bits, NaN and infinities, strings with lone surrogates) as validated raw JSON leaves written back verbatim on the wire, so the upstream body matches Python's json.dumps * fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust * fix(messages): log python's plain provider header names on rust streams * fix(messages): match python's request params, logged params and validation on rust Read a non-bool stream by Python truthiness and send it as given, log the caller's stream flag as Python's handler does, raise the raw Python errors main raises for malformed messages, tools, model_info and hook-set litellm_params or providers, read the tool custom-field drop from drops_unsupported_params, log the caller's nested fields that additional_drop_params strips and the Python values of raw JSON leaves, keep a missing stop_reason absent, and map provider-config rejections (missing max_tokens, unknown reasoning effort) as Python's HTTP handler maps them, with Python's text, effort order, model and provider * fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust * fix(messages): match python on hooked streams, empty cache writes and mcp cache hits * fix(messages): join already buffered stream reads and trim the env api key as python does on rust * fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust A stream cut by a transport error now hands the caller the bytes already buffered before raising, as aiohttp does for a caller that reads them before the event loop sees the cut, so the frames and their usage reach the caller and the failure log. Provider replies keep integers wider than 64 bits exactly and read numbers that overflow a float as infinity, through the response, the response cache and the logged result. The provider client follows up to 20 redirects as httpx does, and running out raises Python's text. A provider_specific_header value the ascii codec cannot encode fails before the send like a caller extra header. The deployment success hook gets the stream as AnthropicMessagesStreamingResponse, as Python's handler hands it. * feat(messages): serve proxy mcp tools on rust * test(messages): cover hook, agentic and interceptor edge cases in the parity suite * fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust * fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust A header value Python's client cannot encode as ASCII now fails before the send whichever header carries it, so a caller api_key with a non-ASCII character raises Python's InternalServerError instead of a raw UnicodeEncodeError after the reply. A caller api_base without an http(s) scheme, or with whitespace, control or non-ASCII characters, is sent through Python's client at the URL Python builds, so its relative-URL and invalid-URL errors match, and a base urlsplit cannot split raises Python's bare ValueError before the send. A provider_specific_header of the wrong shape is scoped through Python's ProviderSpecificHeaderUtils first, so a plain string raises Python's AttributeError. * fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust temperature and top_p now deserialize through recognized::exact_float, which keeps an integer where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent upstream and logged as the integer python forwards, and comparisons still go through scalar.rs cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python TypeError marked as a late rejection, and map_failure hands it to the caller unmapped * test(messages): format the non-string tool_result parity helper * fix(messages): flag and log the call before python's pre-send raises on rust The bridge now hands settings() and map_failure() whether the call is async. A provider_specific_header of the wrong shape sets the aanthropic_messages flag before re-raising, as Python's handler has already flagged the call. A base urlsplit cannot split also logs the resolved model, provider and optional params first, as Python raises it only after its logger is updated. The TypeError raised while capping the thinking budget for a string max_tokens sets the flag too * feat(messages): serve the web search short-circuit override on rust through the intercept host call * fix(messages): give the web search entry kwargs its mutable-ok reason * test(messages): drop the body hook gap the rust short-circuit closes * fix(messages): name the partially unknown handler types on the short-circuit host call |
||
|
Kerry Lu
|
d88a73c870
|
feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429 (#51228)
* feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429 * test(rate-limit): expect rate_limited_by in the exact-headers assertion --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
917fd11966
|
fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082)
* fix(rate-limit): settle a fallback's tokens on the model group that served it The v3 limiter pinned post-call TPM settlement to the model reserved at admission, so when the router fell back to another model group the requested model's counter took the usage and the serving model's counter stayed empty. When the served model group differs from the admitted one, charge the served group and refund the abandoned group's per-model reservation. Same-model retries and alias reloads keep the admission target. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): wait for the primary refund and the fallback charge before reading counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): drop unrelated formatting from the v3 limiter unit test file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
9ebbcc7710
|
feat(anthropic): forward message threads on native /v1/messages (#50976)
* feat(anthropic): forward message threads on native /v1/messages Keep the thread body field on native Anthropic legs, add the message-threads-2026-08-12 beta for it, reject it with a 400 on legs that cannot continue a stored thread, and relay error.details of an upstream Anthropic error so thread_not_found reaches SDK thread helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(anthropic): keep a stale thread 404 from cooling the deployment and type the thread beta ignores Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: sort the cooldown handler imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): dedupe the message-threads beta map rows and annotate test finals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): bind message threads to the key that created them and answer unsupported continues with thread_not_found Every virtual key on a proxy shares one Anthropic credential, so forwarding thread alone let any key continue another key's thread by its message id. A new default proxy hook binds each thread reply id to its virtual key, or to the signed-in identity when there is no key, and answers a continue naming an id it did not bind with the 404 thread_not_found the SDK replays on. Legs without message threads now serve a create without the thread and answer a continue with that 404 before any upstream call, instead of answering it with no history. * fix(proxy): keep a cached message thread reply with the key that got it first A response cache hit replays the original reply id, and binding it again handed the thread to the second key, so the creator's continue got thread_not_found. * fix(anthropic): keep message threads bound on every route and stable across turns Check the thread binding on every route that carries thread except pass-through, keep bindings in their own cache so they never evict rate-limit counters, add no tool or message derived betas on thread turns, answer a create on a leg without threads with a 400 thread_unsupported_request unless drop_params is set, and forward threads on Claude Platform on AWS * fix(anthropic): reject message threads on /v1/messages passthrough deployments A deployment that forwards /v1/messages untranslated to an OpenAI-compatible server inherited the native config's thread support, so `thread` reached a server that keeps no thread state. It now answers the same 400 thread_unsupported_request on create and 404 thread_not_found on continue as a translated leg * fix(anthropic): resend a thread's create betas on continues and check bound ids on pass-through A thread create now gets the betas LiteLLM derives from its tools and messages again, so an advisor tool create that answered 400 on the previous commit answers 200. Anthropic fingerprints the beta set, so the proxy keeps the anthropic-beta header the create sent upstream in the thread binding and adds it to every continue on /v1/messages. It reads that header from the upstream request, since success logging overwrites the logged request details before the success hook runs A continue on the /anthropic pass-through route of a thread the proxy bound to another key now gets the 404 thread_not_found. Ids the proxy never bound still pass through, as before * fix(anthropic): continue unbound threads off /v1/messages and keep every anthropic-beta on chat and Responses * fix(anthropic): skip non-string anthropic-beta list entries instead of raising * test(anthropic): keep the header merge regression test clear of main's appended tests * fix(anthropic): add a thread's create betas to the client's own on continues and keep base header merging on chat and Responses * fix(anthropic): answer an unbound thread id on chat and Responses with the 404 and drop the stale beta-join tests * fix(proxy): check a thread continue sent inside extra_body against its owner Azure AI promotes extra_body into the request body, so another key could continue a thread by putting thread there, past the ownership hook that read only the top-level field * fix(proxy): write a thread's first owner atomically and bind creates sent inside extra_body Two workers that both miss a reply's binding now agree on the first key stored in Redis, since the claim uses SET NX and each worker caches the stored winner. A create sent inside extra_body now binds its reply like a top-level create. The redundant Claude Platform supports_message_threads override is removed, since the provider already resolves to the native Anthropic beta list * fix(proxy): include agent_id in the key-less message thread owner * fix(proxy): tag message thread ownership Redis calls with their key family --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tavren27
|
35e501974e
|
fix(spend): write a background response's settling read in-process when the spend collector is on (#50957)
* fix(passthrough): bill a background response once however many times it is read * fix(passthrough): keep the background response flag on spend events sent to the collector * test(spend): type the background response test helper and its settlement parameter * fix(proxy): keep a background response's settling read in-process instead of adding a spend event field An older collector refuses a spend event with a field it does not know and drops the whole charge, so a newer gateway sending to an older collector lost the background response's cost. The read that settles a background response now stays in-process, the same way video status polls do, and the spend event keeps its shape * fix(proxy): keep a batch retrieve in-process when message redaction hides the logged batch * refactor(proxy): split the is_offloadable_success docstring into one line per reason --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yujonglee
|
11af2eb22d
|
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
joshua-berri
|
076ded32be
|
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session Claude Code sends its background calls straight to Haiku under the same session id, so those requests never reached the auto-router session rollup and the status line under-reported the session. Every billed spend row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total without a shared Redis * fix(auto_router): name the unsupported provider when a turn routes off Anthropic A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model, failed strict native validation and was logged as observation_unavailable, which reads like a broken capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their existing reasons, and no estimate is added, so savings for these turns stay unknown * docs(auto_router): explain when session savings stay unestimated saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected for that window * fix(auto_router): say why a system-only cache prefix has no savings estimate Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the empty body still count it. A live check against claude-opus-5-5 showed a differential count with a placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated and the savings gap for Claude Code is not fixed here * fix(auto_router): declare the session total's Redis key family and test its expiry The session total read and wrote Redis without a service target, so its calls showed up as bare redis spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an active session keeps its total and an idle one restarts * docs(auto_router): note which turns are still estimated inside the history window Turns that took the identical baseline path, and turns whose whole cached prefix the session had already written, are estimated even inside the first cache TTL window, so saved_spend now says so --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
4714aac42b
|
feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter (#50857)
* feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(rate-limit): build model scope settlement targets from one owner tuple Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
523b3efea2
|
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
13df479397
|
fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints (#41831)
* fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): simplify pass-through failure api_base lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): exempt pass-through spend log regression from quality gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse failure payload builder in pass-through exception path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover config-defined pass-through failure spend rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover failure spend route eligibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check configured routes for pass-through failure logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: kenneth <kenneth@berri.ai> |
||
|
zoveki
|
abe3b11f7a
|
fix(proxy): show committed spend in the remaining budget gauges (#49199)
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
43112547fb
|
feat(proxy): total_timeout bounds a whole non-streaming request (#50496)
* feat(proxy): total_timeout bounds a whole non-streaming request A `total_timeout` body field or `x-litellm-total-timeout` header (the header wins) gives a non-streaming /v1/chat/completions, /v1/messages or /v1/responses request one budget for preparation, retries, moderation and post-call processing. Past it the proxy answers 408 and cancels the upstream call, and a retry whose backoff cannot fit ends early with the error it would have retried. A body value on a streaming or background request is refused with a 400, a header value leaves it unbounded. A timed out request is billed like a client disconnect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): ignore any total_timeout header on requests it cannot bound, and never interrupt pre-call admission A globally configured x-litellm-total-timeout that is malformed no longer fails the streaming and background requests it is ignored on. The deadline holds while pre-call admission runs, since admission reserves shared quota before it records it, and an expired deadline then ends the request before dispatch, where the failure path hands the reservation back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): a total_timeout cuts slow pre-call hooks short, holding only the rate limit reservation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
yujonglee
|
574fcea691
|
fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473)
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores RedisCache and the native Redis client each kept their own breaker, so once the cache learned the coordination Redis was wedged, a spend-counter write through the native store still paid a full socket timeout. Every Redis target now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected into both the RedisCache guard and the native client. The Python FSM is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(rust): cargo fmt the shared breaker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(redis): close the breaker when the recovery probe gets a Redis reply error A command error such as WRONGTYPE means Redis answered, but the probe was left unsettled and its drop reopened the breaker for another recovery timeout. Both the RedisCache guard and the native client now settle such a probe as a success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw. * fix(redis): key the shared breaker by server and close the three audit gaps The breaker registry now keys on the Redis server (host and port, unix socket, cluster node set or sentinel service) instead of the full connection target, so caches that differ only in db, credentials or url spelling share one breaker. A READONLY reply from a replica mid-failover no longer counts as Redis answering, so it can't reset a run of failures. A new RedisCache refreshes the breaker state gauge, which the deleted Python breaker did at construction. A shared native client is only reused by a store that passes the same breaker. * refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations lib.rs now only declares modules and re-exports, with resolution, target, tls, auth and endpoint each in their own file. Derived FromPyObject already reads a struct field as the attribute of the same name, so #[pyo3(attribute)] on the breaker and store settings did nothing. * refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot Derived FromPyObject already reads each field as the attribute of the same name. The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in the same order. * refactor(redis): return a HashMap from the three memory-tier snapshots The snapshots read cache-memory's HashMap, so there is no order to keep. PyO3 builds the same dict from a std HashMap, which replaces the hand-written PyDict loops in the spend counter and router usage stores and the IndexMap in the cooldown store. The rate-limit slot registry keeps IndexMap because it hands back a Python dict in the order Python wrote it. * refactor(redis): share Redis breakers and clients through one RedisHosts per host redis-connect's RedisHosts now owns the per-server breaker and the shared clients that python-bridge kept in two process-wide statics. A client takes breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. This drops RedisClient::guarded_by and Breaker::same. python-bridge keeps one RedisHosts per process and calls it with the GIL released. The gateway builds one per connect, so its coordination and response cache on the same server now share a breaker. RedisStoreSettings.breaker is back to breaker settings, as on main. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Krrish Dholakia
|
b3c4a1c3e4
|
feat(spend-logs): turn off prompt storage per key or team (#50135)
* Add per-key and team prompt storage restrictions for spend logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Apply spend-log prompt restrictions to passthrough requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Apply team prompt storage restrictions to queued requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings The spend-event collector decided whether to ship request and response bodies from the global setting alone, so an opted-out key or team still sent prompts through the collector transport. It now uses the same key/team check as the spend log writer. A key or team setting of the string "false" was ignored and prompts kept being stored. It now counts as false, the same way the global setting parses strings. Splits the 42-case tenant storage test into one test for the policy and one for the pass-through logging contexts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows Failure rows, executed batch rows, semantic index embeddings, websearch interception calls and errors rejected before request setup built their logging metadata without the key and team metadata, so a restricted key or team still had prompts stored on those rows. They now stamp it through one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata. The dashboard toggle now shows a saved string "false" as off, matching the server, and its label and help text come from the translation file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mateo Wang
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
moe-berri
|
b02bb99b08
|
feat: add OpenRouter-style -latest model aliases for first-party providers (#50315)
* Add first-party floating model aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Align Moonshot alias parameter test with its target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias proxy budgets and rate limits with targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove target-only team budgets count and block floating aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias quotas with project and organization targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Coalesce colliding floating alias rate limit counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix latest alias Rust fallback and failure accounting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: adapt latest alias checks to typed rate-limit store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
b1b236826a
|
feat(redis): serve the proxy's common Redis call with Rust stores (#50170)
* feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kerry Lu
|
3b20b48473
|
feat(interactions): forward continuation_token and type requires_continuation so callers can resume a long decode (#50151)
* feat(interactions): continue requires_continuation long decodes with continuation_token hops A foreground Interactions request stops at Google's per-hop output cap with status requires_continuation and a continuation_token. litellm now re-posts the request with previous_interaction_id and continuation_token until the status moves on, merging the hops into one interaction (last id, every hop's steps, summed usage with initial_input_tokens from hop 1), for non-streaming and streaming calls. continuation_token is accepted as a request param, and requires_continuation is a known status for billing. * refactor(gemini interactions): validate streaming chunks with model_validate * fix(interactions): keep the held completion across stream hops, stop on a repeated continuation_token, follow the spec redirect * fix(interactions): add grounding_tool_count per tool type, stop on any seen continuation_token, keep streaming past a [DONE] frame when a hop is pending * ci(interactions): point the spec canary at aistudio.google.com, where Google moved the Interactions OpenAPI spec * test(interactions): scripted Google ends each long decode hop with a [DONE] frame * refactor(interactions): pass requires_continuation and continuation_token through instead of looping hops in the proxy The caller drives the continuation, the same as against Google directly. /interactions accepts continuation_token in place of input so a resume request needs no input. * test(proxy): assert the accepted /interactions bodies pass the required-param check * fix(proxy): keep the router default fallback for input on /interactions, continuation_token stands in for it --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
067f93340f
|
refactor(router): delete the python router core, the rust core is the only core (#50178) | ||
|
joshua-berri
|
a12a6cd154
|
feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872)
* fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
joshua-berri
|
906fa7ed28
|
feat(accounting): add opt-in SpendLogs ledger (#49856)
* feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
tessellark
|
8ab0321816
|
fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed (#49824)
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed
A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.
Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.
* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too
A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.
The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number
* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled
Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.
* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction
* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id
* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace
* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row
* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests
* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve
* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction
* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches
* fix(proxy): claim a job and charge the session budget in one Redis call
The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
joshua-berri
|
02b8da2f16
|
refactor: replace accounting event scaffolding with SpendLogs projection (#49843)
* refactor(accounting): replace event scaffolding with SpendLogs projection * test: preserve spend timestamps without removed capture callback * fix: guard accounting event removal across schema update paths * refactor(accounting): retain schema and prepare durable payloads * fix(accounting): preserve dates in serialized payloads * fix(accounting): retain schema push compatibility * fix(accounting): retain span registration for the legacy schema * test(accounting): remove obsolete capture acknowledgement test * fix(accounting): date every daily projection row the way the model rollup does A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(ui): regenerate dashboard API types after rebasing on main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): project only from the stored accounting payload project() now reads entities only from the payload's accounting_entities, so the raw SpendLogs fallback branches are gone. build_accounting_payload() builds the metadata once instead of dumping it to JSON and parsing it back. Guardrail usage dates use a tolerant UTC parse, so a malformed startTime drops only the guardrail rows instead of failing the whole projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
norkelbyte
|
3b35ae340d
|
fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831)
* fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks * fix(rate-limit): count embedding media as media only on models that embed it A data URI sent to a text-only embedding model is read as text, so it keeps the text estimate there. On models flagged supports_embedding_image_input, image and audio use the chat media estimates and PDF and video count at most the model's max_input_tokens. The same estimate now feeds project and key limits, the usage-based, latency and cost routers' deployment tpm, and the deployment itpm check. * fix(proxy): count embedding media only where the backend reads it as media Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as media per item. Cohere counts it as media only when the whole batch is media, and Voyage, OCI and unknown models keep the text estimate. File blocks carrying a data URI count like the bare URI. The router context window pre-call check and the batch limiter now use the same estimate, resolving each deployment or JSONL row model to its backends. The cost map gains supports_embedding_image_input on voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0. * fix(router): hold each embedding media item to the context window on its own A PDF or video data URI alone already reserves at most the model's max_input_tokens, but the context window pre-call check then ran the blanked input through the Responses transform and added a few framing tokens, so a PDF that Vertex accepts got a 400. When every input item is media, the check now returns the largest item's media estimate, since an embedding request embeds each item separately * fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped. * fix(bedrock): bill Cohere embeddings by Bedrock's input token count header * fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage The strict base64 check now scans the payload in place instead of decoding and re-encoding a full copy. Bedrock's billed token header is used only for image requests, so text-only Bedrock Cohere usage matches main. The deployment itpm estimator change is reverted, since the router never passes embedding input to that check. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
864b98acec
|
fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170)
* fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
6e27da8af9
|
fix(router): inspect image content without translating routed requests (#49371)
* fix(router): inspect image content without translating routed requests * fix(router): preserve screenshot image routing * test: make monthly budget reset coverage deterministic --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
joshua-berri
|
be4f3d36ce
|
feat(accounting): capture request usage through the durable meter (#49133)
* feat(accounting): capture request usage through the durable meter * fix(accounting): preserve native Google usage compatibility * fix(accounting): preserve prepared usage observation on failure * fix(accounting): exclude sync acknowledgement policy rejections from capture * test(accounting): verify capture races and failure isolation * fix(accounting): preserve request support with sync acknowledgements * fix(accounting): normalize usage metadata before queue admission * fix(accounting): preserve prior attempt links on successful fallback * fix(accounting): preserve attempt evidence through retries and failures * fix(accounting): depend on prior receipts without resubmitting usage * fix(accounting): preserve pricing inputs and resolved capture attribution * fix(accounting): persist batch usage within capture bounds * test(accounting): cover batch dependency failure and backpressure * fix(accounting): isolate batch persistence and preserve trusted attribution * fix(accounting): share byte admission across batch producers and writes * fix(accounting): reserve batch snapshot reads before admission * fix(accounting): compact repeated batch observations without losing requests --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
quietjules
|
cee72ffdd6
|
fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257)
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits A native Responses WebSocket turn logs its raw events instead of a usage-carrying response object, so the v3 rate limiter read no usage and never charged the key or tag token window. Read the turn's usage from its events the way the cost calculator does, and charge the TPM-limited tags the socket was opened under. * fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage A later response.create turn on an open Responses WebSocket now gets a read-only check of every TPM window it counts toward (key, team, user, org, end user, team member, model-scoped limits and the socket's tags). A turn with no tokens left gets the 429 rate_limit_exceeded frame and the socket stays open. The check never reserves or increments a counter and leaves RPM, budget, PTU and project windows to their own paths. It runs on native and managed sockets. Managed turns carry the socket's call id in their metadata, so they charge the socket's TPM-limited tags. A failed native turn charges the partial usage its failure row already carries. A socket refused at open by a rate limit gets the same refusal frame and closes 1008, logged once without a traceback. /v1/realtime sessions now count toward TPM through the realtime usage combiner, and the v1 limiter reads socket usage through the same helper * fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with. * fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero * fix(proxy): stop charging a realtime session's whole usage into its close minute Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case. * test(proxy): check each per-turn TPM window on its own * fix(proxy): send a fail-closed per-turn socket refusal with its 503 status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
87d91d78b7
|
refactor(router): route the router family through RoutingContext and close the outside surface (#49118)
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points |
||
|
ryan-crabbe-berri
|
5029d027a1
|
test(budgets): age the monthly window past any calendar month so the reset test passes on the 31st (#49372)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
f4d1f45d87
|
fix(proxy): raise live budget counters for failure spend rows (#47741)
* fix(proxy): raise live budget counters for failure spend rows A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook had already released the reservation and never touched the live counters, so an over-budget key kept passing until the counter was next repaired from the database Every charged failure row with a cost now raises the same counters the success path raises. The error waits at most a second for them, so a hung Redis cannot hold it back Fixes #35568 * test(proxy): cover failure spend counters across budgets, endpoints and outages Integration cells for team, user, end-user, tag and window budgets, concurrent streams, billed retries, native /v1/messages, Bedrock guardrail cost, a Redis stall on two workers and disable_spend_updates * test(proxy): check stalled failure counters and use the real spend writer - The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend - Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps - Partial-stream test models are frozen * test(proxy): move the stalled counter bound reason into the assert message --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Filip Sinitsin
|
e81dd348a4
|
fix(proxy)!: log and bill Responses-style bodies sent to /cursor/chat/completions (#30277)
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126) The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests. * test(proxy): pin provider metadata and key billing on /cursor/chat/completions A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land * fix(proxy): pick the Cursor route's metadata field from the body shape A Responses-style body on /cursor/chat/completions keeps proxy metadata in litellm_metadata, the same as /v1/responses, while a Chat Completions body keeps metadata, the same as /v1/chat/completions. Adding the whole route to the internal-metadata list let a client's chat-body tags replace the key's tags in spend logs and skip tag budgets. * fix(proxy): keep get_metadata_variable_name callable with only the request * fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field * fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies * fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string * fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
23ad8870b4
|
fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill (#47554)
* test(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding * fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing Detect raster images from the page content stream, including inline images on pages with empty resources, and walk each form once. Strip only openai/, azure/ and Azure data-zone prefixes, bound the model family scan, read Azure-only chat models from the cost map, and keep cl100k_base for listed non-chat OpenAI models. * fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources Prices a PDF by the model name's last path segment, the same name the proxy counts with, keeps listed non-chat models on Claude pricing, walks each form once per inherited resources dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing only when a message carries a PDF * fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes count like their base model * fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models * fix(token_counter): count reservation and project ITPM with the serving deployment's model Budget reservation and project ITPM counted with the requested group name, so a GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count with each deployment's model minus its provider prefix, the name count_tokens uses, and take the largest. A custom OpenAI tokenizer with no model counts with cl100k again instead of raising, and listed Azure gpt- chat entries price as OpenAI. * perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator pypdf parsed every operator of a page's content stream to find the images it draws, so a compressed page of a million operators took about 30 seconds to price for an OpenAI model. A linear scan of the decoded stream for name Do and inline images finds the same draws. * fix(token-counter): count each team deployment by its PDF pricing and honor base_model Budget reservation and the TPM limiter now look up the deployments that serve the caller's team, by base_model when set. A deployment that reads PDF pages and prices them unlike the group name counts with its own model, any other with the group name. A comment between an XObject name and Do now reads as whitespace, and the timing tests moved to CodSpeed benchmarks. * fix(benchmarks): install pypdf in the benchmark environment The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path. * fix(proxy): count PDF pages for deployments served under multi-segment model names * fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge * fix(proxy): count routing-group and alias members with the caller's team, as routing serves them |
||
|
devin-ai-integration[bot]
|
a0a8ca97d9
|
fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993)
* fix(cost): finalize synchronous response cost before accounting callbacks * fix(cost): keep serving and billing when cost finalization fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): read the finalized cost only off real logging objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): follow the renamed success dispatch and deferred sync callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): keep cache hits free and their headers finalized when logging is deferred Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): preserve completed synchronous charges after hook failures * fix(cost): retain completed usage across cancellation and log suppression * fix(cost): reconcile completed failures and retain cached usage metadata * fix(cost): retain finalized request tags in failure budget updates * fix(cost): honor disabled spend tracking for completed failures * fix(cost): preserve pass-through accounting across cancellation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |