138 commits

Author	SHA1	Message	Date
mrinal-berri	b0c580d391	feat: email key owners before opted-in virtual keys expire (#51482) ... * feat: email owners before opted-in virtual keys expire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: batch expiry alert queries and use branded email template Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: cast expiry alert query bounds as UTC timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: satisfy expiry alert CI contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat: single-day key expiry alerts with global default and daily check - metadata.key_expiry_alert_days is one integer (1-365) instead of a list - general_settings.key_expiry_alert_days sets a default for every key with an expiry and owner email; key metadata overrides it - LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS defaults to 24 hours * fix: isolate key expiry alert setup and check every 6 hours - build the expiry alert manager in its own startup step that logs and returns None on bad config or setup errors, so later background jobs still register - default LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS to 6 hours * refactor: import litellm_enterprise email helpers directly in key expiry alerts Replace the import_module + Protocol module validation with plain imports, matching the key rotation hook and check_batch_cost. litellm_enterprise is absent from the type-check venv, so the imported names carry the same reasoned pyright ignore used in response_api_endpoints. * fix: apply global key expiry default to keys stored with null metadata Keys created without metadata (for example via /user/new) store JSON null, which failed metadata parsing and skipped the global key_expiry_alert_days default. Read null metadata as empty. * fix: guard litellm_enterprise import in key expiry alert job check_unsafe_enterprise_import requires enterprise imports outside the enterprise package to sit in try/except ImportError. Skip the run when the package is absent, matching the key rotation email hook. * feat: run key expiry alert scan on one pod per cycle, every 12 hours Take the shared PodLockManager lock (litellm_key_expiry_alert_job) like key rotation and expired UI session cleanup, so a single pod scans LiteLLM_VerificationToken each cycle; the per-key Redis claim still guards each email. Default LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS to 12 hours. * fix: scan for key expiry alerts when the job lock cannot be read from Redis acquire_lock returns False both when another pod holds the lock and when Redis errors. Skip the cycle only when another pod's id is on the lock key; otherwise scan without the lock, keeping the per-key claim as the duplicate guard, so a Redis blip cannot drop warnings for a 12-hour cycle. Replace the startup wiring assertion with a behaviour test against the proxy's pod lock. * fix: hold the key expiry alert lock for the whole cycle Each pod's timer fires at its own boot time plus jitter, so releasing the lock after a millisecond scan let every pod run each cycle. Take it non-reentrant for nearly the full interval and never release it, like the weekly spend report, so the first pod to fire runs the cycle and the rest skip. * fix: key the key expiry alert lock by clock window Pods' interval timers fire at their own boot time plus jitter, so a fixed lock name with a TTL shorter than the interval let a late-booted pod run twice in one interval, and a TTL equal to the interval made a pod skip its own next run. Name the lock after the clock window (int(now // interval) * interval, as file_usage_caps buckets its counters) with a one-interval TTL: the first pod in each window runs and the rest skip. Release the window if the scan raises so another pod can retry, and run anyway when the Redis circuit breaker is open. * test: keep key expiry lock tests in-process with a fixed clock Simulate the Redis outage with a disconnected in-process fakeredis server (wired like the router_redis fixture) instead of a socket to a closed port, and give the startup-built job the test clock instead of the wall clock, per tests/unit/AGENTS.md. * fix: declare the key expiry alerts service family on process_alerts process_alerts reads the window lock holder from Redis directly, so test_every_function_that_reaches_redis_directly_declares_its_family requires it to carry its family, as _process_key already does. * refactor: align key expiry alerts with existing litellm surfaces - read general_settings.key_expiry_alert_days on every run via _current_general_settings(), like the performance alert and spend capture jobs, so a UI or API edit applies without a restart; an invalid value is logged and ignored - share the event name as VIRTUAL_KEY_EXPIRING_EMAIL_EVENT in litellm/types, like MODEL_DEPRECATION_EMAIL_EVENT - default the Virtual Key Expiring email event off, like the other scheduled emails - name the Redis family for the claim data (key_expiry_alert_claims) and read the window lock under POD_LOCK_TARGET, mirroring reset_budget_job._lease_is_held - give the per-key check its own validator, run with the logging check through raise_on_invalid_key_metadata * chore: drop two comments that restate the code --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 13:47:16 -08:00
norkelbyte	ac1cc5111c	fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions (#51753) ... * fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions * fix(proxy): check the key's max_budget on /queue/chat/completions * fix(proxy): run budget, tag, and request policy checks on queued chat calls * fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls A queued stream now hands its logging object to the stream generator, so a client that leaves mid-stream is billed for the reply it already got, and a stream left before the first model chunk releases its budget reservation. A team's require_trace_id and the failure logging for rejected calls now use the same route check as the other inference checks. * fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses * fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls * fix(proxy): send the call ID on a queued call the provider rejects before streaming * fix(proxy): release a dropped stream's budget hold only when nothing bills it A guardrail that buffers the stream can pull model chunks the client never saw. The early-drop release settled the hold at the input cost first, then the partial bill for those chunks added its full cost on top. The release now runs after partial billing and only when no bill fired * fix(proxy): fold the caller's litellm_metadata into a queued call's metadata A queued call that sent a litellm_metadata bucket, even an empty one, made tag routing and logging read that bucket, so the key and team tags the handler merged into metadata no longer narrowed deployments. The queue handler now folds the bucket into metadata and drops it, as chat does, after stripping the proxy-owned user_api_key_* and internal control fields from both buckets, so a caller can no longer set the queued spend row's organization ID either * fix(proxy): release a queued call's budget hold when the call is cancelled before it answers * fix(proxy): refund a cancelled queued call's hold in full until the model call claims it * fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 08:54:54 -05:00
quietjules	007d1c1b5e	feat(proxy): support budget_limits windows on internal users (#51763) ... * feat(proxy): support internal user budget windows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): make user budget migration repeatable and sync API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bound lock wait for user budget migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): omit unset JSON windows when creating users Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): retain personal budget windows for proxy admins Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): give user budget windows their own counters and keep them on the user User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path * fix(proxy): keep windows on team auto keys and write all users' windows in one transaction * fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets. JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do * fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits * test(proxy): check default window resets against the reset helper instead of the wall clock * fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes * fix(proxy): fail an all-users bulk update whole when its transaction fails The per-user fallback after a failed all-users transaction wrote the newest user and then crashed on a missing user_email attribute, so the call answered 500 with one user changed. Racing window bulk updates past the 5s transaction limit reached it. Answer 500 with no user changed instead --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 07:44:35 -05:00
Ankit Jha	5f381ad472	fix(proxy): keep carried end-user spend when a budget reset is retried (#51143) ... * fix(proxy): keep carried end-user spend when a budget reset is retried End-user pages commit before the budget window advance. When the advance failed, the tier stayed due and the next tick replayed every page, so a rollover-capped row already cut to its carryover (15 -> 5) matched the under-cap zero and lost it, and spend accrued since the first pass was zeroed again. Each end-user reset now stamps the tier window it closes in a new spend_reset_window column and skips rows already stamped with it. Fixes #50898 * fix(proxy): leave never-reset tiers unguarded and test each tier's own window A tier with no budget_reset_at falls back to the old unguarded reset instead of stamping created_at, which an admin clearing the reset time would reuse and silently skip end users. Tests now give tiers distinct windows and cover a second window closing. * test(proxy): give the end-user ledger its own fake prisma client * fix(proxy): pin a reset time on tiers without one before end-user pages commit * fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced * fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset * fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration * fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window * fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time A failed /budget/update used to leave the new reset time written with the stamps still set, so repeating the request found the time unchanged and skipped the clear. The update and the clear now commit together, and a reset time in the future no longer scans the tier's end users * fix(proxy): name each end-user reset window by the tier's last write Stamp end-user resets with the tier's updated_at read on the writer instead of its reset time. A failed window advance leaves updated_at as it was, so the retry still skips the end users already reset, while the advance or any /budget/update moves it and opens a new window. That covers a forced reset on the default end-user tier and drops the stamp clear, so /budget/update is the one-row update it is on main. The writer read also leaves out a tier another sweep already advanced, which a lagging read replica still showed as due * fix(proxy): name end-user reset windows by the tier and its advance count Tiers advanced in one batch share updated_at, so a customer moved between them could skip the new tier's reset, and any unrelated tier edit during a failed advance reopened the window. A budget_reset_count that only the window advance bumps, prefixed with the tier id, names the window instead. * fix(proxy): keep carried spend for customer teams and members on a retried reset Customer team and member pages commit before the window advance like end-user pages, so they now stamp the window they close and a retry skips them too. A member runs on its unexpired override when that has its own duration, else on its template. The window name now joins the tier id, its creation time, its count of closed windows and its reset time, so a duration change or a tier recreated under the same id opens a new window even when the advance failed. * test(integration): list every budget column and relation in the budget update field check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 01:03:19 +00:00
Kerry Lu	db75214bdc	fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry (#51694) ... * fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry The reset job now evicts the customer_org_alias entries of the organizations it resets, and a hierarchy load that began before an eviction can no longer cache what it read. The cache hands a lease out on a miss and refuses the write when the key was evicted since, on this worker and, with enable_redis_auth_cache, on any worker through a generation key checked in Redis. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for a cascade with no customer organizations. * fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation write still let the DEL run and an in-flight load could re-cache what it read. * fix(proxy): delete the auth cache entry before the generation write inside the script A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it. * test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test * fix(proxy): refuse a leased auth cache write for a key no generation key can guard * test(proxy): undo the unrelated reformat of the spend ledger parity tests * test(redis-store): mark the updated memory-only auth cache test with rstest * chore(proxy): drop two pointer comments from the cache lease code --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 14:21:20 -08:00
ishaan-j-berri	df8b8d1b51	feat(customer-portal): complete hierarchy management (#51521) ... * feat(customer-portal): complete hierarchy management * fix(ci): update Soup Sieve across supported Python versions --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>	2026-11-13 17:58:34 +00:00
ishaan-j-berri	ca4c6c0a6d	feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582) ... * fix(proxy): reset customer org, team and member spend on their budget row's window The customer tables carried their own budget_reset_at next to a budget_id whose row already has one. Drop the column from the three tables and let the budget table cascade zero the customer rows it governs, with a member following its unexpired override when that override has a duration and its template otherwise. Customer budget writes stamp budget_reset_at from budget_duration the way /budget/new does, and the management responses report the governing budget's window under the same budget_reset_at field. * feat(budgets): support concurrent customer budget windows * fix(budgets): resolve compatibility and validation checks * feat(proxy): name budget rows, give customer teams an override and members a parent org * fix(budgets): preserve admitted windows during accounting * chore(schema): allow dropping the unreleased customer budget_reset_at columns * fix(budgets): apply standard migration lock timeout * feat(proxy): customer members belong to many teams, charged to the one the request names * test(migrations): allow the unreleased customer budget_reset_at drop * style(ui): prettier the customer dashboard files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(reset-job): name the batch bound on the customer budget IN filters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(budgets): preserve mixed limits and resolve CI regressions * fix(budgets): reuse shared spend verification for customer windows * refactor(ui): isolate customer budget group creation Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild. * fix(proxy): restore Field import and bound the member team lookup after the merge Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(proxy): send parent_team_ids in the budget window customer tests Co-authored-by: Tin <tin@berri.ai> Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): keep rejecting another organization's template after the budget_template_ids rename Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(proxy): cover customer budget increase requests and the cap_policy field Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * chore(ui): regenerate API types for cap_policy and customer budget increase requests Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): hang the increase request back-reference on the customer member model Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): declare increase request list filters with Annotated Query Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Tin <tin@berri.ai> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>	2026-11-13 16:05:35 +00:00
Kerry Lu	172e16acde	feat(proxy): router_settings.fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team model_preferences (#51439) ... * feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call fallback walk reads router settings Key > Team > Global instead of the key's column alone, and skips a fallback the caller may not call. A key or team with router_settings.fallback_on_saturation: true walks that same fallback list as soon as the requested model reaches saturation_threshold in dynamic_rate_limiter_v3, and goes back to the requested model when no fallback admits the request. Responses carry x-litellm-fallback-reason: rate_limit:model_saturation_fallback. Removes the team_metadata.model_preferences steering and the x-litellm-model-steered-from header from #51350, which did the same job through a second knob. * fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call. * fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it cannot pass before saturation. * refactor(proxy): one helper decides which refused fallbacks end the pre-call walk _pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller. * fix(proxy): keep the caller's request object through the pre-call fallback walk Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place. * fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it). * refactor(proxy): read fallback prices through the router registry entry point router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info. * fix(proxy): a fallback's group budget check skips the groups the request already holds The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read. * test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle. * fix(proxy): the saturation handoff only fires under the fallback walk _pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429. * style(proxy): keep the cast-ok reason within the line length * fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only load the end user for fallback checks when a database is connected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request after admission, and _release_pre_call_hold calls it next to the registered limiter. An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the request on to a further alias. It now replays with the model argument the first attempt saw. The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 05:57:18 +00:00
Kerry Lu	d30b2b9b26	feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489) ... * feat(proxy): enforce customer org, team and member budgets and limits on the request path Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget, allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy allowlists in alternate-model steering, and settle spend to the member, team and org rows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): customer hierarchy survives virtual key auth without a legacy end user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(auth): keys pinned to a customer organization may only act for that organization's members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): keep the customer hierarchy spend maps within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): check the key's customer org scope after the key is resolved Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): declare the hierarchy cache key family and keep type counts flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): resolve the customer hierarchy in the shared authorization gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drop a member's cached hierarchy through the declared cache target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(auth): keep the hierarchy reset within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): type the requested models as a sequence Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(spend): add daily customer usage and budget resets (#51493) * feat(spend): add daily customer usage and budget resets * chore(spend): document customer reset query bounds * fix(spend): chart customer queue metrics and update recovery test * fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "feat(spend): add daily customer usage and budget resets (#51493)" This reverts commit ccac47608f. * fix(proxy): evict cached member hierarchies when team, org or template budgets change Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): declare the auth cache dependency alias as a TypeAlias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Reapply "feat(spend): add daily customer usage and budget resets (#51493)" This reverts commit 7542471e17. * fix(spend): reconcile the daily customer spend table with main Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(pagerduty): tolerate failure metadata without customer hierarchy ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(accounting): key daily customer spend rows on main's hashed identity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(pagerduty): read customer ids from failure metadata through a typed lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: moe-berri <moe@berri.ai>	2026-11-12 20:43:44 -08:00
tin-berri	f4f94cc848	feat(budgets): support annual calendar reset periods (#51508) ... * feat(budgets): support annual calendar reset periods * fix(budgets): derive annual spend windows from calendar boundaries * fix(budgets): share timezone-aware calendar window boundaries * fix: preserve budget accounting across calendar boundaries * fix(budgets): isolate calendar changes to annual budgets	2026-11-12 19:40:40 -08:00
ishaan-j-berri	df8b3b498e	feat(proxy): add customer hierarchy usage analytics (#51495)	2026-11-13 00:32:13 +00:00
Yassin Kortam	db6dfb6bb6	chore: move workspace packages under litellm (#51000) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:57 +00:00
tavren27	36c7f10c84	fix(proxy): answer x-litellm-call-id on every token count response (#51037) ... * fix(proxy): answer x-litellm-call-id on every token count response The token count log line names a litellm_call_id, but count responses never carried it, so a client that sent no id of its own could not find its count's log line. Every count route (Anthropic Messages, Responses input_tokens, Gemini countTokens, Bedrock count-tokens, and /utils/token_counter) now answers the id on success and on errors, and a crash answers the JSON 500 the Anthropic and Gemini routes already used. * fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the gateway's generic 500, or 503 when the database is down, now with the x-litellm-call-id header. Responses count rejections carry error.litellm_call_id when include_call_id_in_error_body is on * test(proxy): count through count_request_tokens in the token counter loop tests The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise * fix(proxy): keep a count crash's own exception and log pre-count Responses failures The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line * fix(proxy): keep the token counter's injected args out of its API docs Tests the opt-in error body id on an Anthropic count the proxy rejects before counting * fix(proxy): keep a count route's own HTTP error when adding the call id The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-11 22:26:47 +00:00
mrinal-berri	5e74515c29	feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813) ... * feat(otel): let a team or key OTel v2 destination add its own request metadata keys A team or key callback can now set baggage_metadata_keys in its callback_vars, a comma-separated list of requester_metadata.<key> paths. The tenant fan-out adds them as litellm.metadata.<key> to that destination's spans only, on top of the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other destinations never see them. Values are bound when each span ends, an attribute the span already carries wins, and names of proxy identity fields are rejected when saved, along with lists over 20 keys. Resolves LIT-9545 * fix(otel): refuse request headers in a destination's baggage_metadata_keys The proxy copies the request headers into requester_metadata.headers with credential headers masked, but a header a gateway in front of it added (an IAP or OIDC assertion, an internal token) stays as sent. A team or key admin listing requester_metadata.headers.<name> could export it to an endpoint the team controls. Entries under requester_metadata.headers are now rejected when saved, so exporting a header stays an operator decision in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import. * fix(otel): hold destination metadata to the configured span limits A destination's extra litellm.metadata.* attributes were added to a span view built from a plain mapping, so they skipped the limits the SDK applies in set_attribute: a 1 MiB caller value survived an operator's OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length limit, and keys past the attribute-count limit are dropped and counted instead of evicting an attribute the span already carries. The per-destination views are built in one Final tuple ahead of the forwarding loop, since a Final cannot be assigned inside the loop. * fix(otel): apply destination metadata after the destination view, within the provider's limits The span limits were checked against the span as it ended, before the destination view strips prompt content and backend-only attributes, so a model-call span full of llm.input_messages.* could drop the team's keys on a no_content destination that had room. The values are now bound when the span ends and added as the last step before export, counted against what that destination actually sends, so their dropped count also survives the view. The limits come from the tracer provider the fan-out is attached to, read from the live span at start, instead of a separate env-built SpanLimits. * refactor(otel): consolidate span limits, destination validators and metadata key parsing Reads a span's limits in one guarded place that the emitter and the fan-out share, replaces the two parallel lists of per-destination callback-var validators with one mapping, splits and strips a baggage_metadata_keys list once, drops a filter the post-view merge already enforces, and folds the repeated fan-out test setup into one helper. No behavior change. * ci: retrigger checks stranded by the GitHub Actions outage	2026-11-11 13:57:41 -08:00
joshua-berri	deb7697a8e	fix(auth): enforce model access on Google and image URL routes (#51226) ... * fix(auth): authorize URL-selected Google and image models * fix(auth): preserve server model selection and budget fallbacks * test(auth): isolate URL model authorization settings * fix(auth): type handler model selection boundaries * fix(auth): validate request model selection inputs * fix(auth): tolerate absent request path metadata * fix(auth): support requests without path metadata --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-11-11 12:22:12 -08:00
Kerry Lu	fdafffb842	feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason (#51138) ... * feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-11 11:16:21 -08:00
yujonglee	11af2eb22d	refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) ... * refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 01:57:54 +00:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
ryan-crabbe-berri	87be092af7	fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer (#50736) ... * fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525, first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown Codex versions still get the catalog cut to 1 MiB, since they drop a larger one. * fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those clients counted as too old for the uncut catalog, their custom reasoning levels and stock entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does. * test(proxy): type the Codex catalog byte-limit test * refactor(proxy): gate the uncut Codex catalog on the 0.166 release line Read only the major and minor numbers of client_version instead of a shared semver comparison, and leave the reasoning level and stock entry version checks as they are on main. * refactor(proxy): read every Codex client_version with one helper Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version parser rejects. The catalog byte limit, the per-model minimum version and the custom reasoning level checks now share one reader for Codex's format, which also bounds digit counts so an absurd version can't fail the listing.	2026-11-10 07:10:30 +00:00
Mateo Wang	ac08eb5663	fix: make scheduled key, user, and team spend resets idempotent (#50298) ... * fix: make scheduled spend resets idempotent Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover reset roundoff with shared Redis workers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: isolate reset workers and type raw adapters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: keep updated_at moving on zero-spend budget resets * fix: never leave spend below zero after a budget reset * fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time * fix(proxy): keep a resent budget reset time only once it is due A save that resends the same budget_duration keeps the stored reset time only when that time has already passed, so the scheduled sweep still resets the period it is due for. A row that is not due yet realigns its reset time on save, the same as before. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 22:21:43 -08:00
ryan-crabbe-berri	e099925923	fix(proxy): list images in the Codex catalog only for models known to read them (#50738) ... * fix(proxy): list images in the Codex catalog only for models known to read them A translated Codex entry used to offer image input whenever no deployment declared supports_vision, so Codex let users attach images to models that may not read them. Image input now goes out only when every deployment behind the name says it reads images. * test(proxy): type the vision combination test parameters * refactor(proxy): rename the vision helper and drop the extra vision docstring sentence	2026-11-10 03:46:37 +00:00
ryan-crabbe-berri	16fad64e95	fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments (#50743) ... * fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog. * fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.	2026-11-09 19:00:22 -08:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
jesus-berri	c76a95fc7e	feat(proxy): let a key's logging entry reference a proxy credential (#50572) ... * feat(proxy): let a key's logging entry reference a proxy credential A virtual key's metadata.logging entry can name a proxy credential with litellm_credential_name. At request time the credential's values, which config credential_list resolves through the secret manager at boot, are merged under the entry's callback_vars, with the entry's own values winning. An entry whose credential is not defined is dropped, so it never falls back to the proxy's own callback environment. Only proxy admins may change the logging entries of a key when the stored or requested list holds a reference (403), and a reference to an undefined credential is a 400. The check runs on key generate, update, team bulk update, regenerate and /user/new, and compares decrypted entries so re-saving metadata read back from /key/info is not a change. Resolves LIT-10061 * fix(proxy): drop a key logging entry whose referenced credential holds no logging values * fix(proxy): only require newly added credential references to exist on key writes * fix(proxy): log a dropped credential reference as an error so /key/health reports it * fix(proxy): gate credential references on bulk-created user keys * fix(proxy): keep failing requests on keys with non-object logging metadata A key whose stored metadata.logging is not a list of objects failed every request with a 500 before this branch. The credential reference rewrite turned those values into no entries, so the request quietly succeeded without the key's logging. Raise a TypeError for those shapes again so the request path behaves as it did on main, while empty values and objects with unknown fields still go through the existing skip. * test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging The malformed logging cell expects the base behavior that 7ac4ab1356 restored, a 500 for a key whose stored metadata.logging is not a list of objects and a 200 for an empty list * fix(proxy): keep a 500 for teams with non-object logging metadata On main, a team whose stored metadata.logging held non-object items failed chat requests with a 500, because AddTeamCallback(**item) raised a TypeError. This branch switched to AddTeamCallback.model_validate, whose before-validator raises AttributeError on those items, and the proxy maps AttributeError to a 400 invalid_request_error. Raise a TypeError for non-mapping items before validation so the status matches main again, as the key path already does.	2026-11-09 16:02:46 -08:00
zoveki	abe3b11f7a	fix(proxy): show committed spend in the remaining budget gauges (#49199) ... * fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 20:05:37 +00:00
tavren27	2f430873ae	fix(bedrock): derive the web identity session policy per AWS caller (#50592) ... * fix(bedrock): derive the web identity session policy per AWS caller Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session. * fix(bedrock): let async invoke write its S3 output under the web identity session policy Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com). * fix(opensearch): give OpenSearch vector store search its own web identity session policy * fix(bedrock): give every remaining AWS caller its own web identity session policy Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes * test(batches): expect config export to skip every underscore attribute The session policy class attribute is private, so batches config export no longer returns single-underscore names * fix(bedrock): scope knowledge base ingestion role grants to the roles it creates Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included * fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies * fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through, or realtime The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions, document content, and ingested document ACLs, and a test derives that list from botocore Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and embeddings never open a bidirectional stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:13:53 +00:00
Mateo Wang	7ea4028625	fix(proxy): align ownership and scope handling (#50600)	2026-11-08 07:03:36 +00:00
Kerry Lu	51b49973c9	fix(proxy): hold DB_RECONCILE_LOCK around credential create, update and delete (#50551) ... * fix(proxy): hold MODEL_RECONCILE_LOCK around credential create, update and delete The config reload snapshots the credential table, awaits config rows, then prunes and upserts litellm.credential_list from that snapshot. A credential write landing inside that window was undone in memory until the next reload. The three endpoints now take the same lock the reload holds. Fixes #50273 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): take MODEL_RECONCILE_LOCK before the credential update reads its row Two PATCHes queued behind a reload read the same old row and the second overwrote the first. The relay now keeps enough bytes to spot a split after marker, and the new unit tests are fully typed. * test(credentials): annotate the reconcile lock test locals with Final * refactor(proxy): rename MODEL_RECONCILE_LOCK to DB_RECONCILE_LOCK The lock gates every db-to-memory reconcile and now the credential writes too, so the model-only name undersold what it protects. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 16:02:44 -08:00
yucheng-berri	b603e84c02	fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF (#50032) ... * fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF With OpenSSL enforcing FIPS, hashlib.scrypt raises, so every MCP session token and bridge envelope request failed with a 500. Under FIPS the keys now come from LITELLM_SALT_KEY through SP 800-108r1 KBKDF run by the provider's EVP_KDF. Without FIPS the scrypt keys are unchanged * fix(mcp): keep master key rotation invalidating MCP tokens under FIPS Pass the master key as the KBKDF context, so rotating it changes the three FIPS-mode keys as it changes the scrypt keys * test(mcp): annotate new FIPS key test locals with Final * fix(mcp): derive the FIPS MCP keys only when general_settings.fips_mode is on On a FIPS OpenSSL 1.1.1 host (RHEL 8) scrypt still works, so keying the switch on openssl_enforces_fips() sent those hosts to the OpenSSL 3 only KBKDF and broke MCP sessions and bridge envelopes that worked on main. fips_mode_active() is the same reader B1 adds, set by enforce_fips_mode at startup. * refactor(fips): move the salt key KBKDF into encrypt_decrypt_utils The KBKDF helper derives keys from LITELLM_SALT_KEY, so it now sits next to the other salt key helpers, and fips.py goes back to main's version. * Pin the MCP key KBKDF fetch to the FIPS provider Fetch KBKDF with the property query fips=yes, so a change to the process's default properties after boot can't move the derivation to a non-FIPS provider. The shared fips_openssl test fixture drops the pin, since the test OpenSSL has no FIPS provider.	2026-11-07 13:33:27 -08:00
ryan-crabbe-berri	ab3b12b951	feat(proxy): serve Codex its own release's model catalog, fetched from GitHub (#50318) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release * feat(proxy): serve Codex its own release's model catalog, fetched from GitHub When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the fetch fails, times out after 2 seconds, does not validate, the version has no tag, or LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight most recent versions are kept. The integration tests ask as the bundled release so their expectations hold whether or not the proxy can reach GitHub * fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions Failed versions now live in their own cache so they never push out a fetched catalog, requests for a version already being fetched share that fetch, each worker starts at most six fetches a minute and serves the bundled catalog past that, and the version check only accepts ASCII digits * fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the codebase	2026-11-06 21:58:54 +00:00
Mateo Wang	3f38730d2c	refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) ... * refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type	2026-11-06 13:31:15 -08:00
joshua-berri	de0b389205	feat(router): add the Switchyard stage router as an auto router strategy (#50407) ... * feat(router): add Switchyard Stage Router strategy * feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes ModelCall.respond and fail async. The classifier is now a tagged union of an LLM judge and a decision model, and decision calls go through the router's adecisions so any decisions provider works. Algorithm misbehavior and library errors route to the capable model instead of failing the request, and the session id plus client session headers are always passed so capable holds apply without a classifier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep the Switchyard router inside the type-check gate The gate's environment did not install the optional switchyard extra, so every libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other optional runtime packages, and imports go through switchyard_rust.libsy, which ships py.typed. Step handling moves into helpers so no Final is rebound in the loop, and the dashboard API types are regenerated for the new routing fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): route Switchyard on every endpoint and reject unsafe router saves The router now passes each request's call type to routing strategies, and Switchyard converts the request to chat messages with the existing converter for that call type before translating it. /v1/messages turns with tool or thinking blocks used to fall back to the capable model, and now route on their signals. The Anthropic converter can keep tool_result.is_error on an analysis copy, so Switchyard sees tool failures there, while requests sent to models never carry it Saves of a strategy router that lists itself now fail with a 400 instead of a 500 with a dead row, the capable and efficient models must differ, judge and decision calls are not retried before failing open, the Switchyard routing fields are classified for redaction, and the edit dialog returns the saved router instead of the stale one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): catch self-routing on patch and rename, and trim Switchyard tests The self-routing check now reads the stored params through the decrypting helper and runs when a patch carries no litellm_params, so a PATCH that points a router at itself or renames it onto one of its targets returns 400 instead of a 500 with a dead row. The Switchyard tests share one transcript builder and router fixture, and reformatting noise in unrelated test files is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): make Switchyard a strategy in the existing auto router form Switchyard is now a card in the existing strategy picker. Picking it swaps the tier and classifier fields for the Switchyard fields, and the shared name, team and save button create it through one submit branch, the same flow as the other strategies. The separate add form, the extracted picker and the hidden-form swap are gone, and the edit dialog keeps its own Switchyard form The judge's session_affinity and message_hash_fallback settings are removed, since Switchyard rejects both for the stage router Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): accept partial tool semantics and show server errors on model edit The Switchyard tool semantics field rejected mappings that list only some categories, although the server accepts them. Model PATCH errors were reduced to a generic message, so it now goes through the shared API client like model create and surfaces the server text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): show the routing strategy picker up front in the auto router form Every strategy was hidden behind an Advanced configuration link, so the quick setup read as the only option and never said it creates a heuristic router. The picker now always shows with Heuristics as the labelled default, the quick setup sits under it, and picking another strategy opens that strategy prefilled with the chosen models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): show the server's reason when a model settings or API key update fails Both screens replaced the rejection with a fixed message, so a 400 such as a router routing to itself read only as a generic failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(router): route Switchyard on the Rust router core main replaced the Python router core with the Rust one, which left Switchyard routers registered nowhere. Register them with the other pre-routing strategies, hand each strategy the endpoint the request came through, and evict Switchyard routers on reload and delete like the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form Switchyard session state is keyed by a hash of the caller's key hash and session id, and client session headers no longer reach Switchyard unscoped. The add form carries the quick setup models into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the schema reads API-saved configs with nulls and structured decision instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): make the Switchyard test folder a package * fix(router): read the Switchyard session key hash only from the proxy's own metadata * fix(proxy): keep nested nulls as null when masking deployment params * fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies * fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:07:57 -08:00
simonp9	1f15088fc9	feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408) ... * feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes The native Codex catalog now lists the key's other chat and responses models after ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list refreshes on Codex's timer. A request for a model configured by exact name on another provider goes through /v1/responses without the ChatGPT bearer or account header, while ChatGPT names and wildcard-only matches keep the subscription passthrough. Reasoning items another provider produced are dropped before forwarding to ChatGPT. * fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while `codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native /backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no default, so the provider applies its own default exactly as `codex -m` does * fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may call /v1/responses, drop unpersisted reasoning ids on the passthrough when store is not true, keep any slug the caller's ChatGPT account lists on the subscription via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard precedence over a catch-all deployment. * fix(proxy): keep Codex's native route on the subscription when the account list is unknown A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the account's list is known and leaves it out. A failed catalog read is remembered for 60 s and the Codex version comes only from a User-Agent whose product matches the originator header. The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows, and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so Codex stops refetching or dropping its catalog cache. LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge decodes wrapped thinking blocks with their signature. The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the native route checks the chatgpt/ form of a name the key holds only through that group. A none reasoning effort is dropped only under drop_params. The guide covers app tools, every routing case, and the ETag, and the catalog logs how many models the byte limit left out. * fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API * fix(proxy): keep Codex native turns working when the key's model listing fails A listing callback that raises no longer turns chatgpt.com's catalog or responses answer into a 500. The catalog keeps upstream's body and ETag, and responses keep upstream's X-Models-Etag. Under drop_params, a `none` reasoning effort is now dropped only when a candidate deployment does not explicitly support it, so an alias to a model like openai/gpt-5.4 keeps it. The account slug list and its unavailable marker are keyed on the account id plus the caller's ChatGPT Authorization header, so a failed fetch with one bearer cannot turn off LiteLLM routing for another caller on the same account. The auth step checks the Codex route before importing the Codex module. * fix(proxy): keep Codex model switches working across LiteLLM providers On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400 Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows * fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the native route now hoists them without flattening, like OpenAI and Azure and the chat completions bridge. Other native Responses providers still get flat names, now built with the bridge's namespace_tool_name_map, and each function_call they return gets its tool name and namespace back in the non-streaming body and in streamed output_item.added, output_item.done, and completed/incomplete/failed events. Every other event passes through unchanged * fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools Codex sends its freeform exec tool as a custom tool on every turn, and xAI's Responses API answers 422 for custom, namespace and additional_tools shapes. On the Codex native route, a model whose deployments include a native Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now asks for the chat completions bridge with use_chat_completions_api, which already turns custom and namespace tools into functions, hoists additional_tools, and puts the namespace back on each returned call OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock Mantle still get it on their own Responses API with additional_tools hoisted. The route's namespace flattening and the response-side namespace restore are removed, since no path sends flattened names anymore * fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve A reasoning item another deployment produced, replayed with store false and no blob the target can decrypt, now reaches the target as its readable summary without id or status, and is dropped when nothing readable is left. OpenAI answered 400 on the status field and 404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment replays and stored conversations keep the item whole * refactor(codex): reach encrypted reasoning stripping through the router registry The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise every target must decrypt it * fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog * fix(proxy): drop Codex's none effort for an alias whose native entry names no default * fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop * test(proxy): map the pinned reasoning efforts field in the router registry stub * fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases Codex native routes now send a key to LiteLLM only when admin_only_routes allows it, a native alias with configured service tiers starts with no tier picked, and the account slug cache declares its Redis key family. * fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items * fix(proxy): check vector store grants on file_search tools inside additional_tools items A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input. * fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes * fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply * fix(guardrails): check additional_tools items next to top-level function tools in tool_policy ToolPolicyGuardrail read tools from the request body only when no top-level function tool existed, so a blocked tool defined in an additional_tools item reached Fireworks once the Responses transform moved the item into tools after the pre-call hooks. Check every list on each request, including namespace members inside items. * fix(router): serve decryptable_reasoning from the Rust core's optional checks The merge with main removed the Python router core, so the registry's host has no optional_callbacks and no routing context, and native Codex requests with encrypted reasoning raised RustCoreUnsupported. The filter now lives with the optional checks, which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core Also give the static litellm.Router a declared constructor. Since the Python core was deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...) is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change * fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures. * fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias * fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases Auth no longer asks ChatGPT for the account's model list when the bearer is missing or is the LiteLLM key, so a key sent as the bearer never reaches it. tool_policy now reads the members of a top-level namespace, which is where the native route's hoist puts them. A global alias now reads encrypted content affinity from its target group. A non-list top-level tools value and a non-string function name now reach the provider, which refuses them, instead of raising a 500. The catalog ETag no longer depends on the order a process iterates priced service tiers in, so workers and pods agree on it. * fix(proxy): list a team's models in the order the team set them get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker * fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models and /backend-api/codex/responses only when the JWT auth flow would admit a direct /v1/responses call for that model, so team_allowed_routes, admin_allowed_routes and RBAC route limits hold on the handoff too. The Fireworks Responses transform also leaves additional_tools input items in place when the tools param is not a list, instead of raising while hoisting * fix(fireworks): hoist additional_tools when tools is a tuple litellm.responses passes a tuple tools argument through to the Fireworks transform unchanged, so the non-list guard skipped hoisting and left the additional_tools item in the input. Treat tuples like lists and cover both shapes in the drop_params merge test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 11:14:47 -08:00
yucheng-berri	4086d5ccaa	feat(proxy): hash UI passwords with PBKDF2 in FIPS mode (#49874) ... * feat(proxy): hash UI passwords with PBKDF2 in FIPS mode A FIPS OpenSSL refuses scrypt, so every password write fails there and nobody can sign in with a password. When OpenSSL enforces FIPS, passwords now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is refused with one error log and left as it is, and the boot migration does nothing, because FIPS mode is for new deployments. PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the five request paths run it on one thread instead of the event loop. A single thread keeps a login burst from using up a CPU quota, which is what stalls the loop with asyncio.to_thread. A login that queued for that thread asks the throttle again before it answers, so a burst of guesses can't get past a block that the failures ahead of it just set. The breached password check is off in FIPS mode, because it runs the password through SHA-1. A standard OpenSSL reads False at every new branch, so standard mode keeps scrypt, inline, on the same rows. Resolves LIT-8425 * fix(proxy): take FIPS logins one at a time and recheck the throttle first A login that waited for the KDF thread asked the throttle again only after deriving, so a burst for one user kept that thread busy long after the user was blocked. Logins now wait for their turn on a lock and ask the throttle when it comes, so a blocked login costs no PBKDF2 and only one login at a time holds a place in the KDF queue. Password changes and invite claims no longer wait behind a whole burst of logins. * fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the real password stopped working and the old row text signed in. The migration now leaves a row of that exact shape alone. Any other pbkdf2: text still migrates. FIPS mode now logs a warning when it skips a breach check that is turned on. The one-thread test no longer depends on timing, and the new test variables are annotated Final. * fix(proxy): keep the breached password check on in FIPS mode * fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL * test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt	2026-11-06 01:31:59 -08:00
yujonglee	b1b236826a	feat(redis): serve the proxy's common Redis call with Rust stores (#50170) ... * feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 06:01:11 +00:00
ryan-crabbe-berri	9a473fbbb0	feat(proxy): give Codex accurate model entries for every model in /v1/models (#49825) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release	2026-11-05 19:31:32 -08:00
ryan-crabbe-berri	48ff826cbc	feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496) ... * feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request	2026-11-06 02:50:15 +00:00
yucheng-berri	9f489430a6	fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check (#50305) ... * fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check * test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts	2026-11-05 17:43:32 -08:00
joshua-berri	a12a6cd154	feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872) ... * fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 09:14:10 -08:00
ishaan-j-berri	71fde48016	feat(alerts): show recommended model provenance and provider notices (#50162) ... * feat(alerts): show recommended model provenance and provider notices * fix(ui): avoid nested recommendation source conditionals	2026-11-05 04:34:17 +00:00
Krrish Dholakia	c5da69cd13	feat(ui): show model deprecation banner on the models page (#50080) ... * feat(ui): show model deprecation banner on the models page Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh model deprecations when models change Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): address review feedback on model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover deprecation role gate and agent builder refresh Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): separate deprecation banner toggle from alerts link Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): dedupe and allow dismissing the model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh table once and key banner entries on successor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: quiet successor logs, count banner models, use local dates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): pin banner test date and bucket upcoming entries locally Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover mixed-successor log level through the deprecations endpoint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): move deprecation banner cases into integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep deprecation banner when a refetch fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 18:50:14 -08:00
joshua-berri	906fa7ed28	feat(accounting): add opt-in SpendLogs ledger (#49856) ... * feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:49:15 -08:00
Mateo Wang	80b18d165b	chore(proxy): use neutral example names for baggage tag mappings (#50011)	2026-11-04 14:49:55 +00:00
tin-berri	48bd9b01f6	feat(skills): upload local skill ZIPs from the dashboard (#50008)	2026-11-04 00:56:40 -08:00
yucheng-berri	ffbbb6366b	feat(proxy): add general_settings.fips_mode startup gate (#48600) ... * feat(proxy): add LITELLM_FIPS_MODE startup gate With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification is off, when Python's OpenSSL does not enforce FIPS, or when the startup password migration fails. A value that is not true/false also refuses. * feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS. * fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash The stock cryptography wheel reads the host's FIPS config and reports FIPS on, but its bundled OpenSSL cannot load the host's FIPS provider, so every operation fails. Require a working SHA-256 alongside the FIPS flag. * fix(proxy): refuse TLS verification off anywhere in FIPS mode A model's own ssl_verify: false reached the HTTP client after the startup check passed. get_ssl_verify now raises in FIPS mode whenever verification resolves to off, which also covers the startup check. * refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests * fix(proxy): stop refusing TLS verification off in FIPS mode Skipping certificate verification is a security choice, not a FIPS one: the TLS connection still only uses algorithms the FIPS provider allows. * feat(fips): refuse a database used without FIPS mode, drop the password migration change FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker. * test(fips): close the database test clients with aclose so the engine stops * feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE * refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only * fix(fips): retry the startup marker and state reads once after a dropped connection A dropped connection during the non-FIPS marker read used to count as an unmarked database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup reads use. A read that still fails keeps its old outcome. * fix(fips): refuse a fips_mode key that has no value fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and started without FIPS mode and without a message, while an empty string refused. A key that is present now has to be a boolean; only a config without the key means off. * refactor(fips): move the database check out of the boot gate The boot gate keeps the setting and the two OpenSSL checks. The check that a database matches the FIPS setting moves to its own PR, so a boot without the flag runs no extra database read. * test(fips): pin how a fips_mode string is read through enforce_fips_mode	2026-11-03 14:56:14 -08:00
Emerson Gomes	ecb8ef5ca8	feat(realtime)!: support latest OpenAI audio models (#35600) ... * feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:27:52 +00:00
zoveki	704e73d6fe	fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date (#49707) ... * fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date * fix(proxy): report a deployment's own provider retirement date before the bare model's An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows * fix(proxy): read Azure's retirement schedule for azure_ai deployments An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date. * fix(proxy): let custom and registered base_model rows keep their deprecation date Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate * fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's * fix(proxy): recognize the router's copied cost-map rows by the loaded catalog * fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 02:58:39 +00:00
Etienne Chabert	523b87ec71	fix(proxy): charge budget windows when the cached key or team expired before the request ended (#47608) ... * fix(proxy): charge budget windows when the cached key or team expired before the request ended The post-call spend update reads a key's or team's budget_limits from user_api_key_cache and returned early on a miss, skipping both the window counter increment and the LiteLLM_BudgetWindowSpend row. The cached object expires after user_api_key_cache_ttl (60s by default), so any request that ran longer than that without another request refreshing the entry was counted in the key's lifetime spend but never in its budget windows. With budget reservation disabled nothing else charges the windows, so keys kept running past their window caps On a miss the update now reads budget_limits straight from the key or team row. It does not re-cache the object, since a write from the spend path could race a /key/delete and put a just-deleted key back into the auth cache * test(proxy): assert the budget window fallback never re-caches the key or team * fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback The master key and UI session tokens are logged under names no key row carries, so every one of their requests missed the auth cache and read the database. A missing row is now remembered for the cache TTL in this process only. A failed read is not remembered, and a row that exists is read fresh on every miss so a key update is never served stale. * fix(proxy): skip the budget window row read for a key or team with no windows The fallback remembered only a missing row, so a key or team that exists but holds no budget_limits was read from the database on every request that outlived the key cache. It now remembers both cases under one cache key built in user_api_key_cache.py, for the same TTL as the auth cache * fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added. * test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper * fix(proxy): keep a row read that an update outran from restoring the no-window marker * fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches * fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:41:59 -08:00
tessellark	c720b23238	feat(proxy)!: honor model_info.max_input_tokens in the Codex catalog for stock models (#49701) ... * feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays. * feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it * fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window * test(proxy): stub the max_input_tokens group field and type the Codex window tests * fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:27:58 -08:00
tavren27	a4de720144	fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests the way their hosts bill them (#49074) ... * fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat template does, and add a per-deployment token_count_tool_framing next to token_count_framing for the hidden tool preamble a host bills. The Azure AI Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry the measured framing, overridable in model_info * fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing * fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice * fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing * fix(token_counter): count an explicit null strict like false on Azure Responses * fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302 * fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts * fix(proxy): frame Responses counts for the api_base router defaults send inference to * fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 12:37:35 -08:00