* feat(proxy): publish opted-in pass-through endpoints on the Model Hub
Pass-through endpoints gain display_name and show_in_model_hub. Endpoints that opt in appear on
/public/v1/model_hub and /public/model_hub as rows in mode passthrough, named by their display name
or their path, with the route in pass_through_path. The admin UI exposes both fields on the create
and edit forms and the public hub renders passthrough rows with a route-specific curl example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): carry pass-through methods onto hub rows and key dashboard rows by route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep health, cost and shell quoting correct for pass-through hub rows
Pass-through rows no longer inherit a router model's health check when they share its name, the public table shows n/a instead of Free for their token costs, and the generated curl example escapes shell characters in the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): quote the method in the pass-through hub curl example
A configured method string is placed after curl -X inside escaped double quotes so it cannot alter the copied command
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): build pass-through hub fixtures through a helper
Keeps the dashboard lint budget for inline object arguments under its limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit the pass-through Model Hub surface end to end
Cover forwarding, pagination, validation, auth, updates, duplicate names,
config-file precedence, health-name isolation, repeated reads and two chaos
cells (create/delete churn, a killed proxy worker). Build the stored
pass_through_endpoints list in one shot instead of appending to the list
the config reader handed back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep concurrent pass-through creates from overwriting each other
Creating an endpoint appends to the general_settings list the proxy already holds instead of persisting a private copy, so two admins creating at the same time both land, as on main before this branch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): serialize pass-through endpoint creates behind a process lock
Each create copies the stored list under an asyncio lock before persisting, so simultaneous creates in one worker no longer overwrite each other and a rejected write (config-owned path) never lands in the in-memory settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): collect paginated pass-through rows without mutating a list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): let a pass-through with no stored timeout save from the edit form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the pass-through edit fixture so a null timeout compiles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): read published pass-throughs from the in-memory settings store
The public Model Hub routes read the pass-through list this worker already
serves from proxy_config.settings instead of querying the config table on
every anonymous hub read. Also drops a mutable-ok suppression that no longer
guards a violation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): list a config file pass-through beside database rows on the hub
Main merges database pass-throughs with config file ones on other paths since
#43962, so a database write beside a config entry is accepted rather than
rejected as config-owned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the monkeypatch fixture in the pass-through hub tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): validate pass-through config entries instead of splatting unknown dicts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): list model hub rows as a tagged union of model groups and pass-throughs
Health enrichment and the no-router check now match on ModelGroupRow or
PassThroughRow instead of testing pass_through_path in each route. Both hub
routes share published_hub_rows, the v1 row helper is sync again, and the
config list route goes back to main's inline merge since the hub no longer
needs configured_pass_through_endpoints
* test(ui): find the pass-through hub row by role to stay under the no-node-access budget
* fix(proxy): keep database pass-through endpoints in the settings store across config reloads
A config reload cleared the stored database row before rebuilding the
pass-through list, so without store_model_in_db the list fell back to the
config file entries alone. The Admin UI triggers that reload on page load
through /get/config/callbacks, which dropped UI-created endpoints from the
Model Hub until the next write
* refactor(proxy): make the hub row matches exhaustive with assert_never
* refactor(proxy): return the exhaustive match fallthrough and build hub test inputs in one shot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop the in-process create lock from pass-through endpoints
The lock only covered one worker, and the read-modify-write on the general_settings blob is a separate storage problem tracked as tech debt. Create goes back to main's behavior.
* fix: align pass-through model hub with current lint rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: share worker state for pass-through hub restart coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: show pass-through curl examples in model hub inspector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: skip invalid pass-through hub entries and await peer refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* fix(token-counter): count Anthropic server tool history
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): bound nested results and update proxy coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): include readable web search snippets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When the upstream connection drops or stalls mid-stream on the Rust Messages
path, the caller gets the raw httpx error without litellm_debug_info, so the
llm_exceptions alert names no API base, model group or deployment. Python's
stream iterator attaches that info to the same error, and the Rust host now does
too.
* feat(replicate): add video generation, image generation and text to speech through the predictions API
* fix(replicate): surface poll HTTP errors, keep deployment ssl_verify and native speech input, flatten outputs without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep model family mapping for deployments and versions, preserve video id on status, poll until terminal status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): validate output downloads and poll origin, bind retrieval to the requested model, flatten image extra_body, keep speech input authoritative, use the async image path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep the audited prompt authoritative over image extra_body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(replicate): cover extra_body.prompt not replacing the audited prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): honor the request ssl_verify when polling and downloading speech output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(replicate): build the video usage into VideoObject instead of assigning it afterwards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep the inspected video prompt authoritative over native extra_body prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(replicate): bill native image dimensions and token-metered speech output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): bound output downloads, accept both token metric spellings and build speech usage at construction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): walk bounded-download redirects iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): reject dot and empty model path segments and keep the bounded redirect walk annotated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): drop the unused rebind suppression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): put the rebind suppression on the redirect hop binding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): run pinned versions through the model's own versions endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Absorb #51347, #51222, #51357 and #51275 with the GMI large-context tier.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model
Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false).
When on, the class weights are renormalized over the classes with a team active on the model in the
current window, each class share is divided equally among its active teams, and once the model saturates
each team is held to its own share with a 429 whose rate_limited_by header reads
priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL.
* fix(proxy): keep basedpyright counts flat for the priority class share setting
PriorityReservationSettings is built with model_validate instead of **value, so a new
field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers
type the handler by its implementation class.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way
A registry failure falls back to the class pool instead of skipping the model limits, a team's
counter is keyed by model and team so keys of different priorities spend one share, Responses
WebSocket turns are held to the class share like the team model limit, a share is floored at one
request and one token, and the registry declares the rate limits key family.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share
A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated
Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.
* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates
* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names
* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails
* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests
* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver
An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.
* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take
The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.
* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates
While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.
* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget
While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity
With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could
use everything up to the model's own tpm and rpm while the model was idle.
Setting team_model_limit_max_borrow_multiple in the team's metadata caps the
team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429
whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset
keeps the previous behavior. Values below 1 or non-numeric are logged and
ignored.
* fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check
* fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): close strict JSON schemas on the Messages to Responses bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(anthropic): copy each strict schema subtree once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): close nullable objects in strict JSON schemas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): give MiniMax H3 Max video endpoints their own resolution profile
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): keep the H3 Max test signature on one line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): type the H3 Max test locals and signatures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): opt-in team_model_limit_is_floor turns a team's model_tpm_limit into a guaranteed share that borrows idle capacity
With priority_reservation_settings.team_model_limit_is_floor: true, a team's inherited model_tpm_limit on a model the dynamic rate limiter covers stops being a hard cap in the regular limiter. The dynamic limiter tracks it as a team_model_floor descriptor, lets the team past it while the model is below saturation_threshold, and holds the team to it once the model saturates, with rate_limited_by: team_model_floor on the 429. Default false keeps today's behavior. Key-owned model_tpm_limit, model_rpm_limit, the model-wide tpm and customer limits stay hard.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): name the setting team_model_limit_mode with hard and soft values
soft replaces the team_model_limit_is_floor flag. The 429 header reads rate_limited_by: team_model_soft_limit and the message says the team soft limit was reached.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams covers team model TPM and RPM
Replaces team_model_limit_mode. A team's inherited model_tpm_limit and model_rpm_limit on a model the dynamic
rate limiter covers are enforced by that limiter once the model saturates, so the team borrows idle capacity below
the threshold. The 429 names the limit and reads rate_limited_by: model_per_team_saturated.
Folds in the bot findings: key ownership is checked per requested model, team limits resolve through the model
alias like the regular limiter, the team counter is charged only for requests the pre-call hook tracked, every
response type that carries usage is settled, and tracking below saturation no longer adds a token per request.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the type gate flat in the dynamic limiter's team limit paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle responses usage and null auth metadata, resolve aliases for the borrowing team limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): key the borrowing team counter by resolved model group and read usage through get_usage_object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): hold Responses WebSocket turns to the borrowing team limit once the model saturates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): install the Rust gateway and fail forwarded_proxy when Rust is not in front
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): build the Rust gateway only for the management suites that run forwarded_proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject unknown guardrail names on key and user writes
Key generate, update, regenerate, team bulk update and /user/new now return 400 for a guardrail name that no loaded guardrail answers to and that has no active DB row, instead of storing it so requests run unguarded. Names the key already holds are not rechecked, and the key endpoints keep the enterprise license 403 ahead of the name check.
Resolves LIT-10613
* fix(proxy): read guardrail names from the writer when validating key writes
A guardrail created on another pod may not have reached a lagging read replica yet, so the fallback lookup now reads the writer and a just-created guardrail is not rejected as unknown. Other guardrail table reads keep using the reader.
* feat(proxy): validate guardrail names on team and project writes, reject malformed values loudly
Team and project create and update now return 400 for guardrail names that no loaded guardrail answers to and that have no active DB row, the same check key and user writes use. A metadata.guardrails value that is not a list of names is rejected even when it equals the stored value, with a short message naming the value. The lookup and the license-first wrapper move into common_utils so every entity shares them.
* refactor(proxy): one validate_guardrail_names entry point for guardrail name checks
The name check, its DB lookup and the license-first option now live in litellm/proxy/guardrails/guardrail_name_validation.py and every key, user, team and project write calls validate_guardrail_names directly. The lookup factory, both wrappers and the per-endpoint helpers are gone, and common_utils and guardrail_registry are back to their main versions. Behavior is unchanged.
* fix(proxy): chunk the guardrail lookup and leave legacy dict guardrail settings alone
The guardrail name lookup goes through find_many_in so its IN list has a written bound, and the project test keeps pytest.raises to one statement. A dict-shaped metadata.guardrails, a team's documented {"modify_guardrails": false} or the v1 {name: bool} toggle map, is read at request time, so validation leaves it alone instead of rejecting it.
* fix(proxy): only let the team modify_guardrails permission through as a dict
Stored key, team and project guardrails only run when metadata.guardrails is a list, so a dict such as {"config-guard": true} saved fine and attached nothing. The one stored dict with an effect is a team's documented {"modify_guardrails": bool}, which auth checks read, so that exact shape passes and every other dict gets the Invalid guardrail 400.
* refactor(proxy): move guardrail name validation into guardrail_helpers
validate_guardrail_names and its helpers now live in litellm/proxy/guardrails/guardrail_helpers.py next to the other guardrail helpers, and guardrail_name_validation.py is removed. guardrail_helpers imports LiteLLM_TeamTable and UserAPIKeyAuth from litellm.proxy._types instead of litellm.proxy.proxy_server, so the endpoints can import it at module level without pulling in the proxy server, and its dead sys.path line is dropped. Behavior is unchanged.
* feat(alerts): add default-on monitoring and a five-alert free allowance
* fix(alerts): handle monitoring races and retry-safe migration
* fix(ui): match monitoring upgrade link to status
* feat(ui): restore performance alerts on updated detector
* fix(ui): keep team key detail tabs independent
* fix(ui): preserve team tabs while permissions load
* feat(ui): show concise alerts in the notification bell
* refactor(ui): remove replaced deprecation bell item
* fix(ui): pin notification resource links
* fix(ui): show deprecation failures in notifications
* fix(ui): reset activity state when changing resources
* fix(ui): reset key detail state from every entry point
* feat(alerts): detect daily changes in usage for gateway, models, teams, users and keys
* fix(alerts): drop unused suppressions
* fix(alerts): satisfy strict lint for model usage timing
* fix(alerts): skip only resources that time out or hold invalid data
* fix(alerts): log an unfinished daily sweep at info level
* fix(alerts): keep the deployment-wide run every five minutes
* fix(alerts): record alert and sweep timestamps in UTC
* fix(alerts): repair detector tests and tracing
* refactor(alerts): build the key index in a migration instead of after startup
* refactor(alerts): build the key index after startup again
A concurrent index build inside the boot migration can exceed the 600 second migrate deploy deadline on a large LiteLLM_DailyUserSpend and stop the proxy from starting. This reverts 393db6e4e0.
* fix(alerts): keep keys pending, bound the accounting wait, resume unfinished scopes
Keys waiting for their index no longer mark the day complete. Only deliveries created during the evaluated day hold back its sweep. A day that did not finish restarts the next day from where it stopped instead of a hash boundary that never rotated non-hex IDs.
* refactor(alerts): drop the pending delivery wait, alerts are best effort on flushed usage
* test(alerts): cover detector persistence and recovery
* fix(alerts): recognize Prisma statement timeouts
* feat(proxy): run spend log cleanup from a DB-backed scheduled job table
Spend log retention ran from an in-process APScheduler timer that reset on
every restart and needed Redis for leader election, so pods that restart
more often than the interval never cleaned up at all.
Add LiteLLM_ScheduledJobRun, a generic table with one row per job run.
Every pod polls it, plans the next run idempotently and claims a due run
under a transaction-scoped advisory lock, so exactly one pod runs each
occurrence. Runs carry a lease, a status and a log. Intervals count from
the last finished run and survive restarts. Schedule edits cancel and
replace the pending run. The retention and size-check jobs share a lock
group so they never overlap, and the Redis pod lock is gone.
* fix(proxy): plan an earlier cron edit from now, and cover the job runner
Moving the cleanup cron earlier queued tomorrow's run because the next
occurrence was planned after the pending run being cancelled. Plan from
finished runs only.
Register LiteLLM_ScheduledJobRun as a span table, keep the advisory lock
query inside the tick's span, move the non-string cron startup test to
the schedule parser, and add unit tests for the runner and job wiring.
* test(proxy): annotate new spend log cleanup test locals as Final
* test(proxy): type the parameters of the new spend log cleanup tests
* test(proxy): leave an hour of margin when moving a cron earlier so an hour boundary cannot flake it
* fix(proxy): reject a cron that never fires instead of failing every tick
A crontab like 0 0 31 2 * parses but has no next fire time, so next_run_at raised on every tick and the job was never scheduled again. cron_schedule now returns InvalidSchedule for it, which logs and keeps the pending run like any other bad config
* refactor(proxy): drop the informational log column from LiteLLM_ScheduledJobRun
Nothing read the column back. Run outcomes stay in status, and the reasons
already go to the proxy logs, so JobResult becomes a SUCCEEDED or FAILED
literal and the cleanup returns its RunOutcome instead of a summary report
* fix(gemini): attribute /gemini batchGenerateContent batches to the calling key
A 200 from models/{model}:batchGenerateContent through the Gemini pass-through now registers a managed
batch row for the calling key, team and tags, so CheckBatchCost prices the output file on completion
instead of the job landing as a $0 pass_through_endpoint row with model unknown.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gemini): unit regression for pass-through batch registration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): file batch created through /gemini is registered for the calling key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): bill pass-through batches through the configured gemini deployment
The managed row now points at the gemini/<model> deployment CheckBatchCost fetches the output with, not
the first router id for the model name, and a model with no gemini deployment logs a warning instead of
registering a row that can never be billed. A key without a user keeps user_id unset rather than
default-user. Typed the registration helper and handler so the lint gates stay flat.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
pytest --ignore is a no-op for a file the shell glob already expanded, so the 25
files proxy-infra-root listed under --ignore still ran there as well as in their
own shards (805 duplicate tests, 18% of the shard's time). Switch them to
--deselect, which pytest applies to explicit arguments, and teach
assert_ci_coverage.py to read --deselect and [..] character-class globs.
Split the rest by first letter into proxy-infra-root-a-p and proxy-infra-root-q-z,
and give the three files that spawn a child python (test_component_allowlists,
test_prometheus_metrics_server, test_proxy_cli) their own 2-worker
proxy-subprocess shard so the children stop competing with 4 xdist workers on
a 4 vCPU runner.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill anthropic pass-through calls at the deployment's custom price
The /anthropic route never attached a router deployment's model_info, so
spend fell back to the shared cost-map key, which carries no per-deployment
prices. Attach the model_info of the one deployment in the requested group
that serves that exact Anthropic model and carries custom pricing, so
streaming and non-streaming calls, and dated snapshot responses, bill at it.
Fixes#49596
* fix(proxy): validate deployment rows before reading anthropic pass-through pricing
Read the untyped router rows and request body through a pydantic adapter so
the pricing lookup adds no basedpyright unknown-type errors.
* test(proxy): drive the real anthropic pass-through relay when asserting the deployment price
Stub only the upstream http client and assert the x-litellm-response-cost the
route returns, instead of doubling create_pass_through_route.
* fix(proxy): match an anthropic pass-through deployment by its configured or resolved model name
get_llm_provider rewrites -latest aliases, so a deployment configured with the
alias the caller sends never matched and lost its custom price.
* fix(proxy): price an anthropic pass-through call by the model its custom_body forwards
The relay forwards custom_body when the caller sends one, so resolving the
deployment from the outer model let a caller bill an expensive call at a
cheaper deployment's price.
* fix(proxy): only price an anthropic pass-through call with a deployment the caller's team may use
A teamless key could pick up a team-owned deployment's price because the group
lookup does not filter by team when the key has none.
* fix(proxy): resolve anthropic pass-through pricing from the body forwarded after pre-call hooks
The route stamped the deployment price before the pass-through ran its
pre-call hooks, so a hook that swapped the model billed the original
model's deployment. The route now hands the relay a resolver, and the relay
applies it to the final forwarded body, which already reflects the
custom_body envelope and every hook.
* refactor(proxy): clear pass-through request state keys in one helper
Keeps create_pass_through_route within the C901 ceiling now that the
deployment pricing resolver adds a fourth request.state key.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(auth): allow paid video retrieval after budget exhaustion
* test(auth): check video retrieval boundary on registered route
* fix(auth): exempt video retrieval in post-custom-auth checks
* style(auth): format custom video budget exemption
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(anthropic): time-limit /v1/messages pump waits on a stalled client
The upstream pump waited on client_detached with no timeout, both when the relay queue was full and when waiting for the relay to consume a queued upstream error. client_detached is only set by the relay generator's finally block, so a client that resets its connection while stalled could leave the pump parked forever, holding its request, logging object and collected chunks.
Both waits now give up after ANTHROPIC_MESSAGES_STREAM_CLIENT_STALL_TIMEOUT_SECONDS (default 120), treat the client as gone, and fall through to the existing detached billing path.
* fix(anthropic): bound the detached /v1/messages drain and end a relay whose pump gave up
A provider that stalled after the client left kept the detached pump, the open provider stream and every chunk it
had buffered for billing alive until the provider read timed out, if it ever did. Pumps that never read another
chunk after the disconnect are not counted against the detached-drain cap, so nothing limited them
A detached drain still reading ANTHROPIC_MESSAGES_DETACHED_STREAM_DRAIN_TIMEOUT_SECONDS (default 600) after the
client left now bills what it read and closes the provider stream. The deadline starts when the relay closes or
when the client stall timeout gives up on the client, so it also covers a read that was already stalled
Once the stall timeout gives up, the pump stops feeding the relay. A client that resumed reading then waited on an
empty queue forever. The relay now ends with the incomplete-stream error once the pump is gone. The stall timeout
is read with get_env_float so fractional values parse
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): end a stream cleanly when the pump gives up on its last put
When the upstream had already finished and only the end-of-stream sentinel was left, a client that resumed after
the stall timeout got every chunk through message_stop and then an extra incomplete-stream error event. The relay
now only adds that error when the client has not already received a message_stop or error event
The new test pins the case where the stall timeout fires on the final sentinel put, so the pump exits with the
request billed and a resumed client still sees a clean, complete stream
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): give a stalled /v1/messages client 600 s before the pump gives up on it
The 120 s default could cut off a client that was only slow. 600 s matches the usual request_timeout, so a stalled
client and a stalled provider get the same budget. SSE keepalive pings do not change this, because they block on the
same stalled write. A stuck pump now lives at most about 20 minutes, which keeps only a handful alive at once
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic_messages): keep queued chunks when dropping an unconsumed upstream error
_exception_left_unconsumed emptied the whole relay queue to check whether the
provider error was still in it, so a client that resumed after the error wait
expired lost every chunk buffered ahead of the error. Re-queue everything but
the error, add a regression test for the resumed client, and type the new test
helpers.
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add the CollieAi guardrail integration
* refactor(guardrails): tighten the CollieAi wrapper and its tests
The loader takes its importer as a parameter, so a test covers the fallback for a package that is too old, and the install hint names the Python the package needs
The config model is frozen, the Garden preset's phases and default_on are pinned by a test, and comments the code already says are gone
* fix(guardrails): use the generated Collieai provider key in the dashboard
The dashboard builds provider keys from the proxy's provider list, so collieai becomes Collieai. The static CollieAi key came first in the provider map, which left saved CollieAi guardrails without their display name and logo. The map, the Garden preset and the card now use Collieai, and a test covers the lookup
The preset expectation is a named value now, which keeps the dashboard under its inline object lint budget
* refactor(guardrails): drop the CollieAi wrapper's function docstrings
AGENTS.md keeps comments to complex logic and tool input, and these only restated what each function returns
* fix(guardrails): declare CollieAi's phases, read COLLIEAI_API_KEY and plate its logo
The dashboard offered every mode for CollieAi, and a guardrail saved with a mode the package refuses was reported as created but never loaded. The wrapper now declares pre_call and post_call, so the form offers only those
api_key falls back to COLLIEAI_API_KEY as its description says, and the navy logo gets a plate in dark mode like DeepKeep's
The tests now also cover an entry's own name, phase and default_on, the key fallback, and the modes the dashboard offers
* fix(guardrails): reject CollieAi guardrails the package refuses and stop env-keyed rebuilds
A Garden create whose key or settings the package refuses now answers 400 and
saves nothing, instead of saving a guardrail that checks nothing. Every setting
on the entry now reaches the package, like the class-path route. DB guardrails
keyed with an os.environ/ reference are no longer rebuilt on every DB poll.
The Garden card says which calls CollieAi checks.
* fix(guardrails): load CollieAi stream_scope and import errors as setup errors, default the Garden card off
Pass stream_scope into the package constructor inside the setup-error wrap so a refused scope answers 400 on create and stops a config-file proxy at startup. Drop settings cleared to null so the package default applies. Say an installed package failed to load, with the error, instead of asking to install it, and keep any import-time exception from breaking the guardrail routes. Make api_key optional in the form since COLLIEAI_API_KEY is the fallback, and open the Garden card with Always On off like the other partner presets.
* fix(guardrails): leave CollieAi form fields empty and share one guardrail setup error
The Garden form no longer pre-fills api_base, deadline_ms or forward_conversation_id, so a
create saves only what the admin typed and the package keeps its own defaults. Setup failures
raise GuardrailSetupError from custom_guardrail, the missing-package hint covers the Docker
image, and the card copy no longer repeats the package's eligibility rules.
* test(guardrails): cover env-reference api_base in the DB poll comparison
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(router): list a group's deployments once per registry change on the acompletion path
* perf(router): drop the new docstrings
* perf(router): keep the counter cache's mutable-ok on its annotation
* perf(router): define the counter cache's alias in the registry
* feat(azure_ai): support gpt-image edits on Azure AI Foundry
Foundry hosts gpt-image deployments on the same /openai/deployments/{deployment}/images/edits route as Azure OpenAI, but the azure_ai image-edit factory only knew MAI and FLUX, so gpt-image-2 and the gpt-image-2.5 models raised a ValueError before any request was sent
Reuse AzureImageEditConfig for the request, URL, and multipart handling, and resolve the api_base and credentials with the azure_ai helpers
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(azure_ai): default gpt-image edits to an edit-capable api-version
Foundry gpt-image edits now fall back to 2025-04-01-preview when no api_version is set, since the shared Azure default 2025-02-01-preview answers 404 on /images/edits. The config returns merged headers instead of updating the caller's dict, and the tests check the request each gpt-image model sends
* fix(azure_ai): ignore the Azure OpenAI api-version env for Foundry gpt-image edits
The proxy CLI always sets AZURE_API_VERSION to 2025-02-01-preview, which
Foundry answers with 404 on image edits. gpt-image edits now read only
litellm_params, AZURE_AI_API_VERSION, and litellm.api_version before the
2025-04-01-preview default, matching the other azure_ai edit configs
* fix(azure_ai): route custom Foundry deployments, require a Foundry api_base, and honor caller auth for gpt-image edits
A deployment name the edit factory does not recognize now uses the gpt-image route instead of raising, since Foundry deployment names are free-form. A gpt-image edit with no Foundry api_base raises like the MAI and FLUX configs instead of borrowing AZURE_API_BASE. The shared edit handler passes the call's extra_headers to validate_environment, so a caller's api-key or Authorization header is sent alone
* fix(azure_ai): keep the shared edit handler unchanged and send caller auth headers like azure_ai chat
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
