* fix(otel): distinguish unknown timeout usage from prompt estimates
* fix(otel): handle expected tokenizer failures and cover parent spans
* fix(otel): validate failure usage inputs at typed boundaries
* fix(otel): type Logging.messages for the failed prompt token estimate
Declare Logging.messages as Sequence[object] | None at its single __init__
assignment and drop the list TypeAdapter, so the failure estimate reads a typed
value without new unknown-type diagnostics.
* fix(otel): skip the failure prompt estimate for prompts with attachments
Only estimate the failed prompt when every message is plain text, so files,
documents and images never reach PDF parsing or media decoding on the failure
path. Those requests still report unknown provider usage without an estimate.
* fix(otel): count the failed prompt off the event loop for async requests
Async failure logging now awaits the existing offload_token_count limiter for the
optional prompt estimate, and the sync half of an async request no longer counts
inline, so a slow tokenizer load cannot stall other requests.
* fix(otel): only count the failed prompt for loggers that receive failures
The estimate check now reads general callbacks, the global failure lists through
a typed get_failure_callbacks accessor, and the dynamic failure lists, so a logger
registered only for success never triggers failure token counting.
* fix(otel): return an immutable failure callback snapshot
* fix(otel): keep malformed content blocks from breaking failure logging
Only compare string block types against the text block table, and run the
text-only check inside the estimate's narrow try, so a malformed prompt leaves
usage unknown without an estimate instead of interrupting failure callbacks.
* fix(otel): stop the token counter from debug-logging raw prompts
The failure estimate counts the original prompt, so the counter's debug line
could write prompt text to application logs while message logging was off.
Drop that line and assert the prompt stays out of debug logs on both routes.
* test(otel): pin text-only eligibility and failure callback categories
Cover the public has_only_text_content contract for malformed and non-text
messages, and check get_callbacks_by_type puts sync and async failure
registrations in the failure categories through the public manager APIs.
* fix(otel): keep failure logging when the prompt estimate cannot run
Return no estimate when the offloaded count hits a worker or tokenizer
RuntimeError or OSError, so failure callbacks still run, while cancellation
still propagates. Treat SDK Message objects like their dict form when checking
whether a prompt is text only.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(anthropic_messages): key the first web search round's spend row by its own upstream reply
A non-streaming /v1/messages request answered through websearch_interception
logged its first model call with that call's usage but the follow-up reply's
id, so both spend rows shared one request_id and the first was dropped by
skip_duplicates. The logged row now takes its id and provider-reported cost
from the same upstream body its usage is parsed from, falling back to the
served id when that body has none.
Refs #49651
* refactor(anthropic_messages): read the upstream reply id after reparsing it
Drop the defensive body helper. transform_response already requires a read
JSON body, so the upstream id and reported cost are now read after it
succeeds, which leaves every existing failure path unchanged. Tests gain
return annotations and use the current Sonnet model name.
Refs #49651
* fix(anthropic_messages): key by the upstream reply only when a follow-up was served
Keying every reparsed /v1/messages row by its upstream reply overrode an id a
post-call deployment hook deliberately replaced. The anthropic agentic
executor now records on the outer logging object that it served a follow-up's
reply, and only then does that call's spend row take its id and reported cost
from its own upstream reply.
Refs #49651
* test(anthropic_messages): pin the agentic follow-up row test to a fixed start time
Refs #49651
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
A guardrail such as Pillar aliases the request's metadata to litellm_metadata,
which holds the caller's UserAPIKeyAuth. Since #51119 the Rust host converts
the whole metadata field to JSON before its own user_id filter runs, so the call
failed with "unsupported type UserAPIKeyAuth". The host now passes only
user_id, the one key the Rust validator and Python's AnthropicMetadata keep.
* fix(otel): report the provider's response model under gen_ai_latest_experimental
With OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, gen_ai.response.model now uses
the provider_response_model kept on the response hidden params, then the deployment model when
the response only echoes the client alias. Legacy mode and litellm.model_group keep the alias.
Fixes#51029
* fix(otel): type the semconv response model inputs and attach hidden params in tests
Bind the request model once through as_str so the deployment fallback no longer reads untyped
kwargs. Non-streamed fixtures now attach provider_response_model with set_hidden_params, since
ModelResponse(hidden_params=...) does not keep it on the pydantic private attribute.
* fix(otel): read the request model as object before narrowing it
Binding kwargs.get("model") to a Final[object] stops basedpyright from reporting an unknown
argument passed to as_str.
* fix(otel): validate the untyped request model through a TypeAdapter boundary
TypeAdapter(object).validate_python accepts the untyped kwargs value and returns a known object,
the existing litellm/utils.py and router.py pattern, so as_str no longer receives an Unknown.
* test(otel): stream the aliased response through a real Logging object
Greptile flagged the MagicMock logging_obj as a double on our own code. The streamed fixture now
builds a real Logging with a fixed start time, like test_streaming_handler.py.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Anthropic /v1/messages always runs on Rust since #51119, so the safeguards
contracts no longer assert those calls stay on Python, and the idle reconnect
stream row expects the Rust route header like the keepalive row after #51346.
Vertex and DeepSeek safeguards calls keep their Python assertions.
* feat(proxy): publish opted-in pass-through endpoints on the Model Hub
Pass-through endpoints gain display_name and show_in_model_hub. Endpoints that opt in appear on
/public/v1/model_hub and /public/model_hub as rows in mode passthrough, named by their display name
or their path, with the route in pass_through_path. The admin UI exposes both fields on the create
and edit forms and the public hub renders passthrough rows with a route-specific curl example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): carry pass-through methods onto hub rows and key dashboard rows by route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep health, cost and shell quoting correct for pass-through hub rows
Pass-through rows no longer inherit a router model's health check when they share its name, the public table shows n/a instead of Free for their token costs, and the generated curl example escapes shell characters in the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): quote the method in the pass-through hub curl example
A configured method string is placed after curl -X inside escaped double quotes so it cannot alter the copied command
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): build pass-through hub fixtures through a helper
Keeps the dashboard lint budget for inline object arguments under its limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit the pass-through Model Hub surface end to end
Cover forwarding, pagination, validation, auth, updates, duplicate names,
config-file precedence, health-name isolation, repeated reads and two chaos
cells (create/delete churn, a killed proxy worker). Build the stored
pass_through_endpoints list in one shot instead of appending to the list
the config reader handed back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep concurrent pass-through creates from overwriting each other
Creating an endpoint appends to the general_settings list the proxy already holds instead of persisting a private copy, so two admins creating at the same time both land, as on main before this branch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): serialize pass-through endpoint creates behind a process lock
Each create copies the stored list under an asyncio lock before persisting, so simultaneous creates in one worker no longer overwrite each other and a rejected write (config-owned path) never lands in the in-memory settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): collect paginated pass-through rows without mutating a list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): let a pass-through with no stored timeout save from the edit form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the pass-through edit fixture so a null timeout compiles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): read published pass-throughs from the in-memory settings store
The public Model Hub routes read the pass-through list this worker already
serves from proxy_config.settings instead of querying the config table on
every anonymous hub read. Also drops a mutable-ok suppression that no longer
guards a violation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): list a config file pass-through beside database rows on the hub
Main merges database pass-throughs with config file ones on other paths since
#43962, so a database write beside a config entry is accepted rather than
rejected as config-owned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the monkeypatch fixture in the pass-through hub tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): validate pass-through config entries instead of splatting unknown dicts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): list model hub rows as a tagged union of model groups and pass-throughs
Health enrichment and the no-router check now match on ModelGroupRow or
PassThroughRow instead of testing pass_through_path in each route. Both hub
routes share published_hub_rows, the v1 row helper is sync again, and the
config list route goes back to main's inline merge since the hub no longer
needs configured_pass_through_endpoints
* test(ui): find the pass-through hub row by role to stay under the no-node-access budget
* fix(proxy): keep database pass-through endpoints in the settings store across config reloads
A config reload cleared the stored database row before rebuilding the
pass-through list, so without store_model_in_db the list fell back to the
config file entries alone. The Admin UI triggers that reload on page load
through /get/config/callbacks, which dropped UI-created endpoints from the
Model Hub until the next write
* refactor(proxy): make the hub row matches exhaustive with assert_never
* refactor(proxy): return the exhaustive match fallthrough and build hub test inputs in one shot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop the in-process create lock from pass-through endpoints
The lock only covered one worker, and the read-modify-write on the general_settings blob is a separate storage problem tracked as tech debt. Create goes back to main's behavior.
* fix: align pass-through model hub with current lint rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: share worker state for pass-through hub restart coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: show pass-through curl examples in model hub inspector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: skip invalid pass-through hub entries and await peer refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* fix(token-counter): count Anthropic server tool history
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): bound nested results and update proxy coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): include readable web search snippets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When the upstream connection drops or stalls mid-stream on the Rust Messages
path, the caller gets the raw httpx error without litellm_debug_info, so the
llm_exceptions alert names no API base, model group or deployment. Python's
stream iterator attaches that info to the same error, and the Rust host now does
too.
* feat(replicate): add video generation, image generation and text to speech through the predictions API
* fix(replicate): surface poll HTTP errors, keep deployment ssl_verify and native speech input, flatten outputs without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep model family mapping for deployments and versions, preserve video id on status, poll until terminal status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): validate output downloads and poll origin, bind retrieval to the requested model, flatten image extra_body, keep speech input authoritative, use the async image path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep the audited prompt authoritative over image extra_body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(replicate): cover extra_body.prompt not replacing the audited prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): honor the request ssl_verify when polling and downloading speech output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(replicate): build the video usage into VideoObject instead of assigning it afterwards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): keep the inspected video prompt authoritative over native extra_body prompt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(replicate): bill native image dimensions and token-metered speech output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): bound output downloads, accept both token metric spellings and build speech usage at construction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): walk bounded-download redirects iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): reject dot and empty model path segments and keep the bounded redirect walk annotated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): drop the unused rebind suppression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): put the rebind suppression on the redirect hop binding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(replicate): run pinned versions through the model's own versions endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Absorb #51347, #51222, #51357 and #51275 with the GMI large-context tier.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model
Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false).
When on, the class weights are renormalized over the classes with a team active on the model in the
current window, each class share is divided equally among its active teams, and once the model saturates
each team is held to its own share with a 429 whose rate_limited_by header reads
priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL.
* fix(proxy): keep basedpyright counts flat for the priority class share setting
PriorityReservationSettings is built with model_validate instead of **value, so a new
field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers
type the handler by its implementation class.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way
A registry failure falls back to the class pool instead of skipping the model limits, a team's
counter is keyed by model and team so keys of different priorities spend one share, Responses
WebSocket turns are held to the class share like the team model limit, a share is floored at one
request and one token, and the registry declares the rate limits key family.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share
A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated
Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.
* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates
* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names
* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails
* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests
* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver
An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.
* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take
The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.
* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates
While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.
* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget
While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity
With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could
use everything up to the model's own tpm and rpm while the model was idle.
Setting team_model_limit_max_borrow_multiple in the team's metadata caps the
team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429
whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset
keeps the previous behavior. Values below 1 or non-numeric are logged and
ignored.
* fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check
* fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): close strict JSON schemas on the Messages to Responses bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(anthropic): copy each strict schema subtree once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): close nullable objects in strict JSON schemas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): give MiniMax H3 Max video endpoints their own resolution profile
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): keep the H3 Max test signature on one line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): type the H3 Max test locals and signatures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): opt-in team_model_limit_is_floor turns a team's model_tpm_limit into a guaranteed share that borrows idle capacity
With priority_reservation_settings.team_model_limit_is_floor: true, a team's inherited model_tpm_limit on a model the dynamic rate limiter covers stops being a hard cap in the regular limiter. The dynamic limiter tracks it as a team_model_floor descriptor, lets the team past it while the model is below saturation_threshold, and holds the team to it once the model saturates, with rate_limited_by: team_model_floor on the 429. Default false keeps today's behavior. Key-owned model_tpm_limit, model_rpm_limit, the model-wide tpm and customer limits stay hard.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): name the setting team_model_limit_mode with hard and soft values
soft replaces the team_model_limit_is_floor flag. The 429 header reads rate_limited_by: team_model_soft_limit and the message says the team soft limit was reached.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams covers team model TPM and RPM
Replaces team_model_limit_mode. A team's inherited model_tpm_limit and model_rpm_limit on a model the dynamic
rate limiter covers are enforced by that limiter once the model saturates, so the team borrows idle capacity below
the threshold. The 429 names the limit and reads rate_limited_by: model_per_team_saturated.
Folds in the bot findings: key ownership is checked per requested model, team limits resolve through the model
alias like the regular limiter, the team counter is charged only for requests the pre-call hook tracked, every
response type that carries usage is settled, and tracking below saturation no longer adds a token per request.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the type gate flat in the dynamic limiter's team limit paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle responses usage and null auth metadata, resolve aliases for the borrowing team limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): key the borrowing team counter by resolved model group and read usage through get_usage_object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): hold Responses WebSocket turns to the borrowing team limit once the model saturates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): install the Rust gateway and fail forwarded_proxy when Rust is not in front
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): build the Rust gateway only for the management suites that run forwarded_proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject unknown guardrail names on key and user writes
Key generate, update, regenerate, team bulk update and /user/new now return 400 for a guardrail name that no loaded guardrail answers to and that has no active DB row, instead of storing it so requests run unguarded. Names the key already holds are not rechecked, and the key endpoints keep the enterprise license 403 ahead of the name check.
Resolves LIT-10613
* fix(proxy): read guardrail names from the writer when validating key writes
A guardrail created on another pod may not have reached a lagging read replica yet, so the fallback lookup now reads the writer and a just-created guardrail is not rejected as unknown. Other guardrail table reads keep using the reader.
* feat(proxy): validate guardrail names on team and project writes, reject malformed values loudly
Team and project create and update now return 400 for guardrail names that no loaded guardrail answers to and that have no active DB row, the same check key and user writes use. A metadata.guardrails value that is not a list of names is rejected even when it equals the stored value, with a short message naming the value. The lookup and the license-first wrapper move into common_utils so every entity shares them.
* refactor(proxy): one validate_guardrail_names entry point for guardrail name checks
The name check, its DB lookup and the license-first option now live in litellm/proxy/guardrails/guardrail_name_validation.py and every key, user, team and project write calls validate_guardrail_names directly. The lookup factory, both wrappers and the per-endpoint helpers are gone, and common_utils and guardrail_registry are back to their main versions. Behavior is unchanged.
* fix(proxy): chunk the guardrail lookup and leave legacy dict guardrail settings alone
The guardrail name lookup goes through find_many_in so its IN list has a written bound, and the project test keeps pytest.raises to one statement. A dict-shaped metadata.guardrails, a team's documented {"modify_guardrails": false} or the v1 {name: bool} toggle map, is read at request time, so validation leaves it alone instead of rejecting it.
* fix(proxy): only let the team modify_guardrails permission through as a dict
Stored key, team and project guardrails only run when metadata.guardrails is a list, so a dict such as {"config-guard": true} saved fine and attached nothing. The one stored dict with an effect is a team's documented {"modify_guardrails": bool}, which auth checks read, so that exact shape passes and every other dict gets the Invalid guardrail 400.
* refactor(proxy): move guardrail name validation into guardrail_helpers
validate_guardrail_names and its helpers now live in litellm/proxy/guardrails/guardrail_helpers.py next to the other guardrail helpers, and guardrail_name_validation.py is removed. guardrail_helpers imports LiteLLM_TeamTable and UserAPIKeyAuth from litellm.proxy._types instead of litellm.proxy.proxy_server, so the endpoints can import it at module level without pulling in the proxy server, and its dead sys.path line is dropped. Behavior is unchanged.
* feat(alerts): add default-on monitoring and a five-alert free allowance
* fix(alerts): handle monitoring races and retry-safe migration
* fix(ui): match monitoring upgrade link to status
* feat(ui): restore performance alerts on updated detector
* fix(ui): keep team key detail tabs independent
* fix(ui): preserve team tabs while permissions load
* feat(ui): show concise alerts in the notification bell
* refactor(ui): remove replaced deprecation bell item
* fix(ui): pin notification resource links
* fix(ui): show deprecation failures in notifications
* fix(ui): reset activity state when changing resources
* fix(ui): reset key detail state from every entry point
* feat(alerts): detect daily changes in usage for gateway, models, teams, users and keys
* fix(alerts): drop unused suppressions
* fix(alerts): satisfy strict lint for model usage timing
* fix(alerts): skip only resources that time out or hold invalid data
* fix(alerts): log an unfinished daily sweep at info level
* fix(alerts): keep the deployment-wide run every five minutes
* fix(alerts): record alert and sweep timestamps in UTC
* fix(alerts): repair detector tests and tracing
* refactor(alerts): build the key index in a migration instead of after startup
* refactor(alerts): build the key index after startup again
A concurrent index build inside the boot migration can exceed the 600 second migrate deploy deadline on a large LiteLLM_DailyUserSpend and stop the proxy from starting. This reverts 393db6e4e0.
* fix(alerts): keep keys pending, bound the accounting wait, resume unfinished scopes
Keys waiting for their index no longer mark the day complete. Only deliveries created during the evaluated day hold back its sweep. A day that did not finish restarts the next day from where it stopped instead of a hash boundary that never rotated non-hex IDs.
* refactor(alerts): drop the pending delivery wait, alerts are best effort on flushed usage
* test(alerts): cover detector persistence and recovery
* fix(alerts): recognize Prisma statement timeouts
* feat(proxy): run spend log cleanup from a DB-backed scheduled job table
Spend log retention ran from an in-process APScheduler timer that reset on
every restart and needed Redis for leader election, so pods that restart
more often than the interval never cleaned up at all.
Add LiteLLM_ScheduledJobRun, a generic table with one row per job run.
Every pod polls it, plans the next run idempotently and claims a due run
under a transaction-scoped advisory lock, so exactly one pod runs each
occurrence. Runs carry a lease, a status and a log. Intervals count from
the last finished run and survive restarts. Schedule edits cancel and
replace the pending run. The retention and size-check jobs share a lock
group so they never overlap, and the Redis pod lock is gone.
* fix(proxy): plan an earlier cron edit from now, and cover the job runner
Moving the cleanup cron earlier queued tomorrow's run because the next
occurrence was planned after the pending run being cancelled. Plan from
finished runs only.
Register LiteLLM_ScheduledJobRun as a span table, keep the advisory lock
query inside the tick's span, move the non-string cron startup test to
the schedule parser, and add unit tests for the runner and job wiring.
* test(proxy): annotate new spend log cleanup test locals as Final
* test(proxy): type the parameters of the new spend log cleanup tests
* test(proxy): leave an hour of margin when moving a cron earlier so an hour boundary cannot flake it
* fix(proxy): reject a cron that never fires instead of failing every tick
A crontab like 0 0 31 2 * parses but has no next fire time, so next_run_at raised on every tick and the job was never scheduled again. cron_schedule now returns InvalidSchedule for it, which logs and keeps the pending run like any other bad config
* refactor(proxy): drop the informational log column from LiteLLM_ScheduledJobRun
Nothing read the column back. Run outcomes stay in status, and the reasons
already go to the proxy logs, so JobResult becomes a SUCCEEDED or FAILED
literal and the cleanup returns its RunOutcome instead of a summary report
