* feat(proxy): publish opted-in pass-through endpoints on the Model Hub
Pass-through endpoints gain display_name and show_in_model_hub. Endpoints that opt in appear on
/public/v1/model_hub and /public/model_hub as rows in mode passthrough, named by their display name
or their path, with the route in pass_through_path. The admin UI exposes both fields on the create
and edit forms and the public hub renders passthrough rows with a route-specific curl example
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): carry pass-through methods onto hub rows and key dashboard rows by route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep health, cost and shell quoting correct for pass-through hub rows
Pass-through rows no longer inherit a router model's health check when they share its name, the public table shows n/a instead of Free for their token costs, and the generated curl example escapes shell characters in the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): quote the method in the pass-through hub curl example
A configured method string is placed after curl -X inside escaped double quotes so it cannot alter the copied command
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): build pass-through hub fixtures through a helper
Keeps the dashboard lint budget for inline object arguments under its limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit the pass-through Model Hub surface end to end
Cover forwarding, pagination, validation, auth, updates, duplicate names,
config-file precedence, health-name isolation, repeated reads and two chaos
cells (create/delete churn, a killed proxy worker). Build the stored
pass_through_endpoints list in one shot instead of appending to the list
the config reader handed back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep concurrent pass-through creates from overwriting each other
Creating an endpoint appends to the general_settings list the proxy already holds instead of persisting a private copy, so two admins creating at the same time both land, as on main before this branch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): serialize pass-through endpoint creates behind a process lock
Each create copies the stored list under an asyncio lock before persisting, so simultaneous creates in one worker no longer overwrite each other and a rejected write (config-owned path) never lands in the in-memory settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): collect paginated pass-through rows without mutating a list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): let a pass-through with no stored timeout save from the edit form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the pass-through edit fixture so a null timeout compiles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): read published pass-throughs from the in-memory settings store
The public Model Hub routes read the pass-through list this worker already
serves from proxy_config.settings instead of querying the config table on
every anonymous hub read. Also drops a mutable-ok suppression that no longer
guards a violation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): list a config file pass-through beside database rows on the hub
Main merges database pass-throughs with config file ones on other paths since
#43962, so a database write beside a config entry is accepted rather than
rejected as config-owned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the monkeypatch fixture in the pass-through hub tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): validate pass-through config entries instead of splatting unknown dicts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): list model hub rows as a tagged union of model groups and pass-throughs
Health enrichment and the no-router check now match on ModelGroupRow or
PassThroughRow instead of testing pass_through_path in each route. Both hub
routes share published_hub_rows, the v1 row helper is sync again, and the
config list route goes back to main's inline merge since the hub no longer
needs configured_pass_through_endpoints
* test(ui): find the pass-through hub row by role to stay under the no-node-access budget
* fix(proxy): keep database pass-through endpoints in the settings store across config reloads
A config reload cleared the stored database row before rebuilding the
pass-through list, so without store_model_in_db the list fell back to the
config file entries alone. The Admin UI triggers that reload on page load
through /get/config/callbacks, which dropped UI-created endpoints from the
Model Hub until the next write
* refactor(proxy): make the hub row matches exhaustive with assert_never
* refactor(proxy): return the exhaustive match fallthrough and build hub test inputs in one shot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop the in-process create lock from pass-through endpoints
The lock only covered one worker, and the read-modify-write on the general_settings blob is a separate storage problem tracked as tech debt. Create goes back to main's behavior.
* fix: align pass-through model hub with current lint rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: share worker state for pass-through hub restart coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: show pass-through curl examples in model hub inspector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: skip invalid pass-through hub entries and await peer refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* fix(token-counter): count Anthropic server tool history
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): bound nested results and update proxy coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(token-counter): include readable web search snippets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model
Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false).
When on, the class weights are renormalized over the classes with a team active on the model in the
current window, each class share is divided equally among its active teams, and once the model saturates
each team is held to its own share with a 429 whose rate_limited_by header reads
priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL.
* fix(proxy): keep basedpyright counts flat for the priority class share setting
PriorityReservationSettings is built with model_validate instead of **value, so a new
field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers
type the handler by its implementation class.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way
A registry failure falls back to the class pool instead of skipping the model limits, a team's
counter is keyed by model and team so keys of different priorities spend one share, Responses
WebSocket turns are held to the class share like the team model limit, a share is floored at one
request and one token, and the registry declares the rate limits key family.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share
A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated
Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.
* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates
* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names
* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails
* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests
* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver
An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.
* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take
The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.
* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates
While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.
* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget
While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity
With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could
use everything up to the model's own tpm and rpm while the model was idle.
Setting team_model_limit_max_borrow_multiple in the team's metadata caps the
team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429
whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset
keeps the previous behavior. Values below 1 or non-numeric are logged and
ignored.
* fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check
* fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): opt-in team_model_limit_is_floor turns a team's model_tpm_limit into a guaranteed share that borrows idle capacity
With priority_reservation_settings.team_model_limit_is_floor: true, a team's inherited model_tpm_limit on a model the dynamic rate limiter covers stops being a hard cap in the regular limiter. The dynamic limiter tracks it as a team_model_floor descriptor, lets the team past it while the model is below saturation_threshold, and holds the team to it once the model saturates, with rate_limited_by: team_model_floor on the 429. Default false keeps today's behavior. Key-owned model_tpm_limit, model_rpm_limit, the model-wide tpm and customer limits stay hard.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): name the setting team_model_limit_mode with hard and soft values
soft replaces the team_model_limit_is_floor flag. The 429 header reads rate_limited_by: team_model_soft_limit and the message says the team soft limit was reached.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams covers team model TPM and RPM
Replaces team_model_limit_mode. A team's inherited model_tpm_limit and model_rpm_limit on a model the dynamic
rate limiter covers are enforced by that limiter once the model saturates, so the team borrows idle capacity below
the threshold. The 429 names the limit and reads rate_limited_by: model_per_team_saturated.
Folds in the bot findings: key ownership is checked per requested model, team limits resolve through the model
alias like the regular limiter, the team counter is charged only for requests the pre-call hook tracked, every
response type that carries usage is settled, and tracking below saturation no longer adds a token per request.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the type gate flat in the dynamic limiter's team limit paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle responses usage and null auth metadata, resolve aliases for the borrowing team limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): key the borrowing team counter by resolved model group and read usage through get_usage_object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): hold Responses WebSocket turns to the borrowing team limit once the model saturates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject unknown guardrail names on key and user writes
Key generate, update, regenerate, team bulk update and /user/new now return 400 for a guardrail name that no loaded guardrail answers to and that has no active DB row, instead of storing it so requests run unguarded. Names the key already holds are not rechecked, and the key endpoints keep the enterprise license 403 ahead of the name check.
Resolves LIT-10613
* fix(proxy): read guardrail names from the writer when validating key writes
A guardrail created on another pod may not have reached a lagging read replica yet, so the fallback lookup now reads the writer and a just-created guardrail is not rejected as unknown. Other guardrail table reads keep using the reader.
* feat(proxy): validate guardrail names on team and project writes, reject malformed values loudly
Team and project create and update now return 400 for guardrail names that no loaded guardrail answers to and that have no active DB row, the same check key and user writes use. A metadata.guardrails value that is not a list of names is rejected even when it equals the stored value, with a short message naming the value. The lookup and the license-first wrapper move into common_utils so every entity shares them.
* refactor(proxy): one validate_guardrail_names entry point for guardrail name checks
The name check, its DB lookup and the license-first option now live in litellm/proxy/guardrails/guardrail_name_validation.py and every key, user, team and project write calls validate_guardrail_names directly. The lookup factory, both wrappers and the per-endpoint helpers are gone, and common_utils and guardrail_registry are back to their main versions. Behavior is unchanged.
* fix(proxy): chunk the guardrail lookup and leave legacy dict guardrail settings alone
The guardrail name lookup goes through find_many_in so its IN list has a written bound, and the project test keeps pytest.raises to one statement. A dict-shaped metadata.guardrails, a team's documented {"modify_guardrails": false} or the v1 {name: bool} toggle map, is read at request time, so validation leaves it alone instead of rejecting it.
* fix(proxy): only let the team modify_guardrails permission through as a dict
Stored key, team and project guardrails only run when metadata.guardrails is a list, so a dict such as {"config-guard": true} saved fine and attached nothing. The one stored dict with an effect is a team's documented {"modify_guardrails": bool}, which auth checks read, so that exact shape passes and every other dict gets the Invalid guardrail 400.
* refactor(proxy): move guardrail name validation into guardrail_helpers
validate_guardrail_names and its helpers now live in litellm/proxy/guardrails/guardrail_helpers.py next to the other guardrail helpers, and guardrail_name_validation.py is removed. guardrail_helpers imports LiteLLM_TeamTable and UserAPIKeyAuth from litellm.proxy._types instead of litellm.proxy.proxy_server, so the endpoints can import it at module level without pulling in the proxy server, and its dead sys.path line is dropped. Behavior is unchanged.
* feat(alerts): add default-on monitoring and a five-alert free allowance
* fix(alerts): handle monitoring races and retry-safe migration
* fix(ui): match monitoring upgrade link to status
* feat(ui): restore performance alerts on updated detector
* fix(ui): keep team key detail tabs independent
* fix(ui): preserve team tabs while permissions load
* feat(ui): show concise alerts in the notification bell
* refactor(ui): remove replaced deprecation bell item
* fix(ui): pin notification resource links
* fix(ui): show deprecation failures in notifications
* fix(ui): reset activity state when changing resources
* fix(ui): reset key detail state from every entry point
* feat(alerts): detect daily changes in usage for gateway, models, teams, users and keys
* fix(alerts): drop unused suppressions
* fix(alerts): satisfy strict lint for model usage timing
* fix(alerts): skip only resources that time out or hold invalid data
* fix(alerts): log an unfinished daily sweep at info level
* fix(alerts): keep the deployment-wide run every five minutes
* fix(alerts): record alert and sweep timestamps in UTC
* fix(alerts): repair detector tests and tracing
* refactor(alerts): build the key index in a migration instead of after startup
* refactor(alerts): build the key index after startup again
A concurrent index build inside the boot migration can exceed the 600 second migrate deploy deadline on a large LiteLLM_DailyUserSpend and stop the proxy from starting. This reverts 393db6e4e0.
* fix(alerts): keep keys pending, bound the accounting wait, resume unfinished scopes
Keys waiting for their index no longer mark the day complete. Only deliveries created during the evaluated day hold back its sweep. A day that did not finish restarts the next day from where it stopped instead of a hash boundary that never rotated non-hex IDs.
* refactor(alerts): drop the pending delivery wait, alerts are best effort on flushed usage
* test(alerts): cover detector persistence and recovery
* fix(alerts): recognize Prisma statement timeouts
* feat(proxy): run spend log cleanup from a DB-backed scheduled job table
Spend log retention ran from an in-process APScheduler timer that reset on
every restart and needed Redis for leader election, so pods that restart
more often than the interval never cleaned up at all.
Add LiteLLM_ScheduledJobRun, a generic table with one row per job run.
Every pod polls it, plans the next run idempotently and claims a due run
under a transaction-scoped advisory lock, so exactly one pod runs each
occurrence. Runs carry a lease, a status and a log. Intervals count from
the last finished run and survive restarts. Schedule edits cancel and
replace the pending run. The retention and size-check jobs share a lock
group so they never overlap, and the Redis pod lock is gone.
* fix(proxy): plan an earlier cron edit from now, and cover the job runner
Moving the cleanup cron earlier queued tomorrow's run because the next
occurrence was planned after the pending run being cancelled. Plan from
finished runs only.
Register LiteLLM_ScheduledJobRun as a span table, keep the advisory lock
query inside the tick's span, move the non-string cron startup test to
the schedule parser, and add unit tests for the runner and job wiring.
* test(proxy): annotate new spend log cleanup test locals as Final
* test(proxy): type the parameters of the new spend log cleanup tests
* test(proxy): leave an hour of margin when moving a cron earlier so an hour boundary cannot flake it
* fix(proxy): reject a cron that never fires instead of failing every tick
A crontab like 0 0 31 2 * parses but has no next fire time, so next_run_at raised on every tick and the job was never scheduled again. cron_schedule now returns InvalidSchedule for it, which logs and keeps the pending run like any other bad config
* refactor(proxy): drop the informational log column from LiteLLM_ScheduledJobRun
Nothing read the column back. Run outcomes stay in status, and the reasons
already go to the proxy logs, so JobResult becomes a SUCCEEDED or FAILED
literal and the cleanup returns its RunOutcome instead of a summary report
* fix(gemini): attribute /gemini batchGenerateContent batches to the calling key
A 200 from models/{model}:batchGenerateContent through the Gemini pass-through now registers a managed
batch row for the calling key, team and tags, so CheckBatchCost prices the output file on completion
instead of the job landing as a $0 pass_through_endpoint row with model unknown.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gemini): unit regression for pass-through batch registration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): file batch created through /gemini is registered for the calling key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): bill pass-through batches through the configured gemini deployment
The managed row now points at the gemini/<model> deployment CheckBatchCost fetches the output with, not
the first router id for the model name, and a model with no gemini deployment logs a warning instead of
registering a row that can never be billed. A key without a user keeps user_id unset rather than
default-user. Typed the registration helper and handler so the lint gates stay flat.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill anthropic pass-through calls at the deployment's custom price
The /anthropic route never attached a router deployment's model_info, so
spend fell back to the shared cost-map key, which carries no per-deployment
prices. Attach the model_info of the one deployment in the requested group
that serves that exact Anthropic model and carries custom pricing, so
streaming and non-streaming calls, and dated snapshot responses, bill at it.
Fixes#49596
* fix(proxy): validate deployment rows before reading anthropic pass-through pricing
Read the untyped router rows and request body through a pydantic adapter so
the pricing lookup adds no basedpyright unknown-type errors.
* test(proxy): drive the real anthropic pass-through relay when asserting the deployment price
Stub only the upstream http client and assert the x-litellm-response-cost the
route returns, instead of doubling create_pass_through_route.
* fix(proxy): match an anthropic pass-through deployment by its configured or resolved model name
get_llm_provider rewrites -latest aliases, so a deployment configured with the
alias the caller sends never matched and lost its custom price.
* fix(proxy): price an anthropic pass-through call by the model its custom_body forwards
The relay forwards custom_body when the caller sends one, so resolving the
deployment from the outer model let a caller bill an expensive call at a
cheaper deployment's price.
* fix(proxy): only price an anthropic pass-through call with a deployment the caller's team may use
A teamless key could pick up a team-owned deployment's price because the group
lookup does not filter by team when the key has none.
* fix(proxy): resolve anthropic pass-through pricing from the body forwarded after pre-call hooks
The route stamped the deployment price before the pass-through ran its
pre-call hooks, so a hook that swapped the model billed the original
model's deployment. The route now hands the relay a resolver, and the relay
applies it to the final forwarded body, which already reflects the
custom_body envelope and every hook.
* refactor(proxy): clear pass-through request state keys in one helper
Keeps create_pass_through_route within the C901 ceiling now that the
deployment pricing resolver adds a fourth request.state key.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(auth): allow paid video retrieval after budget exhaustion
* test(auth): check video retrieval boundary on registered route
* fix(auth): exempt video retrieval in post-custom-auth checks
* style(auth): format custom video budget exemption
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(guardrails): add the CollieAi guardrail integration
* refactor(guardrails): tighten the CollieAi wrapper and its tests
The loader takes its importer as a parameter, so a test covers the fallback for a package that is too old, and the install hint names the Python the package needs
The config model is frozen, the Garden preset's phases and default_on are pinned by a test, and comments the code already says are gone
* fix(guardrails): use the generated Collieai provider key in the dashboard
The dashboard builds provider keys from the proxy's provider list, so collieai becomes Collieai. The static CollieAi key came first in the provider map, which left saved CollieAi guardrails without their display name and logo. The map, the Garden preset and the card now use Collieai, and a test covers the lookup
The preset expectation is a named value now, which keeps the dashboard under its inline object lint budget
* refactor(guardrails): drop the CollieAi wrapper's function docstrings
AGENTS.md keeps comments to complex logic and tool input, and these only restated what each function returns
* fix(guardrails): declare CollieAi's phases, read COLLIEAI_API_KEY and plate its logo
The dashboard offered every mode for CollieAi, and a guardrail saved with a mode the package refuses was reported as created but never loaded. The wrapper now declares pre_call and post_call, so the form offers only those
api_key falls back to COLLIEAI_API_KEY as its description says, and the navy logo gets a plate in dark mode like DeepKeep's
The tests now also cover an entry's own name, phase and default_on, the key fallback, and the modes the dashboard offers
* fix(guardrails): reject CollieAi guardrails the package refuses and stop env-keyed rebuilds
A Garden create whose key or settings the package refuses now answers 400 and
saves nothing, instead of saving a guardrail that checks nothing. Every setting
on the entry now reaches the package, like the class-path route. DB guardrails
keyed with an os.environ/ reference are no longer rebuilt on every DB poll.
The Garden card says which calls CollieAi checks.
* fix(guardrails): load CollieAi stream_scope and import errors as setup errors, default the Garden card off
Pass stream_scope into the package constructor inside the setup-error wrap so a refused scope answers 400 on create and stops a config-file proxy at startup. Drop settings cleared to null so the package default applies. Say an installed package failed to load, with the error, instead of asking to install it, and keep any import-time exception from breaking the guardrail routes. Make api_key optional in the form since COLLIEAI_API_KEY is the fallback, and open the Garden card with Always On off like the other partner presets.
* fix(guardrails): leave CollieAi form fields empty and share one guardrail setup error
The Garden form no longer pre-fills api_base, deadline_ms or forward_conversation_id, so a
create saves only what the admin typed and the package keeps its own defaults. Setup failures
raise GuardrailSetupError from custom_guardrail, the missing-package hint covers the Docker
image, and the card copy no longer repeats the package's eligibility rules.
* test(guardrails): cover env-reference api_base in the DB poll comparison
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): wait for child readiness before supervisor cancellation
* test(proxy): sync supervisor cancellation test on a socketpair handshake
Replace the readiness marker poll and timed sleeps with one-byte socketpair
messages. The child reports its SIGTERM handler is installed, reports the
stop, and holds its exit until the parent releases it, so the assertion only
passes when the cancelled supervisor waits for the child. A pytest-timeout
marker bounds a deadlock.
* test(proxy): drive supervisor cancellation cleanup with a controlled child
Replace the real child and socket handshake with a scripted process returned
by asyncio.create_subprocess_exec and a stop observed through os.killpg. The
child only exits after the stop and a checkpoint, so returncode is 0 only when
the cancelled supervisor stops the child and waits for it without being
cancelled.
* test(proxy): start the supervisor cancellation only once the reporter waits
The scripted child signals when a wait begins, and the test cancels only after
the exit reporter is blocked on the stop, so cancellation reaches it without
relying on task start order.
* test(proxy): await background failure spend counters before asserting
Failure spend counters finish in a background task when the counter store is
slower than the wait budget, so the tests read stale values on a slow runner.
Hold the counter store on a release event with a zero wait budget, capture the
background task, and await it before asserting the counters.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(fips): hash MCP advisory-lock keys with SHA-256 in fips_mode and compute the OCI body digest as a security hash
* test(fips): move the lock-key tests into test_mcp_env_vars.py and type the OCI digest test
* feat(bedrock): search knowledge bases by image and return image hits
vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins
Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream
A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search
Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits
* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery
Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image
A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked
A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch
* fix(vector-stores): redact query images and keep store search params out of image checks
* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches
* fix(vector_stores): keep a caller's image query through store and router params
A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off
* fix(cost): leave image search responses out of the cost error text
* fix(vector_stores): accept line-wrapped base64 in query_image data
Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent
* fix(vector_stores): drop rewrite_query false from the chat file_search tool
False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): price connection tests using their resolved mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): avoid deployment pricing on alternate probes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): preserve pricing for equivalent model names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid authentication when comparing health check models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover health probe pricing without a deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): pin the clock in the rpm rate limit test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(messages): apply modify_params rewrites on rust
* feat(messages): mint anthropic workload identity tokens on rust
* feat(messages): apply prompt caching injection on rust
* feat(messages): serve the easy python-only detectors on rust
* feat(messages): retry rejected thinking signatures on rust
* feat(messages): serve response cache reads on rust
* feat(messages): retry rejected strict tools on rust
* refactor(messages): delete the python path for anthropic messages
Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust
without LITELLM_RUST or litellm.rust(True) and never falls back to Python.
A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a
detector that raises) gets UnsupportedParamsError naming the feature, raised
from one place in litellm/messages/dispatch.py. Other providers keep the
Python fallback. A call whose arguments the handler signature rejects still
gets Python's TypeError.
Deleted Python code that only served anthropic Messages:
- workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses
were never eligible; the Rust host mints the token itself)
- the anthropic Foundry beta-row and affinity-base branch in
async_anthropic_messages_handler, and the "forward" mode of the beta filter
that only that branch used
The proxy MCP tools detector now uses the light tool precheck so a function
tool never imports the gateway handler, since a raising detector now refuses.
* feat(messages): forward message threads on rust
The Rust path for anthropic /v1/messages dropped the thread body field, so
a create or continue turn reached Anthropic without its thread or the
message-threads-2026-08-12 beta. It now matches the Python handler from
#50976:
- the host forwards thread with the other body params
- a non-null thread adds the message-threads beta
- a well-formed continue turn keeps advisor blocks in history and derives
no compaction, per-turn-control or tool-change beta from its messages
- the recorded httpx_response carries the request that was sent, for JSON
and streamed calls, so the proxy's thread ownership hook resends the
create turn's betas on a continue as it does for Python
Upstream error.details already reached the exception message unchanged, so
thread_not_found relays and keeps the deployment out of cooldown on both
paths. Parity cases cover create, continue, streaming, sync, retries,
thread_not_found and the ownership hook round trip.
* feat(messages): serve mock responses on rust
* feat(messages): send caller clients and unsupported tls through python on rust
* feat(messages): run pre-request hooks on rust
* feat(messages): run agentic loop hooks on rust
* feat(messages): run the safeguards classifier on rust
* feat(messages): run registered interceptors on rust
* fix(messages): match python stream cache gate, logging and interceptor order on rust
Store a streamed Messages response in the cache only when Python would: a
message_stop event line, no provider error frame and at least one frame with
output. Leave dict results untouched by mark_rust_response so hidden params
never reach result dicts, logging objects or hook digests. Ask interceptors
whether they handle the call after the cache lookup, not at decode time, so a
cache hit never calls can_handle. Emit the track_llm_api_timing service event
for the provider exchange. Bill and log a stream superseded by a router
fallback as the failure of its error frame, with the partial usage warning.
Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the
autorouter baseline cache sees retried requests.
* fix(messages): forward python-only json values to anthropic as python does
Accept any max_tokens, sampling scalar and messages collection that
Python main accepts, and carry values serde_json cannot hold (ints beyond
64 bits, NaN and infinities, strings with lone surrogates) as validated
raw JSON leaves written back verbatim on the wire, so the upstream body
matches Python's json.dumps
* fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust
* fix(messages): log python's plain provider header names on rust streams
* fix(messages): match python's request params, logged params and validation on rust
Read a non-bool stream by Python truthiness and send it as given, log the
caller's stream flag as Python's handler does, raise the raw Python errors
main raises for malformed messages, tools, model_info and hook-set
litellm_params or providers, read the tool custom-field drop from
drops_unsupported_params, log the caller's nested fields that
additional_drop_params strips and the Python values of raw JSON leaves,
keep a missing stop_reason absent, and map provider-config rejections
(missing max_tokens, unknown reasoning effort) as Python's HTTP handler
maps them, with Python's text, effort order, model and provider
* fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust
* fix(messages): match python on hooked streams, empty cache writes and mcp cache hits
* fix(messages): join already buffered stream reads and trim the env api key as python does on rust
* fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust
A stream cut by a transport error now hands the caller the bytes already
buffered before raising, as aiohttp does for a caller that reads them before
the event loop sees the cut, so the frames and their usage reach the caller and
the failure log. Provider replies keep integers wider than 64 bits exactly and
read numbers that overflow a float as infinity, through the response, the
response cache and the logged result. The provider client follows up to 20
redirects as httpx does, and running out raises Python's text. A
provider_specific_header value the ascii codec cannot encode fails before the
send like a caller extra header. The deployment success hook gets the stream as
AnthropicMessagesStreamingResponse, as Python's handler hands it.
* feat(messages): serve proxy mcp tools on rust
* test(messages): cover hook, agentic and interceptor edge cases in the parity suite
* fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust
* fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust
A header value Python's client cannot encode as ASCII now fails before the send
whichever header carries it, so a caller api_key with a non-ASCII character
raises Python's InternalServerError instead of a raw UnicodeEncodeError after
the reply. A caller api_base without an http(s) scheme, or with whitespace,
control or non-ASCII characters, is sent through Python's client at the URL
Python builds, so its relative-URL and invalid-URL errors match, and a base
urlsplit cannot split raises Python's bare ValueError before the send. A
provider_specific_header of the wrong shape is scoped through Python's
ProviderSpecificHeaderUtils first, so a plain string raises Python's
AttributeError.
* fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust
temperature and top_p now deserialize through recognized::exact_float, which keeps an integer
where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent
upstream and logged as the integer python forwards, and comparisons still go through scalar.rs
cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python
cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between
instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort
reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python
TypeError marked as a late rejection, and map_failure hands it to the caller unmapped
* test(messages): format the non-string tool_result parity helper
* fix(messages): flag and log the call before python's pre-send raises on rust
The bridge now hands settings() and map_failure() whether the call is async.
A provider_specific_header of the wrong shape sets the aanthropic_messages
flag before re-raising, as Python's handler has already flagged the call. A
base urlsplit cannot split also logs the resolved model, provider and optional
params first, as Python raises it only after its logger is updated. The
TypeError raised while capping the thinking budget for a string max_tokens
sets the flag too
* feat(messages): serve the web search short-circuit override on rust through the intercept host call
* fix(messages): give the web search entry kwargs its mutable-ok reason
* test(messages): drop the body hook gap the rust short-circuit closes
* fix(messages): name the partially unknown handler types on the short-circuit host call
* feat(guardrails): write guardrail changes to the audit log
Creating, updating, patching or deleting a guardrail, and registering, approving or rejecting a team guardrail submission, now each write one row to LiteLLM_AuditLog with the guardrail before and after the change. Credentials the guardrails table encrypts at rest are redacted from those rows, and the routes take the litellm-changed-by header like key, team and model routes do. The Admin UI audit page gets a Guardrails table filter.
Resolves LIT-10639
* fix(guardrails): audit an allowlisted guardrail snapshot and drop litellm-changed-by
Guardrail audit rows now keep the guardrail id, name, info, status, team and the provider, mode and default_on settings, and name (never show) any other provider setting an update changed. Provider settings can hold credentials under any name, for example an apikey header, so no other value is written. The changer always comes from the caller's API key, so the litellm-changed-by header is gone. PUT audits the saved guardrail instead of the copy the live sync adds headers to.
Resolves LIT-10639
* fix(guardrails): never let the audit snapshot fail a saved guardrail change
The audit snapshot now reads guardrail settings stored as a JSON string and records empty settings when it cannot read them, so a saved approve or reject can no longer turn into a 500. Only the provider, mode and default_on settings are ever copied, so this cannot leak a credential. The create entry also records the active status the new guardrail is saved with.
Resolves LIT-10639
* test(guardrails): run the guardrail audit tests through the real registry and in-memory handler
The audit tests no longer replace GUARDRAIL_REGISTRY or the in-memory handler, and no test edits its inputs. Only the Prisma guardrails table is faked, returning exactly what the registry writes, so the PUT test now checks the real live sync adding x-api-key to the saved guardrail's headers.
Resolves LIT-10639
The test spawned a real Python child that installs a SIGTERM handler, slept
0.3s, then cancelled the supervisor and expected the child to exit 0. CI runs
with --cov, and pytest-cov starts coverage in every spawned Python process,
so the child is often still starting when SIGTERM lands and dies with -15.
open_children now takes the process spawner the same way it takes the
platform, and only needs the platform to signal and stop children. The test
uses a fake child whose wait() blocks on an event and a fake platform that
records stop_group, so it checks the shielded stop-then-wait with no real
process and no sleep. It also checks the child is still being waited on,
because anyio's wait_all_tasks_blocked on asyncio can return one loop tick
before a task group finishes.
* fix(fallback): serialize fallback writes and wait for replicas on read-back
POST and DELETE /fallback now read, edit and write router_settings in one
writer transaction under the same per-param advisory lock the config section
writer takes, so concurrent writes on any replica no longer drop each other's
rules. The cache is invalidated and the router updated only after commit.
The Terraform provider now polls the read after a litellm_fallback create or
update until the proxy serves the written list, within new create/update
timeouts, instead of clearing the ID on a stale 404 or saving a stale list.
Refresh 404 still removes the resource, and error responses are not retried.
Fixes#47888
* fix(fallback): bound provider requests by the apply context and inject the fallback repository
The provider's fallback write and read-back requests now carry the apply
context through a context-aware MakeRequest variant, so a cancelled apply
sends nothing and the create/update timeout also ends a hung request.
The POST and DELETE /fallback routes take a RouterFallbackRepositoryDep that
returns a RouterFallbackStore. Router settings parsing moved into
RouterFallbackRepository, which edits the rules through the locked
ConfigRepository operation.
Unit tests now check the outcome of the race and rollback through a fake
lock instead of pinning the lock SQL. They also cover a racing config
section write, a 404 delete inside the lock, route resolution of the
repository and the no-database path.
* refactor(fallback): keep atomic edits in the config repository
* fix(terraform): release hung test handlers and audit MakeRequestWithContext calls
The hung-request cancellation test now releases its handler through a test
owned channel before closing the server, since the server does not notice
the client's cancel while the POST body is unread.
The endpoint audit now reads the method and path of MakeRequestWithContext
calls from their shifted argument positions, and skips only MakeRequest's own
forwarding call inside the helper files, so POST /fallback counts as covered
again and dynamic paths elsewhere still fail closed.
* test(fallback): keep stored-rule regressions with their helpers
* fix(fallback): read saved rules without Terraform changes
* fix(fallback): preserve valid rules when stored entries are malformed
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
The API Reference page was already behind a deprecation banner. This deletes
the /ui/api-reference route and its components, the Developer Tools sidebar
entry and page description, the legacy ?page=api_ref redirects, and the i18n
strings only that page used.
Old ?page=api_ref and ?page=api-reference bookmarks now fall through to the
dashboard like any other unknown page id, since there is no replacement page.
Tests that listed the page are updated: the sidebar and legacy-route vitest
files, the uiHref examples, the e2e sidebar fixtures and Internal Viewer nav
check, and the UI path detection unit test.
* feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429
* test(rate-limit): expect rate_limited_by in the exact-headers assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer x-litellm-call-id on every token count response
The token count log line names a litellm_call_id, but count responses
never carried it, so a client that sent no id of its own could not find
its count's log line. Every count route (Anthropic Messages, Responses
input_tokens, Gemini countTokens, Bedrock count-tokens, and
/utils/token_counter) now answers the id on success and on errors, and
a crash answers the JSON 500 the Anthropic and Gemini routes already used.
* fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies
A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the
gateway's generic 500, or 503 when the database is down, now with the
x-litellm-call-id header. Responses count rejections carry error.litellm_call_id
when include_call_id_in_error_body is on
* test(proxy): count through count_request_tokens in the token counter loop tests
The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise
* fix(proxy): keep a count crash's own exception and log pre-count Responses failures
The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line
* fix(proxy): keep the token counter's injected args out of its API docs
Tests the opt-in error body id on an Anthropic count the proxy rejects before counting
* fix(proxy): keep a count route's own HTTP error when adding the call id
The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused
/customer/delete only dropped the committed twin, so spend:end_user:<id> kept the old spend and a
re-created customer with the same id stayed over budget. Zero the counter and its DB floor marker on
every worker after the delete commits, the same way a key spend reset does.
* fix(proxy): drop a deleted customer's spend counter before zeroing it
A Redis that rejects writes but still serves reads, such as one at its memory limit, kept the old
counter when the zeroing SET failed, so the re-created customer stayed refused. Delete the counter
first, which Redis still allows in that state, then write zero and the floor.
* test(proxy): type the customer delete spend counter test parameters
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys
A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.
Resolves LIT-9545
* fix(otel): refuse request headers in a destination's baggage_metadata_keys
The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.
* fix(otel): hold destination metadata to the configured span limits
A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.
The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.
* fix(otel): apply destination metadata after the destination view, within the provider's limits
The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.
* refactor(otel): consolidate span limits, destination validators and metadata key parsing
Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.
* ci: retrigger checks stranded by the GitHub Actions outage
* fix(proxy): stream the guardrail block response when the block lands after an SSE keepalive ping
* docs(proxy): keep the keepalive docstring's reasons together
* fix(proxy): stream the /v1/messages block after a keepalive ping and always end a late block stream
A passthrough guardrail block on a streamed /v1/messages request that landed
after the first keepalive ping still ended as an api_error event. The route now
hands its block stream to the keepalive relay, as chat and text completions do.
If a route's block stream itself fails after a ping, the relay sends the usual
late failure frame instead of raising, so the stream always ends.
The passthrough tests now drive the real routes with a real guardrail.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
