Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to
ModelInfo so a model deployment can carry the inputs for provisioned-throughput
flat-cost attribution. ModelInfo validates per-field bounds (positive count,
non-negative rate, effective_to after effective_from); model/new and
model/{id}/update enforce the cross-field invariant (count and rate set together,
team_id required) on the effective model_info so partial updates validate the
merged result, and v1/model/info returns the fields.
LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a
sentinel api_key constant; the daily rollup that writes them lands in a follow-up
PR. Adding the optional model_info fields is backward compatible; models without
them are unaffected.
ptu_effective_from is required alongside the count and rate rather than optional. Flat
cost accrues from that instant, so an absent start has to be inferred, and inferring it
let a deployment configured today be billed for days it did not exist. Both PTU validators
also run over the merged view before any write on the update path, beside the premium check the create path
already runs there: the team ACL update below autocommits, so a validator raising further
down left the team mutated and the deployment row never written.
The update path validates the model_info a patch would store rather than the patch
alone. An invariant holds over the deployment as it will exist, not over whichever
subset of fields a caller sent, and validating the patch rejected raising the rate on
an already configured model because that patch carries no start of its own.
Zero-estimate cap bypass (cursor): a key quiet during the estimate
lookback gets cost_estimate 0.0, which the spend cap treated the same
as 'no estimate' and left uncapped — a later traffic spike on exactly
that job would bill until ends_at. Only a NULL estimate (rows predating
estimates) is uncapped now; $0 still gets the $1 floor.
Concurrent-start race (cursor): the find_first-then-create check passes
on both sides of a race, giving one key two active jobs and double
judge spend. A partial unique index (api_key_id WHERE status IN
(pending, running)) — raw SQL in the unshipped migration, since
schema.prisma cannot express partial indexes — makes the DB the
arbiter; the losing create surfaces as the same 409 as the advisory
check. The old find_many('desc') + reversed() insertion in the logger
cache already prefers the newest job for any legacy duplicates.
Request-path DB read (greptile): an expired job snapshot awaited
find_many inside the success callback, so every N seconds one request
per pod paid a synchronous Prisma read. The lookup is now sync-only:
it serves the current snapshot and kicks a detached refresh task when
stale. Cost: a cold pod's first ~1 refresh-window of samples are
missed (acceptable for a sampled eval); stale-if-error semantics keep
a DB blip from disabling the feature.
Also from review: a collapsed previous-job row said 'no verdicts' for
jobs with thousands of verdicts, because the list endpoint omits
results by design — it now says 'view results' when completed_count>0.
Co-Authored-By: Claude <noreply@anthropic.com>
A poll of a Vertex passthrough batch wrote nothing to the managed-object row,
so status and file_object stayed frozen at the create-time snapshot and
GET /v1/batches served a stale status and an empty output file id for the life
of the batch. Only the create may claim a batch, but every observation of one
may refresh its state.
store_unified_object_id takes create_if_missing, which the poll clears: it
refreshes status and file_object through update_many, and leaves a row that is
absent absent rather than creating one owned by the observer, since created_by
and team_id are written by whoever reaches the create branch. The update payload
is now shared with the upsert so it cannot drift into writing api_key,
request_tags, created_by or team_id.
The passthrough identity re-assertion that was previously part of this PR ships
separately in #36121, so this PR keeps only the batch attribution work.
The creating key owns user_api_key_alias only when it actually has one. Guarding
the overwrite on the presence of a key rather than on a resolved alias nulled the
field out for every key generated without key_alias, and for any key rotated or
deleted before its batch finished, losing the creating user's alias that the spend
row previously carried. The guard now matches the team-alias line below it.
The per-key TTL cache did one indexed find_first per distinct key hash
per 30s — fine at small scale, but on a proxy serving 10k active keys
that is hundreds of small reads per second across pods, all to discover
that almost every key has no job.
Cache the entire active-job set instead: one find_many per pod per TTL
(the set is admin-started and capped at one job per key, so it is
single-digit rows), served to every key as an in-memory dict hit. DB
load is now flat and constant in the number of keys, idle or active.
On a DB blip the stale snapshot is kept and the next TTL retries, so a
blip degrades freshness rather than disabling the feature. Concurrent
requests share one refresh behind a lock instead of stampeding.
Adds @@index([status]) for the status-only find_many, folded into the
unshipped shadow eval migration.
Co-Authored-By: Claude <noreply@anthropic.com>
A shadow eval samples ongoing traffic, so a job without an end date keeps
billing judge calls until someone remembers to stop it — and the upfront
estimate silently priced exactly one week regardless. Jobs now take a
duration_days (1-30, default 7): the start endpoint stamps ends_at, the
estimate scales trailing volume to the requested window, and the logger
completes a job past its window through the same guarded update + cache
eviction path as the spend cap (generalized into _finalize_job). The
existing shadow eval migration is amended in place since it has not
shipped anywhere yet.
The start form no longer asks anyone to paste a key hash: the key is a
type-to-search combobox backed by /key/list alias substring search that
submits the token, the auto-router is a filter-as-you-type combobox fed
by the configured auto-router deployments, and duration is a select.
Active job cards show when the job will end.
The judge model field is now labelled as such, with guidance: judging
two answers blind needs solid comprehension and reliable JSON, not
frontier reasoning — a mid-tier model (Claude Sonnet / GPT-4o class) is
recommended, nano/mini-class judges give unreliable verdicts, and
frontier reasoning models add cost without changing outcomes. Same
guidance mirrored into the API field description.
Co-Authored-By: Claude <noreply@anthropic.com>
The column was declared Float? with no default, so every row started NULL.
The verdict writer increments it, and NULL + x is NULL in Postgres, meaning
judge spend never accumulated and the UI always showed no spend.
Makes the column non-null with a default of 0 across all three schema copies
and the migration, and tightens the response model to a plain float.
Resolves two real conflicts (the PR's actual base branch is
litellm_internal_staging, not main):
- litellm/types/management_endpoints/auto_router_endpoints.py: kept both
the Mapping and Literal imports, both used by pre-existing types.
- tests/e2e/proxy_client.py: kept upstream's more detailed create_model()
docstring covering multi-replica propagation.
Everything else auto-merged cleanly. Regenerated the OpenAPI schema
(schema.d.ts) to pick up upstream's tier_turns addition to the auto-router
benchmarks response.
Co-Authored-By: Claude <noreply@anthropic.com>
Core components:
- Added LiteLLM_ShadowEvalJob and LiteLLM_ShadowEvalVerdict tables to schema
- Migration: 20260807000000_add_shadow_eval_job (SQL for both tables + indexes)
- Types: StartShadowEvalRequest, GetShadowEvalJobResponse, etc. (pydantic models)
- ShadowEvalLogger: CustomLogger integration that will fire background shadow tasks
TODOs for next phase:
1. Wire config parsing (read shadow_eval from model_info or key settings)
2. Implement router call (_call_router_for_shadow) to get classifier tier + model
3. Build endpoints (POST /auto_router/shadow_eval/start, GET .../job/{id})
4. Implement verdict accumulation + per-tier result binning
5. Hook ShadowEvalLogger into proxy initialization
Still unresolved (implementation details):
- Cost estimation accuracy (judge model pricing per region)
- Sub-sampling strategy for high-traffic keys
- Job lifecycle: incremental verdict writes vs batch
- Retry logic for failed router/judge calls
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(auto-router): track turns per complexity tier (LIT-5302)
Stamps complexity tier at decision time (rollup never re-derives from routed
model, since tier->model mapping is mutable config). Records per-tier turn
counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router
in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup
for dashboard turns/share metrics.
Addresses Greptile/Bugbot findings:
- Missing _SessionAggRow.tier_turns field: added with field_validator to
parse jsonb text cast and handle NULL. Would 500 every benchmarks read.
- Missing ::text cast on tier parameter: Postgres fails type inference on
parameterized CASE/IS NULL without explicit cast. Added to all usages.
- Docstring false claim (only complexity routers produce tiers): quality
router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents
cross-contamination. Rewrote docstring to clarify isolation.
- Comment convention violations: stripped per CLAUDE.md rule.
- Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior
tests for SQL semantics against real Postgres. 12 mutations killed.
Fixed fragile complexity_router test that broke on nested function calls.
No API change; extends existing GET /auto_router/benchmarks response only.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): address review findings on tier turns tracking
- Guard router_type update so a mid-session reconfigure can't pool
foreign tier names into tier_turns
- Keep pinned turns attributed to the tier that actually serves them
- Drop stray -- AlterTable comment from hand-written migration
- Drop the now-unnecessary ::text/json.loads round-trip; prisma
already returns tier_turns as a parsed dict
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): satisfy type-discipline lint gate
- tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001,
mutable collection in annotation); these are read-only after
construction
- _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict
literal)
- default-fallback branch: replace the reassigned-without-Final
fallback_tier with a Final default_model_first flag and a single
ternary assignment (LIT010)
Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and
type_check_gate.py all pass against the litellm_internal_staging
merge-base; full test_complexity_router.py (374), auto_router
management-endpoint tests (26), db-layer rollup tests (31), and the
live-Postgres proxy_behavior rollup suite (17) all pass.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
heal_incomplete_nodeenv_cache() stats a $HOME-derived path with a bare
Path.is_dir(). pathlib only swallows ENOENT-shaped errnos, so a cache
directory the process cannot search raises PermissionError instead of
answering False. Images bake that cache under the build user's home, whose
mode is 0700, so a container started under any other uid dies there before
the Prisma CLI is ever invoked, and the migration never runs.
Tolerate OSError while inspecting the cache, matching the guard
nodeenv_cache_dir() already carries, so an unreachable cache means there is
nothing to heal rather than a crash. This restores the never-raises
contract ensure_prisma_toolchain() documents.
Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
SGR has had two independent definitions. The admin UI derived it from
SpendLogs, so it counted what litellm's logging callbacks observed and could
attribute and price. BillableRequestMetricsMiddleware counted what the proxy
actually answered at the ASGI edge, but only exported to OTLP for enterprise
metering. The two disagree by design in places, and the SpendLogs figure goes
quiet whenever spend logging is disabled or the callbacks are bypassed.
This adds LiteLLM_DailyGatewayRequests, written by the middleware, and points
the dashboard's Successful Requests tile at it.
Requests fold into an in-memory map at record time rather than going through a
queue like the spend path. A count is a pure aggregate, and every dimension of
the key is chosen by the proxy from a closed set: the date, the category, and a
route that the classifier maps to one of a fixed list of strings rather than
passing the raw path through. Nothing a caller sends can add a key, so the fold
and the table are bounded by (days x categories x routes) however much traffic
arrives; the spend queue blocks once full, which is not acceptable in the
response path. A scheduler job drains it on the existing batch interval, and a
failed flush merges its counts back so a database blip undercounts nothing.
The middleware previously returned early when no billing recorder was
injected, which is the unlicensed case. The new sink is not license-gated, so
that early return now requires both sinks to be absent. The billing recorder
keeps its 2xx-only gate; the sink takes every status so failed_requests is
real. The sink is not told which deployment served the request, unlike the
billing recorder. That id is a sha256 over litellm_params, credentials
included, so a caller who puts a credential in the request body mints a fresh
one per distinct value. No configuration is needed for that: api_base and
base_url are on _BANNED_REQUEST_BODY_PARAMS and need allow_client_side_
credentials, but api_key is not on that list, and both reach the same
_handle_clientside_credential branch. The read endpoint aggregates the
dimension away regardless, so the key is better off without it.
The new table carries no key, user or team dimension, so /gateway/daily/activity
is restricted to proxy admin roles and the per-key and per-model breakdowns
keep reading the daily spend tables. The old path is left running and marked
with TODOs.
A fetched result carries the range key it was fetched for, and the render
selects it only when that key matches the range on screen. Both the gateway
counts and the spend aggregate go through that rule: the request tiles read the
first and fall through to the second, so stamping only one of them would leave
the tile showing a superseded range by the other route.
The paginated pages behind that aggregate are reached through a failure flag,
so the flag is stamped too. A flag left over from the previous range would let
those pages through while a new range is in flight, which is the same defect
one fallback further down.
The Prisma CLI is a Node program that installs a private Node runtime on its
first invocation. That one-time install shared the 60s budget that bounds each
migration command, so on a slow or cold machine it was killed before it could
finish. Prisma then decides whether to reinstall by testing the cache
directory for existence alone, and a killed install leaves that directory
behind, so every later attempt skipped the install and failed on a node binary
that was never written. The existing four-attempt retry loop could not help:
each attempt hit the same missing binary, which turned a slow start into a
container that never migrated again.
Migrations now prepare the toolchain as its own step under its own budget, and
a cache directory that exists without a node binary is deleted first so an
interrupted install reinstalls instead of persisting. Both budgets are
overridable, LITELLM_PRISMA_BOOTSTRAP_TIMEOUT for the install and
LITELLM_PRISMA_COMMAND_TIMEOUT for each Prisma command, and every previously
hardcoded timeout now goes through one helper rather than thirteen literals.
The per-command default stays at 60s.
An override is only honoured when it parses as a finite positive number.
Infinity and NaN parse as floats and survive a plain positivity check, and
subprocess treats either as no deadline at all, so a value like `inf` or a
fat-fingered `1e400` would have silently disabled the timeout it was meant to
configure.
* fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db
The model cost map and Anthropic beta headers reload schedules kept their
last-run time in a per-pod module global, so GET /schedule/*/status reported
last_run null after any restart and the Admin UI showed the reload as never
having run. The reload check also only ran from the add_deployment job, which
is registered only when store_model_in_db is true, so config-file deployments
stored a schedule that never fired.
Persist last_run_at and reload_requested_at as dedicated columns on
LiteLLM_Config, owned by the reload job and manual reload endpoints, while the
schedule endpoints own the param_value JSON (interval_hours); no writer can
clobber another's fields. Serve status entirely from the row. Register the
check as its own periodic_reload_job outside the store_model_in_db gate.
Replace the force_reload boolean with a reload_requested_at timestamp each pod
compares against its own in-memory last reload, so a manual reload reaches
every pod exactly once instead of being cleared by the first poller. Run the
blocking fetches via asyncio.to_thread, and stamp last_run_at with update_many
so a schedule cancelled mid-poll is not resurrected.
* fix(proxy): compare reload requests against pod data age seeded at boot
A pod that had never reloaded kept its in-memory clock at None, and with no
interval configured nothing ever set it, so every manual reload request was
ignored by every pod except the one serving the click (Greptile P1 on the
previous commit). Seed the per-pod timestamp at boot as the time its data was
loaded and reload whenever a request or the interval is older than that, which
also removes both None special cases from the due predicate. A schedule whose
row has no last_run_at fires on the next tick so the first run does not wait a
full interval.
* fix(proxy): scope reload persistence to the model cost map and seed the pod clock from the actual load time
Revert the Anthropic beta headers reload path to its previous JSON-flag
implementation so this PR only changes the price data reload; the beta headers
path keeps working exactly as before and can migrate to the shared module in a
follow-up. The unused columns on its config row are inert.
Seed model_cost_map_loaded_at from the timestamp get_model_cost_map records at
the actual import-time fetch instead of ProxyConfig construction time, closing
the startup window where a manual reload request stamped between the fetch and
the constructor compared as older than the pod's data and was skipped
(Greptile P1 on the previous commit).
* refactor(proxy): drop the legacy force_reload backfill from the reload tracking migration
The backfill only carried over a manual reload clicked in the seconds before an
upgrade, and every upgrade restarts the pods, which re-fetch the cost map at
import and so already deliver what that request asked for. Removing it makes
the migration schema-only, so prisma db push and prisma migrate deploy leave
the database in the same state instead of diverging on a data statement that
only one of them runs.
* fix(proxy): stamp reload timestamps at the precision they are stored at
Postgres stores these columns as TIMESTAMP(3) while Python stamps microseconds,
so a pod comparing its in-memory clock against the persisted copy of the same
instant read as newer and skipped the reload request it had just recorded.
Truncate every stamp to milliseconds at the source, and floor the boot seed the
same way, so the in-memory value and its persisted copy compare exactly.
* fix(proxy): identify manual reloads by revision instead of comparing timestamps
Comparing a request timestamp against each pod's data age made correctness depend
on clock resolution: Postgres stores TIMESTAMP(3) while Python stamps microseconds,
and two events inside the same millisecond are indistinguishable no matter how the
comparison is written.
Replace reload_requested_at with a reload_revision counter the manual reload
endpoint increments atomically in the database. Each pod records the revision it
last applied and reloads whenever the row's differs, so a request reaches every pod
exactly once regardless of clock skew or precision, and concurrent requests publish
distinct revisions instead of overwriting one another. A pod adopts the current
revision on its first poll, since data it loaded at boot already satisfies any
earlier request. Interval reloads still key off the pod's own data age, where hour
scale comparisons make precision irrelevant.
* fix(proxy): seed the applied reload revision at startup
A pod adopted whatever revision it found on its first poll, so a manual reload
published while the pod was starting was marked applied without ever being
served and the pod kept the prices it fetched at import. Read the row once at
startup instead, right after that fetch, and treat a missing row as revision 0
* style(tests): revert incidental reformatting of test_proxy_server.py
An earlier ruff format run reflowed the whole file from its 88-column
formatting, adding ~1150 lines of churn unrelated to this PR. Replay only
the real test changes onto the original formatting
* fix(proxy): serve an outstanding reload request on a booting pod
Seeding the applied revision at startup left a window: a manual reload
published after the import-time cost map fetch but before startup read the
row was marked applied without ever being fetched, stranding that pod on
stale prices when no interval was configured. A pod now starts unapplied and
serves any outstanding request on its first poll, which costs one redundant
fetch per boot and removes the window along with the seeding step
* fix(proxy): accept a reload interval still encoded as JSON text
param_value is written with safe_dumps, and a raw row read can return it
decoded or as a string depending on the driver. Strict validation rejected
the string, so the schedule read as disabled and an admin's configured
reloads silently stopped. Mirrors the guard ConfigRepository.get_param
already carries for the same column
* fix(proxy): cancel a reload schedule without resetting the revision
* fix(proxy): null the interval in JSON so cancelling keeps the revision
prisma rejects a null literal for a Json? column, so update_many writes an
interval-less object instead. The fake config table now rejects the same input
the database does, which is what the live run caught and the mock did not.
Also records the run before adopting the revision, so a failed status write
leaves the request unserved for the next poll rather than reporting a run that
never landed.
* fix(ui): match the CI-generated user_role union order in schema.d.ts
* feat(spend): add net auto-router savings to the cost-optimization dashboard
The dashboard credited compression and prompt caching but said nothing about the
optimization that picks the model, so the driver with the largest lever on a bill
was the one an operator could not see.
Savings are the counterfactual: without a router a deployment runs one model, and
it has to be one that can carry the hardest request, so the baseline is the
priciest model in the router's hardest configured tier. A cheap tier is a choice
the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model`
overrides it for operators who would genuinely have run something else. Both are
provider-qualified before pricing, because a bare name can resolve to a different
vendor's rates or to nothing at all, and a deployment is priced by its `base_model`
where it has one, which is how Azure deployments are priced everywhere else.
Both arms price the request's real usage through `generic_cost_per_token` rather
than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers
and regional uplifts stay consistent with what was actually billed. `prompt_tokens`
already includes the cache buckets, so charging them again at the input rate would
price the same tokens twice.
Cache state is what makes this hard. The baseline serves every turn, so whether it
had the prompt cached is whether the conversation was already underway. On a
continuing conversation it wrote the prompt earlier and would only read it now, so
this request's write is what switching cost and counts against the saving. On a
first turn nothing was cached for any model, the baseline would have written the
same prompt, and both arms carry the write at their own rates. Charging the write
to both cases understates a first turn to a few percent of its value, and because
the write premium is fixed by prompt size while the saving grows with completion
length, it can render a profitable route as a loss.
That shape is read off the conversation rather than remembered: a second human ask
means an earlier turn was served. No cache, no session id, and no dependence on a
caller sending a session header. It cannot see a switch on a turn the router did
not classify, and it reads a few-shot prompt's synthetic turns as prior
conversation; both err toward charging the write, which under-claims.
The baseline and the shape ride on the existing `routing_decision` record, which is
already carried from the router to the spend log, already classified for redaction,
and already written-or-cleared per attempt. A fallback that re-enters the hook
therefore cannot leave either fact behind to be attributed to a deployment that
never routed, and no new metadata key crosses the trust boundary.
The result is signed. Whether a switch pays off is a race between the rate gap and
the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly
the routing behaviour an operator needs to see. The donut plots only drivers that
saved, while the card and range total keep the sign.
Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup
tables, declared `NotRequired` because rows queued by a pod on the previous release
carry no such key. It is summed by the rollup merge the cross-pod Redis drain also
runs, and carried through the aggregation query, the per-row accumulation and the
response model, so the dashboard reads a value the API actually sends. Tests
enumerate the drivers from the response model itself and assert each is summed,
accumulated, carried and totalled, so one added later cannot be half-wired.
* fix(spend): let the baseline pay for a continuing turn's own growth
`_baseline_usage` moved every cache-creation token into the baseline's read bucket
whenever the conversation was underway. That is right for a switch, where the
baseline never left the model it was on and really would only read, but wrong for a
turn that stayed put: the prompt grew, and the tokens written are that growth. They
are new to every model, so the baseline would have paid to write them too. Forgiving
it that write made the counterfactual cheaper than it was and shrank the reported
saving on ordinary steady-state traffic, by about 2% per turn.
The selected arm was never involved; it has always been priced on the real usage.
The error sat entirely on the baseline.
The condition is that the request read more than it wrote, not that it read anything.
A switch onto a model already holding a small prefix of this prompt still writes most
of it, and that write is the switch's own cost; keying off a nonzero read would have
handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing
the two buckets separates a warm continuation, which reads far more than it writes,
from a cold arrival, which does the reverse, and it leaves the existing invariant
intact: a request reading 0 and one reading 1 both still land in the same place.
* fix(spend): price each arm under the key litellm billed it, and see agent turns
Two ways the savings number read the wrong thing, both from identifying a model by
its name when the name is not what it costs.
The counterfactual was ranked and priced on the public rate for the model a
deployment names. A deployment may not be charged that rate: the router registers
its configured prices under the deployment's own id and deliberately keeps them off
the shared model-name key so deployments sharing a backend model do not pollute each
other. So a hardest-tier deployment configured above its public rate lost the
ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays.
Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision,
the resolver the real request is billed through, rather than a second rule here that
would have to re-learn that per-second and tiered overrides count, that a partial
override still counts, and that a deployment configured at zero is priced at zero
rather than treated as unpriced.
The arm being subtracted had the same fault and a sharper edge. It priced the spend
log's `model`, which on Azure is the deployment name, absent from the cost map, so
the whole driver silently read zero for that traffic. It no longer re-derives
anything: `model_map_information.model_map_key` is what litellm actually billed the
request under, recorded at request time by that same resolver with `base_model` and
custom pricing already applied.
Separately, the conversation-shape discriminator counted human asks, and an agent
loop can run twenty turns on one of them. Its tool traffic rides `tool_result`
blocks on user turns that flatten to empty text, and `tool` roles that are never
read, so a long agentic conversation looked like its own first turn and was handed
the arithmetic that leaves the cache write on both arms. That is the one direction
this must never fail in, because it inflates. An assistant turn is the direct
evidence that something answered earlier, and it is blind to how the tool plumbing
is spelled on either surface.
* fix(spend): give the cost-key resolver both inputs the selected arm needs
The served model was resolved through one input at a time, and each choice broke the
half the other fixed.
`model_map_key` is the served model already resolved through `base_model`, which is
the only way an Azure deployment name reaches the cost map at all; without it the
selected arm priced a name absent from the map, returned nothing, and the whole
driver silently read zero for that traffic. But it is built without
`router_model_id`, so it never carries a deployment's own price overrides, and a
custom-priced deployment was compared at its public rate while the baseline used the
real override. On a deployment configured well above its public rate that inverted
the answer outright: a route that lost $21.88 reported saving $0.10.
`_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a
deployment stays its decision rather than a rule restated here.
* fix(spend): same model is only the same cost when it is the same deployment
The short-circuit compared resolved model identity, so two deployments of one model
collapsed to "no switch" and reported zero. They are not the same cost: a deployment
can carry a negotiated rate, and routing from the dear one to the list-price one is a
real saving the dashboard reported as $0.00 against a true $21.93.
Both arms now carry the key litellm prices them under, so the comparison is between
deployments rather than between names.
* refactor(spend): price from resolved rates, not from a name we keep re-resolving
Four review rounds landed on one mechanism: which identifier prices a deployment.
base_model, then the deployment id, then cache-only overrides. Each round added a
clause to a resolution rule that should not exist, and a wrong primitive fails once
per input shape, so each shape arrived as its own finding.
`Router.get_deployment_model_info` already owns this. It merges a deployment's
configured prices over the built-in map, folds in `base_model` defaults for
deployments whose name is not a model, and falls back to the model name when nothing
is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure)
was that function re-implemented badly.
`generic_cost_per_token` now accepts already-resolved rates instead of demanding a
name it looks up itself, which is what forced the name-bending in the first place.
Both arms resolve through the owner and pass what they got: the counterfactual by the
deployment the router would have used, the served request by the deployment that
served it. The invented cost-key resolver is gone, and `Baseline` carries a
deployment id rather than a key we chose on litellm's behalf.
Net 64 insertions against 79 deletions.
* test(spend): follow _most_expensive onto the router that prices its candidates
Ranking moved through `Router.get_deployment_model_info`, since what a deployment
costs is the router's answer to give; these four cases were still calling the old
free-function signature.
* fix(spend): rank baseline candidates by what a request costs, not by two rates
"Most expensive" was decided by comparing output rate then input rate. That is a
property of a rate, not of a request: a deployment dearer per output token can be
cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and
recorded the wrong counterfactual.
Candidates are now costed on one reference request through the same engine the
savings themselves use, which leaves cache read and write rates, tiered tables and
every other billing dimension to that engine rather than to another rule restated
here. The reference request is cache-heavy because auto-routed traffic is.
* fix(spend): pick the baseline against the request that ran, not a stand-in for one
Ranking happened in the pre-routing hook, where the request has not executed yet, so
candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it
cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest
tier holding a deployment with non-proportional configured rates could be ranked for
a request nothing like the one served.
The mix is known on the spend path, so the ranking belongs there. The routing
decision now carries the tier's candidates rather than a winner already chosen, and
the baseline is resolved against the usage that actually happened. The reference
workload is gone; nothing here assumes a traffic shape any more.
The router is passed in rather than imported from `proxy_server` inside the
computation, so the savings stay a pure function of their arguments and the caller
owns where the router comes from. That also makes the spend path testable without a
running proxy, which the previous shape was not.
* refactor(spend): measure savings against one configured model, not a derived one
The counterfactual was derived per request: enumerate the hardest tier's
deployments, resolve each one's effective pricing, price them all, take the dearest.
That machinery produced a review finding per input shape it had not anticipated,
and every answer it gave was one an operator could have stated in a line of config.
So they state it. `litellm_settings.autorouter_savings_baseline_model` names the
model the traffic would have run on without a router, for every auto-router on the
proxy, and unset means the driver is off rather than a model nobody named being
guessed at. `savings_baseline.py` and its tests are deleted outright, along with the
tier enumeration, the candidate list on the routing decision, and the per-deployment
override that shadowed it.
Cache-state handling is untouched: the baseline is still priced on this request's own
read and write split, so a switch still pays for re-warming the cache and a first
turn still charges the write to both arms.
45 insertions against 482 deletions.
* refactor(router): compute the conversation shape once and pass it down
`_classify_and_route` re-derived it from the messages the hook had already resolved,
so an ordinary routed request walked the turn list twice for one boolean. The hook
computes it and hands it over, which is also where the affinity-hit path already got
it from.
Also moves `_get_llm_router` below the imports it sat among.
* fix(router): drop the dead conversation_continuing parameter off the hook
It was added to `async_pre_routing_hook` by mistake and immediately overwritten by
the value the hook computes, so it never did anything. It also widened a signature
every pre-routing strategy shares with the protocol in `types/router.py`, leaving
this one router diverged from `AutoRouter` and the interface for no reason.
Also records why an unreadable request counts as continuing: no messages is no
evidence a turn was served, so it pays the cache write and under-claims rather than
being handed a first turn's larger saving on nothing.
* fix(spend): charge a baseline its input rate for cache buckets it cannot price
A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket.
* refactor(spend): build the daily upsert payloads in one shot
`common_data` and `update_data` were constructed and then appended to: `request_id`
conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that
grows after its literal cannot be reasoned about by reading the literal, which is the
whole point of building it at once.
The conditional key resolves to a spreadable value before either payload, so both are
single expressions and the tag branch appears once instead of twice.
Not wrapped in MappingProxyType, though it was suggested: these go straight to
prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested
node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through
to the serializer and raises `TypeError: Type <class 'mappingproxy'> not
serializable` inside the batch upsert, where the surrounding except would log it and
leave the rollups silently unwritten.
* fix(spend): keep the one-shot upsert payloads under the type-discipline budget
Building both payloads as single literals traded a mutation for two dict literals,
and LIT002 counts construction rather than mutation, so the change the review asked
for is the one the gate charges for.
The empty branch is the avoidable half: it is the same value every time, so it moves
to a module constant built once instead of a literal per transaction, and it is a
read-only mapping so none of the call sites that spread it can fill it in later.
Logical replication consumers need FULL replica identity to reconstruct the
old row of an UPDATE or DELETE, and prisma leaves every table it creates at
the postgres default. Operators had to re-apply the setting by hand after
each migration run.
Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM
table at the end of a successful migration run, through the prisma CLI so the
dependency-free proxy-extras package stays that way. Tables that are already
FULL are skipped, foreign tables in the same schema are left alone, and a
database that refuses the ALTER is reported rather than failing the run.
Resolves LIT-3022
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to
LiteLLM_SpendLogs with a start_time-only predicate the composite
(tool_name, start_time) index cannot serve, and the dedup total query
left the outer SpendLogs scan unwindowed, so every dashboard load
walked both per-request tables end to end.
- clamp the window to the most recent 30 days ending at end_date; the
response start_date reflects the effective window and the dashboard
notes the clamp
- index SpendLogToolIndex on start_time (all schema copies + migration)
- window the SpendLogs side of both queries (1s margin: the two writers
can disagree by ~1ms on the same request)
- expire SpendLogToolIndex rows on the spend-log retention cutoff via a
parametrized batch-delete engine shared with the SpendLogs cleanup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(spend): track prompt compression saved tokens in daily spend aggregates
Native compression interception now records tokens_before/after/saved into the
request litellm_metadata so savings land in the SpendLog metadata JSON under a
typed compression_savings key. A single normalizer
(extract_compression_saved_tokens) sums that key with Headroom guardrail
tokens_saved; the two writers are disjoint and run at different stages, so
summing never double-counts. The spend-log redactor now preserves purely
numeric compression stats inside guardrail_response so Headroom savings
survive the store_prompts_in_spend_logs=false default. compression_saved_tokens
is threaded through BaseDailySpendTransaction, queue aggregation, the daily
upsert blocks, a new BigInt column on all six daily spend tables, and the
daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups)
* fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader
* feat(spend): aggregate compression and prompt caching dollar savings in daily rollups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): update daily spend aggregation fixtures for savings columns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add Cost Optimization dashboard page
New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows.
Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Config.yaml-declared OAuth2 MCP servers using Dynamic Client Registration have no LiteLLM_MCPServerTable row, so the DCR persist path called update_mcp_server, which returns None for a missing row, then update_server(None), which dereferenced .approval_status and raised AttributeError. The exception was swallowed to a warning while /register still returned 200, so the minted client was never stored and every access-token expiry forced a full re-authorization
Persist the acquired DCR client (client_id, client_secret, token_endpoint_auth_method, redirect_uris, encrypted at rest) in a dedicated LiteLLM_MCPServerOAuthClient store keyed by server_id when the server has no row, overlay it onto the in-memory config server so the refresh_token grant can authenticate within the process, and rehydrate it when the registry syncs from the database (which runs after the DB connects, unlike config load) so restarts and other pods pick it up. The store is encrypted at rest and is re-encrypted by the master-key rotation path alongside the server rows, through a shared helper so the two sites cannot diverge. The DB-backed server path is unchanged, and guarding the None return removes the swallowed-crash footgun
Resolves the config.yaml DCR persistence regression introduced in v1.92.0 by #31912
Adds an admin-configured issuer to MCP servers. When set, OAuth metadata is
fetched from the issuer's own origin and adopted only when the document
self-attests that same issuer (RFC 8414 §3.3), making token_endpoint,
registration_endpoint, and scopes authoritative for the pinned issuer instead
of a document the MCP resource server chose. This closes the mix-up where a
compromised resource echoes a pinned authorization_url to smuggle its own
token endpoint and inflated scopes past the corroboration gate. Discovery is
same-authority against the issuer origin, fails closed on a §3.3 mismatch, and
does not fall back to resource-rooted discovery. Rows without an issuer keep
the existing corroboration-gate behavior unchanged.
Backend + schema only; UI field and live-proxy proof follow.
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'
key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.
Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move key_scope helper to components root
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(keys): persist key_type on virtual keys so the UI reads scope directly
Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken
The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard
OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be
configured through config.yaml; the create/update REST API and the dashboard had no way to
express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end.
These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url
and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the
credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first
and falls back to the credentials blob so servers persisted before the columns existed still load.
client_id and client_secret continue to ride the existing encrypted credentials path.
On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field
section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous
catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of
mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from
non-admin and virtual-key responses, matching how token_url is treated.
* fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange
The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code:
when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could
not mint a token), the user must re-authorize via the browser flow. token_exchange has no
gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was
undefined (a compile error) and, per this file's convention and its tests, meant authorization_code;
renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an
authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode
instead and drop the now-unused isTokenExchange
* fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes
Switching an existing oauth2 server to oauth2_token_exchange left the old flow's
token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url
as the configured exchange endpoint, so the stale value both suppressed the RFC
9728/8414 discovery this PR adds and sent the exchange grant (client credentials
plus the user's subject token) to the previous flow's token endpoint
update_mcp_server now mirrors its existing stale-credentials rule for the
flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow,
token_exchange_endpoint, audience, subject_token_type): when auth_type changes,
each one is cleared unless the same request explicitly provides it, so a
deliberate override in the switch request still wins. Updates that keep the
auth_type never touch these columns, which keeps legacy OBO rows that use
token_url as their exchange endpoint working
The edit form sends explicit nulls for the previous flow's fields on an auth type
switch; antd preserves unmounted field values by default, so without this the old
token_url would be re-sent verbatim and read as an explicit override. Transitions
are detected against the persisted auth_type, so saves that keep the auth type
send nothing extra
Reported by Cursor Bugbot on the PR
* fix(mcp): lift legacy blob token-exchange settings into their columns on every write
The three token-exchange settings live in dedicated columns but also exist on
MCPCredentials as the pre-column REST shape. Writes now lift incoming blob
values into the columns (an explicit top-level value wins, including an
explicit null) and strip them from the stored blob; the same-auth credentials
merge migrates legacy rows the same way. The read-time column-or-blob fallback
then only ever serves rows current code has never written, so clearing a column
to re-enable RFC 9728/8414 discovery can no longer be silently undone by a
stale blob copy.
Also asserts the auth-switch clearing fires on the external fields_set path
(PUT /v1/mcp/server).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(mcp): single source for the RFC 8693 default subject_token_type
The default was applied at four egress build sites plus two model defaults,
each with its own copy of the literal. All sites now share
DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is
deliberately not used: Prisma writes explicit values on insert, so a column
default would rarely apply, and NULL-means-RFC-default keeps existing rows
correct.
Also documents two review decisions in place: the audience column keeps the
RFC 8693 parameter name (RFC 8707 resource indicators are already a separate
concept named resource in the v2 egress types), and the migration's
out-of-order timestamp is safe under prisma migrate deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: fix import sort order in outbound_credentials/types.py (I001 strict budget)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials
The migrate-on-write in the credentials merge lifts blob values into null
columns, which is correct for legacy rows but could repopulate a column an
admin had cleared in an earlier no-credentials update (that path never touched
the blob, so the stale copy survived to be lifted later). An explicit
token-exchange column write (set or clear) now migrates the row even when the
update carries no credentials: untouched null columns are lifted, every blob
copy is stripped, and unrelated blob keys stay as-is. A cleared column can then
never be resurrected, because no write path leaves a blob copy behind.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(mcp): state the blob-to-column lift contract on the legacy credential keys
The three token-exchange keys on MCPCredentials are the pre-column REST shape
(the only REST shape from 2026-05 until this PR). Document on both the blob
type and the request models that the dedicated columns are authoritative and
that writes lift blob values into them and strip the stored copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers
The other two token-exchange fields were cleared while subject_token_type was
left visible. It is a public RFC 8693 URN with no disclosure value, but the
sanitizers' rule is that these views receive no token-exchange config at all —
cleared for uniformity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add an optional per-server max_concurrent_requests that caps how many
tool calls LiteLLM sends to one MCP server at once, so batch-processing
backends are not overwhelmed by unbounded parallel dispatch. Excess calls
queue on a per-server asyncio.Semaphore instead of being rejected. Unset
or non-positive means unlimited, preserving existing behavior.
Resolves LIT-2749
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
* Mark xAI models retiring on 2026-05-15 (#28788)
Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is
retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3
with various reasoning efforts; callers continuing to use the old slugs
will be billed at grok-4.3 pricing):
grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort)
grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none)
grok-4-fast-reasoning -> grok-4.3 (low effort)
grok-4-fast-non-reasoning -> grok-4.3 (none)
grok-4-0709 -> grok-4.3 (low effort)
grok-code-fast-1{,-0825} -> grok-build-0.1
grok-3 -> grok-4.3 (none)
Only the direct xai/ slugs are tagged; third-party hosts (azure_ai,
oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The
grok-3 retirement list explicitly names only the base grok-3 slug — the
-mini / -fast / -beta / -latest variants are not listed, so they remain
untouched.
* feat(moonshot): advertise json_schema response support on live models (#29683)
litellm.responses() already routes Moonshot through the responses->chat-completions
bridge, and Moonshot honors response_format json_schema on chat completions. The
cost-map entries left supports_response_schema unset, so discovery layers that gate
on that flag dropped Moonshot from structured-output / responses listings even though
the capability works end to end.
Set supports_response_schema on the nine models currently live on api.moonshot.ai:
kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants,
and moonshot-v1-auto. Verified against the live API that each honors json_schema and
that litellm.responses() returns schema-valid structured output through the bridge.
* chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685)
Thirteen Moonshot/Kimi models in the cost map no longer resolve on
api.moonshot.ai (all return 404). Stamp each with its deprecation_date from
platform.kimi.ai/docs/models rather than deleting the entries, so historical
cost calculation keeps resolving the names while tooling can surface the
retirement.
Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context
variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the
moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot
publishes no discontinuation date for them).
* fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687)
Kimi reasoning models reject every temperature except 1; a request with
temperature=0.2 returns "invalid temperature: only 1 is allowed for this model".
litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd.
Drop the temperature param entirely for reasoning models (gated on
supports_reasoning, the same signal transform_request already uses) so the model
default is used; the non-reasoning moonshot-v1 models keep the existing clamp.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(mcp): add per-server timeout configuration (#29672)
* feat(mcp): add per-server timeout configuration
* fix(mcp): address timeout field review comments
- use is not None guard instead of or for 0.0 edge case
- copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table)
- add timeout Float? column to all three schema.prisma files
- extend round-trip test to cover _build_mcp_server_table direction
- add test for zero timeout not treated as falsy
* fix(mcp): forward timeout in _build_temporary_mcp_server_record
* fix(mcp): return 504 instead of 500 when per-server timeout fires
* test(mcp): add 504 timeout regression test; fix black formatting
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7
AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the
existing us./eu./au./global. profiles for Claude Opus 4.7
(ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is
missing from model_prices_and_context_window.json. Tokyo-region
users currently get an "unknown model" error when routing through
the JP geo profile.
Adds the entry to both the canonical file and the bundled backup,
mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches
the other regional profiles (10% premium over base/global).
Regression test pins all six documented profiles (base, global, us, eu,
au, jp) and asserts pricing parity between jp. and au. variants.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(soniox): add soniox audio transcription integration (#29508)
* feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650)
The OpenMeter callback resolves the CloudEvent subject from kwargs["user"]
first, then falls back to the key-bound user_api_key_user_id. For
multi-tenant proxy deployments, a client can set `"user": "..."` in the
request body and cause their usage to be attributed to that arbitrary
string — a billing-attribution forgery risk.
Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward
compatibility). When set to "false", the request-supplied `user` field is
ignored and the subject is resolved solely from user_api_key_user_id.
Matches the existing env-var-driven config pattern in this file
(OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE).
* feat(search): add you_com as a search provider (#28370)
* feat(search): add you_com as a search provider
Registers You.com Search API as a first-class `search_provider` in the
`search_tools` registry, alongside Tavily, Exa, Perplexity, etc.
- New adapter: litellm/llms/you_com/search/transformation.py
- POSTs to https://ydc-index.io/v1/search
- Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key)
- Maps Perplexity unified spec: max_results -> count,
search_domain_filter -> include_domains, country -> country
- Flattens results.web + results.news into a single SearchResult list;
snippet prefers snippets[0], falls back to description; page_age -> date
- Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired
into ProviderConfigManager.get_provider_search_config()
- Pricing entry: model_prices_and_context_window.json (placeholder $0.0;
happy to adjust to maintainers' preferred public number)
- Docs: example router config snippet and example proxy yaml updated
- Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests
(payload shape, domain filter mapping, snippet fallback, news flattening,
missing-api-key error)
Refs upstream expansion signal: #15942
* review fixups: normalize api_base, lowercase country, scope env-var to test
Addresses Greptile inline review comments on #28370:
- get_complete_url: strip trailing slashes from api_base *before* the
endswith("/v1/search") check, so a custom base like ".../v1/search/"
doesn't become ".../v1/search/v1/search".
- transform_search_request: .lower() country before sending, matching
Tavily's convention so callers using the unified spec form ("US") get
consistent behavior across providers.
- Tests: replace direct os.environ writes with an autouse monkeypatch
fixture so YOUCOM_API_KEY is set per-test and removed afterwards.
The missing-key test now uses monkeypatch.delenv. New test asserts the
trailing-slash normalization above.
Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note
that documentation changes belong in the litellm-docs repo.
* support keyless free tier (api.you.com/v1/agents/search) as default
You.com offers an IP-throttled keyless endpoint that returns the same
response shape as the keyed one (~100 queries/day, no signup). This is a
significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG
providers already in the search_tools registry.
Behavior:
- YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search
(X-API-Key header)
- no key -> free: POST https://api.you.com/v1/agents/search
(no auth)
- YOUCOM_API_BASE override -> honored as-is
Tests:
- New: test_you_com_search_keyless_free_tier - asserts URL + absence of
X-API-Key when no key is configured.
- New: test_you_com_search_validate_environment_keyless - asserts the
config no longer raises when the key is absent.
- Removed: test_you_com_search_raises_without_api_key (the precondition
no longer holds).
- Existing payload/domain-filter/etc tests still cover keyed mode via
the autouse YOUCOM_API_KEY fixture.
Verified both endpoints accept POST + return identical JSON shape:
results.web[] / results.news[] with title, url, snippets, description,
page_age.
* register you_com in provider_endpoints_support.json
Adding `litellm/llms/you_com/` requires a corresponding entry in
provider_endpoints_support.json or the
code-quality/check_provider_folders_documented CI check fails.
Follows the compact tavily/serper pattern - endpoints: { search: true }.
Local run of the check now reports "All 114 provider folders are documented".
* move tests under tests/test_litellm/llms/ so CI exercises them
The litellm CI workflows scope unit tests to `tests/test_litellm/...`
(see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so
tests living under `tests/search_tests/` are never run in CI - which is
why codecov reports 0% patch coverage for the new adapter even though
the unit tests exist and pass locally.
Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so
the test-unit-llm-providers job picks it up. 7/7 tests still pass at
the new location.
(Sibling search-only providers - tavily, exa_ai, brave, etc. - still
live only in `tests/search_tests/` and would benefit from the same
move, but that is out of scope for this PR.)
* fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug
The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises
Content-Encoding: gzip but returns a body that httpx's decoder rejects
with `zlib.error: Error -3 while decompressing data: incorrect header
check`, surfacing as litellm.APIConnectionError in user code. curl works
because it doesn't request compression by default.
Pin Accept-Encoding: identity in validate_environment so the upstream
server skips compression entirely. Harmless on the keyed endpoint
(ydc-index.io/v1/search) which negotiates content-encoding correctly.
The header uses setdefault so a caller-supplied Accept-Encoding still
takes precedence. (Server-side bug has been flagged to the You.com team
separately - once fixed there, this workaround can be removed.)
New unit test: test_you_com_search_pins_identity_accept_encoding.
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* docs: fix README typo (#29419)
Correct clear spelling mistakes in documentation without changing behavior.
Confidence: high
Scope-risk: narrow
Tested: git diff --check; uvx codespell on changed files
Not-tested: Full docs build not run; text-only changes
* Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480)
* fix(langfuse): pass ssl_verify to Langfuse httpx client
* fix_langfuse_
* add unit tests
* addressed comments
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(models): add minimax/MiniMax-M3 to model cost map (#29412)
Add MiniMax's new flagship MiniMax-M3 to the native minimax provider:
512K context, 128K max output, native multimodal (supports_vision),
reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output
2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so
cache_creation_input_token_cost is omitted.
Updated both the root model_prices_and_context_window.json (remote
source) and the bundled litellm/model_prices_and_context_window_backup.json
(local fallback), keeping them in sync.
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394)
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log
* fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation
* feat(provider): Add Neosantara provider as OpenAI Compatible (#29646)
* Add Neosantara provider
* Register Neosantara provider enum
* Address Neosantara provider review feedback
* Add Neosantara packaged endpoint support
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix: address greptile and veria review feedback
- langfuse: guard httpx_client injection behind version check (>= 2.7.3)
- soniox: propagate audio_transcription_duration in _hidden_params for spend tracking
- soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base
- mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError
* chore(mcp): add migration for per-server timeout column
* fix(test): add tool_use_system_prompt_tokens to model prices schema validator
* fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key
* fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs
The search flow resolves api_key in validate_environment but never passed it
into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the
env) set the X-API-Key header yet still selected the keyless free-tier endpoint.
Forward api_key through both the search entrypoint and the http handler so the
keyed endpoint is chosen.
HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox
poll and transcript-fetch GETs silently used the client global default instead
of the caller timeout. Add a per-request timeout to get() and forward the
configured timeout from the Soniox handler.
* fix(soniox): price stt-async-v4 per second so transcriptions are billed
The handler stores audio_transcription_duration in _hidden_params, but the
model carried only token cost fields and the response has no token usage, so
the transcription cost path fell through to cost_per_second and returned $0.
An authenticated caller could transcribe Soniox audio without decrementing
their budget. Switch the entry to output_cost_per_second at Soniox's published
$0.10/hour async rate so the stored duration produces a real charge.
* fix(langfuse): use a dedicated httpx client for the SDK injection
The httpx_client handed to the Langfuse SDK came from _get_httpx_client(),
which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that
client on teardown it would invalidate the shared client used by every other
LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL
verification and client certificate from LiteLLM's configuration.
* fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var
* fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779)
* fix(cohere): support max_completion_tokens on cohere v2 chat
The default cohere_chat route resolves to CohereV2ChatConfig, which did not
list or map max_completion_tokens, so get_optional_params raised
UnsupportedParamsError for the standard OpenAI parameter (the modern
replacement for the deprecated max_tokens). The v1 config already maps it to
cohere's max_tokens; mirror that in v2 and add v2 regression tests.
* fix(cohere): make max_completion_tokens take precedence over max_tokens on v2
When both max_tokens and max_completion_tokens are supplied, prefer
max_completion_tokens explicitly rather than relying on dict iteration order,
and cover both orderings with a regression test.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Dan Lemon <dan@danlemon.com>
Co-authored-by: Saswat <saswatds@users.noreply.github.com>
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com>
Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: kape <168134658+kapelame@users.noreply.github.com>
Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com>
Co-authored-by: Just R <remixingmagelang@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
