mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
303 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yuneng Jiang
|
34c0c707c0
|
revert(spend-logs): drop the endTime backfill migration for spend log timestamps
Reverts #37554, which added 20260819000000_backfill_spend_log_timestamps |
||
|
tin-berri
|
2dcd453860
|
feat(shadow_eval)!: gate the per-key budget on dollar spend instead of turns (#37555) | ||
|
Mateo Wang
|
6292489192
|
fix(spend-logs): backfill created_at/updated_at from endTime instead of migration time (#37554) | ||
|
mateo-berri
|
31090d122e |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_infer_single_worker_redis_banner
# Conflicts: # litellm/proxy/proxy_server.py |
||
|
tin-berri
|
a613773fca
|
feat(auto-router)!: scope shadow eval jobs to multiple keys (#37251)
* feat(auto-router): scope shadow eval jobs to multiple keys A shadow eval job now covers a set of keys instead of exactly one, and each key carries its own max_turns budget, so one key exhausting its budget leaves its siblings sampling. The existing job row already is the per-key unit (api_key_id, max_turns, stopped_at, and the one-active-per-key-and-direction partial unique index all live on it), so multi-key is grouping rather than schema surgery: a new group_id column ties N sibling rows written atomically by one create_many, the API's job id becomes the group id, and pre-existing jobs backfill group_id = id so their ids keep resolving. The sampler hot path is untouched; its test file has a zero-line diff Results come back pooled plus a per-key breakdown and responses list every key with its own budget, stop state and read-time labels. The dashboard is adapted minimally to the new shapes (the picker stays single-key and submits a one-key list); the multi-select picker and per-key table land in the stacked UI PR * fix(shadow_eval): derive completed from spent budgets and record operator stops * fix(shadow_eval): stamp stops atomically and freeze counts at the stamp The stop endpoint wrote stopped_by and stopped_at as two separate updates, so a failure between them left a job reading stopped while its unstamped legs kept sampling, and the retry got 400 already stopped. One UPDATE now stamps stopped_by and every missing stopped_at together, preserving the stopped_at a leg earned from its own budget via COALESCE Attempt counts now exclude attempts that land after a leg's stopped_at, so an in-flight attempt finishing just after an operator stop can never push a legacy pre-stopped_by job over its budget and flip it from stopped to completed at read time * fix(shadow_eval): backfill stopped_by so legacy stops never read as completions * chore(ui): regenerate api types for the shadow eval stop fields * fix(shadow_eval): let the stop statement pick one winner under racing stops Two operators can both pass the derived-status guard in the race window. The stop UPDATE now claims only legs with stopped_by still null and the endpoint judges by its row count, so exactly one caller ever gets the 200 and the loser gets the same already-stopped 400 a late caller gets * refactor(shadow_eval): make the stop statement the whole state machine The status guard ran before the UPDATE, so a stop racing the last budgeted attempt still claimed the job and it read stopped forever instead of completed. The statement now claims the job only while a leg still samples inside the window with no stop recorded, and the endpoint reads once after writing: a racing operator, a same-instant budget spend, and a repeat stop all get the 400 naming the status the job actually holds. The pre-write guard and the hand-built response go away * chore(ui): regenerate api types for the stop route description |
||
|
Tianhe Zhang
|
607e4a4e30 | feat(spend-logs): add lifecycle timestamps | ||
|
mateo-berri
|
8ba2263d4c |
perf(guardrails): aggregate usage units in one sorted pass
The flush and the usage endpoints summed units with a scan per distinct key, quadratic in rows times keys; group sorted rows instead. Skip payloads without a request_id like the metrics path, type the flush key as a NamedTuple, and drop the (guardrail_id, date) index that the primary key already covers |
||
|
mateo-berri
|
55e80849d1 | feat(guardrails): track bedrock guardrail usage units per invocation | ||
|
mateo-berri
|
ed33687422 | feat(proxy): auto-suppress the no-Redis banner for confirmed single-worker deployments | ||
|
tin-berri
|
2d3c3e3098
|
feat(shadow_eval): add reverse-direction shadow eval jobs (#36865)
Shadow eval only answered "should this key adopt this auto-router". Once a key is on the router it is invisible to the feature, because the sampling gate skips any request the shadowed router already served, so post-adoption quality regressions go unmeasured. Reverse mode inverts the arms: sample the traffic the router did serve and duplicate it against a fixed baseline_model, judged by the same blind pairwise judge. Same job table, same attempt rows, same aggregates. real_* stays the arm the caller was served and shadow_* the duplicated one, so in reverse real_model is the router's pick and shadow_model is the baseline. The active-job slot becomes one per (key, direction) so both directions can run at once, and tier attribution in reverse reads the control request's routing decision rather than the shadow call's write-back. |
||
|
tin-berri
|
d8fda675cc
|
feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state) (#36587) | ||
|
ryan-crabbe-berri
|
b144b15d48
|
fix(proxy): add config_updated_at audit timestamp for virtual keys (#36488)
* fix(proxy): add config_updated_at audit timestamp for virtual keys updated_at carries Prisma's @updatedAt, so every batched spend flush rewrites it and it cannot distinguish config changes from usage. Add an additive config_updated_at column stamped only by key management writes (update, bulk update, regenerate, block, unblock) via a shared helper, expose it on key responses, and switch the key page's Last Updated to it with a created_at fallback. * test(proxy): assert config_updated_at survives key archival * refactor(proxy): rename config_updated_at to settings_updated_at |
||
|
yucheng-berri
|
e5386c10a7
|
feat(ptu): configure provisioned-throughput flat cost on a model deployment (#35341)
Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to
ModelInfo so a model deployment can carry the inputs for provisioned-throughput
flat-cost attribution. ModelInfo validates per-field bounds (positive count,
non-negative rate, effective_to after effective_from); model/new and
model/{id}/update enforce the cross-field invariant (count and rate set together,
team_id required) on the effective model_info so partial updates validate the
merged result, and v1/model/info returns the fields.
LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a
sentinel api_key constant; the daily rollup that writes them lands in a follow-up
PR. Adding the optional model_info fields is backward compatible; models without
them are unaffected.
ptu_effective_from is required alongside the count and rate rather than optional. Flat
cost accrues from that instant, so an absent start has to be inferred, and inferring it
let a deployment configured today be billed for days it did not exist. Both PTU validators
also run over the merged view before any write on the update path, beside the premium check the create path
already runs there: the team ACL update below autocommits, so a validator raising further
down left the team mutated and the deployment row never written.
The update path validates the model_info a patch would store rather than the patch
alone. An invariant holds over the deployment as it will exist, not over whichever
subset of fields a caller sent, and validating the patch rejected raising the rate on
an already configured model because that patch carries no start of its own.
|
||
|
yucheng-berri
|
efc4e6f28c
|
fix(batches): keep batch state in sync on a poll without claiming attribution (#34456)
A poll of a Vertex passthrough batch wrote nothing to the managed-object row, so status and file_object stayed frozen at the create-time snapshot and GET /v1/batches served a stale status and an empty output file id for the life of the batch. Only the create may claim a batch, but every observation of one may refresh its state. store_unified_object_id takes create_if_missing, which the poll clears: it refreshes status and file_object through update_many, and leaves a row that is absent absent rather than creating one owned by the observer, since created_by and team_id are written by whoever reaches the create branch. The update payload is now shared with the upsert so it cannot drift into writing api_key, request_tags, created_by or team_id. The passthrough identity re-assertion that was previously part of this PR ships separately in #36121, so this PR keeps only the batch attribution work. The creating key owns user_api_key_alias only when it actually has one. Guarding the overwrite on the presence of a key rather than on a resolved alias nulled the field out for every key generated without key_alias, and for any key rotated or deleted before its batch finished, losing the creating user's alias that the spend row previously carried. The guard now matches the team-alias line below it. |
||
|
tin-berri
|
3238ce8406
|
feat(auto-router): track turns per complexity tier (LIT-5302) (#36209)
* feat(auto-router): track turns per complexity tier (LIT-5302)
Stamps complexity tier at decision time (rollup never re-derives from routed
model, since tier->model mapping is mutable config). Records per-tier turn
counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router
in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup
for dashboard turns/share metrics.
Addresses Greptile/Bugbot findings:
- Missing _SessionAggRow.tier_turns field: added with field_validator to
parse jsonb text cast and handle NULL. Would 500 every benchmarks read.
- Missing ::text cast on tier parameter: Postgres fails type inference on
parameterized CASE/IS NULL without explicit cast. Added to all usages.
- Docstring false claim (only complexity routers produce tiers): quality
router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents
cross-contamination. Rewrote docstring to clarify isolation.
- Comment convention violations: stripped per CLAUDE.md rule.
- Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior
tests for SQL semantics against real Postgres. 12 mutations killed.
Fixed fragile complexity_router test that broke on nested function calls.
No API change; extends existing GET /auto_router/benchmarks response only.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): address review findings on tier turns tracking
- Guard router_type update so a mid-session reconfigure can't pool
foreign tier names into tier_turns
- Keep pinned turns attributed to the tier that actually serves them
- Drop stray -- AlterTable comment from hand-written migration
- Drop the now-unnecessary ::text/json.loads round-trip; prisma
already returns tier_turns as a parsed dict
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): satisfy type-discipline lint gate
- tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001,
mutable collection in annotation); these are read-only after
construction
- _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict
literal)
- default-fallback branch: replace the reassigned-without-Final
fallback_tier with a Final default_model_first flag and a single
ternary assignment (LIT010)
Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and
type_check_gate.py all pass against the litellm_internal_staging
merge-base; full test_complexity_router.py (374), auto_router
management-endpoint tests (26), db-layer rollup tests (31), and the
live-Postgres proxy_behavior rollup suite (17) all pass.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Yassin Kortam
|
de00655363
|
fix(migrations): keep the toolchain heal from raising on an unreadable nodeenv cache (#35986)
heal_incomplete_nodeenv_cache() stats a $HOME-derived path with a bare Path.is_dir(). pathlib only swallows ENOENT-shaped errnos, so a cache directory the process cannot search raises PermissionError instead of answering False. Images bake that cache under the build user's home, whose mode is 0700, so a container started under any other uid dies there before the Prisma CLI is ever invoked, and the migration never runs. Tolerate OSError while inspecting the cache, matching the guard nodeenv_cache_dir() already carries, so an unreachable cache means there is nothing to heal rather than a crash. This restores the never-raises contract ensure_prisma_toolchain() documents. |
||
|
tin-berri
|
32deaff015
|
feat(spend): rebuild the auto-router benchmarks backend as a per-session rollup (#35910)
Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one conditional upsert at spend-write time, classifying each turn (same model, first visit, return to tier, out of order) against the row's own columns so nothing is read before the write. The upsert's placeholders and argument tuple both derive from the transaction dataclass's own field order, so the SQL and the call site cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's cache interaction is derived once from its usage record (savings.py owns the extraction; compute_savings_spend derives cache reads from usage_object itself), hits are counted order-independently so the overall hit rate matches its covered denominator, caller-chosen session ids are bounded before entering the primary key, and a poisoned statement drops only its own session's remaining turns. Return misses inside the recorded TTL are named for what the telemetry shows (within_ttl) rather than a presumed cause, since a provider can evict early. Savings ride each router's derived baseline by default, so the response carries no deployment-wide baseline label. Rollup retention has its own maximum_autorouter_session_retention_period setting, pattern-identical to the spend-logs knob and running in the same cleanup job on its own cutoff. Every drain trigger sizes the queues through one owner and the enqueue honors disable_spend_logs beside the tool-usage queue it mirrors. |
||
|
Yassin Kortam
|
09dd167b5a
|
feat(sgr): make the gateway middleware the source of truth for successful requests (#35717)
SGR has had two independent definitions. The admin UI derived it from SpendLogs, so it counted what litellm's logging callbacks observed and could attribute and price. BillableRequestMetricsMiddleware counted what the proxy actually answered at the ASGI edge, but only exported to OTLP for enterprise metering. The two disagree by design in places, and the SpendLogs figure goes quiet whenever spend logging is disabled or the callbacks are bypassed. This adds LiteLLM_DailyGatewayRequests, written by the middleware, and points the dashboard's Successful Requests tile at it. Requests fold into an in-memory map at record time rather than going through a queue like the spend path. A count is a pure aggregate, and every dimension of the key is chosen by the proxy from a closed set: the date, the category, and a route that the classifier maps to one of a fixed list of strings rather than passing the raw path through. Nothing a caller sends can add a key, so the fold and the table are bounded by (days x categories x routes) however much traffic arrives; the spend queue blocks once full, which is not acceptable in the response path. A scheduler job drains it on the existing batch interval, and a failed flush merges its counts back so a database blip undercounts nothing. The middleware previously returned early when no billing recorder was injected, which is the unlicensed case. The new sink is not license-gated, so that early return now requires both sinks to be absent. The billing recorder keeps its 2xx-only gate; the sink takes every status so failed_requests is real. The sink is not told which deployment served the request, unlike the billing recorder. That id is a sha256 over litellm_params, credentials included, so a caller who puts a credential in the request body mints a fresh one per distinct value. No configuration is needed for that: api_base and base_url are on _BANNED_REQUEST_BODY_PARAMS and need allow_client_side_ credentials, but api_key is not on that list, and both reach the same _handle_clientside_credential branch. The read endpoint aggregates the dimension away regardless, so the key is better off without it. The new table carries no key, user or team dimension, so /gateway/daily/activity is restricted to proxy admin roles and the per-key and per-model breakdowns keep reading the daily spend tables. The old path is left running and marked with TODOs. A fetched result carries the range key it was fetched for, and the render selects it only when that key matches the range on screen. Both the gateway counts and the spend aggregate go through that rule: the request tiles read the first and fall through to the second, so stamping only one of them would leave the tile showing a superseded range by the other route. The paginated pages behind that aggregate are reached through a failure flag, so the flag is stamped too. A flag left over from the previous range would let those pages through while a new range is in flight, which is the same defect one fallback further down. |
||
|
Mateo Wang
|
45ed654603
|
Merge pull request #35962 from BerriAI/litellm_fix_basedpyright_over_limits
fix(lint): bring basedpyright rule counts back under their budget limits |
||
|
mateo-berri
|
469d5126f6 | fix(lint): bring basedpyright rule counts back under their budget limits | ||
|
Yassin Kortam
|
0659738b3e
|
fix(migrations): recover from an interrupted Prisma toolchain install (#35832)
The Prisma CLI is a Node program that installs a private Node runtime on its first invocation. That one-time install shared the 60s budget that bounds each migration command, so on a slow or cold machine it was killed before it could finish. Prisma then decides whether to reinstall by testing the cache directory for existence alone, and a killed install leaves that directory behind, so every later attempt skipped the install and failed on a node binary that was never written. The existing four-attempt retry loop could not help: each attempt hit the same missing binary, which turned a slow start into a container that never migrated again. Migrations now prepare the toolchain as its own step under its own budget, and a cache directory that exists without a node binary is deleted first so an interrupted install reinstalls instead of persisting. Both budgets are overridable, LITELLM_PRISMA_BOOTSTRAP_TIMEOUT for the install and LITELLM_PRISMA_COMMAND_TIMEOUT for each Prisma command, and every previously hardcoded timeout now goes through one helper rather than thirteen literals. The per-command default stays at 60s. An override is only honoured when it parses as a finite positive number. Infinity and NaN parse as floats and survive a plain positivity check, and subprocess treats either as no deadline at all, so a value like `inf` or a fat-fingered `1e400` would have silently disabled the timeout it was meant to configure. |
||
|
ryan-crabbe-berri
|
9ea5cfce0e
|
fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db (#35165)
* fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db The model cost map and Anthropic beta headers reload schedules kept their last-run time in a per-pod module global, so GET /schedule/*/status reported last_run null after any restart and the Admin UI showed the reload as never having run. The reload check also only ran from the add_deployment job, which is registered only when store_model_in_db is true, so config-file deployments stored a schedule that never fired. Persist last_run_at and reload_requested_at as dedicated columns on LiteLLM_Config, owned by the reload job and manual reload endpoints, while the schedule endpoints own the param_value JSON (interval_hours); no writer can clobber another's fields. Serve status entirely from the row. Register the check as its own periodic_reload_job outside the store_model_in_db gate. Replace the force_reload boolean with a reload_requested_at timestamp each pod compares against its own in-memory last reload, so a manual reload reaches every pod exactly once instead of being cleared by the first poller. Run the blocking fetches via asyncio.to_thread, and stamp last_run_at with update_many so a schedule cancelled mid-poll is not resurrected. * fix(proxy): compare reload requests against pod data age seeded at boot A pod that had never reloaded kept its in-memory clock at None, and with no interval configured nothing ever set it, so every manual reload request was ignored by every pod except the one serving the click (Greptile P1 on the previous commit). Seed the per-pod timestamp at boot as the time its data was loaded and reload whenever a request or the interval is older than that, which also removes both None special cases from the due predicate. A schedule whose row has no last_run_at fires on the next tick so the first run does not wait a full interval. * fix(proxy): scope reload persistence to the model cost map and seed the pod clock from the actual load time Revert the Anthropic beta headers reload path to its previous JSON-flag implementation so this PR only changes the price data reload; the beta headers path keeps working exactly as before and can migrate to the shared module in a follow-up. The unused columns on its config row are inert. Seed model_cost_map_loaded_at from the timestamp get_model_cost_map records at the actual import-time fetch instead of ProxyConfig construction time, closing the startup window where a manual reload request stamped between the fetch and the constructor compared as older than the pod's data and was skipped (Greptile P1 on the previous commit). * refactor(proxy): drop the legacy force_reload backfill from the reload tracking migration The backfill only carried over a manual reload clicked in the seconds before an upgrade, and every upgrade restarts the pods, which re-fetch the cost map at import and so already deliver what that request asked for. Removing it makes the migration schema-only, so prisma db push and prisma migrate deploy leave the database in the same state instead of diverging on a data statement that only one of them runs. * fix(proxy): stamp reload timestamps at the precision they are stored at Postgres stores these columns as TIMESTAMP(3) while Python stamps microseconds, so a pod comparing its in-memory clock against the persisted copy of the same instant read as newer and skipped the reload request it had just recorded. Truncate every stamp to milliseconds at the source, and floor the boot seed the same way, so the in-memory value and its persisted copy compare exactly. * fix(proxy): identify manual reloads by revision instead of comparing timestamps Comparing a request timestamp against each pod's data age made correctness depend on clock resolution: Postgres stores TIMESTAMP(3) while Python stamps microseconds, and two events inside the same millisecond are indistinguishable no matter how the comparison is written. Replace reload_requested_at with a reload_revision counter the manual reload endpoint increments atomically in the database. Each pod records the revision it last applied and reloads whenever the row's differs, so a request reaches every pod exactly once regardless of clock skew or precision, and concurrent requests publish distinct revisions instead of overwriting one another. A pod adopts the current revision on its first poll, since data it loaded at boot already satisfies any earlier request. Interval reloads still key off the pod's own data age, where hour scale comparisons make precision irrelevant. * fix(proxy): seed the applied reload revision at startup A pod adopted whatever revision it found on its first poll, so a manual reload published while the pod was starting was marked applied without ever being served and the pod kept the prices it fetched at import. Read the row once at startup instead, right after that fetch, and treat a missing row as revision 0 * style(tests): revert incidental reformatting of test_proxy_server.py An earlier ruff format run reflowed the whole file from its 88-column formatting, adding ~1150 lines of churn unrelated to this PR. Replay only the real test changes onto the original formatting * fix(proxy): serve an outstanding reload request on a booting pod Seeding the applied revision at startup left a window: a manual reload published after the import-time cost map fetch but before startup read the row was marked applied without ever being fetched, stranding that pod on stale prices when no interval was configured. A pod now starts unapplied and serves any outstanding request on its first poll, which costs one redundant fetch per boot and removes the window along with the seeding step * fix(proxy): accept a reload interval still encoded as JSON text param_value is written with safe_dumps, and a raw row read can return it decoded or as a string depending on the driver. Strict validation rejected the string, so the schedule read as disabled and an admin's configured reloads silently stopped. Mirrors the guard ConfigRepository.get_param already carries for the same column * fix(proxy): cancel a reload schedule without resetting the revision * fix(proxy): null the interval in JSON so cancelling keeps the revision prisma rejects a null literal for a Json? column, so update_many writes an interval-less object instead. The fake config table now rejects the same input the database does, which is what the live run caught and the mock did not. Also records the run before adopting the revision, so a failed status write leaves the request unserved for the next poll rather than reporting a run that never landed. * fix(ui): match the CI-generated user_role union order in schema.d.ts |
||
|
tin-berri
|
9e3a8df6c0
|
feat(spend): add net auto-router savings to the cost-optimization dashboard (#35521)
* feat(spend): add net auto-router savings to the cost-optimization dashboard The dashboard credited compression and prompt caching but said nothing about the optimization that picks the model, so the driver with the largest lever on a bill was the one an operator could not see. Savings are the counterfactual: without a router a deployment runs one model, and it has to be one that can carry the hardest request, so the baseline is the priciest model in the router's hardest configured tier. A cheap tier is a choice the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model` overrides it for operators who would genuinely have run something else. Both are provider-qualified before pricing, because a bare name can resolve to a different vendor's rates or to nothing at all, and a deployment is priced by its `base_model` where it has one, which is how Azure deployments are priced everywhere else. Both arms price the request's real usage through `generic_cost_per_token` rather than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers and regional uplifts stay consistent with what was actually billed. `prompt_tokens` already includes the cache buckets, so charging them again at the input rate would price the same tokens twice. Cache state is what makes this hard. The baseline serves every turn, so whether it had the prompt cached is whether the conversation was already underway. On a continuing conversation it wrote the prompt earlier and would only read it now, so this request's write is what switching cost and counts against the saving. On a first turn nothing was cached for any model, the baseline would have written the same prompt, and both arms carry the write at their own rates. Charging the write to both cases understates a first turn to a few percent of its value, and because the write premium is fixed by prompt size while the saving grows with completion length, it can render a profitable route as a loss. That shape is read off the conversation rather than remembered: a second human ask means an earlier turn was served. No cache, no session id, and no dependence on a caller sending a session header. It cannot see a switch on a turn the router did not classify, and it reads a few-shot prompt's synthetic turns as prior conversation; both err toward charging the write, which under-claims. The baseline and the shape ride on the existing `routing_decision` record, which is already carried from the router to the spend log, already classified for redaction, and already written-or-cleared per attempt. A fallback that re-enters the hook therefore cannot leave either fact behind to be attributed to a deployment that never routed, and no new metadata key crosses the trust boundary. The result is signed. Whether a switch pays off is a race between the rate gap and the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly the routing behaviour an operator needs to see. The donut plots only drivers that saved, while the card and range total keep the sign. Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup tables, declared `NotRequired` because rows queued by a pod on the previous release carry no such key. It is summed by the rollup merge the cross-pod Redis drain also runs, and carried through the aggregation query, the per-row accumulation and the response model, so the dashboard reads a value the API actually sends. Tests enumerate the drivers from the response model itself and assert each is summed, accumulated, carried and totalled, so one added later cannot be half-wired. * fix(spend): let the baseline pay for a continuing turn's own growth `_baseline_usage` moved every cache-creation token into the baseline's read bucket whenever the conversation was underway. That is right for a switch, where the baseline never left the model it was on and really would only read, but wrong for a turn that stayed put: the prompt grew, and the tokens written are that growth. They are new to every model, so the baseline would have paid to write them too. Forgiving it that write made the counterfactual cheaper than it was and shrank the reported saving on ordinary steady-state traffic, by about 2% per turn. The selected arm was never involved; it has always been priced on the real usage. The error sat entirely on the baseline. The condition is that the request read more than it wrote, not that it read anything. A switch onto a model already holding a small prefix of this prompt still writes most of it, and that write is the switch's own cost; keying off a nonzero read would have handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing the two buckets separates a warm continuation, which reads far more than it writes, from a cold arrival, which does the reverse, and it leaves the existing invariant intact: a request reading 0 and one reading 1 both still land in the same place. * fix(spend): price each arm under the key litellm billed it, and see agent turns Two ways the savings number read the wrong thing, both from identifying a model by its name when the name is not what it costs. The counterfactual was ranked and priced on the public rate for the model a deployment names. A deployment may not be charged that rate: the router registers its configured prices under the deployment's own id and deliberately keeps them off the shared model-name key so deployments sharing a backend model do not pollute each other. So a hardest-tier deployment configured above its public rate lost the ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays. Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision, the resolver the real request is billed through, rather than a second rule here that would have to re-learn that per-second and tiered overrides count, that a partial override still counts, and that a deployment configured at zero is priced at zero rather than treated as unpriced. The arm being subtracted had the same fault and a sharper edge. It priced the spend log's `model`, which on Azure is the deployment name, absent from the cost map, so the whole driver silently read zero for that traffic. It no longer re-derives anything: `model_map_information.model_map_key` is what litellm actually billed the request under, recorded at request time by that same resolver with `base_model` and custom pricing already applied. Separately, the conversation-shape discriminator counted human asks, and an agent loop can run twenty turns on one of them. Its tool traffic rides `tool_result` blocks on user turns that flatten to empty text, and `tool` roles that are never read, so a long agentic conversation looked like its own first turn and was handed the arithmetic that leaves the cache write on both arms. That is the one direction this must never fail in, because it inflates. An assistant turn is the direct evidence that something answered earlier, and it is blind to how the tool plumbing is spelled on either surface. * fix(spend): give the cost-key resolver both inputs the selected arm needs The served model was resolved through one input at a time, and each choice broke the half the other fixed. `model_map_key` is the served model already resolved through `base_model`, which is the only way an Azure deployment name reaches the cost map at all; without it the selected arm priced a name absent from the map, returned nothing, and the whole driver silently read zero for that traffic. But it is built without `router_model_id`, so it never carries a deployment's own price overrides, and a custom-priced deployment was compared at its public rate while the baseline used the real override. On a deployment configured well above its public rate that inverted the answer outright: a route that lost $21.88 reported saving $0.10. `_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a deployment stays its decision rather than a rule restated here. * fix(spend): same model is only the same cost when it is the same deployment The short-circuit compared resolved model identity, so two deployments of one model collapsed to "no switch" and reported zero. They are not the same cost: a deployment can carry a negotiated rate, and routing from the dear one to the list-price one is a real saving the dashboard reported as $0.00 against a true $21.93. Both arms now carry the key litellm prices them under, so the comparison is between deployments rather than between names. * refactor(spend): price from resolved rates, not from a name we keep re-resolving Four review rounds landed on one mechanism: which identifier prices a deployment. base_model, then the deployment id, then cache-only overrides. Each round added a clause to a resolution rule that should not exist, and a wrong primitive fails once per input shape, so each shape arrived as its own finding. `Router.get_deployment_model_info` already owns this. It merges a deployment's configured prices over the built-in map, folds in `base_model` defaults for deployments whose name is not a model, and falls back to the model name when nothing is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure) was that function re-implemented badly. `generic_cost_per_token` now accepts already-resolved rates instead of demanding a name it looks up itself, which is what forced the name-bending in the first place. Both arms resolve through the owner and pass what they got: the counterfactual by the deployment the router would have used, the served request by the deployment that served it. The invented cost-key resolver is gone, and `Baseline` carries a deployment id rather than a key we chose on litellm's behalf. Net 64 insertions against 79 deletions. * test(spend): follow _most_expensive onto the router that prices its candidates Ranking moved through `Router.get_deployment_model_info`, since what a deployment costs is the router's answer to give; these four cases were still calling the old free-function signature. * fix(spend): rank baseline candidates by what a request costs, not by two rates "Most expensive" was decided by comparing output rate then input rate. That is a property of a rate, not of a request: a deployment dearer per output token can be cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and recorded the wrong counterfactual. Candidates are now costed on one reference request through the same engine the savings themselves use, which leaves cache read and write rates, tiered tables and every other billing dimension to that engine rather than to another rule restated here. The reference request is cache-heavy because auto-routed traffic is. * fix(spend): pick the baseline against the request that ran, not a stand-in for one Ranking happened in the pre-routing hook, where the request has not executed yet, so candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest tier holding a deployment with non-proportional configured rates could be ranked for a request nothing like the one served. The mix is known on the spend path, so the ranking belongs there. The routing decision now carries the tier's candidates rather than a winner already chosen, and the baseline is resolved against the usage that actually happened. The reference workload is gone; nothing here assumes a traffic shape any more. The router is passed in rather than imported from `proxy_server` inside the computation, so the savings stay a pure function of their arguments and the caller owns where the router comes from. That also makes the spend path testable without a running proxy, which the previous shape was not. * refactor(spend): measure savings against one configured model, not a derived one The counterfactual was derived per request: enumerate the hardest tier's deployments, resolve each one's effective pricing, price them all, take the dearest. That machinery produced a review finding per input shape it had not anticipated, and every answer it gave was one an operator could have stated in a line of config. So they state it. `litellm_settings.autorouter_savings_baseline_model` names the model the traffic would have run on without a router, for every auto-router on the proxy, and unset means the driver is off rather than a model nobody named being guessed at. `savings_baseline.py` and its tests are deleted outright, along with the tier enumeration, the candidate list on the routing decision, and the per-deployment override that shadowed it. Cache-state handling is untouched: the baseline is still priced on this request's own read and write split, so a switch still pays for re-warming the cache and a first turn still charges the write to both arms. 45 insertions against 482 deletions. * refactor(router): compute the conversation shape once and pass it down `_classify_and_route` re-derived it from the messages the hook had already resolved, so an ordinary routed request walked the turn list twice for one boolean. The hook computes it and hands it over, which is also where the affinity-hit path already got it from. Also moves `_get_llm_router` below the imports it sat among. * fix(router): drop the dead conversation_continuing parameter off the hook It was added to `async_pre_routing_hook` by mistake and immediately overwritten by the value the hook computes, so it never did anything. It also widened a signature every pre-routing strategy shares with the protocol in `types/router.py`, leaving this one router diverged from `AutoRouter` and the interface for no reason. Also records why an unreadable request counts as continuing: no messages is no evidence a turn was served, so it pays the cache write and under-claims rather than being handed a first turn's larger saving on nothing. * fix(spend): charge a baseline its input rate for cache buckets it cannot price A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket. * refactor(spend): build the daily upsert payloads in one shot `common_data` and `update_data` were constructed and then appended to: `request_id` conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that grows after its literal cannot be reasoned about by reading the literal, which is the whole point of building it at once. The conditional key resolves to a spreadable value before either payload, so both are single expressions and the tag branch appears once instead of twice. Not wrapped in MappingProxyType, though it was suggested: these go straight to prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through to the serializer and raises `TypeError: Type <class 'mappingproxy'> not serializable` inside the batch upsert, where the surrounding except would log it and leave the rollups silently unwritten. * fix(spend): keep the one-shot upsert payloads under the type-discipline budget Building both payloads as single literals traded a mutation for two dict literals, and LIT002 counts construction rather than mutation, so the change the review asked for is the one the gate charges for. The empty branch is the avoidable half: it is the same value every time, so it moves to a module constant built once instead of a literal per transaction, and it is a read-only mapping so none of the call sites that spread it can fill it in later. |
||
|
Yassin Kortam
|
87c2e03af8
|
feat(db): opt-in REPLICA IDENTITY FULL after prisma migrations (#35267)
Logical replication consumers need FULL replica identity to reconstruct the old row of an UPDATE or DELETE, and prisma leaves every table it creates at the postgres default. Operators had to re-apply the setting by hand after each migration run. Setting LITELLM_SET_REPLICA_IDENTITY_FULL now re-asserts it on every LiteLLM table at the end of a successful migration run, through the prisma CLI so the dependency-free proxy-extras package stays that way. Tables that are already FULL are skipped, foreign tables in the same schema are left alone, and a database that refuses the ALTER is reported rather than failing the run. Resolves LIT-3022 |
||
|
Tin Chi Lo
|
c8b0530c30 |
fix(proxy): roll up tool spend daily instead of scanning SpendLogs
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
|
||
|
Tin Chi Lo
|
26f6ff24d8 |
fix(proxy): cap tool spend window at 30 days and bound every SpendLogs read
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to LiteLLM_SpendLogs with a start_time-only predicate the composite (tool_name, start_time) index cannot serve, and the dedup total query left the outer SpendLogs scan unwindowed, so every dashboard load walked both per-request tables end to end. - clamp the window to the most recent 30 days ending at end_date; the response start_date reflects the effective window and the dashboard notes the clamp - index SpendLogToolIndex on start_time (all schema copies + migration) - window the SpendLogs side of both queries (1s margin: the two writers can disagree by ~1ms on the same request) - expire SpendLogToolIndex rows on the spend-log retention cutoff via a parametrized batch-delete engine shared with the SpendLogs cleanup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Tin Chi Lo
|
cfa074d970 | fix(mcp): make the EMA retention gate authoritative across pods and tighten the assertion store | ||
|
Tin Chi Lo
|
83e147b553 | feat(mcp): store the enterprise IdP identity assertion at SSO login for EMA egress | ||
|
tin-berri
|
3f3295b33f
|
feat(spend): track prompt compression saved tokens in daily spend aggregates (#33810)
* feat(spend): track prompt compression saved tokens in daily spend aggregates Native compression interception now records tokens_before/after/saved into the request litellm_metadata so savings land in the SpendLog metadata JSON under a typed compression_savings key. A single normalizer (extract_compression_saved_tokens) sums that key with Headroom guardrail tokens_saved; the two writers are disjoint and run at different stages, so summing never double-counts. The spend-log redactor now preserves purely numeric compression stats inside guardrail_response so Headroom savings survive the store_prompts_in_spend_logs=false default. compression_saved_tokens is threaded through BaseDailySpendTransaction, queue aggregation, the daily upsert blocks, a new BigInt column on all six daily spend tables, and the daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups) * fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader * feat(spend): aggregate compression and prompt caching dollar savings in daily rollups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): update daily spend aggregation fixtures for savings columns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): add Cost Optimization dashboard page New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows. Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tin Chi Lo
|
99b85a3f2c |
fix(mcp): persist config.yaml DCR clients in a server-scoped store
Config.yaml-declared OAuth2 MCP servers using Dynamic Client Registration have no LiteLLM_MCPServerTable row, so the DCR persist path called update_mcp_server, which returns None for a missing row, then update_server(None), which dereferenced .approval_status and raised AttributeError. The exception was swallowed to a warning while /register still returned 200, so the minted client was never stored and every access-token expiry forced a full re-authorization Persist the acquired DCR client (client_id, client_secret, token_endpoint_auth_method, redirect_uris, encrypted at rest) in a dedicated LiteLLM_MCPServerOAuthClient store keyed by server_id when the server has no row, overlay it onto the in-memory config server so the refresh_token grant can authenticate within the process, and rehydrate it when the registry syncs from the database (which runs after the DB connects, unlike config load) so restarts and other pods pick it up. The store is encrypted at rest and is re-encrypted by the master-key rotation path alongside the server rows, through a shared helper so the two sites cannot diverge. The DB-backed server path is unchanged, and guarding the None return removes the swallowed-crash footgun Resolves the config.yaml DCR persistence regression introduced in v1.92.0 by #31912 |
||
|
Tin Chi Lo
|
8e73ff057f |
feat(mcp): issuer-anchored OAuth discovery (RFC 8414 §3.3) as the trust anchor
Adds an admin-configured issuer to MCP servers. When set, OAuth metadata is fetched from the issuer's own origin and adopted only when the document self-attests that same issuer (RFC 8414 §3.3), making token_endpoint, registration_endpoint, and scopes authoritative for the pinned issuer instead of a document the MCP resource server chose. This closes the mix-up where a compromised resource echoes a pinned authorization_url to smuggle its own token endpoint and inflated scopes past the corroboration gate. Discovery is same-authority against the issuer origin, fails closed on a §3.3 mismatch, and does not fall back to resource-rooted discovery. Rows without an issuer keep the existing corroboration-gate behavior unchanged. Backend + schema only; UI field and live-proxy proof follow. |
||
|
devin-ai-integration[bot]
|
001457af8b
|
fix(keys): persist key_type so the UI shows correct key scope instead of "All Proxy Models" (#33115)
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models' key_type is not persisted on a key (the proxy maps it to allowed_routes and drops it), so the keys tables only inspected the models list and rendered 'All Proxy Models' for any key with an empty models array, including SCIM, Management and Read-only keys that cannot call a single model. Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a scope tooltip for those recognized scopes; unrestricted, AI-API and custom keys keep the existing model-list rendering. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): move key_scope helper to components root Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(keys): persist key_type on virtual keys so the UI reads scope directly Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy, and proxy-extras schemas plus an additive migration) and stop dropping the value in handle_key_type, so management/read_only/llm_api/default keys store their type alongside the derived allowed_routes. Surface it on the key read and create response models. The dashboard now prefers the persisted key_type for the no-inference buckets and keeps the allowed_routes derivation as the fallback for keys created before the column existed (key_type null), so no backfill is required. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(keys): add key_type column to LiteLLM_DeletedVerificationToken The deleted-token archive model inherits key_type from the verification token, so regenerate/delete flows write key_type into LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration) so the archive insert does not fail with FieldNotFoundError. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tin Chi Lo
|
41a43d5283 | feat(mcp): add dcr_bridge column and plumbing for client-forwarded auth modes | ||
|
tin-berri
|
db2402754a
|
feat(mcp): let users select the entra_obo token_exchange profile in the UI and API (#32144)
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
tin-berri
|
ff6dc33291
|
feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard (#31772)
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be configured through config.yaml; the create/update REST API and the dashboard had no way to express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end. These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first and falls back to the credentials blob so servers persisted before the columns existed still load. client_id and client_secret continue to ride the existing encrypted credentials path. On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from non-admin and virtual-key responses, matching how token_url is treated. * fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code: when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could not mint a token), the user must re-authorize via the browser flow. token_exchange has no gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was undefined (a compile error) and, per this file's convention and its tests, meant authorization_code; renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode instead and drop the now-unused isTokenExchange * fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes Switching an existing oauth2 server to oauth2_token_exchange left the old flow's token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url as the configured exchange endpoint, so the stale value both suppressed the RFC 9728/8414 discovery this PR adds and sent the exchange grant (client credentials plus the user's subject token) to the previous flow's token endpoint update_mcp_server now mirrors its existing stale-credentials rule for the flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow, token_exchange_endpoint, audience, subject_token_type): when auth_type changes, each one is cleared unless the same request explicitly provides it, so a deliberate override in the switch request still wins. Updates that keep the auth_type never touch these columns, which keeps legacy OBO rows that use token_url as their exchange endpoint working The edit form sends explicit nulls for the previous flow's fields on an auth type switch; antd preserves unmounted field values by default, so without this the old token_url would be re-sent verbatim and read as an explicit override. Transitions are detected against the persisted auth_type, so saves that keep the auth type send nothing extra Reported by Cursor Bugbot on the PR * fix(mcp): lift legacy blob token-exchange settings into their columns on every write The three token-exchange settings live in dedicated columns but also exist on MCPCredentials as the pre-column REST shape. Writes now lift incoming blob values into the columns (an explicit top-level value wins, including an explicit null) and strip them from the stored blob; the same-auth credentials merge migrates legacy rows the same way. The read-time column-or-blob fallback then only ever serves rows current code has never written, so clearing a column to re-enable RFC 9728/8414 discovery can no longer be silently undone by a stale blob copy. Also asserts the auth-switch clearing fires on the external fields_set path (PUT /v1/mcp/server). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(mcp): single source for the RFC 8693 default subject_token_type The default was applied at four egress build sites plus two model defaults, each with its own copy of the literal. All sites now share DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is deliberately not used: Prisma writes explicit values on insert, so a column default would rarely apply, and NULL-means-RFC-default keeps existing rows correct. Also documents two review decisions in place: the audience column keeps the RFC 8693 parameter name (RFC 8707 resource indicators are already a separate concept named resource in the v2 egress types), and the migration's out-of-order timestamp is safe under prisma migrate deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fix import sort order in outbound_credentials/types.py (I001 strict budget) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials The migrate-on-write in the credentials merge lifts blob values into null columns, which is correct for legacy rows but could repopulate a column an admin had cleared in an earlier no-credentials update (that path never touched the blob, so the stale copy survived to be lifted later). An explicit token-exchange column write (set or clear) now migrates the row even when the update carries no credentials: untouched null columns are lifted, every blob copy is stripped, and unrelated blob keys stay as-is. A cleared column can then never be resurrected, because no write path leaves a blob copy behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(mcp): state the blob-to-column lift contract on the legacy credential keys The three token-exchange keys on MCPCredentials are the pre-column REST shape (the only REST shape from 2026-05 until this PR). Document on both the blob type and the request models that the dedicated columns are authoritative and that writes lift blob values into them and strip the stored copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers The other two token-exchange fields were cleared while subject_token_type was left visible. It is a public RFC 8693 URN with no disclosure value, but the sanitizers' rule is that these views receive no token-exchange config at all — cleared for uniformity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Krrish Dholakia
|
28ddad271e
|
feat(proxy): add key-level budget_fallbacks to reroute requests when a per-model budget is exceeded (#31783) | ||
|
tin-berri
|
58de920921
|
feat(mcp): bound outbound tool-call concurrency per MCP server (#31641)
Add an optional per-server max_concurrent_requests that caps how many tool calls LiteLLM sends to one MCP server at once, so batch-processing backends are not overwhelmed by unbounded parallel dispatch. Excess calls queue on a per-server asyncio.Semaphore instead of being rejected. Unset or non-positive means unlimited, preserving existing behavior. Resolves LIT-2749 |
||
|
Krrish Dholakia
|
cca71a07c2
|
feat(mcp): add mcp_tool_search virtual tools for large tool catalogs (#31777)
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
|
||
|
Mateo Wang
|
4ec4ab99d0
|
feat(mcp): per-server env vars with global + per-user scopes (#28917) | ||
|
Sameer Kankute
|
d671a09c20
|
Litellm oss staging 050626 (#29774)
* Mark xAI models retiring on 2026-05-15 (#28788) Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3 with various reasoning efforts; callers continuing to use the old slugs will be billed at grok-4.3 pricing): grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort) grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none) grok-4-fast-reasoning -> grok-4.3 (low effort) grok-4-fast-non-reasoning -> grok-4.3 (none) grok-4-0709 -> grok-4.3 (low effort) grok-code-fast-1{,-0825} -> grok-build-0.1 grok-3 -> grok-4.3 (none) Only the direct xai/ slugs are tagged; third-party hosts (azure_ai, oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The grok-3 retirement list explicitly names only the base grok-3 slug — the -mini / -fast / -beta / -latest variants are not listed, so they remain untouched. * feat(moonshot): advertise json_schema response support on live models (#29683) litellm.responses() already routes Moonshot through the responses->chat-completions bridge, and Moonshot honors response_format json_schema on chat completions. The cost-map entries left supports_response_schema unset, so discovery layers that gate on that flag dropped Moonshot from structured-output / responses listings even though the capability works end to end. Set supports_response_schema on the nine models currently live on api.moonshot.ai: kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants, and moonshot-v1-auto. Verified against the live API that each honors json_schema and that litellm.responses() returns schema-valid structured output through the bridge. * chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685) Thirteen Moonshot/Kimi models in the cost map no longer resolve on api.moonshot.ai (all return 404). Stamp each with its deprecation_date from platform.kimi.ai/docs/models rather than deleting the entries, so historical cost calculation keeps resolving the names while tooling can surface the retirement. Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot publishes no discontinuation date for them). * fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687) Kimi reasoning models reject every temperature except 1; a request with temperature=0.2 returns "invalid temperature: only 1 is allowed for this model". litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd. Drop the temperature param entirely for reasoning models (gated on supports_reasoning, the same signal transform_request already uses) so the model default is used; the non-reasoning moonshot-v1 models keep the existing clamp. Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(mcp): add per-server timeout configuration (#29672) * feat(mcp): add per-server timeout configuration * fix(mcp): address timeout field review comments - use is not None guard instead of or for 0.0 edge case - copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table) - add timeout Float? column to all three schema.prisma files - extend round-trip test to cover _build_mcp_server_table direction - add test for zero timeout not treated as falsy * fix(mcp): forward timeout in _build_temporary_mcp_server_record * fix(mcp): return 504 instead of 500 when per-server timeout fires * test(mcp): add 504 timeout regression test; fix black formatting * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(soniox): add soniox audio transcription integration (#29508) * feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650) The OpenMeter callback resolves the CloudEvent subject from kwargs["user"] first, then falls back to the key-bound user_api_key_user_id. For multi-tenant proxy deployments, a client can set `"user": "..."` in the request body and cause their usage to be attributed to that arbitrary string — a billing-attribution forgery risk. Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward compatibility). When set to "false", the request-supplied `user` field is ignored and the subject is resolved solely from user_api_key_user_id. Matches the existing env-var-driven config pattern in this file (OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE). * feat(search): add you_com as a search provider (#28370) * feat(search): add you_com as a search provider Registers You.com Search API as a first-class `search_provider` in the `search_tools` registry, alongside Tavily, Exa, Perplexity, etc. - New adapter: litellm/llms/you_com/search/transformation.py - POSTs to https://ydc-index.io/v1/search - Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key) - Maps Perplexity unified spec: max_results -> count, search_domain_filter -> include_domains, country -> country - Flattens results.web + results.news into a single SearchResult list; snippet prefers snippets[0], falls back to description; page_age -> date - Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired into ProviderConfigManager.get_provider_search_config() - Pricing entry: model_prices_and_context_window.json (placeholder $0.0; happy to adjust to maintainers' preferred public number) - Docs: example router config snippet and example proxy yaml updated - Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests (payload shape, domain filter mapping, snippet fallback, news flattening, missing-api-key error) Refs upstream expansion signal: #15942 * review fixups: normalize api_base, lowercase country, scope env-var to test Addresses Greptile inline review comments on #28370: - get_complete_url: strip trailing slashes from api_base *before* the endswith("/v1/search") check, so a custom base like ".../v1/search/" doesn't become ".../v1/search/v1/search". - transform_search_request: .lower() country before sending, matching Tavily's convention so callers using the unified spec form ("US") get consistent behavior across providers. - Tests: replace direct os.environ writes with an autouse monkeypatch fixture so YOUCOM_API_KEY is set per-test and removed afterwards. The missing-key test now uses monkeypatch.delenv. New test asserts the trailing-slash normalization above. Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note that documentation changes belong in the litellm-docs repo. * support keyless free tier (api.you.com/v1/agents/search) as default You.com offers an IP-throttled keyless endpoint that returns the same response shape as the keyed one (~100 queries/day, no signup). This is a significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG providers already in the search_tools registry. Behavior: - YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search (X-API-Key header) - no key -> free: POST https://api.you.com/v1/agents/search (no auth) - YOUCOM_API_BASE override -> honored as-is Tests: - New: test_you_com_search_keyless_free_tier - asserts URL + absence of X-API-Key when no key is configured. - New: test_you_com_search_validate_environment_keyless - asserts the config no longer raises when the key is absent. - Removed: test_you_com_search_raises_without_api_key (the precondition no longer holds). - Existing payload/domain-filter/etc tests still cover keyed mode via the autouse YOUCOM_API_KEY fixture. Verified both endpoints accept POST + return identical JSON shape: results.web[] / results.news[] with title, url, snippets, description, page_age. * register you_com in provider_endpoints_support.json Adding `litellm/llms/you_com/` requires a corresponding entry in provider_endpoints_support.json or the code-quality/check_provider_folders_documented CI check fails. Follows the compact tavily/serper pattern - endpoints: { search: true }. Local run of the check now reports "All 114 provider folders are documented". * move tests under tests/test_litellm/llms/ so CI exercises them The litellm CI workflows scope unit tests to `tests/test_litellm/...` (see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so tests living under `tests/search_tests/` are never run in CI - which is why codecov reports 0% patch coverage for the new adapter even though the unit tests exist and pass locally. Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so the test-unit-llm-providers job picks it up. 7/7 tests still pass at the new location. (Sibling search-only providers - tavily, exa_ai, brave, etc. - still live only in `tests/search_tests/` and would benefit from the same move, but that is out of scope for this PR.) * fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises Content-Encoding: gzip but returns a body that httpx's decoder rejects with `zlib.error: Error -3 while decompressing data: incorrect header check`, surfacing as litellm.APIConnectionError in user code. curl works because it doesn't request compression by default. Pin Accept-Encoding: identity in validate_environment so the upstream server skips compression entirely. Harmless on the keyed endpoint (ydc-index.io/v1/search) which negotiates content-encoding correctly. The header uses setdefault so a caller-supplied Accept-Encoding still takes precedence. (Server-side bug has been flagged to the You.com team separately - once fixed there, this workaround can be removed.) New unit test: test_you_com_search_pins_identity_accept_encoding. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * docs: fix README typo (#29419) Correct clear spelling mistakes in documentation without changing behavior. Confidence: high Scope-risk: narrow Tested: git diff --check; uvx codespell on changed files Not-tested: Full docs build not run; text-only changes * Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480) * fix(langfuse): pass ssl_verify to Langfuse httpx client * fix_langfuse_ * add unit tests * addressed comments --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(models): add minimax/MiniMax-M3 to model cost map (#29412) Add MiniMax's new flagship MiniMax-M3 to the native minimax provider: 512K context, 128K max output, native multimodal (supports_vision), reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output 2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so cache_creation_input_token_cost is omitted. Updated both the root model_prices_and_context_window.json (remote source) and the bundled litellm/model_prices_and_context_window_backup.json (local fallback), keeping them in sync. * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394) * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log * fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation * feat(provider): Add Neosantara provider as OpenAI Compatible (#29646) * Add Neosantara provider * Register Neosantara provider enum * Address Neosantara provider review feedback * Add Neosantara packaged endpoint support --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: address greptile and veria review feedback - langfuse: guard httpx_client injection behind version check (>= 2.7.3) - soniox: propagate audio_transcription_duration in _hidden_params for spend tracking - soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base - mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError * chore(mcp): add migration for per-server timeout column * fix(test): add tool_use_system_prompt_tokens to model prices schema validator * fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key * fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs The search flow resolves api_key in validate_environment but never passed it into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the env) set the X-API-Key header yet still selected the keyless free-tier endpoint. Forward api_key through both the search entrypoint and the http handler so the keyed endpoint is chosen. HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox poll and transcript-fetch GETs silently used the client global default instead of the caller timeout. Add a per-request timeout to get() and forward the configured timeout from the Soniox handler. * fix(soniox): price stt-async-v4 per second so transcriptions are billed The handler stores audio_transcription_duration in _hidden_params, but the model carried only token cost fields and the response has no token usage, so the transcription cost path fell through to cost_per_second and returned $0. An authenticated caller could transcribe Soniox audio without decrementing their budget. Switch the entry to output_cost_per_second at Soniox's published $0.10/hour async rate so the stored duration produces a real charge. * fix(langfuse): use a dedicated httpx client for the SDK injection The httpx_client handed to the Langfuse SDK came from _get_httpx_client(), which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that client on teardown it would invalidate the shared client used by every other LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL verification and client certificate from LiteLLM's configuration. * fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var * fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779) * fix(cohere): support max_completion_tokens on cohere v2 chat The default cohere_chat route resolves to CohereV2ChatConfig, which did not list or map max_completion_tokens, so get_optional_params raised UnsupportedParamsError for the standard OpenAI parameter (the modern replacement for the deprecated max_tokens). The v1 config already maps it to cohere's max_tokens; mirror that in v2 and add v2 regression tests. * fix(cohere): make max_completion_tokens take precedence over max_tokens on v2 When both max_tokens and max_completion_tokens are supplied, prefer max_completion_tokens explicitly rather than relying on dict iteration order, and cover both orderings with a regression test. --------- Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com> Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Dan Lemon <dan@danlemon.com> Co-authored-by: Saswat <saswatds@users.noreply.github.com> Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com> Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: kape <168134658+kapelame@users.noreply.github.com> Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com> Co-authored-by: Just R <remixingmagelang@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com> |
||
|
michelligabriele
|
3f79222350
|
fix(proxy): persist oauth2_flow on MCP server registration (#29690) | ||
|
Mateo Wang
|
6d6eda8101
|
[internal copy of #28008] Support MCP OAuth passthrough and issuer-scoped JWT auth (#28356)
* fix(proxy): point /metrics 401 at the opt-out flag Operators upgrading past |
||
|
Sameer Kankute
|
36c494fdd2
|
Litellm oss staging (#28161)
* fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455) Squash-merged by litellm-agent from Anai-Guo's PR. * feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508) Squash-merged by litellm-agent from yimao's PR. * fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711) Squash-merged by litellm-agent from krisxia0506's PR. * fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503) Squash-merged by litellm-agent from krisxia0506's PR. * Fix Gemini MIME detection for extensionless GCS URIs (#27278) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107) Squash-merged by litellm-agent from voidborne-d's PR. * feat(chart): add support for autoscaling behavior in HPA (#27990) Squash-merged by litellm-agent from FabrizioCafolla's PR. * feat(proxy): add blocked flag to models for pause/resume from the UI (#27927) Squash-merged by litellm-agent from Cyberfilo's PR. * fix: pass socket timeouts to Redis cluster clients (#27920) Squash-merged by litellm-agent from tomdee's PR. * Fix/cache token (#28009) Squash-merged by litellm-agent from escon1004's PR. * fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080) Squash-merged by litellm-agent from Divyansh8321's PR. * fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617) * fix: reset org and tag budgets (#27326) * reset org budgets * reset tag budgets --------- Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> * fix(ui): omit allowed_routes from key edit save when unchanged (#27553) * fix(ui): omit allowed_routes from key edit save when unchanged When a team admin opens Edit Settings on a key with key_type=AI APIs and saves without changing anything, the UI re-sends the existing allowed_routes value, which the backend's _check_allowed_routes_caller_permission gate rejects for non-proxy-admins (LIT-2681). Strip allowed_routes from the patch in handleSubmit when it deep-equals the original keyData.allowed_routes. The backend treats absence as "leave alone," so no-op saves now succeed for non-admins. Admins explicitly editing the field still send the new value. * fix(ui): order-insensitive allowed_routes diff + cover null-original case Address Greptile review: - Switch the "is allowed_routes unchanged" check to a Set-based comparison so a server-side reorder of the array doesn't register as a user edit and re-trigger LIT-2681. - Add two regression tests: (1) keyData.allowed_routes is null and the form is untouched — patch should strip the field; (2) server returned routes in a different order than the user originally entered — patch should still recognize the value as unchanged. * chore(ui): strip ticket refs and tighten comments in key edit fix - Remove internal-tracker references from in-code comments - Tighten the WHY comment in handleSubmit to two lines - Drop redundant test-block comments — test names already describe the case * fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc * fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests GuardrailRaisedException and BlockedPiiEntityError both lacked a status_code attribute. When these exceptions reached the proxy exception handler (getattr(e, 'status_code', 500)), the fallback defaulted to HTTP 500 — making intentional guardrail blocks indistinguishable from server errors and causing unnecessary client retries. Changes: - Add status_code=400 (keyword-only) to GuardrailRaisedException - Add status_code=400 (keyword-only) to BlockedPiiEntityError - Update _is_guardrail_intervention() to recognize both exceptions so downstream loggers record 'guardrail_intervened' instead of 'guardrail_failed_to_respond' - Add 6 unit tests for default/custom status codes and getattr pattern - Strengthen existing blocked-action test with status_code assertion Fixes #24348 --------- Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> * fix(router/proxy): address Greptile P1+P2 review comments on PR #28161 - router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429) when a specifically-addressed deployment is administratively blocked; 429 misleads retry-enabled clients into spinning forever against a paused model - proxy_server: compute get_fully_blocked_model_names() once before both branches in model_list() instead of duplicating the call in each branch - deepseek: upgrade silent debug log to warning when injecting placeholder reasoning_content so callers are clearly notified of degraded multi-turn quality - tests: update two blocked-deployment assertions to expect ServiceUnavailableError Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address bug detection findings (cache token order, mutable defaults) Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bugs in async pass-through, anthropic cache token detection, rerank tests - async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments - cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0 - dashscope rerank tests: pass request to httpx.Response constructions for consistency Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix code qa * fix(vertex_ai/gemini): strip MIME parameters from GCS contentType GCS object metadata's contentType field can include parameters such as 'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases so downstream get_file_extension_from_mime_type sees a bare MIME type. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai/gemini): clarify mime-type error message string concatenation Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Vincent <yimao1231@gmail.com> Co-authored-by: Kris Xia <xiajiayi0506@gmail.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Tom Denham <tom@tomdee.co.uk> Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com> Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com> Co-authored-by: robin-fiddler <robin@fiddler.ai> Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Krrish Dholakia
|
8bbc61e03c
|
fix: harden /key/update authorization checks (#27878)
* fix: patch Host-header auth bypass in get_request_route Starlette reconstructs request.url from the Host header. A malformed Host like `localhost/?x=1` causes Starlette to build the full URL as `http://localhost/?x=1/health`, which url-parses to path="/". Since "/" is in LiteLLMRoutes.public_routes, all protected routes became reachable without authentication. Fix: read scope["path"] (set by uvicorn from the HTTP request line, not derivable from headers) instead of request.url.path. Sub-path deployments are handled via scope["app_root_path"] / scope["root_path"], mirroring Starlette's own base_url construction logic. Affected variants confirmed fixed: Host: localhost/?x=1 Host: localhost:4000/?x=1 Host: localhost/#test Host: localhost:4000/#test Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * style: reduce comments in route fix Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block credential fields in RAG ingest vector_store options Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.) in ingest_options.vector_store are now rejected at the API boundary with a 400 error. Credentials must be configured server-side. Previously any authenticated user could supply a vertex_credentials dict with type=external_account pointing credential_source.file at an arbitrary path (e.g. /proc/1/environ) and token_url at an attacker-controlled server. google-auth's identity_pool.Credentials refresh() would read the file and POST its contents to the attacker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block /key/update self-escalation by assigned users Non-admin users who were assigned a key (created_by != caller) could update any non-budget field — models, rpm_limit, guardrails, etc. — without admin authorization, allowing privilege self-escalation. Gate: only the key creator (created_by == caller) may edit their own key without admin check; budget changes always require admin regardless of creator status. All other callers must pass _check_key_admin_access. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block user-controlled api_base in RAG ingest vector_store options A user-supplied api_base in ingest_options.vector_store caused the server to forward its configured provider credentials (Gemini, OpenAI) to an attacker-controlled endpoint via SSRF. Add api_base to the blocked credential params set alongside api_key and the existing credential fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check Any authenticated internal_user could POST arbitrary provider config (aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have the server forward its credentials to an attacker-controlled endpoint. - Gate the endpoint on PROXY_ADMIN role (403 for all other roles) - Call is_request_body_safe() to reject banned params even for admins - Convert ValueError from safety check to HTTP 400 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: apply banned-param check to /utils/transform_request Without is_request_body_safe(), any authenticated user could pass aws_sts_endpoint, api_base, or aws_web_identity_token to /utils/transform_request and have the server forward its configured provider credentials to an attacker-controlled endpoint during SDK credential resolution. Applies the same banned-param blocklist already used by LLM endpoints. Endpoint remains accessible to all authenticated users. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter Any frontmatter key not in ["model","input","output"] flowed into optional_params and was merged into the LLM call data dict, bypassing is_request_body_safe. An attacker with any bearer key could set api_base in YAML to redirect the outbound LLM request — including the provider API key — to an attacker-controlled host. Fix: call is_request_body_safe on the constructed data dict after optional_params are merged, before invoking ProxyBaseLLMRequestProcessing. ValueError from the banned-param check is surfaced as HTTP 400. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Update litellm/proxy/rag_endpoints/endpoints.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix: coerce nested config strings before banned-param check _NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently skipped litellm_embedding_config when delivered as a JSON string via multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.) nested inside the stringified value were invisible to is_request_body_safe. _NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: replace substring match with prefix match in is_llm_api_route mapped_pass_through_routes used `_llm_passthrough_route in route` (substring) so any admin-only path whose URL contained a provider name (openai, anthropic, azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the admin gate in non_proxy_admin_allowed_routes_check. Confirmed live: non-admin key could GET /credentials/by_name/openai (read masked provider API key) and DELETE /credentials/openai (delete credential). Fix: use exact match or startswith(prefix + "/") — the same pattern used everywhere else in RouteChecks — so only routes that actually start with a passthrough prefix are allowed through. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: stabilize PR #27878 test failures - key_management_endpoints: extend can_skip_admin_check to team keys so team members with /key/update permission can update non-budget fields. can_team_member_execute_key_management_endpoint already validates team membership + permission and raises if unauthorized; reaching the admin check on a team key means the caller was authorized. - test: set created_by on mock key in test_update_key_non_budget_fields_allowed_for_internal_user so caller_is_creator resolves correctly (MagicMock default ≠ user_id). - auth_utils.get_request_route: guard against non-dict request.scope (e.g. MagicMock in unit tests) to prevent a MagicMock leaking into UserAPIKeyAuth.request_route and failing Pydantic validation. - ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard in test-unit-proxy-db.yml to satisfy the shard-coverage check. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(lint): add explicit str() cast in get_request_route for MyPy scope.get() returns Any|None which MyPy cannot coerce to str implicitly. Wrap both scope.get() calls in str() to satisfy the type checker. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: guard bare-/ root_path strip + make total_spend migration idempotent auth_utils.get_request_route: when Starlette sets scope["app_root_path"] to "/" (e.g. behind some middleware), the old stripping logic would remove the leading slash from every path ("/team/new" → "team/new"), breaking route matching and causing auth to misclassify protected routes. Skip stripping when root_path is bare "/". migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration is safe to replay when a prior partial run already created the column. Without this guard, prisma migrate deploy fails on CI DBs that were partially migrated, causing all subsequent DB operations (including /team/new) to 500. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: require creator still owns key for personal-key bypass in /key/update caller_is_creator now requires both created_by == caller AND user_id == caller. Previously checking only created_by let a demoted admin who originally created a key for another user continue editing non-budget fields on it after reassignment, bypassing _check_key_admin_access. Adds regression test: creator whose key was reassigned is blocked (403). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix: extract auth checks to fix PLR0915 + broaden max_budget assertion internal_user_endpoints._update_single_user_helper exceeded 50 statements (PLR0915). Extract authorization checks into _check_user_update_authz helper to bring statement count under the limit. test_validate_max_budget: assert "negative" (substring of both the local "cannot be negative" and the CI "non-negative finite number" messages) so the test is stable regardless of which exact wording the function uses. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
||
|
Sameer Kankute
|
18f77ff7bc
|
feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough (#27834)
* feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough Adds an opt-in per-server flag that lets clients (e.g. VS Code) complete PKCE directly with an upstream OAuth2 MCP server, instead of LiteLLM double-gating with its own API-key/SSO check. Only honored when auth_type=oauth2 and the operator explicitly sets the flag; mixed-target or non-oauth2 requests fail closed. - Adds the field to Pydantic models, Prisma schema, and a migration - New MCPRequestHandler._target_servers_delegate_auth_to_upstream gate that runs only when no x-litellm-api-key is present, so authenticated users still get user_id resolution + stored-credential lookup - Anonymous callers now see delegate servers in get_allowed_mcp_servers (scoped to delegate servers only; the upstream still enforces auth) - mcp_management_endpoints: allow anonymous /authorize and /token for delegate servers so VS Code can complete PKCE without a LiteLLM session - UI toggle (shown only for oauth2) + payload/view wiring - Tests covering: oauth2 on/off, non-oauth2 with flag, mixed targets, no resolvable target, explicit key precedence, and 401 emission Co-authored-by: Cursor <cursoragent@cursor.com> * Enforce oauth2 for delegated MCP auth bypass Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): close secondary Authorization bypass for delegate servers The delegate-auth bypass gated only on the primary `x-litellm-api-key` header, so a LiteLLM key sent via `Authorization: Bearer sk-...` (the secondary header) was silently dropped — skipping spend tracking and rate limiting. Gate on the resolved litellm_api_key (which considers both headers) so the bypass fires only when neither is present. Also update the existing "Authorization header present" test to reflect that an upstream OAuth token now flows through the existing oauth2 fallback (LiteLLM auth attempt → fail → anonymous), not via the delegate branch. Co-authored-by: Cursor <cursoragent@cursor.com> * Avoid duplicate MCP OAuth credential lookup Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): block delegate bypass for M2M and internal-only servers Two security issues flagged in code review: 1. High – client_credentials (M2M) servers must not be delegatable: LiteLLM auto-fetches the upstream token using stored credentials, so allowing anonymous bypass would let any external caller invoke tools authenticated as LiteLLM's service account. Fix: check `server.has_client_credentials` in `_target_servers_delegate_auth_to_upstream`, the anonymous allow-list in `get_allowed_mcp_servers`, and `_mcp_oauth_user_api_key_auth`. 2. Medium – internal-only servers exposed to public internet: The anonymous delegate allow-list was not filtering by `available_on_public_internet`, so external callers with an upstream OAuth token could invoke tools on servers marked internal-only. Fix: add `available_on_public_internet` guard to the anonymous delegate server list in `get_allowed_mcp_servers`. Tests added for both cases. Co-authored-by: Cursor <cursoragent@cursor.com> * Require public MCP delegate auth servers Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): align delegate auth path parsing with downstream routing `_extract_target_server_names_from_path` used a naive segments-based split while `server.py::_get_mcp_servers_in_path` uses a regex that allows server names with one embedded slash and comma-separated lists. With the old parser, a request to `/mcp/<delegated>/<garbage>` was parsed as targeting `<delegated>` by the auth gate (bypassing LiteLLM auth) while the routing layer parsed it as `<delegated>/<garbage>` — when that name did not resolve, the request fell back to the anonymous allow-list, which can include `allow_all_keys` servers that normally require a LiteLLM key. Replace the parser with the same regex logic as `_get_mcp_servers_in_path` so auth gating sees the exact target name(s) downstream routing sees. Add regression tests covering parser parity and the specific extra-path-segment bypass attempt. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(mcp): close header/path TOCTOU in MCP delegate auth gate `_target_servers_delegate_auth_to_upstream` and `_target_servers_use_oauth2` trusted the `x-mcp-servers` header when present, but `server.py::extract_mcp_auth_context` overrides that header with the path-derived list for `/mcp/...` routes. An attacker could set `x-mcp-servers: <delegated>` while pointing the URL path at a non-delegate server, flipping the auth gate without changing the target downstream routing actually uses. Extract a shared `_resolve_target_server_names` helper that mirrors the downstream override (path-derived names for `/mcp/...` routes, header value otherwise). Add regression tests covering the TOCTOU attempt and the helper's path-vs-header precedence. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix delegated MCP OAuth test mock Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): drop unreachable /{server}/mcp branch in auth path parser `_extract_target_server_names_from_path` also matched the ``/{server_name}/mcp`` form, but the downstream parser ``_get_mcp_servers_in_path`` only handles ``/mcp/...`` — and ``dynamic_mcp_route`` in ``proxy_server`` rewrites ``/{name}/mcp`` to ``/mcp/{name}`` on the scope before the MCP handler runs. Parsing the un-rewritten form on the auth side was therefore unreachable in production, and contradicted the docstring's claim of mirroring the downstream parser — exactly the kind of mismatch that risks a future header/path TOCTOU if any new entry point skips the rewrite. Drop the branch; the canonical ``/mcp/...`` path matches both parsers. Update the regression test to assert the new behavior. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP path auth target resolution Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): require auth for refresh_token grants on delegate-auth servers `_mcp_oauth_user_api_key_auth` gates the unauthenticated PKCE flow for ``delegate_auth_to_upstream`` servers, but the bypass applied to BOTH ``/authorize`` and ``/token`` regardless of grant type. ``mcp_token`` accepts ``grant_type=refresh_token`` as well as ``authorization_code``, and ``exchange_token_with_server`` attaches the server's stored ``client_secret`` to whatever is forwarded upstream. An unauthenticated caller holding a refresh token issued to that OAuth client could mint fresh upstream access tokens through LiteLLM. Limit the anonymous bypass on ``/token`` to ``grant_type=authorization_code`` (the only grant PKCE actually protects via ``code_verifier``); fall through to normal LiteLLM auth for ``refresh_token`` and any other grant. ``/authorize`` continues to allow anonymous PKCE redirects. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(ui): clear delegate_auth_to_upstream when switching off oauth2 The ``delegate_auth_to_upstream`` form field is rendered inside an ``isOAuth2 && (...)`` conditional, so the Form.Item unmounts when the user changes ``auth_type`` away from ``oauth2``. The follow-up ``form.setFieldValue("delegate_auth_to_upstream", false)`` runs after the field has already deregistered, so ``onFinish`` receives ``undefined`` and the fallback ``?? mcpServer.delegate_auth_to_upstream`` preserved the old ``true``. The flag then persisted in the database for a non-oauth2 server and silently re-activated if ``auth_type`` was later switched back to ``oauth2``. In the edit payload, force the flag to ``false`` whenever ``auth_type !== oauth2``; only trust the form value (and the existing DB fallback) when the server is actually oauth2. Backend defense-in-depth already ignores the flag for non-oauth2 servers, but the DB state should stay clean too. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP delegate auth reset on edit Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude <claude@anthropic.com> |
||
|
user
|
83971a8712 | fix(proxy): normalize managed resource team owner field | ||
|
user
|
799d79160a
|
fix(proxy): match Prisma index names + extend listing to team for user-keyed callers
Two follow-ups to the managed-resource isolation fix: 1. Rename the new composite indexes to match Prisma's auto-generated naming convention (`<Table>_created_by_team_id_created_at_idx`). The previous `*_team_owner_created_at_idx` names left `prisma migrate diff` reporting an outstanding `RENAME INDEX`, failing `test_aaaasschema_migration_check`. 2. Make `build_owner_filter` return an OR clause when the caller has both a `user_id` and a `team_id`, so listings include team-shared resources the same way `can_access_resource` already permits reading them. Without this a user could fetch a team-shared resource by id but never see it in their list view. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
user
|
84fede37b4
|
fix(proxy): isolate managed resources for service-account API keys
Service-account API keys are issued without a `user_id`, and managed file/batch/vector-store ownership checks compared `resource.created_by == user_api_key_dict.user_id`. Because Python evaluates `None == None` as True, any service-account key passed ownership checks for any resource also created without a user id, and listing endpoints skipped the `created_by` filter entirely when the caller had no user id — returning every tenant's records. Replace the bare equality with an identity-aware helper: - Admins (PROXY_ADMIN, PROXY_ADMIN_VIEW_ONLY) keep their unscoped view. - Callers with a `user_id` are scoped to records they created. - Callers without a `user_id` but with a `team_id` are scoped to records created within their team via a new `created_by_team_id` column. - Callers with no admin role and no identifying ids are denied — the listing path returns an empty page without issuing a query. Schema migration adds `created_by_team_id` to LiteLLM_ManagedFileTable, LiteLLM_ManagedObjectTable, and LiteLLM_ManagedVectorStoreTable, plus indexes for the new filter. Writes in BaseManagedResource and the enterprise managed_files hook now stamp the column from `user_api_key_dict.team_id`. Reads in `can_user_access_unified_resource_id`, `can_user_call_unified_file_id`, `can_user_call_unified_object_id`, `list_user_resources`, `list_user_batches`, and `get_user_created_file_ids` all delegate to the new helper. Tests cover the helper in isolation, the base-class listing/access paths, and the enterprise file-access hook (including a regression test for the original `None == None` bypass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sameer Kankute
|
6588564a88
|
Merge pull request #26691 from BerriAI/litellm_team_search_credentials_metadata
feat(proxy): add team-level search provider credentials |
||
|
ishaan-berri
|
4a7af1ff68
|
feat(proxy): durable agent workflow run tracking via /v1/workflows/runs (#26793)
* feat(schema): add workflow run tracking tables (LiteLLM_WorkflowRun, LiteLLM_WorkflowEvent, LiteLLM_WorkflowMessage) * feat(proxy): add /v1/workflows/runs endpoints for durable agent workflow tracking * feat(proxy): register workflow management router in proxy_server * docs(workflows): add README for workflow run tracking API * test(workflows): add unit tests for /v1/workflows/runs endpoints * fix(workflows): atomic event+status update via tx(), run_id 404 guard, sequence retry on collision * test(workflows): add tx mock, 404 on unknown run_id, retry-on-collision tests * fix(workflows): constrain status to Literal enum, rename total→count in list responses * add tenant isolation and bounded limits to workflow endpoints * add created_by column and index to LiteLLM_WorkflowRun * add ownership and bounded-limit tests for workflow endpoints * Fix workflow run ownership for null owners * guard prisma import in workflow_management_endpoints * sync schema.prisma copies with workflow run models * black: format workflow_management_endpoints.py --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |