Commit graph

45730 commits

Author SHA1 Message Date
Yuneng Jiang
02046fb7b7
fix(ui): keep a deleted-from search query instead of blanking the box
The paginated search select diffs the input against the selected option's
label to recover what the user just typed, so a query started from a picked
value searches for the new text rather than the label plus the new text.
That diff can only express an insertion: it walks a common prefix and a
common suffix and returns what sits between them. A deletion leaves nothing
between them, so every deletion-only edit returned the empty string.

Backspacing once in a field showing a selected label therefore threw the
edit away. The empty result was stored as the query, the controlled input
re-rendered blank, and the server was asked for the unfiltered page instead
of the text the user left in the box.

An edit that yields no insertion but did change the value is a deletion, and
there the remaining text is the query the user means. Insertions and
whole-selection replacements are untouched.
2026-08-29 15:52:48 -07:00
ryan-crabbe-berri
e263c09e4f test(e2e): cover model access group budgets against a live proxy
Four cases in tests/e2e/quota_management/budgets, driving real OpenAI calls
through a group whose shared pool is drained to exhaustion: the spender key
stays blocked, a key that spent nothing of its own is blocked by the same
pool, a sibling group with no budget keeps serving, and the budget read
reports the spend drawn against the group.

Adds set/get/delete access group budget to BudgetClient and the four
matching rows to the coverage registry.
2026-08-29 15:46:49 -07:00
mateo-berri
af179be681 fix(together_ai): stop writing context_length as max_output_tokens in the serverless sync
The Together catalog exposes only context_length, so the sync was recording
every chat model's context window as its output ceiling. New entries now carry
max_input_tokens and the legacy max_tokens from the catalog and get an output
ceiling only from a reviewed capability rule. GLM-5.2 and GLM-5.3-Flash rules
carry the documented 128K ceiling, and the 26 other inflated together_ai chat
entries drop max_output_tokens in both registry copies.
2026-08-29 15:26:44 -07:00
mateo-berri
71a951691a fix(anthropic): cap reasoning_effort thinking budget below max_tokens on /v1/messages
A deployment carrying reasoning_effort in its litellm_params on the
/v1/messages passthrough mapped the effort to a legacy thinking block
whose budget_tokens was forwarded as is, so any request whose max_tokens
sat at or below that budget was rejected upstream with a 400. The mapped
budget now runs through the same cap the adaptive-to-legacy branch and
the chat path already use: it is clamped to max_tokens - 1, and dropped
with a warning when even the minimum budget cannot fit.

The cap helper becomes public since three call sites outside
AnthropicConfig use it.
2026-08-29 15:24:05 -07:00
ryan-crabbe-berri
40a7fe9221 fix(budgets): make the model access group ceiling exclusive
A pool whose recorded spend has reached max_budget has nothing left to give, so
the next request is refused rather than admitted. This departs from the tag
check it otherwise mirrors and matches where keys and organizations already
draw the line.

A non-positive budget now means no budget here too, so the read-time check and
the reservation path agree on what counts as unbudgeted.
2026-08-29 15:20:10 -07:00
tin-berri
36ea28b092
fix(anthropic): emit signature-only thinking blocks on the /v1/messages bridge (#38809) 2026-08-29 15:04:22 -07:00
ryan-crabbe-berri
d7c0bc1e6d test(budgets): clear the test-quality violations this branch added
The four model access group callback tests now share one helper, so nine
patches of proxy_server internals become three, and both mock-echo assertions
go with them. The delete_access_group tests share a context manager for the
same reason.

test_group_exactly_at_its_max_budget_passes gained the assertion it was
missing: it now proves the group reached the spend comparison, which a group
skipped for a missing budget row would not. The route-allowed patch beside it
was dead, so it is gone.

What is left is suppressed with the collaborator each one cannot inject.
2026-08-29 14:44:00 -07:00
tin-berri
5c034fda74
fix(ui): allow in-place editing of classifier numeric inputs (#38803)
Backspacing the last digit of Context Window Size instantly refilled the default (3), since onChange mapped empty input to null and the handler coalesced null back to the default. The same defect affected Timeout (ms) and Context Character Budget. Add per-field raw draft state so an empty or partial value stays visible while focused, commit only finite values (rounded, clamped to each field's minimum), and clear the draft on blur so an abandoned edit falls back to the committed value. 0 stays a valid committed value for both context controls. Add stable ids and label associations; update tests to query by label
2026-08-29 14:43:20 -07:00
Mateo Wang
20cfccaf5f
Merge pull request #37208 from BerriAI/litellm_managed_batches_observability
fix(batches): aggregate reasoning tokens and per-line pass/fail counts
2026-08-29 14:43:03 -07:00
mateo-berri
cfb7a26327 fix(registry): add gemma 4 capability flags verified against the gemini api 2026-08-29 14:38:21 -07:00
mateo-berri
9448293903 fix(openai): flatten tool schema unions only for models whose validator rejects them
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
2026-08-29 14:38:08 -07:00
Mateo Wang
c62c2afa09
Merge pull request #38234 from BerriAI/litellm_request_timeouts
fix(proxy): give every `requests` call a timeout so a silent server cannot hang the caller
2026-08-29 14:30:46 -07:00
mateo-berri
ed5761daef fix(ci): keep earlier objections when the duplicate notice is re-posted
The detector fires on issue edits and posts a fresh notice each time, so the sweep now counts replies from the first notice on and a thumbs down on any notice
2026-08-29 17:30:39 -04:00
mateo-berri
539bc8ef92 fix(ci): close only identical-title duplicates, dry-run the sweep, reopen on reply
The merged detector's 0.6 flag threshold had become the close bar, and 6 of the 7 real flagged pairs at 85% or more were not duplicates. The sweep now closes only when an older open issue has the identical normalized title, measures the grace period from the latest bot notice, and leaves the issue open when anyone replies or gives the notice a thumbs down.

A reporter cannot reopen an issue the bot closed, so a reporter comment after the automatic close reopens it, drops the duplicate label, and asks for a human look. Manual dispatch defaults to a dry run and takes a grace_period_days input, the runner supplies the repository, the dead python closer is gone, and the decision core has bun tests on a PR-triggered job.
2026-08-29 17:30:01 -04:00
mubashir1osmani
f118511f55 fix(ci): honour a duplicate-notice thumbs down from anyone
The notice tells every reader that a thumbs down keeps the issue open, but the
sweep only counted the reaction when it came from the issue author. A maintainer
or another affected user could follow the instruction exactly and still watch the
issue close, which made the notice a promise the sweep did not keep.

Any thumbs down now spares the issue. That buys back nothing an abuser did not
already have: a plain comment stops the clock for anyone, so restricting the
reaction only ever penalised people who did what they were told.

Drops the issue and reaction author fields, since nothing reads them now.
2026-08-29 17:29:56 -04:00
mubashir1osmani
f4542d9605 fix(ci): pin the Bun runtime instead of tracking latest
The setup step ran `bun-version: latest`, carried over from the upstream layout,
and the step after it holds an issues: write token. A compromised Bun release
would have executed privileged in that job and could rewrite or close issues.

Pinned to 1.4.0, the release the passing runs already resolved to. setup-bun
takes no checksum input, so pinning the action by sha and the runtime by exact
version is as far as this can be hardened without hand-rolling the download.
2026-08-29 17:29:56 -04:00
mubashir1osmani
3ea11b64e6 refactor(ci): run the duplicate sweep as a Bun TypeScript script
Moves the sweep out of inline workflow JavaScript and into scripts/, following
the layout anthropics/claude-code uses for the same job: a checked-out repo, a
sha-pinned setup-bun step, and `bun run scripts/auto-close-duplicates.ts`.

The script mirrors that repo's file shape, keeping the same request helper,
interfaces, per-issue debug logging, and top-level catch, so the two read the
same way side by side.

Two things stay deliberately different. Candidates come from the notice marker's
digits-only field rather than a regex over the comment prose, because titles are
attacker-controlled and are interpolated into that same comment. The label is
also added on its own endpoint instead of alongside the state change, since
sending labels with a PATCH replaces every label already on the issue.
2026-08-29 17:29:56 -04:00
mubashir1osmani
af340c0240 fix(ci): read duplicate candidates from the marker, not the notice prose
The notice interpolates each candidate's title, and the sweep scanned the whole
comment for issue references and took the lowest. Titles are attacker-controlled,
so filing a candidate titled "... see #1" redirected the closure: any later report
matching that candidate would be closed as a duplicate of #1 instead.

The detector now emits the candidate numbers as a digits-only field inside the
marker, built from the API's number field, and the sweep reads only that. Prose is
never parsed, so nothing a reporter can type reaches the target selection.
2026-08-29 17:29:55 -04:00
mubashir1osmani
e2ffb6b01c feat(ci): close duplicate issues after a 3-day grace period
Duplicate detection already labelled and commented on new issues, and then
closed them outright at 0.85 title similarity. That gave the reporter no chance
to push back, and a title-similarity match is not strong enough evidence to
close on its own.

Detection now only flags. A new daily sweep closes a flagged issue three days
later, and only if nobody engaged with the flag. Replying to it, thumbs-downing
it, or applying an opt-out label all keep the issue open. The notice says all of
that up front, so the reporter knows what happens and how to stop it.

The two workflows hand off through an HTML marker in the comment body rather
than its prose, so rewording the notice cannot silently break the sweep. The
sweep lists by label instead of walking the whole backlog: 1663 open issues
against 23 carrying the label meant a comments request each, which would burn
the Actions token's hourly budget for a handful of matches.

Candidates are taken as the lowest issue number, not the first one listed. The
detector orders by score rather than age, so the first candidate can be newer
than the issue being closed, and folding an original report into a later one is
backwards. An issue whose only candidates are newer is skipped.

Closures use state_reason=duplicate rather than not_planned, which reads as
"see the other issue" instead of "we are not doing this".

Also drops {{html_url}} from the notice. The detection action only exposes
number, title and accuracy, so that placeholder had been rendering empty and
every "similar issue" link in the comment pointed nowhere.
2026-08-29 17:29:55 -04:00
ryan-crabbe-berri
acf3ed7d9b fix(budgets): narrow model access group spend counters to the served deployment
The database writer already intersects the auth-matched groups with the ones
the served deployment declares, but the live spend counters got the unnarrowed
set. A caller granted two pools that both cover a model group debited both
counters while only one row moved, so the in-memory ceiling could block a pool
its persisted spend never touched.

Narrow once at the callback so both consumers read the same set.
2026-08-29 14:25:56 -07:00
mateo-berri
68404d8ff4 fix(registry): drop xai video entries that have no video adapter 2026-08-29 14:20:48 -07:00
ryan-crabbe-berri
13bed620e9 Merge commit '085d5f5d4e' into litellm_window_spend_reader 2026-08-29 14:18:51 -07:00
ryan-crabbe-berri
085d5f5d4e docs(proxy): state that the cross-pod window seed gap over-counts, never under-counts 2026-08-29 14:18:41 -07:00
ryan-crabbe-berri
7a2f6c4cc2 fix(budgets): let the model access group spend helper swallow its own failure
The helper already logs through spend_log_error, so re-raising only for
_batch_database_updates to catch and log again was double reporting. Isolate
inside the helper instead and drop the caller's wrapper.
2026-08-29 14:17:45 -07:00
ryan-crabbe-berri
6b2e7f8a1f refactor(budgets): declare route dependencies with Annotated instead of argument defaults 2026-08-29 14:10:58 -07:00
Mateo Wang
2963b47cda test: patch the Logging handler instead of the class in the poller error-file test 2026-08-29 14:09:33 -07:00
Mateo Wang
128d7e5278 refactor(batches): make count_error_file_failed_requests public for the poller import 2026-08-29 14:09:18 -07:00
mateo-berri
b777947364 fix(openai): bound $ref expansion and keep root plus branch required when flattening tool schemas 2026-08-29 14:09:08 -07:00
ryan-crabbe-berri
06d0665f50 fix(budgets): drop the bespoke access group cache TTL and format the repository
The cache TTL for a group's budget row was a new
DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL env var defaulting to 600 seconds, which
nobody asked for and which the docs gate rightly rejected as undocumented.
Every other management object cached in auth_checks, tags included, already
reads get_management_object_ttl, so it honors general_settings
user_api_key_cache_ttl and falls back to the shared default. Group budgets now
do the same, which drops a constant, drops an env var, and makes the row expire
on the same operator knob as keys and teams.

Also formats ModelAccessGroupBudgetRepository, and teaches the FakeBatch double
in the unit of work tests about the new table. That double is read while the
cascade unit of work is constructed rather than inside the block, so three tests
that never mention access groups were failing at the async with. The new test
alongside it walks the dataclass fields, so the next dependent added to the
cascade is covered without anyone remembering to update a list.
2026-08-29 14:00:56 -07:00
ryan-crabbe-berri
fb9e76fab3 chore(budgets): refresh the lazy OpenAPI snapshot and allowlist the model access group budget routes 2026-08-29 14:00:12 -07:00
devin-ai-integration[bot]
194a3cc202
ci: build the benchmark environment outside the CodSpeed runner (#38426)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 13:58:04 -07:00
Devin AI
5024c4d520 fix: update stale source URLs in model cost map
119 entries pointed at 404ing or permanently-moved pages (Pylon #7777).
Replaced with verified working equivalents (200-checked or permanent
redirect targets).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 20:56:21 +00:00
ryan-crabbe-berri
b5ec80d903 Merge commit 'a5f47a271a' into litellm_window_spend_reader 2026-08-29 13:55:53 -07:00
ryan-crabbe-berri
a5f47a271a fix(proxy): re-queue budget window spend increments when the commit fails
Budget enforcement trusts a current LiteLLM_BudgetWindowSpend row without
reconciling it against LiteLLM_SpendLogs, so an increment dropped after a
failed commit let the entity spend past its window limit after the next
counter reseed. Failed increments now go back on the in-memory queue, or
back to the Redis buffer, and retry on the next scheduler tick like every
other spend category.
2026-08-29 13:55:36 -07:00
mateo-berri
40f5d53c04 fix(mcp): harden connector import auth handling and registration
Send a non-Bearer Authorization header verbatim via auth_type authorization
instead of wrapping it as a bearer credential, and always drop the
Authorization header from static_headers so the plaintext copy cannot shadow
the encrypted credential at request time. Register each imported server with
the in-memory manager before the best-effort reload, matching the manual add
path. Let get_all_mcp_servers propagate read failures instead of returning [],
which silently disabled the import dedupe and allowed duplicate imports.
2026-08-29 13:41:22 -07:00
devin-ai-integration[bot]
645792955d
feat(proxy): cyberark conjur secret manager configuration via Admin UI (#38445)
* feat(proxy): CyberArk Conjur secret manager configuration via Admin UI

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): mock networking base-url helpers in AdminPanel test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): restore deployment CyberArk env config on delete and roll back on persist failure

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): reinit env-configured hashicorp vault manager after cyberark persist rollback

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 13:36:08 -07:00
Mateo Wang
8dd9c4acb1
Merge pull request #30782 from emerzon/litellm_veo_31_lite
feat(vertex-ai): add veo 3.1 lite model metadata
2026-08-29 13:36:02 -07:00
Mateo Wang
306daf13b5
Merge pull request #38752 from BerriAI/litellm_deflake_20260829
fix: bound Hugging Face config fetch and keep embedding tests off the network
2026-08-29 13:33:06 -07:00
mateo-berri
4d5205c355 fix(proxy): give the remaining CLI clients a request timeout
The keys, credentials, models, model groups, and chat clients still sent
requests with no timeout, so a proxy that accepts the connection and
never answers pinned the caller forever. They now default to the same
30 seconds as their teams and users siblings, with chat on the OpenAI
SDK's 600 second default, and Client wires its timeout through to all of
them. S113 cannot see Session methods, so each client gets a
hanging-server regression test instead.
2026-08-29 13:32:39 -07:00
mateo-berri
8aba6e9203 Merge branch 'litellm_internal_staging' into litellm_request_timeouts 2026-08-29 13:32:33 -07:00
ryan-crabbe-berri
2c8efca0d3 Merge commit '56dd4e06ac' into litellm_window_spend_reader 2026-08-29 13:30:10 -07:00
ryan-crabbe-berri
56dd4e06ac test(proxy): satisfy the test-quality gate for the window spend writer tests 2026-08-29 13:29:59 -07:00
mateo-berri
08574d8e07 fix(openai): resolve local $refs and nested combinators when flattening tool schemas 2026-08-29 13:24:19 -07:00
mateo-berri
2bd7b58640 fix(registry): correct xai retired slug pricing, bedrock grok caching, and unsourced entries
Reprice ten more retired xAI slugs (grok-3 and grok-3-mini families,
grok-4-1-fast) to the grok-4.3 rates they now bill at, with family-correct
deprecation dates. Restore cache_read_input_token_cost on the Bedrock Grok 4.6
entries so implicit cache hits bill at the cache-read rate while explicit
cachePoint stays unsupported. Drop the unsourced 1080p video rate and the
gemini/ live native-audio entry the Gemini API 404s on. Add Groq qwen3.8-27b
tool-use flags per Groq docs. Extend the xai and gemini tests to lock all of
this in
2026-08-29 13:24:09 -07:00
Mateo Wang
38145c2082 test: undo the drive-by reformat below the poller error-file regression test 2026-08-29 13:17:43 -07:00
yassin
0557a95259 fix(mcp): lift Authorization headers into bearer credentials on import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 20:11:45 +00:00
mateo-berri
9b8ad46f37 fix(openai): flatten top-level anyOf/oneOf/allOf in Responses API tool schemas
OpenAI's function-calling validator rejects tool parameters carrying
oneOf/anyOf/allOf/enum/const/not at the top level, while the ChatGPT
backend Codex talks to natively accepts them, so an MCP tool declaring a
top-level union 400s through the proxy. Merge the branches into the
object schema for OpenAI itself only, walking the namespace-nested tools
current Codex builds send, on both /v1/responses and /v1/responses/compact
2026-08-29 13:11:04 -07:00
Mateo Wang
e0ed0a4c7a
Merge pull request #35017 from BerriAI/litellm_lit_4913_headroom_streaming_ccr
fix(headroom): resolve CCR retrieval on streaming /chat/completions
2026-08-29 13:01:35 -07:00
devin-ai-integration[bot]
f0340fef16
feat(mcp_gateway): add RFC 7662 introspection for gateway session tokens (#38726)
* feat(mcp_gateway): add RFC 7662 introspection for gateway session tokens

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp_gateway): omit Bearer token_type for refresh introspection and allow mcp-scoped keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp_gateway): cover introspection of RS256-signed session tokens

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp_gateway): load the discoverable router on a cold /introspect request

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(openapi): regenerate lazy snapshot and schema.d.ts for /introspect

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 12:56:04 -07:00
Mateo Wang
9ed7de6c02
Merge pull request #38670 from BerriAI/devin_ai_38659_cohere_embed_dispatch
fix(bedrock): route all cohere.embed models to the cohere embedding config
2026-08-29 12:55:32 -07:00