The first pass accepted any empty completion whose finish_reason was "length",
which also swallowed a fallback that produced nothing at all. Require the
response to have billed completion tokens as well, so empty content is
accepted only when the budget was demonstrably spent on non-visible reasoning.
Asserts on completion_tokens rather than reasoning_tokens because the latter
is provider-optional; with empty content and a refusal of null, consumed
completion tokens are reasoning by elimination, since visible text would be
content. Both counts are reported in the failure message.
Folds the three body accessors onto one _parsed helper instead of re-parsing
per call, and adds completion_tokens_of / reasoning_tokens_of alongside.
Multi-window budgets (budget_limits on keys/teams) currently keep window
spend only in cache. Every cold or expired counter recomputes the window
by aggregating LiteLLM_SpendLogs, which has no usable index for that
query and saturates the DB on large tables (#35766).
This adds a LiteLLM_BudgetWindowSpend table holding one row per
configured window, keyed (entity_type, entity_id, window_duration),
with window_start identifying the period the spend belongs to.
Follow-up PRs maintain these rows from the spend update writer and move
window budget enforcement reads onto them.
The LLM classifier's cost was recorded on the routing decision but never
reached any savings surface: per-request autorouter_savings stayed gross
and the session rollup recorded only the served request's spend, so
/auto_router/benchmarks overstated savings and understated routed spend.
Net the classifier cost into the savings figure at its one computation
owner and fold it into the rollup turn's spend, keeping
baseline_spend = spend + saved_spend. The response header's numeric
guard now shares the same reader.
Fixes#38816
max_tokens=64 caps reasoning plus visible output on gpt-5.5, so the fallback
target can legitimately return finish_reason="length" with empty content.
litellm-e2e build 90 hit exactly that: the response cost of $0.002005 backs
out to 64 completion tokens at gpt-5.5's $3e-05/token, i.e. the whole budget
spent reasoning about "say hi" with none left to answer. The fallback itself
worked -- 200, served by gpt-5.5-2026-04-23, x-litellm-attempted-fallbacks
present -- so the only thing that failed was an assertion about OpenAI's token
budgeting rather than about routing.
Raise the reliability helper's budget to 512 and accept empty content only
when finish_reason is "length". Empty content under any other finish_reason
still fails, so the tests keep catching a fallback that returns nothing for a
reason we do control.
The relaxed assertion lives in the helper both reliability fallback tests
share, so test_timeout_routes_to_fallback is covered too; it has the same
shape and had not tripped yet.
* fix(proxy): run SMTP send_email off the event loop with a connection timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): format utils.py and update _create_smtp_connection tests for timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep malformed SMTP_TIMEOUT inside the email error boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: exclude misaligned circleci coverage flag from merged codecov report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci for codecov and benchmarks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: disable carryforward for the circleci codecov flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: exclude carried-forward coverage from the codecov patch status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: stop carrying forward the dead circleci codecov flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Multi-window budgets (budget_limits on keys/teams) currently keep window
spend only in cache. Every cold or expired counter recomputes the window
by aggregating LiteLLM_SpendLogs, which has no usable index for that
query and saturates the DB on large tables (#35766).
This adds a LiteLLM_BudgetWindowSpend table holding one row per
configured window, keyed (entity_type, entity_id, window_duration),
with window_start identifying the period the spend belongs to.
Follow-up PRs maintain these rows from the spend update writer and move
window budget enforcement reads onto them.
The paginated search select diffs the input against the selected option's
label to recover what the user just typed, so a query started from a picked
value searches for the new text rather than the label plus the new text.
That diff can only express an insertion: it walks a common prefix and a
common suffix and returns what sits between them. A deletion leaves nothing
between them, so every deletion-only edit returned the empty string.
Backspacing once in a field showing a selected label therefore threw the
edit away. The empty result was stored as the query, the controlled input
re-rendered blank, and the server was asked for the unfiltered page instead
of the text the user left in the box.
An edit that yields no insertion but did change the value is a deletion, and
there the remaining text is the query the user means. Insertions and
whole-selection replacements are untouched.
Four cases in tests/e2e/quota_management/budgets, driving real OpenAI calls
through a group whose shared pool is drained to exhaustion: the spender key
stays blocked, a key that spent nothing of its own is blocked by the same
pool, a sibling group with no budget keeps serving, and the budget read
reports the spend drawn against the group.
Adds set/get/delete access group budget to BudgetClient and the four
matching rows to the coverage registry.
The Together catalog exposes only context_length, so the sync was recording
every chat model's context window as its output ceiling. New entries now carry
max_input_tokens and the legacy max_tokens from the catalog and get an output
ceiling only from a reviewed capability rule. GLM-5.2 and GLM-5.3-Flash rules
carry the documented 128K ceiling, and the 26 other inflated together_ai chat
entries drop max_output_tokens in both registry copies.
A deployment carrying reasoning_effort in its litellm_params on the
/v1/messages passthrough mapped the effort to a legacy thinking block
whose budget_tokens was forwarded as is, so any request whose max_tokens
sat at or below that budget was rejected upstream with a 400. The mapped
budget now runs through the same cap the adaptive-to-legacy branch and
the chat path already use: it is clamped to max_tokens - 1, and dropped
with a warning when even the minimum budget cannot fit.
The cap helper becomes public since three call sites outside
AnthropicConfig use it.
A pool whose recorded spend has reached max_budget has nothing left to give, so
the next request is refused rather than admitted. This departs from the tag
check it otherwise mirrors and matches where keys and organizations already
draw the line.
A non-positive budget now means no budget here too, so the read-time check and
the reservation path agree on what counts as unbudgeted.
The four model access group callback tests now share one helper, so nine
patches of proxy_server internals become three, and both mock-echo assertions
go with them. The delete_access_group tests share a context manager for the
same reason.
test_group_exactly_at_its_max_budget_passes gained the assertion it was
missing: it now proves the group reached the spend comparison, which a group
skipped for a missing budget row would not. The route-allowed patch beside it
was dead, so it is gone.
What is left is suppressed with the collaborator each one cannot inject.
Backspacing the last digit of Context Window Size instantly refilled the default (3), since onChange mapped empty input to null and the handler coalesced null back to the default. The same defect affected Timeout (ms) and Context Character Budget. Add per-field raw draft state so an empty or partial value stays visible while focused, commit only finite values (rounded, clamped to each field's minimum), and clear the draft on blur so an abandoned edit falls back to the committed value. 0 stays a valid committed value for both context controls. Add stable ids and label associations; update tests to query by label
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
The database writer already intersects the auth-matched groups with the ones
the served deployment declares, but the live spend counters got the unnarrowed
set. A caller granted two pools that both cover a model group debited both
counters while only one row moved, so the in-memory ceiling could block a pool
its persisted spend never touched.
Narrow once at the callback so both consumers read the same set.
The helper already logs through spend_log_error, so re-raising only for
_batch_database_updates to catch and log again was double reporting. Isolate
inside the helper instead and drop the caller's wrapper.
The cache TTL for a group's budget row was a new
DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL env var defaulting to 600 seconds, which
nobody asked for and which the docs gate rightly rejected as undocumented.
Every other management object cached in auth_checks, tags included, already
reads get_management_object_ttl, so it honors general_settings
user_api_key_cache_ttl and falls back to the shared default. Group budgets now
do the same, which drops a constant, drops an env var, and makes the row expire
on the same operator knob as keys and teams.
Also formats ModelAccessGroupBudgetRepository, and teaches the FakeBatch double
in the unit of work tests about the new table. That double is read while the
cascade unit of work is constructed rather than inside the block, so three tests
that never mention access groups were failing at the async with. The new test
alongside it walks the dataclass fields, so the next dependent added to the
cascade is covered without anyone remembering to update a list.
Budget enforcement trusts a current LiteLLM_BudgetWindowSpend row without
reconciling it against LiteLLM_SpendLogs, so an increment dropped after a
failed commit let the entity spend past its window limit after the next
counter reseed. Failed increments now go back on the in-memory queue, or
back to the Redis buffer, and retry on the next scheduler tick like every
other spend category.
Send a non-Bearer Authorization header verbatim via auth_type authorization
instead of wrapping it as a bearer credential, and always drop the
Authorization header from static_headers so the plaintext copy cannot shadow
the encrypted credential at request time. Register each imported server with
the in-memory manager before the best-effort reload, matching the manual add
path. Let get_all_mcp_servers propagate read failures instead of returning [],
which silently disabled the import dedupe and allowed duplicate imports.
* feat(proxy): CyberArk Conjur secret manager configuration via Admin UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): mock networking base-url helpers in AdminPanel test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): restore deployment CyberArk env config on delete and roll back on persist failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reinit env-configured hashicorp vault manager after cyberark persist rollback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
