The Together catalog exposes only context_length, so the sync was recording
every chat model's context window as its output ceiling. New entries now carry
max_input_tokens and the legacy max_tokens from the catalog and get an output
ceiling only from a reviewed capability rule. GLM-5.2 and GLM-5.3-Flash rules
carry the documented 128K ceiling, and the 26 other inflated together_ai chat
entries drop max_output_tokens in both registry copies.
A deployment carrying reasoning_effort in its litellm_params on the
/v1/messages passthrough mapped the effort to a legacy thinking block
whose budget_tokens was forwarded as is, so any request whose max_tokens
sat at or below that budget was rejected upstream with a 400. The mapped
budget now runs through the same cap the adaptive-to-legacy branch and
the chat path already use: it is clamped to max_tokens - 1, and dropped
with a warning when even the minimum budget cannot fit.
The cap helper becomes public since three call sites outside
AnthropicConfig use it.
A pool whose recorded spend has reached max_budget has nothing left to give, so
the next request is refused rather than admitted. This departs from the tag
check it otherwise mirrors and matches where keys and organizations already
draw the line.
A non-positive budget now means no budget here too, so the read-time check and
the reservation path agree on what counts as unbudgeted.
The four model access group callback tests now share one helper, so nine
patches of proxy_server internals become three, and both mock-echo assertions
go with them. The delete_access_group tests share a context manager for the
same reason.
test_group_exactly_at_its_max_budget_passes gained the assertion it was
missing: it now proves the group reached the spend comparison, which a group
skipped for a missing budget row would not. The route-allowed patch beside it
was dead, so it is gone.
What is left is suppressed with the collaborator each one cannot inject.
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
The database writer already intersects the auth-matched groups with the ones
the served deployment declares, but the live spend counters got the unnarrowed
set. A caller granted two pools that both cover a model group debited both
counters while only one row moved, so the in-memory ceiling could block a pool
its persisted spend never touched.
Narrow once at the callback so both consumers read the same set.
The cache TTL for a group's budget row was a new
DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL env var defaulting to 600 seconds, which
nobody asked for and which the docs gate rightly rejected as undocumented.
Every other management object cached in auth_checks, tags included, already
reads get_management_object_ttl, so it honors general_settings
user_api_key_cache_ttl and falls back to the shared default. Group budgets now
do the same, which drops a constant, drops an env var, and makes the row expire
on the same operator knob as keys and teams.
Also formats ModelAccessGroupBudgetRepository, and teaches the FakeBatch double
in the unit of work tests about the new table. That double is read while the
cascade unit of work is constructed rather than inside the block, so three tests
that never mention access groups were failing at the async with. The new test
alongside it walks the dataclass fields, so the next dependent added to the
cascade is covered without anyone remembering to update a list.
Budget enforcement trusts a current LiteLLM_BudgetWindowSpend row without
reconciling it against LiteLLM_SpendLogs, so an increment dropped after a
failed commit let the entity spend past its window limit after the next
counter reseed. Failed increments now go back on the in-memory queue, or
back to the Redis buffer, and retry on the next scheduler tick like every
other spend category.
Send a non-Bearer Authorization header verbatim via auth_type authorization
instead of wrapping it as a bearer credential, and always drop the
Authorization header from static_headers so the plaintext copy cannot shadow
the encrypted credential at request time. Register each imported server with
the in-memory manager before the best-effort reload, matching the manual add
path. Let get_all_mcp_servers propagate read failures instead of returning [],
which silently disabled the import dedupe and allowed duplicate imports.
* feat(proxy): CyberArk Conjur secret manager configuration via Admin UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): mock networking base-url helpers in AdminPanel test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): restore deployment CyberArk env config on delete and roll back on persist failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reinit env-configured hashicorp vault manager after cyberark persist rollback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The keys, credentials, models, model groups, and chat clients still sent
requests with no timeout, so a proxy that accepts the connection and
never answers pinned the caller forever. They now default to the same
30 seconds as their teams and users siblings, with chat on the OpenAI
SDK's 600 second default, and Client wires its timeout through to all of
them. S113 cannot see Session methods, so each client gets a
hanging-server regression test instead.
Reprice ten more retired xAI slugs (grok-3 and grok-3-mini families,
grok-4-1-fast) to the grok-4.3 rates they now bill at, with family-correct
deprecation dates. Restore cache_read_input_token_cost on the Bedrock Grok 4.6
entries so implicit cache hits bill at the cache-read rate while explicit
cachePoint stays unsupported. Drop the unsourced 1080p video rate and the
gemini/ live native-audio entry the Gemini API 404s on. Add Groq qwen3.8-27b
tool-use flags per Groq docs. Extend the xai and gemini tests to lock all of
this in
OpenAI's function-calling validator rejects tool parameters carrying
oneOf/anyOf/allOf/enum/const/not at the top level, while the ChatGPT
backend Codex talks to natively accepts them, so an MCP tool declaring a
top-level union 400s through the proxy. Merge the branches into the
object schema for OpenAI itself only, walking the namespace-nested tools
current Codex builds send, on both /v1/responses and /v1/responses/compact
Two problems, both caught in review.
The new table only landed in the root schema.prisma. Client generation reads
litellm/proxy/schema.prisma and packaging reads the copy under
litellm-proxy-extras, so the generated client had no
litellm_modelaccessgroupbudgettable and every budget read and write against
it would have failed at runtime. The root is the source of truth; both
copies are now byte-identical to it.
Nothing incremented spend:model_access_group:{group} after a call. Only the
reservation path ever wrote it, so with disable_budget_reservation the
read-time check was reading a counter nobody maintained and falling back to
the row's spend, which is cached for up to DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL.
A caller could run well past the pool inside that window, which is precisely
the case the read-time check exists to cover.
increment_spend_counters now takes the matched groups and charges them
through _init_and_increment_unreserved_spend_counter, so a group already
covered by a reservation is skipped rather than counted twice. The cost
callback sources the names with get_request_model_access_groups, the same
reader the spend writer uses.
Regenerates the lazy OpenAPI snapshot and dashboard schema.d.ts from the merged tree.
Passes list/dict copies into NewMCPServerRequest from the connector conversion and switches the
server-managed NewMCPServerRequest fields to keyword defaults so the basedpyright budget gate stays flat.
A model access group could gate which models a caller reaches but never how
much that group of callers could spend in total. Capping a shared pool meant
setting a per-entity budget on every key by hand, which caps each key
separately and still leaves no way to read what the group cost.
Spend is attributed to a group only when the group's name appears on an
allowlist the caller was granted (key, team, team-member scope, project or
org) and that group serves the requested model. Asking for a model that
merely belongs to a group attributes nothing, because nothing about the
caller named the group. Levels are unioned rather than ranked, so a team
granted "*" whose member is scoped to one group still counts as gated by
that group.
Enforcement runs on both paths tags already use: a reservation counter on
the pre-call path and a read-time max_budget check inside the existing
concurrent budget gather, so the ceiling still holds under
disable_budget_reservation.
Adds LiteLLM_ModelAccessGroupBudgetTable, which is the only place a group is
ever a row: the groups themselves stay free-text strings in
model_info.access_groups, so a row exists only once someone gives that group
a budget. GET, PUT and DELETE /access_group/{name}/budget manage it, and
/access_group/{name}/info now carries the spend and budget alongside the
models.
