The database writer already intersects the auth-matched groups with the ones
the served deployment declares, but the live spend counters got the unnarrowed
set. A caller granted two pools that both cover a model group debited both
counters while only one row moved, so the in-memory ceiling could block a pool
its persisted spend never touched.
Narrow once at the callback so both consumers read the same set.
The helper already logs through spend_log_error, so re-raising only for
_batch_database_updates to catch and log again was double reporting. Isolate
inside the helper instead and drop the caller's wrapper.
The cache TTL for a group's budget row was a new
DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL env var defaulting to 600 seconds, which
nobody asked for and which the docs gate rightly rejected as undocumented.
Every other management object cached in auth_checks, tags included, already
reads get_management_object_ttl, so it honors general_settings
user_api_key_cache_ttl and falls back to the shared default. Group budgets now
do the same, which drops a constant, drops an env var, and makes the row expire
on the same operator knob as keys and teams.
Also formats ModelAccessGroupBudgetRepository, and teaches the FakeBatch double
in the unit of work tests about the new table. That double is read while the
cascade unit of work is constructed rather than inside the block, so three tests
that never mention access groups were failing at the async with. The new test
alongside it walks the dataclass fields, so the next dependent added to the
cascade is covered without anyone remembering to update a list.
Two problems, both caught in review.
The new table only landed in the root schema.prisma. Client generation reads
litellm/proxy/schema.prisma and packaging reads the copy under
litellm-proxy-extras, so the generated client had no
litellm_modelaccessgroupbudgettable and every budget read and write against
it would have failed at runtime. The root is the source of truth; both
copies are now byte-identical to it.
Nothing incremented spend:model_access_group:{group} after a call. Only the
reservation path ever wrote it, so with disable_budget_reservation the
read-time check was reading a counter nobody maintained and falling back to
the row's spend, which is cached for up to DEFAULT_MODEL_ACCESS_GROUP_CACHE_TTL.
A caller could run well past the pool inside that window, which is precisely
the case the read-time check exists to cover.
increment_spend_counters now takes the matched groups and charges them
through _init_and_increment_unreserved_spend_counter, so a group already
covered by a reservation is skipped rather than counted twice. The cost
callback sources the names with get_request_model_access_groups, the same
reader the spend writer uses.
A model access group could gate which models a caller reaches but never how
much that group of callers could spend in total. Capping a shared pool meant
setting a per-entity budget on every key by hand, which caps each key
separately and still leaves no way to read what the group cost.
Spend is attributed to a group only when the group's name appears on an
allowlist the caller was granted (key, team, team-member scope, project or
org) and that group serves the requested model. Asking for a model that
merely belongs to a group attributes nothing, because nothing about the
caller named the group. Levels are unioned rather than ranked, so a team
granted "*" whose member is scoped to one group still counts as gated by
that group.
Enforcement runs on both paths tags already use: a reservation counter on
the pre-call path and a read-time max_budget check inside the existing
concurrent budget gather, so the ceiling still holds under
disable_budget_reservation.
Adds LiteLLM_ModelAccessGroupBudgetTable, which is the only place a group is
ever a row: the groups themselves stay free-text strings in
model_info.access_groups, so a row exists only once someone gives that group
a budget. GET, PUT and DELETE /access_group/{name}/budget manage it, and
/access_group/{name}/info now carries the spend and budget alongside the
models.
Adds the durable row that a model access group budget hangs off. Model
access groups live only as free-text strings inside
model_info.access_groups, so unlike tags there is no existing row to
carry a budget_id.
Foundation only: schema, migration, repository, entity type, spend
transaction bucket, auth carrier field and registry cache keys. Nothing
reads or writes these yet.
The suggester pins temperature=0.2 for tool-selection determinism and passed no
drop_params, so an operator-supplied reasoning model whose only accepted temperature is 1
made litellm raise UnsupportedParamsError and the whole suggestion fail. The default
gpt-4o-mini is unaffected; the failure needs the caller to name a model.
Every other internal LLM call the proxy makes on a user's behalf already opts in through
judge_acompletion, which sets drop_params=True on both dispatch paths. This was the one
caller outside that contract, so the sampling preference is now advisory here too and the
call degrades instead of dying.
Resolves LIT-6352
The request and response tree passed the library's light palette in every
theme, so in dark mode the string values rendered dark green and the
punctuation rendered black on a near black surface. Pick the palette from
the resolved theme instead, and let the tree inherit the themed surface
rather than painting the library's own background.
Team-scoped deployments keep the internal model_name_{team_id}_{uuid} routing key and expose the public name in model_info.team_public_model_name. The dashboard links team model chips with the public name, so the exact filter now matches either name via the existing helper.
The where clause now uses the exact name string directly and skips the DB query when the typed search cannot occur in that name, so no new mutable literals are added (LIT002 gate).
With model=<group>&search=<term>, the router list was narrowed to the group but the DB query only matched the substring, so other groups' rows leaked into the page and total_count.
* feat(mcp): support RS256 signing for MCP gateway session tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format session token modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): enforce key strength on rotated public keys and unique kids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pass the selected group as the exact model= param on /v2/model/info rather than as the substring search, so a group like gpt-4 no longer pulls gpt-4o rows into the page and count. Drop two comments that restated helper behavior.
* fix(health): honor allow_requests_on_db_unavailable in readiness probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound readiness DB check and pass reconnect timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): bound whole readiness DB check with one deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): keep readiness deadline fallback within lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(health): suppress TQ008 for proxy-global readiness patches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): release reconnect lock when a waiting reconnect is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
The customStyle background only reaches the <pre>, so the prism theme's own
background stayed on the inner <code>. In dark mode that left a white card
with a dark box hugging every line. Let the block inherit the themed surface
from its wrapper instead.
* test: add logging e2e coverage (s3_v2, gcs_bucket, team langfuse callback, datadog failure)
Five new live e2e scenarios raising Logging & Guardrails registry coverage:
s3_v2 success and failure objects read back from the real S3 bucket,
gcs_bucket success record read back through the GCS JSON API (with
nextPageToken pagination and per-request bearer minting), team-scoped
Langfuse callback delivery with non-team isolation, and DataDog failure
event delivery queried by indexed model_group. datadog_reader gains
query-based variants of the marker search; the langfuse cell is a new
registry row. Bucket readers settle past a full flush interval so a
late duplicate cannot hide from the exactly-one assertions
* test: cover clock-skew day prefix in gcs read-back and retry team callback propagation
* test: key the s3 failure read-back on the provider error, not payload absence
* chore: rerun ci
* chore: rerun ci after config sync
* chore: rerun ci with pr lane env
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* test: add guardrail e2e coverage (presidio masking, bedrock post and during call, moderation on messages) (#38553)
* test: add guardrail e2e coverage (presidio masking, bedrock post/during, moderation on messages)
* test: require the phone placeholder positively in the presidio masking predicate
* test: count only the 400 verdict body as a bedrock post_call block
* test(e2e): exempt the guardrail config echo from the post_call leak assertion
* test(e2e): pin the fail-closed contract for an unknown guardrail name (skipped, product gap)
* test(e2e): tolerate the readiness 503 from a transient db blip in the callback-config probes
