The counting runs at the response boundary, so a metrics failure there would
turn a served 429 into no response at all. The guard existed but nothing
exercised it, which is the one uncovered line the patch report was pointing at.
The mark is set when `ProxyRateLimitError` is constructed, which is the one
point every raise site passes through. That also means a rejection the proxy
recovers from stays marked: `_pre_call_with_fallbacks` catches the local rate
limit and retries against a fallback model, so a provider 429 on that fallback
was counted as this pod shedding load. That is the exact conflation the mark
exists to prevent, reintroduced through the back door.
The mark is now cleared before each fallback attempt, so only a rejection this
proxy actually returns counts.
The restore before `raise original_exc` states the postcondition at the raise
site. Current control flow already satisfies it, because reaching that line
means every fallback raised its own `ProxyRateLimitError` and re-marked on
construction, so no test distinguishes it. It is kept so the guarantee does not
depend on that incidental re-marking.
Removing 503 from the counted set left the metric still advertising it, so a
consumer would have kept alerting on a status that can no longer appear. The
documentation now names the counted status and says why the two excluded kinds
are excluded: upstream rate limits arrive as 429 as well, and the proxy's 503s
mean a budget could not be verified, so a dependency is unreachable rather than
this pod being at capacity.
The test parses the advertised list rather than substring-matching it, and
clears the collector registry first, since a second PrometheusLogger in one
process otherwise trips the duplicate-collector guard depending on test order.
`_SHED_STATUSES` carried 503 alongside 429, but nothing ever marks a 503 as
shed by this proxy, so the entry was unreachable and read as though those
responses were counted.
Leaving it and marking the 503 paths would be wrong. Both of them are
fail-closed budget rejections, raised when spend cannot be verified against
Redis or the database. That is a dependency being unreachable, not this pod
running out of capacity, and the two call for opposite responses. Folding them
in would rebuild exactly the conflation that dropping provider 429s removed.
The status label stays, so a future rejection kind can be added deliberately
along with whatever marks it.
`is_global_limit_enforced` compares the registered limiter against the v1
handler, which is the only one that reads `global_max_parallel_requests`.
basedpyright called that comparison statically impossible, because
`PROXY_HOOKS` inferred its value type from the dict literal alone, and the
v1 handler is not in the literal. It is swapped in under
LEGACY_MULTI_INSTANCE_RATE_LIMITING, and the enterprise hooks are merged in
at the bottom of the module.
So the registry's inferred type never described what it actually holds, which
made the feature-flag swap and the enterprise merge unsound too. Annotating it
as `dict[str, type[CustomLogger]]` states the real contract and lets the
identity check type-check. Verified at runtime: all 10 registered entries are
CustomLogger subclasses.
Identity against v1 is kept deliberately, rather than "anything but v3", so an
unrecognised limiter reports the ceiling as unbounded instead of claiming one
it may not apply.
Three defects, each reproduced on a live proxy before the fix.
The ceiling gauge was published inside the database-gated startup branch, so a
prometheus proxy with no DATABASE_URL never published and the registered gauge
rendered as 0, which reads as "no requests allowed" on a proxy serving every
request. Publishing is not conditional on a database: concurrency is bounded per
worker either way.
The shed counter counted any 429 or 503 seen at the ASGI layer, but litellm
forwards upstream rate limits with the same 429 it uses for its own, so a
provider throttling us was recorded as this pod shedding load, inverting the
throttle-or-scale decision. Requests the proxy itself declines are now marked at
ProxyRateLimitError, the one class litellm raises for that, and only marked
responses count.
The ceiling gauge is also republished when general_settings is reloaded, since
it previously went stale for the life of the process.
Operators could see how many requests were in flight on a pod but not how many
the proxy was shedding, nor what ceiling was actually being applied, so there
was no way to tell "throttle upstream" apart from "add pods".
Shed responses are counted at the ASGI layer rather than at each limiter, so no
rejection path can be missed, and the count is per worker for the same reason
the in-flight gauge is. 500s are excluded: that is the proxy failing, not
declining.
The ceiling gauge reports what is actually in force. global_max_parallel_requests
is only read by the v1 limiter, which is off by default, so the gauge reports
+Inf when nothing bounds concurrency rather than echoing a configured number
that no limiter applies. A registered gauge always exposes a value, so leaving
it unset would have rendered as 0 and read as "no requests allowed".
Refs LIT-5435
The review flagged the added commentary against the repo rule that comments are
for genuinely complex logic, suppressions and TODOs. Measured, this stack was
running about 9.7 percent comment lines against a 6 percent baseline for
litellm/, so the point stands. The design rationale these blocks carried now
lives in the PR description, where it does not have to be kept in sync with the
code. Suppression reasons and the short notes on genuinely counterintuitive
behaviour are kept.
Two defects the review caught, both in the case this telemetry exists for.
The last-run clock was refreshed on every event, including runs that never
executed. A job repeatedly missed or skipped kept looking recently run, so the
"time since last run" alert the docs recommend would stay quiet through exactly
the outage it is meant to catch. Only a run that executed moves the clock now;
the skip is still counted, just not as a run.
Start times also leaked on a missed run. MAX_INSTANCES is emitted by the
scheduler before it submits, so there is nothing to release, and the previous
code treated MISSED the same way. MISSED comes from the executor after the
submit (`apscheduler/executors/base.py`), so its start time was recorded and
never freed, growing one entry per miss for the life of the process. The
comment claiming neither follows a submission was wrong for MISSED.
Also corrects the injected clock's annotation, which used `Final` in a parameter
position where it is not valid, and adds the missing `Final` on the lock
telemetry's logger lookup.
MISSED is one of the four outcomes the job metric advertises and had no test,
so nothing would have caught it being dropped or mislabelled. It arrives with no
submission of its own, which is the case worth pinning.
The never-raise guard was only reached through a real scheduler, and a listener
running on the scheduler's own thread is invisible to coverage, so it is now
driven directly as well.
`acquire_lock` caught exceptions to tell a failed attempt from losing the
election, but an outage does not always raise. `RedisCache.async_set_cache`
catches connection errors, records a swallowed-failure marker and returns None,
and None is also how redis reports SET NX losing the race. Both arrived as
`not_acquired`, so a Redis outage was published as ordinary contention: the
distinction the split was added to make, lost in the most important case.
The marker is the only thing that separates them, so it is compared across the
attempt. It is a per-task ContextVar, so a concurrent caller's failure cannot be
misread as this one's. `swallowed_redis_failure_count` exposes it rather than
having callers reach for the private ContextVar.
Verified against a stub that mimics RedisCache exactly: an outage now records
error, losing the race still records not_acquired, and success still records
acquired.
The cron-lock metric documented three result values while the code emitted a
fourth, `error`, added when a failed attempt was split from losing the
election. A consumer building alerts from the documented set would silently
drop every Redis-outage attempt, which is the case the split existed to
surface.
Rather than adding the missing word, the four outcomes are now a
`LockAttemptResult` enum that both sides derive from: the lock manager is typed
to emit only its members, and the metric documentation is generated from them,
so the two cannot drift again.
The test parses the advertised list out of the documentation rather than
substring-matching it, since `error` also appears in the prose that follows and
would have made a looser assertion pass against the very documentation that
prompted this.
Three defects surfaced in review.
Durations were keyed by job id alone, but APSCHEDULER_MAX_INSTANCES and
coalesce are both env-overridable, so several runs of one job can be in flight
or one submission can carry several run times. Runs are now keyed by job id and
scheduled run time, so concurrent runs stop consuming each other's start times.
A failed lock attempt was reported as ordinary contention, because the inner
method returns False both when another pod wins and when Redis errors. The
wrapper now owns the no-Redis and error exits and emits a distinct result, with
the bool | None contract to callers unchanged.
update_spend no longer reports an item count. The spend-log queue is drained by
_monitor_spend_logs_queue as well, so a count derived from queue depth here
could credit that task's work or hide work this job did. The listener still
publishes counts for any job that can report one it owns.
Nothing recorded which background job ran on a pod, when, for how long, whether
it succeeded, or how much work it moved, so during an incident job activity
could only be inferred from database load. A job that overran its interval and
started being skipped left no trace at all.
One APScheduler listener instruments every registered job at once rather than
each job growing its own instrumentation, and reports max_instances skips as a
first-class result so a job falling behind its schedule is visible. The
single-owner cron lock outcome becomes a metric that separates winning the lock
from losing it from having no Redis to elect with.
Three integrations registered their export job without an explicit id, leaving
APScheduler to generate a uuid that would have grown the job_name label without
bound; they now pin the id they already had a constant for.
Refs LIT-5435
The review flagged the added commentary against the repo rule that comments are
for genuinely complex logic, suppressions and TODOs. Measured, this stack was
running about 9.7 percent comment lines against a 6 percent baseline for
litellm/, so the point stands. The design rationale these blocks carried now
lives in the PR description, where it does not have to be kept in sync with the
code. Suppression reasons and the short notes on genuinely counterintuitive
behaviour are kept.
Pool contention raises a typed prisma error carrying `code == "P2024"`, which
the attribute check already handled. When the database itself is unreachable the
engine reports the same P2024 as a raw `EngineRequestError` instead, with no
`code` attribute and the code recorded only in the JSON body it was built from,
so those went uncounted. Seen on a live proxy at connection limit 1 with
Postgres paused.
Prisma classifies both as P2024, so which layer surfaced it should not decide
whether the counter moves. The match is on prisma's own `error_code` field
rather than the message text, so an error that merely mentions the code cannot
trip it, and a different engine code is rejected.
The two cases stay distinguishable in the metrics that matter: saturation holds
busy at max with waiters queued, while an unreachable database drops open
connections instead.
`success_callback: ["prometheus"]` is the registration the docs show, and it
never reaches a callback list. The logger is constructed lazily on the first
request and cached in `_in_memory_loggers`, so `get_instance` searching only
the callback lists returned None for the whole life of such a proxy, and every
pool metric registered and sat at zero.
Verified directly: after the lazy construction the cache holds the logger while
`get_instance` still returned None. It now returns that same object.
An earlier review raised this and I refuted it on the strength of a live proxy
showing 103 metric families after the first request. That measurement was about
litellm's own metrics, which the logging path records on the instance directly,
not through `get_instance`, so it did not cover this path. The finding was
correct.
The regression test drives the string registration and the lazy construction
rather than placing an object on the list, which is what the previous test did
and why it passed throughout.
Checking whether a sample was due and consuming that interval were separate
steps, so every caller in a concurrent burst of database calls saw the same
due-ness and dispatched its own task. Only one did real work, but the rest
still allocated a task on the auth hot path.
try_claim does both in one synchronous step, and the dispatched task now takes
the sample it was already granted instead of claiming again, which would have
failed and left no sample taken at all.
The restart guard zeroed the counter deltas but left the waiter baseline in
place. A fresh engine's waiter gauge carries no latch, so subtracting the
pre-restart baseline hid real waiters and could report zero during the very
saturation that caused the restart.
A proxy without prometheus was still taking the throttled query-engine read
every interval and discarding the result. The client resolver now checks for a
collector first, so that deployment does no extra work while the interval is
still consumed, which keeps the throttle from retrying on every database call.
Also condenses the rationale comments this change added down to what the code
cannot say for itself, per the repo's comment policy.
Two defects surfaced in review, both reproduced on a live proxy first.
`PrometheusLogger.get_instance` searched only `litellm.callbacks`, so the
equally supported `litellm_settings.success_callback: ["prometheus"]` left every
pool metric registered and permanently at zero. It now resolves through
`logging_callback_manager`, which covers all five callback lists.
Decorated database helpers nest, `get_object_permission` is called from inside
`get_key_object` and both carry the decorator, so one P2024 was counted once per
enclosing `except`. The exception is now marked the first time it is counted.
Operators could see latency and database CPU symptoms during an incident but
could not tell whether the proxy had run out of connections, because nothing
exposed the pool. Ten bounded-cardinality metrics now bridge the Prisma query
engine's own pool counters into Prometheus, keeping the time a query spent
waiting for a slot separate from the time it spent executing.
The configured maximum is derived from the engine as busy + idle rather than
parsed out of DATABASE_URL, so no credential is read on this path.
Sampling rides on database work instead of a scheduled job, and the exhaustion
counter is incremented at the error site, so both keep reporting through the
window a delayed exporter would erase.
Refs LIT-5435
detect-backend-changes diffed the event payload's base.sha against the
checked-out ref. Those are two different points in time: actions/checkout
resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base
branch advances, so the diff picked up whatever landed on staging between the
event firing and the job starting. On a recent UI-only pull request three
backend commits from staging were attributed to the branch, and every backend
shard ran in full
Ask the API which files the pull request touches instead. That is the same set
the Files changed tab shows, and it is immune to either endpoint moving. The
shell body moves into .github/scripts/detect_backend_changes.sh so it can be
exercised directly, and the fail-open paths now also cover an API failure, a
file list past the API's 3000-entry listing ceiling, and a classifier that
prints something unexpected
Every dynamic tracer-provider build called Resource.create, which scans the entry
points of every installed distribution, roughly 3ms and 200 file opens. The dynamic
providers reach it from the async logging path, which runs on the event loop serving
requests, so past the provider cache bound every request paid it and delayed the
requests in flight alongside it
The value derives only from the logger's config and process environment, so it is
built once per logger and reused. This logger's own init-time providers share it,
which also removes redundant startup builds. ArizeLogger overrides _init_tracing and
still builds its own, so it keeps one extra build
Refs LIT-5437
The checkbox and its visible text both sat inside the max_budget
FormField label, which renders a single <label htmlFor> pointing at the
number input, so clicking the text focused Max Budget instead of ticking
the box. The checkbox only carried an aria-label, so it had no label of
its own to catch the click.
Wrap the checkbox and its text in their own <label>, the association the
antd checkbox wrapper used to provide. The accessible name now comes
from that label, so the aria-label is dropped rather than duplicated,
and the outer field label still points at the number input.
* test(ui): cover the two modals no test would catch breaking
Both files sit in the antd Modal migration's blind spot. EditSSOSettingsModal's
test replaced antd wholesale with a stub Modal and asserted the stub's own
data-testid markup, so it proved nothing about the modal a user sees and would
have stayed green through any regression. routing_groups had no test at all.
Rewrite the first against the real antd Modal, querying by dialog role and
accessible name so the assertions hold under either library, and add an
integration test for the second that drives the row menu and the delete
confirmation end to end.
Modal width drops out of the SSO assertions: antd carries it as an inline style
and shadcn as a max-width class, so either form couples the test to the library
rather than to anything a user perceives.
* refactor(ui): move the straightforward antd Modals onto the shared Dialog
Twenty files whose Modal only used title, open, width, footer, className and
onCancel, so each one maps onto Dialog without judgement calls. Width becomes a
max-width class, the body gets the house scroll cap so tall content stays
reachable, and destroyOnHidden goes away because Base UI unmounts a closed
dialog on its own.
EditMembership needed a real fix rather than a translation. Clearing the form
after a submit resolved only ever worked by accident: the reset set every field
to undefined, which react-hook-form does not push out to a subscribed
Controller, and the fields looked cleared only because antd's Modal happened to
re-render the subtree afterwards. Dialog does not, so the stale values showed
through. emptyMemberFormValues now returns the empty value each control
actually understands, an empty string, null or an empty list, and the reset
lands whatever renders around it. Its test asserted the undefined shape while
describing the behaviour it was missing, so it now checks the values instead.
* refactor(ui): migrate the antd Modals that needed a judgement call
Sixteen more files. Most carried a prop that does not translate literally:
maskClosable becomes disablePointerDismissal, afterOpenChange becomes
onOpenChangeComplete, and closable={false} becomes showCloseButton={false}.
The styles prop went away everywhere it appeared. All but one instance set the
body to 24px and the header to 24px with no border, which is what DialogContent
already renders, so keeping it would have meant writing the default back by
hand.
Several Modals passed onOk alongside footer={null}, so antd rendered no OK
button and the handler could never fire. Each of those handlers was a
character-for-character copy of the neighbouring onCancel, so they are gone
rather than translated.
Rich titles now sit inside DialogHeader with DialogTitle carrying the heading
text, instead of the whole header block being nested inside DialogTitle. That
had put an h2 inside another h2, which is invalid and gave one dialog two
headings.
UserEnvVarsModal loses its formGeneration counter. Remounting the form when the
modal finished opening only mattered because antd kept a closed modal's
children mounted; Base UI unmounts them, so reopening is blank on its own. Its
test helper had encoded that remount as a timing assumption, so the file now
states the requirement outright and checks that reopening shows an empty field.
CreateMCPServer's cancel test read the tool list while the modal was closed,
which only worked because forceRender kept it mounted. It now asserts what a
user can actually observe: the panel is gone while closed, and reopening brings
back an empty URL and no tools.
Unmounting an open Base UI dialog leaves its scroll lock on <html> and <body>,
which survives cleanup() and makes every later test in the file see a locked
page where popups compute pointer-events: none and clicks quietly do nothing.
The shared setup now releases it.
* refactor(ui): finish the antd Modal migration onto the shared Dialog
Fifteen files whose Modal relied on antd's built-in footer. okText, cancelText,
onOk, okButtonProps, cancelButtonProps and confirmLoading collapse into two
explicit buttons in a DialogFooter, with danger becoming the destructive
variant and the various loading flags becoming disabled plus aria-busy. The one
okButtonProps that also hand-set a red background drops it, since the variant
already carries that.
add_guardrail_form keeps its own chrome, so its DialogContent turns off the
built-in close button and the padding, and its heading becomes the DialogTitle.
Under antd it passed title={null} and had no accessible name at all.
Modals that positioned themselves near the top of the viewport needed
translate-y-0 alongside top-8, because DialogContent centres itself with a
transform that top alone does not undo.
TeamGuardrailsTab's test reached its Mode select by index into every combobox on
the page. A modal dialog hides the rest of the page from assistive technology,
which antd never did, so the count changed and the index pointed at the wrong
control. It asks for the field by label now.
The mask-dismissal test drove antd's .ant-modal-wrap class directly; it uses the
overlay slot our own component exposes, and still fails if
disablePointerDismissal is dropped.
CreateUserButton stays on antd. Its Modal converts cleanly, but the colocated
test file then fails a varying handful of cases, and the cause sits in the test
file rather than the component, so it wants its own change.
Pruning suppressions for the touched files also cleared four
react-hooks/set-state-in-effect entries on CreateMCPServer that were already
stale before this branch.
* test(ui): pick Base UI select options through the shared helper
React 19's flush timing loses the race this test was relying on: the option
lands in the DOM one render before its positioner drops pointer-events: none,
so user-event refused the click. tests/test-utils already exports
chooseSelectOption for exactly this, added alongside the React 19 upgrade.
The chart shipped ghcr.io/berriai/litellm-database as its image default,
with a comment offering it as the "optimized image with database". That
distinction no longer exists: Dockerfile and docker/Dockerfile.database
differ only in comment text and one builder-stage apk package, and both
published images bake the prisma CLI, engines, schema.prisma and
prisma_migration.py into /app, so either one runs the migrations job.
Point the default at the canonical image the release notes, the cosign
verification instructions and the chart's own README already name, and
update the chart's tests and README so nothing still refers to the
legacy repository.
* refactor(ui): migrate the remaining dashboard pages off antd
Converts the teams, usage, guardrails, vector stores, cost tracking,
agents, policies, login and onboarding screens onto the shadcn
primitives, including the team info tab shell and the virtual keys
hover cards.
* fix(ui): close out the antd migration's failing type checks and tests
Alert and Badge were missing the success and info variants their call
sites already used. Combobox dropped disabled because Base UI merges the
primitive's own props over the render child, so the flag never reached
the input, and the guardrails status filter had no accessible name, which
left two comboboxes indistinguishable to the tests.
The remaining test updates swap antd's title-based queries for the roles
the shadcn controls expose.
The reasoning override's floor was pinned to tier_boundaries.simple_medium,
so an operator could not restore the unconditional promotion nor raise the bar
independently of the SIMPLE/MEDIUM cut. Setting reasoning_override_min_score
was accepted and echoed back by /model/info, because the config model allows
extra keys, while routing ignored it.
Resolve the floor through one accessor that falls back to simple_medium when
the field is unset, so moving that boundary still moves the floor with it, and
an explicit 0 is a real floor rather than an absent one. Record the resolved
value on the routing decision so a logged row states the floor that applied,
which is also what lets the Admin UI stop hardcoding the copy PR #37500 added.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
`lite up` already patches ~/.claude/settings.json, but only for as long as it
runs in the foreground, and it restores the original file on exit. Users
proxying Claude Code through LiteLLM therefore have to re-wire it by hand after
every login.
--config-claude makes that write persistent. It reuses the settings shape
`lite up` writes (env.ANTHROPIC_BASE_URL plus an apiKeyHelper invocation),
preserves every unrelated key, creates the file when missing, and writes it
atomically with owner-only permissions. Plain `lite login` is unchanged.
Reaching the credential through apiKeyHelper rather than copying it into the
file means a later login refreshes it with no further action, and keeps the
short-lived CLI token out of settings.json entirely.
The shared parts of the settings-file handling move from up.py into a new
claude_settings.py, since up.py imports auth.py and so auth.py cannot import
up.py back. That module now also owns the registry of commands that can be
temporarily managing the file, so the persistent write refuses while either
`lite up` or `lite autoroute up` holds a backup it would later restore over
this write.
Because this write has no backup and no `lite down`, it is stricter than
`lite up` about the user's file: it writes through a symlinked settings.json
rather than replacing the link with a regular file, and it refuses rather than
silently discarding a non-object `env` value.
Also fixes the apiKeyHelper command itself: --base-url belongs to the
top-level `lite` group, so `lite auth print-token --base-url X` is rejected by
click with "No such option". Every settings file `lite up` has written carries
that malformed command, which makes the helper return nothing and every Claude
Code request lose its token. The existing tests only string-matched the
generated command, so the new tests parse it through the real CLI instead.
