Settings refresh on a timer and the publish helper runs on each one, so a proxy
with the setting stored in the database repeated the same warning for the life
of the process. It is now said once per configured value, through a cached warn
rather than a module-level flag, so no mutable state is introduced.
Also adds the missing `Final` on the shed recorder's logger lookup.
`success_callback: ["prometheus"]` is the registration the docs show, and it
builds the logger on the first request. Startup had already published the
ceiling by then, with no logger to publish to, so the gauge registered at its
zero default and stayed there. A proxy serving every request reported a ceiling
of zero, which is the reading `+Inf` was introduced to prevent.
Reproduced on a live proxy with that config and a limit of 7 enforced: the gauge
read 0.0 after a successful request. With the ceiling restored on construction
it reads 7.0, and an unconfigured proxy still reads +Inf.
Set on the instance rather than through the module helper, since during
`__init__` the logger is not yet reachable by the lookup that helper uses.
The counting runs at the response boundary, so a metrics failure there would
turn a served 429 into no response at all. The guard existed but nothing
exercised it, which is the one uncovered line the patch report was pointing at.
The mark is set when `ProxyRateLimitError` is constructed, which is the one
point every raise site passes through. That also means a rejection the proxy
recovers from stays marked: `_pre_call_with_fallbacks` catches the local rate
limit and retries against a fallback model, so a provider 429 on that fallback
was counted as this pod shedding load. That is the exact conflation the mark
exists to prevent, reintroduced through the back door.
The mark is now cleared before each fallback attempt, so only a rejection this
proxy actually returns counts.
The restore before `raise original_exc` states the postcondition at the raise
site. Current control flow already satisfies it, because reaching that line
means every fallback raised its own `ProxyRateLimitError` and re-marked on
construction, so no test distinguishes it. It is kept so the guarantee does not
depend on that incidental re-marking.
Removing 503 from the counted set left the metric still advertising it, so a
consumer would have kept alerting on a status that can no longer appear. The
documentation now names the counted status and says why the two excluded kinds
are excluded: upstream rate limits arrive as 429 as well, and the proxy's 503s
mean a budget could not be verified, so a dependency is unreachable rather than
this pod being at capacity.
The test parses the advertised list rather than substring-matching it, and
clears the collector registry first, since a second PrometheusLogger in one
process otherwise trips the duplicate-collector guard depending on test order.
`_SHED_STATUSES` carried 503 alongside 429, but nothing ever marks a 503 as
shed by this proxy, so the entry was unreachable and read as though those
responses were counted.
Leaving it and marking the 503 paths would be wrong. Both of them are
fail-closed budget rejections, raised when spend cannot be verified against
Redis or the database. That is a dependency being unreachable, not this pod
running out of capacity, and the two call for opposite responses. Folding them
in would rebuild exactly the conflation that dropping provider 429s removed.
The status label stays, so a future rejection kind can be added deliberately
along with whatever marks it.
Three defects, each reproduced on a live proxy before the fix.
The ceiling gauge was published inside the database-gated startup branch, so a
prometheus proxy with no DATABASE_URL never published and the registered gauge
rendered as 0, which reads as "no requests allowed" on a proxy serving every
request. Publishing is not conditional on a database: concurrency is bounded per
worker either way.
The shed counter counted any 429 or 503 seen at the ASGI layer, but litellm
forwards upstream rate limits with the same 429 it uses for its own, so a
provider throttling us was recorded as this pod shedding load, inverting the
throttle-or-scale decision. Requests the proxy itself declines are now marked at
ProxyRateLimitError, the one class litellm raises for that, and only marked
responses count.
The ceiling gauge is also republished when general_settings is reloaded, since
it previously went stale for the life of the process.
Operators could see how many requests were in flight on a pod but not how many
the proxy was shedding, nor what ceiling was actually being applied, so there
was no way to tell "throttle upstream" apart from "add pods".
Shed responses are counted at the ASGI layer rather than at each limiter, so no
rejection path can be missed, and the count is per worker for the same reason
the in-flight gauge is. 500s are excluded: that is the proxy failing, not
declining.
The ceiling gauge reports what is actually in force. global_max_parallel_requests
is only read by the v1 limiter, which is off by default, so the gauge reports
+Inf when nothing bounds concurrency rather than echoing a configured number
that no limiter applies. A registered gauge always exposes a value, so leaving
it unset would have rendered as 0 and read as "no requests allowed".
Refs LIT-5435
Two defects the review caught, both in the case this telemetry exists for.
The last-run clock was refreshed on every event, including runs that never
executed. A job repeatedly missed or skipped kept looking recently run, so the
"time since last run" alert the docs recommend would stay quiet through exactly
the outage it is meant to catch. Only a run that executed moves the clock now;
the skip is still counted, just not as a run.
Start times also leaked on a missed run. MAX_INSTANCES is emitted by the
scheduler before it submits, so there is nothing to release, and the previous
code treated MISSED the same way. MISSED comes from the executor after the
submit (`apscheduler/executors/base.py`), so its start time was recorded and
never freed, growing one entry per miss for the life of the process. The
comment claiming neither follows a submission was wrong for MISSED.
Also corrects the injected clock's annotation, which used `Final` in a parameter
position where it is not valid, and adds the missing `Final` on the lock
telemetry's logger lookup.
MISSED is one of the four outcomes the job metric advertises and had no test,
so nothing would have caught it being dropped or mislabelled. It arrives with no
submission of its own, which is the case worth pinning.
The never-raise guard was only reached through a real scheduler, and a listener
running on the scheduler's own thread is invisible to coverage, so it is now
driven directly as well.
`acquire_lock` caught exceptions to tell a failed attempt from losing the
election, but an outage does not always raise. `RedisCache.async_set_cache`
catches connection errors, records a swallowed-failure marker and returns None,
and None is also how redis reports SET NX losing the race. Both arrived as
`not_acquired`, so a Redis outage was published as ordinary contention: the
distinction the split was added to make, lost in the most important case.
The marker is the only thing that separates them, so it is compared across the
attempt. It is a per-task ContextVar, so a concurrent caller's failure cannot be
misread as this one's. `swallowed_redis_failure_count` exposes it rather than
having callers reach for the private ContextVar.
Verified against a stub that mimics RedisCache exactly: an outage now records
error, losing the race still records not_acquired, and success still records
acquired.
The cron-lock metric documented three result values while the code emitted a
fourth, `error`, added when a failed attempt was split from losing the
election. A consumer building alerts from the documented set would silently
drop every Redis-outage attempt, which is the case the split existed to
surface.
Rather than adding the missing word, the four outcomes are now a
`LockAttemptResult` enum that both sides derive from: the lock manager is typed
to emit only its members, and the metric documentation is generated from them,
so the two cannot drift again.
The test parses the advertised list out of the documentation rather than
substring-matching it, since `error` also appears in the prose that follows and
would have made a looser assertion pass against the very documentation that
prompted this.
Three defects surfaced in review.
Durations were keyed by job id alone, but APSCHEDULER_MAX_INSTANCES and
coalesce are both env-overridable, so several runs of one job can be in flight
or one submission can carry several run times. Runs are now keyed by job id and
scheduled run time, so concurrent runs stop consuming each other's start times.
A failed lock attempt was reported as ordinary contention, because the inner
method returns False both when another pod wins and when Redis errors. The
wrapper now owns the no-Redis and error exits and emits a distinct result, with
the bool | None contract to callers unchanged.
update_spend no longer reports an item count. The spend-log queue is drained by
_monitor_spend_logs_queue as well, so a count derived from queue depth here
could credit that task's work or hide work this job did. The listener still
publishes counts for any job that can report one it owns.
Nothing recorded which background job ran on a pod, when, for how long, whether
it succeeded, or how much work it moved, so during an incident job activity
could only be inferred from database load. A job that overran its interval and
started being skipped left no trace at all.
One APScheduler listener instruments every registered job at once rather than
each job growing its own instrumentation, and reports max_instances skips as a
first-class result so a job falling behind its schedule is visible. The
single-owner cron lock outcome becomes a metric that separates winning the lock
from losing it from having no Redis to elect with.
Three integrations registered their export job without an explicit id, leaving
APScheduler to generate a uuid that would have grown the job_name label without
bound; they now pin the id they already had a constant for.
Refs LIT-5435
Pool contention raises a typed prisma error carrying `code == "P2024"`, which
the attribute check already handled. When the database itself is unreachable the
engine reports the same P2024 as a raw `EngineRequestError` instead, with no
`code` attribute and the code recorded only in the JSON body it was built from,
so those went uncounted. Seen on a live proxy at connection limit 1 with
Postgres paused.
Prisma classifies both as P2024, so which layer surfaced it should not decide
whether the counter moves. The match is on prisma's own `error_code` field
rather than the message text, so an error that merely mentions the code cannot
trip it, and a different engine code is rejected.
The two cases stay distinguishable in the metrics that matter: saturation holds
busy at max with waiters queued, while an unreachable database drops open
connections instead.
`success_callback: ["prometheus"]` is the registration the docs show, and it
never reaches a callback list. The logger is constructed lazily on the first
request and cached in `_in_memory_loggers`, so `get_instance` searching only
the callback lists returned None for the whole life of such a proxy, and every
pool metric registered and sat at zero.
Verified directly: after the lazy construction the cache holds the logger while
`get_instance` still returned None. It now returns that same object.
An earlier review raised this and I refuted it on the strength of a live proxy
showing 103 metric families after the first request. That measurement was about
litellm's own metrics, which the logging path records on the instance directly,
not through `get_instance`, so it did not cover this path. The finding was
correct.
The regression test drives the string registration and the lazy construction
rather than placing an object on the list, which is what the previous test did
and why it passed throughout.
Checking whether a sample was due and consuming that interval were separate
steps, so every caller in a concurrent burst of database calls saw the same
due-ness and dispatched its own task. Only one did real work, but the rest
still allocated a task on the auth hot path.
try_claim does both in one synchronous step, and the dispatched task now takes
the sample it was already granted instead of claiming again, which would have
failed and left no sample taken at all.
The restart guard zeroed the counter deltas but left the waiter baseline in
place. A fresh engine's waiter gauge carries no latch, so subtracting the
pre-restart baseline hid real waiters and could report zero during the very
saturation that caused the restart.
A proxy without prometheus was still taking the throttled query-engine read
every interval and discarding the result. The client resolver now checks for a
collector first, so that deployment does no extra work while the interval is
still consumed, which keeps the throttle from retrying on every database call.
Also condenses the rationale comments this change added down to what the code
cannot say for itself, per the repo's comment policy.
Two defects surfaced in review, both reproduced on a live proxy first.
`PrometheusLogger.get_instance` searched only `litellm.callbacks`, so the
equally supported `litellm_settings.success_callback: ["prometheus"]` left every
pool metric registered and permanently at zero. It now resolves through
`logging_callback_manager`, which covers all five callback lists.
Decorated database helpers nest, `get_object_permission` is called from inside
`get_key_object` and both carry the decorator, so one P2024 was counted once per
enclosing `except`. The exception is now marked the first time it is counted.
Operators could see latency and database CPU symptoms during an incident but
could not tell whether the proxy had run out of connections, because nothing
exposed the pool. Ten bounded-cardinality metrics now bridge the Prisma query
engine's own pool counters into Prometheus, keeping the time a query spent
waiting for a slot separate from the time it spent executing.
The configured maximum is derived from the engine as busy + idle rather than
parsed out of DATABASE_URL, so no credential is read on this path.
Sampling rides on database work instead of a scheduled job, and the exhaustion
counter is incremented at the error site, so both keep reporting through the
window a delayed exporter would erase.
Refs LIT-5435
detect-backend-changes diffed the event payload's base.sha against the
checked-out ref. Those are two different points in time: actions/checkout
resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base
branch advances, so the diff picked up whatever landed on staging between the
event firing and the job starting. On a recent UI-only pull request three
backend commits from staging were attributed to the branch, and every backend
shard ran in full
Ask the API which files the pull request touches instead. That is the same set
the Files changed tab shows, and it is immune to either endpoint moving. The
shell body moves into .github/scripts/detect_backend_changes.sh so it can be
exercised directly, and the fail-open paths now also cover an API failure, a
file list past the API's 3000-entry listing ceiling, and a classifier that
prints something unexpected
Every dynamic tracer-provider build called Resource.create, which scans the entry
points of every installed distribution, roughly 3ms and 200 file opens. The dynamic
providers reach it from the async logging path, which runs on the event loop serving
requests, so past the provider cache bound every request paid it and delayed the
requests in flight alongside it
The value derives only from the logger's config and process environment, so it is
built once per logger and reused. This logger's own init-time providers share it,
which also removes redundant startup builds. ArizeLogger overrides _init_tracing and
still builds its own, so it keeps one extra build
Refs LIT-5437
The reasoning override's floor was pinned to tier_boundaries.simple_medium,
so an operator could not restore the unconditional promotion nor raise the bar
independently of the SIMPLE/MEDIUM cut. Setting reasoning_override_min_score
was accepted and echoed back by /model/info, because the config model allows
extra keys, while routing ignored it.
Resolve the floor through one accessor that falls back to simple_medium when
the field is unset, so moving that boundary still moves the floor with it, and
an explicit 0 is a real floor rather than an absent one. Record the resolved
value on the routing decision so a logged row states the floor that applied,
which is also what lets the Admin UI stop hardcoding the copy PR #37500 added.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
`lite up` already patches ~/.claude/settings.json, but only for as long as it
runs in the foreground, and it restores the original file on exit. Users
proxying Claude Code through LiteLLM therefore have to re-wire it by hand after
every login.
--config-claude makes that write persistent. It reuses the settings shape
`lite up` writes (env.ANTHROPIC_BASE_URL plus an apiKeyHelper invocation),
preserves every unrelated key, creates the file when missing, and writes it
atomically with owner-only permissions. Plain `lite login` is unchanged.
Reaching the credential through apiKeyHelper rather than copying it into the
file means a later login refreshes it with no further action, and keeps the
short-lived CLI token out of settings.json entirely.
The shared parts of the settings-file handling move from up.py into a new
claude_settings.py, since up.py imports auth.py and so auth.py cannot import
up.py back. That module now also owns the registry of commands that can be
temporarily managing the file, so the persistent write refuses while either
`lite up` or `lite autoroute up` holds a backup it would later restore over
this write.
Because this write has no backup and no `lite down`, it is stricter than
`lite up` about the user's file: it writes through a symlinked settings.json
rather than replacing the link with a regular file, and it refuses rather than
silently discarding a non-object `env` value.
Also fixes the apiKeyHelper command itself: --base-url belongs to the
top-level `lite` group, so `lite auth print-token --base-url X` is rejected by
click with "No such option". Every settings file `lite up` has written carries
that malformed command, which makes the helper return nothing and every Claude
Code request lose its token. The existing tests only string-matched the
generated command, so the new tests parse it through the real CLI instead.
An unknown reasoning split now falls back to reasoning_tokens=0 in the
chat-to-responses usage translation, since the OpenAI SDK requires
output_tokens_details with an int reasoning_tokens, and the streaming
chunk builder caps the tokenized reasoning estimate at completion_tokens
and fills text_tokens with the remainder
Greptile flagged that lazy per-slug pool initialization could double-build
under concurrent replay calls, splitting consumption across a discarded
pool. Pools are now built once at ReplaySource construction and per-key
consumption is a single atomic deque pop, with a barrier-synchronized
regression test that fails 10/10 under the lazy-init mutant
A trusted hostname over plain http would expose the bearer token or a
replayable SigV4 signature to network observers. Credentials now only ride
https, with localhost exempt so local MCP stubs keep working.
Web-search connector 1.1.0 and later return the machine-readable results in result.structuredContent and may leave the text block as prose, which the parser dropped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
