Two defects the review caught, both in the case this telemetry exists for.
The last-run clock was refreshed on every event, including runs that never
executed. A job repeatedly missed or skipped kept looking recently run, so the
"time since last run" alert the docs recommend would stay quiet through exactly
the outage it is meant to catch. Only a run that executed moves the clock now;
the skip is still counted, just not as a run.
Start times also leaked on a missed run. MAX_INSTANCES is emitted by the
scheduler before it submits, so there is nothing to release, and the previous
code treated MISSED the same way. MISSED comes from the executor after the
submit (`apscheduler/executors/base.py`), so its start time was recorded and
never freed, growing one entry per miss for the life of the process. The
comment claiming neither follows a submission was wrong for MISSED.
Also corrects the injected clock's annotation, which used `Final` in a parameter
position where it is not valid, and adds the missing `Final` on the lock
telemetry's logger lookup.
MISSED is one of the four outcomes the job metric advertises and had no test,
so nothing would have caught it being dropped or mislabelled. It arrives with no
submission of its own, which is the case worth pinning.
The never-raise guard was only reached through a real scheduler, and a listener
running on the scheduler's own thread is invisible to coverage, so it is now
driven directly as well.
`acquire_lock` caught exceptions to tell a failed attempt from losing the
election, but an outage does not always raise. `RedisCache.async_set_cache`
catches connection errors, records a swallowed-failure marker and returns None,
and None is also how redis reports SET NX losing the race. Both arrived as
`not_acquired`, so a Redis outage was published as ordinary contention: the
distinction the split was added to make, lost in the most important case.
The marker is the only thing that separates them, so it is compared across the
attempt. It is a per-task ContextVar, so a concurrent caller's failure cannot be
misread as this one's. `swallowed_redis_failure_count` exposes it rather than
having callers reach for the private ContextVar.
Verified against a stub that mimics RedisCache exactly: an outage now records
error, losing the race still records not_acquired, and success still records
acquired.
The cron-lock metric documented three result values while the code emitted a
fourth, `error`, added when a failed attempt was split from losing the
election. A consumer building alerts from the documented set would silently
drop every Redis-outage attempt, which is the case the split existed to
surface.
Rather than adding the missing word, the four outcomes are now a
`LockAttemptResult` enum that both sides derive from: the lock manager is typed
to emit only its members, and the metric documentation is generated from them,
so the two cannot drift again.
The test parses the advertised list out of the documentation rather than
substring-matching it, since `error` also appears in the prose that follows and
would have made a looser assertion pass against the very documentation that
prompted this.
Three defects surfaced in review.
Durations were keyed by job id alone, but APSCHEDULER_MAX_INSTANCES and
coalesce are both env-overridable, so several runs of one job can be in flight
or one submission can carry several run times. Runs are now keyed by job id and
scheduled run time, so concurrent runs stop consuming each other's start times.
A failed lock attempt was reported as ordinary contention, because the inner
method returns False both when another pod wins and when Redis errors. The
wrapper now owns the no-Redis and error exits and emits a distinct result, with
the bool | None contract to callers unchanged.
update_spend no longer reports an item count. The spend-log queue is drained by
_monitor_spend_logs_queue as well, so a count derived from queue depth here
could credit that task's work or hide work this job did. The listener still
publishes counts for any job that can report one it owns.
Nothing recorded which background job ran on a pod, when, for how long, whether
it succeeded, or how much work it moved, so during an incident job activity
could only be inferred from database load. A job that overran its interval and
started being skipped left no trace at all.
One APScheduler listener instruments every registered job at once rather than
each job growing its own instrumentation, and reports max_instances skips as a
first-class result so a job falling behind its schedule is visible. The
single-owner cron lock outcome becomes a metric that separates winning the lock
from losing it from having no Redis to elect with.
Three integrations registered their export job without an explicit id, leaving
APScheduler to generate a uuid that would have grown the job_name label without
bound; they now pin the id they already had a constant for.
Refs LIT-5435
Pool contention raises a typed prisma error carrying `code == "P2024"`, which
the attribute check already handled. When the database itself is unreachable the
engine reports the same P2024 as a raw `EngineRequestError` instead, with no
`code` attribute and the code recorded only in the JSON body it was built from,
so those went uncounted. Seen on a live proxy at connection limit 1 with
Postgres paused.
Prisma classifies both as P2024, so which layer surfaced it should not decide
whether the counter moves. The match is on prisma's own `error_code` field
rather than the message text, so an error that merely mentions the code cannot
trip it, and a different engine code is rejected.
The two cases stay distinguishable in the metrics that matter: saturation holds
busy at max with waiters queued, while an unreachable database drops open
connections instead.
`success_callback: ["prometheus"]` is the registration the docs show, and it
never reaches a callback list. The logger is constructed lazily on the first
request and cached in `_in_memory_loggers`, so `get_instance` searching only
the callback lists returned None for the whole life of such a proxy, and every
pool metric registered and sat at zero.
Verified directly: after the lazy construction the cache holds the logger while
`get_instance` still returned None. It now returns that same object.
An earlier review raised this and I refuted it on the strength of a live proxy
showing 103 metric families after the first request. That measurement was about
litellm's own metrics, which the logging path records on the instance directly,
not through `get_instance`, so it did not cover this path. The finding was
correct.
The regression test drives the string registration and the lazy construction
rather than placing an object on the list, which is what the previous test did
and why it passed throughout.
Checking whether a sample was due and consuming that interval were separate
steps, so every caller in a concurrent burst of database calls saw the same
due-ness and dispatched its own task. Only one did real work, but the rest
still allocated a task on the auth hot path.
try_claim does both in one synchronous step, and the dispatched task now takes
the sample it was already granted instead of claiming again, which would have
failed and left no sample taken at all.
The restart guard zeroed the counter deltas but left the waiter baseline in
place. A fresh engine's waiter gauge carries no latch, so subtracting the
pre-restart baseline hid real waiters and could report zero during the very
saturation that caused the restart.
A proxy without prometheus was still taking the throttled query-engine read
every interval and discarding the result. The client resolver now checks for a
collector first, so that deployment does no extra work while the interval is
still consumed, which keeps the throttle from retrying on every database call.
Also condenses the rationale comments this change added down to what the code
cannot say for itself, per the repo's comment policy.
Two defects surfaced in review, both reproduced on a live proxy first.
`PrometheusLogger.get_instance` searched only `litellm.callbacks`, so the
equally supported `litellm_settings.success_callback: ["prometheus"]` left every
pool metric registered and permanently at zero. It now resolves through
`logging_callback_manager`, which covers all five callback lists.
Decorated database helpers nest, `get_object_permission` is called from inside
`get_key_object` and both carry the decorator, so one P2024 was counted once per
enclosing `except`. The exception is now marked the first time it is counted.
Operators could see latency and database CPU symptoms during an incident but
could not tell whether the proxy had run out of connections, because nothing
exposed the pool. Ten bounded-cardinality metrics now bridge the Prisma query
engine's own pool counters into Prometheus, keeping the time a query spent
waiting for a slot separate from the time it spent executing.
The configured maximum is derived from the engine as busy + idle rather than
parsed out of DATABASE_URL, so no credential is read on this path.
Sampling rides on database work instead of a scheduled job, and the exhaustion
counter is incremented at the error site, so both keep reporting through the
window a delayed exporter would erase.
Refs LIT-5435
detect-backend-changes diffed the event payload's base.sha against the
checked-out ref. Those are two different points in time: actions/checkout
resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base
branch advances, so the diff picked up whatever landed on staging between the
event firing and the job starting. On a recent UI-only pull request three
backend commits from staging were attributed to the branch, and every backend
shard ran in full
Ask the API which files the pull request touches instead. That is the same set
the Files changed tab shows, and it is immune to either endpoint moving. The
shell body moves into .github/scripts/detect_backend_changes.sh so it can be
exercised directly, and the fail-open paths now also cover an API failure, a
file list past the API's 3000-entry listing ceiling, and a classifier that
prints something unexpected
Every dynamic tracer-provider build called Resource.create, which scans the entry
points of every installed distribution, roughly 3ms and 200 file opens. The dynamic
providers reach it from the async logging path, which runs on the event loop serving
requests, so past the provider cache bound every request paid it and delayed the
requests in flight alongside it
The value derives only from the logger's config and process environment, so it is
built once per logger and reused. This logger's own init-time providers share it,
which also removes redundant startup builds. ArizeLogger overrides _init_tracing and
still builds its own, so it keeps one extra build
Refs LIT-5437
The reasoning override's floor was pinned to tier_boundaries.simple_medium,
so an operator could not restore the unconditional promotion nor raise the bar
independently of the SIMPLE/MEDIUM cut. Setting reasoning_override_min_score
was accepted and echoed back by /model/info, because the config model allows
extra keys, while routing ignored it.
Resolve the floor through one accessor that falls back to simple_medium when
the field is unset, so moving that boundary still moves the floor with it, and
an explicit 0 is a real floor rather than an absent one. Record the resolved
value on the routing decision so a logged row states the floor that applied,
which is also what lets the Admin UI stop hardcoding the copy PR #37500 added.
* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
`lite up` already patches ~/.claude/settings.json, but only for as long as it
runs in the foreground, and it restores the original file on exit. Users
proxying Claude Code through LiteLLM therefore have to re-wire it by hand after
every login.
--config-claude makes that write persistent. It reuses the settings shape
`lite up` writes (env.ANTHROPIC_BASE_URL plus an apiKeyHelper invocation),
preserves every unrelated key, creates the file when missing, and writes it
atomically with owner-only permissions. Plain `lite login` is unchanged.
Reaching the credential through apiKeyHelper rather than copying it into the
file means a later login refreshes it with no further action, and keeps the
short-lived CLI token out of settings.json entirely.
The shared parts of the settings-file handling move from up.py into a new
claude_settings.py, since up.py imports auth.py and so auth.py cannot import
up.py back. That module now also owns the registry of commands that can be
temporarily managing the file, so the persistent write refuses while either
`lite up` or `lite autoroute up` holds a backup it would later restore over
this write.
Because this write has no backup and no `lite down`, it is stricter than
`lite up` about the user's file: it writes through a symlinked settings.json
rather than replacing the link with a regular file, and it refuses rather than
silently discarding a non-object `env` value.
Also fixes the apiKeyHelper command itself: --base-url belongs to the
top-level `lite` group, so `lite auth print-token --base-url X` is rejected by
click with "No such option". Every settings file `lite up` has written carries
that malformed command, which makes the helper return nothing and every Claude
Code request lose its token. The existing tests only string-matched the
generated command, so the new tests parse it through the real CLI instead.
An unknown reasoning split now falls back to reasoning_tokens=0 in the
chat-to-responses usage translation, since the OpenAI SDK requires
output_tokens_details with an int reasoning_tokens, and the streaming
chunk builder caps the tokenized reasoning estimate at completion_tokens
and fills text_tokens with the remainder
Greptile flagged that lazy per-slug pool initialization could double-build
under concurrent replay calls, splitting consumption across a discarded
pool. Pools are now built once at ReplaySource construction and per-key
consumption is a single atomic deque pop, with a barrier-synchronized
regression test that fails 10/10 under the lazy-init mutant
A trusted hostname over plain http would expose the bearer token or a
replayable SigV4 signature to network observers. Credentials now only ride
https, with localhost exempt so local MCP stubs keep working.
Web-search connector 1.1.0 and later return the machine-readable results in result.structuredContent and may leave the text block as prose, which the parser dropped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
An OpenAPI-backed MCP tool whose upstream answered 401 came back as a
successful tool result carrying the upstream's rejection as its content, so a
caller saw {"error":"invalid_token"} presented as data and the gateway recorded
the request in its own spend log as call_mcp_tool | success.
Three layers each erased the outcome. The request function returned
response.text whatever the status, _handle_local_mcp_tool caught every exception
and returned it as ordinary TextContent, and both dispatch sites then stamped
isError=False unconditionally. Fixing only the first, which is the obvious fix,
changes nothing, because the two above it still map failure onto the
success-shaped value.
The status is now classified where the response is held: a 401 becomes
MCPUpstreamAuthError so the caller is told to re-authenticate, and every other
non-2xx becomes MCPOpenApiUpstreamError, which carries the status and drops the
upstream body rather than serving it as tool content. _handle_local_mcp_tool no
longer swallows, and the call_tool arm keeps the auth error's type. Nothing new
renders these: call_mcp_tool and call_tool_rest_api already turn them into an
isError result naming the status and into a real 401 with WWW-Authenticate, and
the OpenAPI path simply never reached them.
The result is now byte-identical to the regular MCP path for the same failure.
Two or more reasoning keyword matches promoted a request straight to the
REASONING tier no matter what the weighted score said, so "hi, step by step,
pros and cons" scored 0.100 and still bought the most expensive tier.
Require the score to clear the simple_medium boundary before the override
applies. Promotion from MEDIUM or COMPLEX is unchanged; only prompts the
scorer already placed in the cheapest band stay there.
