APScheduler anchors an interval job at now + interval, so every scheduled
background job registered in one proxy startup shares a single firing instant
for the life of the process, and every replica a rollout brought up together
shares that instant too. Each tick the spend flushes, budget reset sweep,
config-in-DB reload, credential reload and cost pollers all hit Postgres at the
same moment, on every pod, competing with request-path auth and budget queries.
Shift each eligible job by a deterministic offset derived from
sha256(job_id, identity), where identity covers the pod and the worker process.
The offset lives in the trigger rather than in a one-off next_run_time, because
a cron trigger recomputes each fire from the wall clock and would otherwise snap
straight back onto the shared instant. An interval job is never offset by more
than one of its own periods.
Only schedules LiteLLM chose are shifted: interval jobs always, cron jobs only
when the id is one of the product's own defaults, so an operator-supplied
crontab keeps the instant it asks for. general_settings.scheduled_job_stagger
turns it off, widens the window, replaces the identity, or pins a job. The
applied offsets are logged once at startup and each fire logs its scheduled
instant against its actual start.
Resolves LIT-5433
A capped pre-wait still burns the first daily pass when the router takes
longer than the cap to appear (a >10 minute boot), and reads the router
in two places. Folding the poll into the loop makes the first alert
unconditional on boot duration and keeps a single read per pass.
Cue grouping previously assumed space-separated Latin-style text. Chinese
and Japanese audio fused entire utterances into one unbreakable word (and
therefore one giant cue) because CJK scripts carry no spaces, and Arabic,
Urdu, Hindi and Armenian sentence terminators never triggered a cue break.
Words now also split at CJK character boundaries with basic kinsoku
handling so punctuation stays attached, the sentence-end set covers
script-specific terminators, and the cue length budget counts East Asian
wide characters as double width so CJK cues match the same two-line
subtitle footprint as Latin text.
The previous cue grouping broke every 15 subword tokens or 5s, which
produced uniform ~3s cues that split mid-word and bridged silence gaps,
so subtitles did not track the actual speech. Cues are now built from
whole words and break on sentence-final punctuation, speaker changes,
silence gaps >= 700ms, a 84-char budget, or a 7s duration cap, with
timestamps taken directly from token timings. Untimestamped translation
tokens are excluded from cues so translated text is never mixed into
original-language subtitles.
The comment restated what the gate does and carried incident detail that would drift,
including a claim about downstream callbacks that the evidence does not support. The
rationale belongs in the regression test, which fails if the copy is ever reintroduced
ahead of the gate, rather than in prose that can rot silently
Also corrects that test's docstring for the same overclaim: the raise aborts the handler
body at the redaction call, and what that costs a given deployment was not established
perform_redaction deepcopies the result before inspecting it, but every shape it does not
recognize falls through to the placeholder return at the end of that block, so the copy is
built and then discarded. Binary and HTTP response bodies land in exactly that case: batch
output, file content and audio responses hold an unpicklable `_thread.lock`, so
copy.deepcopy raises TypeError
The raise lands inside the try in Logging.success_handler that also wraps the callback
loop, so the handler body aborts at the redaction call and everything after it is skipped.
It surfaces only as "[Non-Blocking] Exception occurred while success logging cannot pickle
'_thread.lock' object", which is why it can run unnoticed. The async handler body reaches
perform_redaction the same way. Only deployments with message redaction enabled are
affected, since perform_redaction runs only when turn_off_message_logging resolves true
Deciding redactability before copying fixes the crash as a consequence rather than catching
it, and keeps the deepcopy off large batch bodies it was never going to help. Behaviour for
every recognized shape is unchanged: the copy still shields the caller's object from
in-place redaction
Observed on a live gateway with turn_off_message_logging enabled, where every managed-batch
output download logged that error; after this change the error no longer appears
/v1/messages and other litellm_metadata endpoints store proxy metadata,
including x-litellm-tags header tags, under litellm_metadata instead of
metadata. The pre-routing hook read request tags with a hardcoded
metadata bucket, so it never saw the tags that selected the marker and
cleared the consumed-tags stamp, and tag filtering then 401'd the routed
tier. Resolve the bucket from the request kwargs instead, matching how
the stamp write and the tag-filter read already resolve it.
The outer wrap_sse_stream_with_keepalive_pings layer duplicated the
keepalive engine that PR #34423 already runs inside async_data_generator
for chat completions and responses streams, and it kept pinging
deployments whose operator set keepalive_seconds: 0 as a hard disable.
sse_keepalive_ping_interval_seconds is now the global fallback inside
_resolve_keepalive_seconds, so deployment and request values keep
precedence, an explicit 0 still disables, the [1, 300]s clamp applies,
and router-less proxies arm the wrap when the global default is set.
The Rust messages bridge logs a parsed Anthropic response without an
httpx_response, so the fallback transform dropped the request speed and
billed fast-mode calls at the standard rate. Thread optional_params
speed into transform_parsed_response and add a regression test for the
parsed-response branch.
