The proxy-server-core matrix entry named test_proxy_server_caching.py and
test_proxy_server_langfuse.py by path. This PR deletes both, so pytest exited 5
with "no tests collected" and the whole shard failed without running the four
files that do exist.
assert-shard-coverage did not catch it because it only checks one direction:
every file under tests/proxy_unit_tests/ must appear in some shard. It never
checks that every path a shard names still exists, so a stale entry passes.
After this change no shard names a missing path and no file is left without a
shard. The shard collects 85 tests.
Python keeps only the last binding for a name, so when a file defines the same
test twice the earlier one is unreachable. pytest cannot collect a function that
no longer exists, so nothing reports it and the file still looks like it covers
the scenario.
These ten are cases where the two definitions have different bodies, meaning a
real test was replaced rather than duplicated. Each is renamed to say what it
actually covers, which makes it reachable again:
- test_gemini_frequency_penalty: the dead copy checks the parameter is listed in
get_supported_openai_params for vertex_ai; the survivor checks get_optional_params
maps a value for gemini. Different function and different provider.
- test_async_log_success_event_adds_to_queue and the failure variant: the dead
copies run without mocking asyncio.create_task, so they exercise the real task
path the survivors mock out.
- test_async_send_batch_triggers_tasks: the dead copy asserts send is not awaited
directly; the survivor asserts create_task was called.
- test_model_id_in_required_metrics: the dead copy checks the model_id label on
twelve further metrics the survivor dropped.
- test_anthropic_messages_pt_file_block_preserves_cache_control: the dead copy
passes model and llm_provider explicitly and uses real base64 PDF content.
- test_translate_streaming_openai_chunk_to_anthropic_with_thinking: the dead copy
covers thinking_delta; the survivor covers signature_delta.
- test_client_initialization and test_client_without_api_key: the dead copies
assert the resource clients are wired with the right base URL and key; the
survivors only construct the object.
- test_client_initialization_strips_trailing_slash: the dead copy constructs
ModelsManagementClient directly rather than going through Client.
Verification: collecting the seven touched files gives 401 node IDs before and
411 after, the ten new names and nothing else, with nothing lost. All ten pass.
Running the touched files in full gives 299 passed, and test_optional_params.py
goes from 111 passed to 112.
Two further shadowed definitions were left alone rather than renamed: the dead
copies of test_prompt_caching and test_cost_calculator_with_base_model_with_router
have no assertions at all, one being a bare pass and the other a lone import, so
restoring them would add tests that cannot fail.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
APScheduler anchors an interval job at now + interval, so every scheduled
background job registered in one proxy startup shares a single firing instant
for the life of the process, and every replica a rollout brought up together
shares that instant too. Each tick the spend flushes, budget reset sweep,
config-in-DB reload, credential reload and cost pollers all hit Postgres at the
same moment, on every pod, competing with request-path auth and budget queries.
Shift each eligible job by a deterministic offset derived from
sha256(job_id, identity), where identity covers the pod and the worker process.
The offset lives in the trigger rather than in a one-off next_run_time, because
a cron trigger recomputes each fire from the wall clock and would otherwise snap
straight back onto the shared instant. An interval job is never offset by more
than one of its own periods.
Only schedules LiteLLM chose are shifted: interval jobs always, cron jobs only
when the id is one of the product's own defaults, so an operator-supplied
crontab keeps the instant it asks for. general_settings.scheduled_job_stagger
turns it off, widens the window, replaces the identity, or pins a job. The
applied offsets are logged once at startup and each fire logs its scheduled
instant against its actual start.
Resolves LIT-5433
A capped pre-wait still burns the first daily pass when the router takes
longer than the cap to appear (a >10 minute boot), and reads the router
in two places. Folding the poll into the loop makes the first alert
unconditional on boot duration and keeps a single read per pass.
Cue grouping previously assumed space-separated Latin-style text. Chinese
and Japanese audio fused entire utterances into one unbreakable word (and
therefore one giant cue) because CJK scripts carry no spaces, and Arabic,
Urdu, Hindi and Armenian sentence terminators never triggered a cue break.
Words now also split at CJK character boundaries with basic kinsoku
handling so punctuation stays attached, the sentence-end set covers
script-specific terminators, and the cue length budget counts East Asian
wide characters as double width so CJK cues match the same two-line
subtitle footprint as Latin text.
The previous cue grouping broke every 15 subword tokens or 5s, which
produced uniform ~3s cues that split mid-word and bridged silence gaps,
so subtitles did not track the actual speech. Cues are now built from
whole words and break on sentence-final punctuation, speaker changes,
silence gaps >= 700ms, a 84-char budget, or a 7s duration cap, with
timestamps taken directly from token timings. Untimestamped translation
tokens are excluded from cues so translated text is never mixed into
original-language subtitles.
The comment restated what the gate does and carried incident detail that would drift,
including a claim about downstream callbacks that the evidence does not support. The
rationale belongs in the regression test, which fails if the copy is ever reintroduced
ahead of the gate, rather than in prose that can rot silently
Also corrects that test's docstring for the same overclaim: the raise aborts the handler
body at the redaction call, and what that costs a given deployment was not established
perform_redaction deepcopies the result before inspecting it, but every shape it does not
recognize falls through to the placeholder return at the end of that block, so the copy is
built and then discarded. Binary and HTTP response bodies land in exactly that case: batch
output, file content and audio responses hold an unpicklable `_thread.lock`, so
copy.deepcopy raises TypeError
The raise lands inside the try in Logging.success_handler that also wraps the callback
loop, so the handler body aborts at the redaction call and everything after it is skipped.
It surfaces only as "[Non-Blocking] Exception occurred while success logging cannot pickle
'_thread.lock' object", which is why it can run unnoticed. The async handler body reaches
perform_redaction the same way. Only deployments with message redaction enabled are
affected, since perform_redaction runs only when turn_off_message_logging resolves true
Deciding redactability before copying fixes the crash as a consequence rather than catching
it, and keeps the deepcopy off large batch bodies it was never going to help. Behaviour for
every recognized shape is unchanged: the copy still shields the caller's object from
in-place redaction
Observed on a live gateway with turn_off_message_logging enabled, where every managed-batch
output download logged that error; after this change the error no longer appears
/v1/messages and other litellm_metadata endpoints store proxy metadata,
including x-litellm-tags header tags, under litellm_metadata instead of
metadata. The pre-routing hook read request tags with a hardcoded
metadata bucket, so it never saw the tags that selected the marker and
cleared the consumed-tags stamp, and tag filtering then 401'd the routed
tier. Resolve the bucket from the request kwargs instead, matching how
the stamp write and the tag-filter read already resolve it.
The outer wrap_sse_stream_with_keepalive_pings layer duplicated the
keepalive engine that PR #34423 already runs inside async_data_generator
for chat completions and responses streams, and it kept pinging
deployments whose operator set keepalive_seconds: 0 as a hard disable.
sse_keepalive_ping_interval_seconds is now the global fallback inside
_resolve_keepalive_seconds, so deployment and request values keep
precedence, an explicit 0 still disables, the [1, 300]s clamp applies,
and router-less proxies arm the wrap when the global default is set.
