The comment restated what the gate does and carried incident detail that would drift,
including a claim about downstream callbacks that the evidence does not support. The
rationale belongs in the regression test, which fails if the copy is ever reintroduced
ahead of the gate, rather than in prose that can rot silently
Also corrects that test's docstring for the same overclaim: the raise aborts the handler
body at the redaction call, and what that costs a given deployment was not established
perform_redaction deepcopies the result before inspecting it, but every shape it does not
recognize falls through to the placeholder return at the end of that block, so the copy is
built and then discarded. Binary and HTTP response bodies land in exactly that case: batch
output, file content and audio responses hold an unpicklable `_thread.lock`, so
copy.deepcopy raises TypeError
The raise lands inside the try in Logging.success_handler that also wraps the callback
loop, so the handler body aborts at the redaction call and everything after it is skipped.
It surfaces only as "[Non-Blocking] Exception occurred while success logging cannot pickle
'_thread.lock' object", which is why it can run unnoticed. The async handler body reaches
perform_redaction the same way. Only deployments with message redaction enabled are
affected, since perform_redaction runs only when turn_off_message_logging resolves true
Deciding redactability before copying fixes the crash as a consequence rather than catching
it, and keeps the deepcopy off large batch bodies it was never going to help. Behaviour for
every recognized shape is unchanged: the copy still shields the caller's object from
in-place redaction
Observed on a live gateway with turn_off_message_logging enabled, where every managed-batch
output download logged that error; after this change the error no longer appears
/v1/messages and other litellm_metadata endpoints store proxy metadata,
including x-litellm-tags header tags, under litellm_metadata instead of
metadata. The pre-routing hook read request tags with a hardcoded
metadata bucket, so it never saw the tags that selected the marker and
cleared the consumed-tags stamp, and tag filtering then 401'd the routed
tier. Resolve the bucket from the request kwargs instead, matching how
the stamp write and the tag-filter read already resolve it.
The outer wrap_sse_stream_with_keepalive_pings layer duplicated the
keepalive engine that PR #34423 already runs inside async_data_generator
for chat completions and responses streams, and it kept pinging
deployments whose operator set keepalive_seconds: 0 as a hard disable.
sse_keepalive_ping_interval_seconds is now the global fallback inside
_resolve_keepalive_seconds, so deployment and request values keep
precedence, an explicit 0 still disables, the [1, 300]s clamp applies,
and router-less proxies arm the wrap when the global default is set.
The Rust messages bridge logs a parsed Anthropic response without an
httpx_response, so the fallback transform dropped the request speed and
billed fast-mode calls at the standard rate. Thread optional_params
speed into transform_parsed_response and add a regression test for the
parsed-response branch.
A stream can terminate with a response.failed event that still reports
consumed tokens; those were rebuilt as None and logged at zero spend.
Parse response.failed alongside completed and incomplete, matching the
buffered path, which prices any terminal response that reports usage.
The Anthropic batch create never persisted the creating key's hashed token or its
request tags on the managed object, so when CheckBatchCost billed the batch hours
later there was nothing to attribute it to. Key spend, key budgets and tag spend
never moved for batch usage.
Persist both from the create, the way the Vertex passthrough already does, and
register the batch only from the collection route. An id-scoped route cannot
rebuild the unified object id, because it embeds the model and the model comes
from the create's request body, so it could only claim a row it did not create or
fail the model_object_id unique constraint.
The shared metadata helpers, the route predicate and the registration-result
logging now live in batch_attribution instead of being copied per provider. The
Anthropic write previously logged success unconditionally, before the
fire-and-forget task had run.
Resolves LIT-5288
Streams that terminate with response.incomplete (e.g. max_output_tokens
reached) carry real usage in the terminal event but were rebuilt as None
and logged at zero spend, letting callers bypass budget enforcement.
Parse response.incomplete alongside response.completed when
reconstructing the streamed response.
Addresses review feedback: the ResponseCompletedEvent SSE parsing now lives
in OpenAIResponsesAPIConfig next to the other Responses stream event handling,
and the proxy logging handler calls it. Adds coverage for streams that end
without a response.completed event.
Register remote MCP servers without awaiting OAuth metadata, warm discovery in the background, and share bounded request-time retries with per-server cooldowns.
Preserve the existing discovered-tool boundary for explicit server calls.
Co-Authored-By: Codex
* feat(router): make routing groups callable as virtual models and list them in /v1/models
* fix(router): traffic-scoped cooldown exemption, live model_names on delete, group-info cache invalidation
* fix(router): share one recognized-model predicate across proxy gates, resolve aliases in group cooldown, read metadata via the dual-bucket owner
* fix(router): close the gate and cache families for callable groups, strip member access_groups from group rows, prove cooldown wiring end to end
* refactor(router): cache materialized group rows under the model-group cache owner and drop the redundant wiring test
* fix(router): warn-and-shadow on group name collisions, name-level test coverage for group helpers, faithful router doubles in a2a and cursor tests
* test(router): pin group cooldown metadata across the retry path
Gemini image usage carries prompt_tokens and friends as extra fields on
ResponseAPIUsage, which collided with the bridge's explicit kwargs and
raised TypeError. Exclude keys the bridge already sets explicitly.
Every TypedDict field must carry a ReadOnly[...] qualifier (PEP 705),
nesting freely with Required/NotRequired/Annotated. Detection covers the
class form (including same-module transitive subclasses) and the
functional form. The 4519 existing violations across litellm/ are
grandfathered via type-discipline-budget.json; suppress deliberate
writable keys with # writable-ok: <reason>.
Drop the transform overrides that swapped response.usage to the chat
shape, which broke the /v1/responses client contract. Provider extras
like server_side_tool_usage_details already survive validation via
ResponseAPIUsage extra fields, so the shared usage bridge now carries
them onto the bridged chat Usage generically. The web_search_call
output gate also reads dict output items, since items that fail SDK
validation stay plain dicts, and the chat path gains billing tests.
* fix(alerting): dedupe scheduled Slack spend reports across pods
Every pod ran its own weekly/monthly spend report jobs, prometheus
fallback stats cron, and daily report loop, so deployments with
multiple replicas or uvicorn workers received one copy per pod.
Gate each scheduled send behind the shared PodLockManager redis lock.
The lock is never released: its TTL (the full reporting window for the
weekly interval job, whose per-pod anchors drift by boot time and
jitter) doubles as a sent-this-window marker. acquire_lock returning
None (no redis wired) proceeds, preserving single-pod behavior.
Also generalize the pod lock could-not-acquire log line, which claimed
to be about spend tracking for every consumer.
Fixes#14809
* fix(alerting): harden spend report locks after adversarial review
Weekly lock TTL gets an hour haircut: with ttl equal to the interval,
the winner re-fires just before its own key expires, reacquires without
a TTL refresh, and the key then lapses in time for a trailing pod to
re-send. Job/lock ids move to litellm/constants.py per convention, and
spend_report_frequency now rejects non-positive day counts, which
previously coerced to an every-second schedule and would now compute a
negative lock TTL that silently never sends.
Adds the missing test coverage the review flagged: startup_event's
pod_lock_manager wiring (identity-asserted), the prometheus closure's
positive path, and the ungated immediate prometheus send pinned to
exactly one await.
* test(alerting): consolidate spend_report_frequency validator coverage
Drops a duplicate non-positive-days test and parametrizes the survivor
over the suffix half of the validator too
* fix(alerting): route the startup prometheus fallback send through the pod lock
Greptile caught that the boot-time send still ran once per pod when
PROMETHEUS_URL is set, the same duplication class this PR removes
* fix(alerting): make report lock acquisition non-reentrant
Greptile caught that a pod booting within an hour of the fallback stats
cron sent twice: the startup send takes the lock, then the cron fire
hits acquire_lock's reacquire branch, which returns True for the
holder. Window-marker gates now pass allow_reentrant=False so a live
lock blocks everyone including its holder; leader-election consumers
keep the reentrant default
* test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer
_initialize_slack_alerting_jobs now reads it for the pod lock manager,
and spec=ProxyLogging blocks instance-only attributes
