* fix(ui): align spend and budget columns
* fix(ui): preserve sub-threshold money formatting
Co-Authored-By: Codex
* fix(ui): use two-decimal summary amounts
Co-Authored-By: Codex
* test(ui): tolerate organization lookup in access checks
Scope denied-role assertions to the protected page endpoints so the
organization membership lookup does not make the tests fail.
Generated with AI
Co-Authored-By: Claude Code
Co-Authored-By: Codex
The proxy-server-core matrix entry named test_proxy_server_caching.py and
test_proxy_server_langfuse.py by path. This PR deletes both, so pytest exited 5
with "no tests collected" and the whole shard failed without running the four
files that do exist.
assert-shard-coverage did not catch it because it only checks one direction:
every file under tests/proxy_unit_tests/ must appear in some shard. It never
checks that every path a shard names still exists, so a stale entry passes.
After this change no shard names a missing path and no file is left without a
shard. The shard collects 85 tests.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
APScheduler anchors an interval job at now + interval, so every scheduled
background job registered in one proxy startup shares a single firing instant
for the life of the process, and every replica a rollout brought up together
shares that instant too. Each tick the spend flushes, budget reset sweep,
config-in-DB reload, credential reload and cost pollers all hit Postgres at the
same moment, on every pod, competing with request-path auth and budget queries.
Shift each eligible job by a deterministic offset derived from
sha256(job_id, identity), where identity covers the pod and the worker process.
The offset lives in the trigger rather than in a one-off next_run_time, because
a cron trigger recomputes each fire from the wall clock and would otherwise snap
straight back onto the shared instant. An interval job is never offset by more
than one of its own periods.
Only schedules LiteLLM chose are shifted: interval jobs always, cron jobs only
when the id is one of the product's own defaults, so an operator-supplied
crontab keeps the instant it asks for. general_settings.scheduled_job_stagger
turns it off, widens the window, replaces the identity, or pins a job. The
applied offsets are logged once at startup and each fire logs its scheduled
instant against its actual start.
Resolves LIT-5433
/v1/messages and other litellm_metadata endpoints store proxy metadata,
including x-litellm-tags header tags, under litellm_metadata instead of
metadata. The pre-routing hook read request tags with a hardcoded
metadata bucket, so it never saw the tags that selected the marker and
cleared the consumed-tags stamp, and tag filtering then 401'd the routed
tier. Resolve the bucket from the request kwargs instead, matching how
the stamp write and the tag-filter read already resolve it.
The outer wrap_sse_stream_with_keepalive_pings layer duplicated the
keepalive engine that PR #34423 already runs inside async_data_generator
for chat completions and responses streams, and it kept pinging
deployments whose operator set keepalive_seconds: 0 as a hard disable.
sse_keepalive_ping_interval_seconds is now the global fallback inside
_resolve_keepalive_seconds, so deployment and request values keep
precedence, an explicit 0 still disables, the [1, 300]s clamp applies,
and router-less proxies arm the wrap when the global default is set.
The Rust messages bridge logs a parsed Anthropic response without an
httpx_response, so the fallback transform dropped the request speed and
billed fast-mode calls at the standard rate. Thread optional_params
speed into transform_parsed_response and add a regression test for the
parsed-response branch.
A stream can terminate with a response.failed event that still reports
consumed tokens; those were rebuilt as None and logged at zero spend.
Parse response.failed alongside completed and incomplete, matching the
buffered path, which prices any terminal response that reports usage.
The Anthropic batch create never persisted the creating key's hashed token or its
request tags on the managed object, so when CheckBatchCost billed the batch hours
later there was nothing to attribute it to. Key spend, key budgets and tag spend
never moved for batch usage.
Persist both from the create, the way the Vertex passthrough already does, and
register the batch only from the collection route. An id-scoped route cannot
rebuild the unified object id, because it embeds the model and the model comes
from the create's request body, so it could only claim a row it did not create or
fail the model_object_id unique constraint.
The shared metadata helpers, the route predicate and the registration-result
logging now live in batch_attribution instead of being copied per provider. The
Anthropic write previously logged success unconditionally, before the
fire-and-forget task had run.
Resolves LIT-5288
LiteLLMCompletionStreamingIterator overrides __init__ without calling
super().__init__(), so completed_response was only set once the stream
reached RESPONSE_COMPLETED. On a mid-stream provider error the router's
_extract_partial_responses_usage read source_iterator.completed_response
during fallback recovery and raised AttributeError, masking the real
provider error (e.g. Anthropic 529) and bypassing configured retries and
fallbacks. Initialize the attribute to None so recovery degrades to no
partial usage instead of crashing.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Streams that terminate with response.incomplete (e.g. max_output_tokens
reached) carry real usage in the terminal event but were rebuilt as None
and logged at zero spend, letting callers bypass budget enforcement.
Parse response.incomplete alongside response.completed when
reconstructing the streamed response.
Addresses review feedback: the ResponseCompletedEvent SSE parsing now lives
in OpenAIResponsesAPIConfig next to the other Responses stream event handling,
and the proxy logging handler calls it. Adds coverage for streams that end
without a response.completed event.
AWS no longer serves `anthropic.claude-3-sonnet-20240229-v1:0`. The streaming
path returns a plain 404, "Model with the provided id
anthropic.claude-3-sonnet-20240229-v1:0 is not found", and the non-streaming
path answers 500 for the same reason. Our own cost map has carried a
2026-07-30 deprecation date for it since #36538
That accounts for 20 failures across local_testing_part1, local_testing_part2
and llm_translation_testing. litellm maps both statuses correctly, so the
tests are what went stale, not the client
Replacement is `us.anthropic.claude-sonnet-4-5-20250929-v1:0`: a like-for-like
Sonnet, and the newest Bedrock Sonnet this repo exercises against the real API
in tests/e2e. Newer ids exist in the cost map, but nothing in the repo calls
them live, so picking one would be an unverified guess about model access on
the CI account
Scope is limited to the tests that actually issue a request. The occurrences
that assert on the model string itself, or that feed mocked transformations,
keep the old id so their assertions stay meaningful
* feat(router): make routing groups callable as virtual models and list them in /v1/models
* fix(router): traffic-scoped cooldown exemption, live model_names on delete, group-info cache invalidation
* fix(router): share one recognized-model predicate across proxy gates, resolve aliases in group cooldown, read metadata via the dual-bucket owner
* fix(router): close the gate and cache families for callable groups, strip member access_groups from group rows, prove cooldown wiring end to end
* refactor(router): cache materialized group rows under the model-group cache owner and drop the redundant wiring test
* fix(router): warn-and-shadow on group name collisions, name-level test coverage for group helpers, faithful router doubles in a2a and cursor tests
* test(router): pin group cooldown metadata across the retry path
Gemini image usage carries prompt_tokens and friends as extra fields on
ResponseAPIUsage, which collided with the bridge's explicit kwargs and
raised TypeError. Exclude keys the bridge already sets explicitly.
`supports_native_structured_output` was set only on the bare `deepseek.v3.2`
and `zai.glm-5` entries, so the cross-region inference profiles and the
region-pinned ids resolved to None. The flag gates the native
`outputConfig.textFormat` branch in BedrockConverseConfig, so callers
addressing the same model as `us.deepseek.v3.2` or
`bedrock/us-west-2/deepseek.v3.2` silently fell back to synthetic tool
injection. `us.` is the form Bedrock steers callers toward, so the most
common way to reach these models was the one missing the capability.
Adds the flag to the 12 affected ids and keeps the packaged backup in sync.
test_get_model_info_bedrock_models already caught the region-pinned ids, but
it filters on `litellm_provider == "bedrock"` and the cross-region profiles
carry `bedrock_converse`, so reverting just `us.deepseek.v3.2` and
`eu.deepseek.v3.2` left it green. The new parity test covers the prefixed
profiles and fails on exactly that mutation.
* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
