The two writes that hand the skip list to the next attempt now carry a
`# rebind-ok` reason, which is the sanctioned escape hatch for an unavoidable
parameter mutation and matches how `log_retry` already writes into the same
kwargs dict a few lines above
`get_excluded_filtered_deployments`'s docstring said returning the unfiltered
list would re-include the deployment that just failed. The retry skip does
exactly that on purpose, so the docstring now says each caller decides what an
empty result means
The reliability registry cell the new e2e test claims is marked
`fail_before_fix: proven`: the same config returns 400 at the merge base and
200 off a sibling deployment at the tip
BadRequestErrorRetries and ContentPolicyViolationErrorRetries did let a retry
happen, but the retry re-picked the deployment that had just refused, since a
400 never puts a deployment in cooldown. On a weighted model group the caller
got the same 400 back after every configured retry, and the existing 401/403
"retry on another deployment" rule broke the same way
A retry after a non-transient status now carries the deployments that already
answered this request in the per-request exclusion list weighted failover
already honors, so the next attempt lands on a sibling. Single-deployment
groups still retry in place, and 408/429/5xx retries are untouched
Adds live e2e coverage for reliability.retry.context_window.succeeds_within_retries
and renames the two litellm.utils deployment filters that are now called from
outside the module
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
The first canary run failed pass 1 because the stage-mirror config had no
openai-text-embedding-3-small while test_llm_api_routes_group_grants_every_llm_endpoint
calls /embeddings with it; the public log named the test, which is the
behavior the previous commit added
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
Anthropic now returns the 64-token 'count to 20' reply in one to three content_block_delta events, measured directly against api.anthropic.com and through proxies at 7672399 and 49a1145 alike, so the incrementality assertion (at least two deltas) failed in litellm-e2e builds 125, 130 and the 278 rerun with no proxy change behind it. A 'count to 100' reply at max_tokens 400 arrived in five to fifty deltas across every measured run
* test(e2e): cover presidio post_call, tool_permission, and weave logging cells
Five registry cells in Logging & Guardrails had no covering test. Each one now
has a live scenario read back from the real destination:
- guardrail.presidio.post_call.masks: an output-scoped Presidio guardrail
anonymizes the PII the model repeats back. The prompt also asks for the
address's local part, which Presidio does not mask, so one response proves the
model saw the raw address (no pre-call masking) while the address itself comes
back as <EMAIL_ADDRESS>
- guardrail.tool_permission.pre_call.blocks / .allows: an allow-list of one tool.
A request declaring an unlisted tool is rejected 400 naming it; a request
declaring the permitted tool is served and carries
x-litellm-applied-guardrails, so the allow half cannot pass by the guardrail
never running
- logging.niche_integrations.success.logs_spend / .failure.logs_spend: a
key-scoped weave_otel callback delivers to the real Weave project, read back
through Weave's query API. Success asserts exactly one call whose
llm.response.cost equals the x-litellm-response-cost header; failure asserts
one ERROR-status call naming the provider exception and carrying no cost
Logging & Guardrails coverage goes 24/59 to 29/59. No registry rows are added.
* test(e2e): make the tool-permission allow case deterministic and scope the Weave read-back
Review follow-ups on the coverage PR.
- the allow scenario forced the outcome to depend on whether the model felt like
calling an optional tool, and checked for the tool name as a substring of the
whole body, which a prose mention would satisfy. It now sends
tool_choice="required" and asserts the parsed response carries exactly one tool
call, for the permitted tool
- the Weave read-back queried the newest 200 calls of a shared project and
filtered client-side, so busy traffic could push the target out of the window
and read as a delivery failure. The query now scopes server-side to the
litellm_request op and to calls started after the request, and pages through
the window with offset
- the reader builds its results as tuples instead of accumulating into lists
Also unblocks the lint gate: `basedpyright tests/e2e` runs only on PRs that touch
tests/e2e, and it has been failing on staging for three FakeItem arguments in
test_junit_properties.py. The stand-in now goes through one typed adapter that
says why, so the gate is green without touching junit_properties.py itself.
* test(e2e): scope the presidio post_call guardrail to email and phone
Running the suite three times in a row caught a real flake: Presidio's broader
recognizers sometimes claim the email's local part as an NRP entity, so the
answer came back as `<NRP>\n<EMAIL_ADDRESS>\n<PHONE_NUMBER>` and the assertion
that the raw local part survives failed. That token is what tells output masking
apart from input masking, so it has to survive.
The post_call guardrail now registers pii_entities_config for EMAIL_ADDRESS and
PHONE_NUMBER only, which is also the narrower thing the scenario means. Verified
against the exact marker that failed, plus two others.
* test(e2e): mark weave logging cells stage red
* test(e2e): use per-test stage red skips for the weave logging cells
test_no_linear_scans_in_router: #39674 renamed heuristic_v2_router_limit_violation
to auto_router_capability_violation, so the allowlist entry stopped matching and the
same admin-only scan tripped the static check. Rename the entry to follow it.
tableScrolling.spec.ts: 9ba6cab889 (LIT-4738) gave the Tags and Model Hub tables
client-side pagination at 25 rows, so the 40 seeded rows no longer render on one
page. Select 50 rows per page before counting, as the Logs case already does.
Four registry cells that had no e2e test. The cache_control suite gains a direct
Anthropic case (the same cache_control prefix the Bedrock and Vertex rows send)
and an OpenAI case, where caching is automatic so the prefix goes out as a plain
system string with a prompt_cache_key; both assert the second identical call
reports cache-read tokens. The shared second-call helper now takes the send
callable so the OpenAI shape fits without a second copy of the retry loop.
The embeddings suite gains a cohere/embed-v4.0 deployment that must return a
non-zero vector, and the passthrough suite gains an OpenAI-format chat through
the raw /openai/v1/chat/completions prefix that must relay a real completion and
log a costed pass_through_endpoint row whose token counts match the usage the
caller was served.
Three deterministic proxy-only cells from the coverage registry that had no e2e
test. A key over its max_budget is reset to 0 through /key/{key}/reset_spend and
must both read back 0 on /key/info and serve traffic again. /key/regenerate with
grace_period keeps the old key valid until the period elapses and rejects it 401
afterwards. A key whose allowed_routes is the llm_api_routes group must reach
/chat/completions and /embeddings while /model/new stays 403.
KeyRegenerateBody gains grace_period and the management client gains
reset_key_spend so the tests stay on the shared typed transport.
Adds TestAnthropicChatCompletions to the chat completions regression suite,
registering a claude-haiku-4-5 deployment via /model/new and asserting the
streamed call delivers real content deltas and a tool-forced call returns a
well-formed get_weather tool_call on both the non-streamed and streamed paths.
Covers three P0 registry cells that had no e2e test.
The completed-batch early return skipped both the cancel and the list
assertion while the lifecycle's covers markers still credited both cells.
List does not depend on the batch being cancellable, so it now runs either
way; cancel on a completed batch stays a documented vacuous pass
Bedrock batch cancel (StopModelInvocationJob) and the managed list view
both work through the proxy since LIT-4774, but the batches e2e still
gated them off and the coverage registry claimed no cell for either.
Flip can_cancel/can_list for the Bedrock provider, assert cancel the
same way the OpenAI leg does, add the two registry cells the gates
select, and update COVERAGE.md
Both of these are e2e assumptions that PRs #31731 and #39532 invalidated, not
product regressions. They have been red in litellm-e2e builds 119-123.
Wildcard readiness probe (6 errors in test_model_access_group_e2e.py)
#31731 made _get_wildcard_models drop a wildcard route from /v1/models
unconditionally; before it, a wildcard with a matching router deployment stayed
in the list and only the no-router / no-deployment fallbacks removed it. The
shared readiness helper polls /v1/models for an exact id match, so registering
openai/gpt-5.4* now times out at model_servable_timeout every run and every
test in the class errors in setup.
return_wildcard_routes=True still re-adds the route, so the poll asks for it.
The flag is a no-op for a concrete model name -- it only ever adds wildcard
entries -- so it is set unconditionally rather than sniffing the name.
Semantic auto-router spend assertion
#39532 bills the routing embedding to the caller's key on purpose, so the key's
spend logs now legitimately carry an openai/text-embedding-3-small row and
_assert_served_only_by rejects it.
Widening the allowlist would have weakened the assertion this test exists for --
that the request reached the target deployment. Instead the embedding row is
split off and asserted separately, which turns the break into coverage for
#39532. The poll gains a predicate so it waits for the embedding row rather than
racing whichever row is written first.
DataTable loses the comment that narrated its sticky header classes. The
table scrolling e2e spec now types every management API response it
reads, seeds rows through an immutable reduce instead of pushing into
arrays, and deletes what it seeded in each test's finally block instead
of draining a shared mutable list in afterEach.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
Virtual Keys, Teams, Request Logs and Tags now hand DataTable a bounded
flex chain and use fillHeight, so the app shell main stays the only page
scroller, the rows scroll under a pinned header and the pagination footer
sits at the bottom of the page. DataTable keeps the sticky header inside
its own scroller in maxBodyHeight mode too, which is what let the header
scroll away with the rows on Keys, Teams and Models. Model Hub, Vector
Stores and the team detail keys tab drop their 75vh boxes and flow with
the page scroller.
Adds an e2e spec that fails on the merge base for every one of those
pages and passes at this tip.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
PR #39604 renamed the Internal Users search box placeholder to "Search by email or ID…" but left searchUsers.spec.ts looking for the old "Search by email…" copy, so e2e_ui_testing has been red on litellm_internal_staging since it merged. Point the locator at the shipped placeholder
`POST /v1/agents/{id}/make_public` appended the agent id to
`litellm.public_agent_groups` and only then called `get_config()`, which
re-applies the DB's `litellm_settings` over the module globals and threw the
append away. The config it saved was therefore a no-op: the endpoint answered
200 with an empty `public_agent_groups`, the agent never reached
`GET /public/agent_hub`, and re-publishing never hit the "already public" 400.
Read the config first, derive the new list from the refreshed globals, save it,
then update the global
Also fixes the e2e model hub spec, which is flaky for a second reason: the
"Make Models Public" modal preselects the groups that are already public, so a
blind click on "Select All" cleared them and left "Next" disabled for the full
15s action timeout. Check the box instead of toggling it, and wait for "Next"
to be enabled before clicking
* fix(ui): paginate request logs by session groups server-side
The logs table server-paginated raw spend logs and then collapsed
multi-call sessions client-side, so a page could render 3 rows while
the footer claimed 25 and sessions straddled pages. Adds an opt-in
group_by_session param to /spend/logs/ui that pages and counts one
representative row per session (DISTINCT ON, newest non-MCP call),
keeps the bounded count contract, enriches whole-session llm/agent
composition counts, and deletes the client-side collapse pipeline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* feat(ui): add a 10 rows-per-page option and default request logs to it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): key session aggregates per api key in the logs enrichment
Grouped pagination splits a reused session id into one row per api key,
but the enrichment still aggregated by session_id alone, so both rows
showed combined spend and counts. The aggregate query now groups by
(session_id, api_key), the count folds into it (the separate group_by
query is deleted), and each row reads its own key's totals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): treat an empty api_key as a real session group value
The spend-log schema defaults api_key to an empty string; truthiness
guards in the enrichment treated it as missing, so keyless multi-call
sessions lost their count and spend. Only None means missing now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The default password policy from #39381 (12+ chars with upper, lower, number, and special) makes globalSetup's POST /user/update with password "test" fail with a 400, so every role login in the UI e2e suite has been failing since that merge. Seed the roles, the noteam@test.local login, and the second-admin spec with policy-compliant passwords and update the scrypt hash in seed.sql to match.
Adds a required-check candidate that selects the tests/e2e test files a PR added or
modified, boots a stage-mirror stack on the runner (migrations, backend, two gateway
processes behind nginx, Postgres, Jaeger, TLS cluster Valkey), and runs those files
three times with retries off. The run job sits behind the e2e-changed GitHub
environment, so a reviewer approves each run before the OIDC token that reads the
provider keys from AWS Secrets Manager exists. Supersedes #34981
