litellm/tests/e2e/ui/tests
yuneng-jiang 14f4c34c61
fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294)
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts

The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.

* fix(ci): keep the policy-load backoff typed as float

* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests

* fix(secrets): load CyberArk policy one request at a time per manager

* test(secrets): pin that non-conflict CyberArk policy failures are not retried

* test(unit): run tests/unit with only an allowlisted host environment

CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.

* test(e2e): name a failed search and the stuck batch status instead of misattributing them

The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.

* fix(ci): scrub the host environment per unit test instead of for the whole pytest process

GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.

* fix(ci): scrub only the host variables present before litellm is imported

The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.

* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue

LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.

The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.

* test(e2e/ui): wait for the logout redirect before visiting a protected page

Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).

* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests

* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read

* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect

* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
2026-09-26 09:25:13 -07:00
..
auth fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
budgets test(e2e/ui): wait for the filtered budget list before clicking a row action 2026-09-17 21:39:22 -07:00
guardrails test(e2e): wait for serving propagation in UI journeys 2026-09-11 10:25:43 -07:00
integrationCritical fix(ui): group cost optimization cache leakage by model group (#43008) 2026-09-24 23:51:51 -07:00
internal-user fix(e2e): expect models filters to persist after reload 2026-09-15 18:21:43 -07:00
internal-viewer test(e2e): assert user-observable behavior instead of DOM structure 2026-08-31 14:04:15 -07:00
login fix(e2e/ui): resolve dashboard base URL from env instead of hardcoding localhost (#34739) 2026-07-27 10:19:32 -07:00
logs fix(ui): paginate request logs by session groups server-side (#39257) 2026-09-02 17:54:17 -07:00
mcp test(e2e-ui): check the MCP Tools tab against the upstream's own tools/list (#42397) 2026-09-21 20:35:30 -07:00
migration test(ui): assert navigation through accessible links and page content 2026-09-06 00:59:18 -07:00
modelHub style: drop explanatory comments from the agent publish fix 2026-09-03 04:27:43 -07:00
modelsPage test(e2e): drop the auto-router select "opens below" spec 2026-09-16 22:43:51 -07:00
navigation test(ui): assert navigation through accessible links and page content 2026-09-06 00:59:18 -07:00
playground test(e2e/ui): address the migrated dashboard controls by role, not antd classes 2026-08-15 15:37:09 -07:00
prompts test(ui): address review feedback 2026-09-17 23:46:53 -07:00
proxy-admin fix(e2e): onboard dashboard fixtures through invitations 2026-09-15 17:44:18 -07:00
settings test(e2e): drop redundant SCIM key cleanup, the throwaway db is the teardown 2026-08-31 15:36:35 -07:00
tables test: repair two CI tests broken by intentional changes 2026-09-05 12:02:48 -07:00
tagManagement test(ui): address review feedback 2026-09-17 23:46:53 -07:00
team-admin fix(e2e): onboard dashboard fixtures through invitations 2026-09-15 17:44:18 -07:00
usage test(e2e/ui): cover the Usage page activity tabs (#39061) 2026-09-01 12:36:17 -07:00
users test(e2e): match the Internal Users search placeholder shipped by #39604 2026-09-03 16:57:10 -07:00