4 commits

Author	SHA1	Message	Date
yuneng-jiang	32a5259407	test(e2e-ui): verify UI mutations against the API instead of trusting the toast (#36632) ... * test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows These three pages carried no e2e coverage, so the manual QA checklist was the only thing standing behind them. Playground: sends a chat from the UI for both configured models, and for both virtual-key sources (the logged-in session, and a key pasted into the panel). This is the only spec that drives the dashboard's own LLM call path rather than an admin CRUD endpoint. Logs: a request the proxy actually served appears in the table, its drawer expands to the real request and response bodies, both copy to the clipboard, the Input card collapses, the JSON view exposes Request/Response, and the End User filter narrows the table to one customer. Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card toggles between table and chart, and the key opens its key-info panel. Router settings: the existing spec proved the UI can record a fallback; the new one proves the fallback is honoured, by pointing a model at an unreachable upstream and asserting the reply comes back anyway. It asserts the un-fallen-back call fails first, so a quietly-working primary cannot fake a pass. Supporting changes: - helpers/traffic.ts generates the traffic these pages render, rather than seeding rows no code produced. Its two wait helpers exist because the Logs and Usage pages read different stores: spend logs are flushed on a timer, and the Usage page reads a background rollup *and* fetches once on mount, so waiting on the DOM there can never converge. - helpers/playground.ts holds the playground controls, now shared with the fallback spec. Everything is scoped to the visible copy of the config panel, which is rendered twice for the docked and collapsed layouts. - run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a spec can be re-run against it without paying for a UI rebuild each iteration. Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped. * test(e2e-ui): cover listing and calling MCP tools Covers the two MCP manual-QA items the create-only spec cannot reach: opening a server's tool list, and calling a tool and seeing its result. Both need an MCP server that actually answers, so this points at DeepWiki's public MCP server -- Streamable HTTP, auth None, so there is no credential to hold and nothing to leak from a public repo. The call is made by the proxy, not the browser; nothing in the e2e chart restricts that egress. The external dependency is real and is left visible: an upstream outage turns these red rather than auto-skipping, because a spec that skips itself on connection trouble also skips when the proxy's MCP client is what broke. E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out. Not yet executed against a live stack. * test(e2e-ui): verify key mutations round-trip instead of trusting the toast The recurring customer report is a form that says "Saved!" and then either no-ops or clobbers an unrelated field. A toast-only assertion passes in both cases, and outside three specs that is all this suite checks. Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing, credentials and routerSettings already use: capture the outgoing request body, then read the resource back through the management API. Applies it to the keys spec: - create: the key is readable from /key/list and owns a team_id, rather than trusting a table row rendered from the create response the UI already held - update limits: TPM/RPM are on the wire AND persisted, and the key's models and team are unchanged -- bumping one field wiping another is the reported failure mode (PR #34452), not a hypothetical - delete: the key is gone from /key/list, not merely toasted as deleted - regenerate: the stored token actually changed /key/list shape is per KeyListResponseObject in litellm/proxy/_types.py. Not yet executed: ports 4000/8090 are held by a parallel run. * test(e2e-ui): let the local harness run on non-default ports Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/ 8090, so the second aborts on "port 4000 is in use" and the only way forward is to stop someone else's stack. PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the historical values so an unset environment behaves exactly as before -- CI, the CircleCI job and the chart's sidecar all keep working untouched. Two details that would otherwise make a relocated stack fail confusingly: - the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000 independently, so the run would build and boot correctly and then test whatever was on the default port. run_e2e.sh now derives it. - the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the proxy's own pod is the contract the e2e chart's sidecar is written against. * test(e2e-ui): cover MCP server edit and delete, verified via the API mcpServers.spec.ts only ever creates a server, and creation is the one MCP operation nobody has complained about. The reports are all on the other side: an alias rename that needs three or four saves to take, a delete that needs two attempts. Both produce a success toast on the failing attempt, so a toast-only assertion cannot tell them from working software. Rename asserts the new alias and the target server_id are on the PUT, then polls /v1/mcp/server until the stored alias matches -- one save has to be enough. Delete asserts the server is really gone from the list. Points at an unreachable URL: these exercise litellm's persistence, never the upstream, so a live MCP server would add a network dependency for nothing. mcpTools.spec.ts is where a real upstream is needed. Both pass against a local stack, as do the mcpTools specs from 5e189e9b1a. Neither reproduced the reported failures on this build -- they guard, they did not catch. * test(e2e-ui): verify team create, invite and delete against the API Three team mutations stopped at a toast, and one of those toasts is matched as loosely as /success/i -- almost any notification satisfied it. - create: the team is readable from /team/list and kept the models chosen in the modal, rather than trusting the UI's own "Team created" - invite: the invited address really appears in members_with_roles, which is the point of the flow - delete: the team is gone from /team/list. The existing assertion was that the row vanished, which is the client dropping it from local state and happens whether or not the delete reached the database. Shapes read off a live proxy: /team/list is a bare array; /team/info nests the record under team_info. All 6 tests pass locally. * test(e2e-ui): verify team-admin member and key mutations against the API The team-admin flows stopped at a success toast. A member add that lands on the wrong team, a remove that takes out the wrong row, and a key that comes back unscoped all produce the same toast as the working case, so the existing assertions could not tell them apart. Each mutation now pins what went on the wire and reads the result back: member add/remove assert team_id and the member identifier on the request, then poll /team/info's roster; the team key asserts team_id on /key/generate and reads /key/list back to confirm the key is owned by the admin's own team rather than orphaned. * test(e2e-ui): verify model add and limit edits against the stored deployment The Models specs checked the rendered result: the TPM/RPM edit asserted the new numbers were visible in view mode, and the two add flows asserted a row showed up in the table. Both render from state the UI already holds, so a save the backend dropped and a save it kept look the same. Each mutation now pins the request and reads the deployment back. The limits edit also asserts the fields it did not touch -- upstream model and team ownership -- are unchanged, because handleModelUpdate rebuilds and PATCHes the whole litellm_params blob, which is how an unrelated field gets clobbered by a save that reports success. The two add flows assert model_name, the routed model and custom_llm_provider on the wire and in storage; a deployment that loses its provider looks correct in the table and is unroutable. The Team-BYOK test is unchanged -- it is skipped without a license, so any change to it would be unverified. * test(e2e-ui): delete the MCP servers these specs create MCP servers outlive the test that made them, the MCP page contacts every server it lists, and most of the ones these specs create point at an unreachable host. They accumulate, and each one makes navigateToPage's networkidle wait a little slower to settle. Measured on a local stack: with eleven leaked servers the whole MCP suite failed on a 30s navigation timeout, including specs that leaked nothing. Deleting the leftovers made all five pass. With per-test cleanup added, a run from a clean slate leaves zero behind and takes 20s instead of 1m24s. mcpServers.spec.ts carried a note that no teardown was needed because the runner brings up a fresh database each time. That holds for CI and is why this went unnoticed; it does not hold for a local stack that is reused. * test(e2e-ui): say why the team-model setup call failed The setup that creates a team-scoped model asserted a bare `ok()`, so a failure read "expected true, received false" and pointed at the UI. The call is enterprise-gated -- creating a model with model_info.team_id returns 403 without LITELLM_LICENSE -- and that is invisible from the old message. It now carries the status and body, which names the cause immediately. The api_base also pointed at the mock's default port rather than the one the harness started; nothing in the test calls the model, but the two should not disagree. * test(e2e-ui): add a model through the UI and serve traffic with it Every existing Add Model test stops at "the row appears in the table", which a deployment that cannot serve a single request also does. The manual-QA item this replaces is the whole loop: fill the form, pass Test Connect, add it, confirm it works. The new test ends by calling the model it just created. That is the only assertion that rules out a dropped api_base, a mangled provider prefix, or a name the router never registers -- all of which look identical in the UI. No provider credential is involved. OpenAI-Compatible is the provider whose form exposes API Base, so the deployment points at the harness's own mock LLM. The mock speaks the OpenAI wire format, so Test Connect performs a real completion against a real endpoint and really succeeds. Also adds teardown for the deployment it creates. A local run throws its database away, but the deployed stack does not, and a leaked deployment shows up in every later Models table and /v2/model/info readback. Both new assertions were mutation-tested: pointing the traffic poll at a name that was never created fails the test, and the wire assertion fails when the typed name is not what reaches /model/new. * test(e2e-ui): print the proxy log when the proxy dies on its own In E2E_KEEP_ALIVE=1 mode the harness blocks until the proxy pid goes away, then printed a bare "Proxy exited." and fell straight into cleanup, which rm -f's the log. The proxy has now exited by itself twice, minutes after a run had finished, leaving nothing to look at. Both startup failure paths already tail -n 100 the log before giving up, so this was the one death that stayed silent Dump the same 100 lines before exiting. A normal Ctrl-C teardown still deletes the log and prints nothing, which is why INT and TERM now exit instead of running cleanup and falling back into the wait loop: under the single trap a SIGTERM deleted the log, resumed the loop, and would then report "tail: no such file", besides running cleanup twice * test(e2e-ui): split the log-drawer copy assertions off the expand test The copy assertions need `navigator.clipboard`, which the browser only exposes in a secure context. Locally the suite runs against http://127.0.0.1 and localhost is trustworthy, so it is there. In CI the run pod is pointed at a plain-HTTP cluster DNS name, where it is undefined -- and InputCard.handleCopy calls writeText unguarded, so the click throws before MessageManager.success and no toast ever renders. That failed all three attempts of litellm-e2e-ui build 10. Measured rather than inferred: on http://127.0.0.1:4100 isSecureContext/typeof navigator.clipboard are true/"object", and on a DNS name resolving to that same address they are false/"undefined", which reproduces the CI failure exactly. Splitting keeps the drawer-rendering coverage running everywhere and confines the skip to the part the browser has actually switched off. The copy assertions still run in full wherever the origin is trustworthy. The underlying product behaviour is left alone deliberately: any deployment served over plain HTTP on a hostname has a copy button that throws and gives no feedback, and that deserves its own fix rather than being papered over from a test. * test(e2e-ui): cut the added comments back to what the code cannot say itself Greptile flagged the helper commentary, and it was right: CLAUDE.md says not to write comments unless they explain very complex business logic, and much of what was added here narrated ordinary test setup and motivation instead. Trims 310 comment lines across the 14 files this branch touched. Kept only the notes that record something unrecoverable from the code: why the request listener is armed before the click, why a locator walks up the DOM, why an assertion exists beyond the toast. Pre-existing comments are left alone. No behaviour change. The only non-comment hunk is a prettier reformat.	2026-08-12 13:39:58 -07:00
Yuneng Jiang	fb353423d8	test(e2e): self-seed the ui suite's password-login users in global setup	2026-08-04 17:38:38 -07:00
mubashir1osmani	328e41b1f9	test(e2e): unblock the ui suite, fix the mcp registration race, park two known product bugs (#34853) ... * test(e2e): let the ui suite run from a read-only cwd The playwright suite never executed on stage. It died in globalSetup before a single test ran, and the reported error was a red herring. /app/e2e/ui is a read-only filesystem in the packaged e2e image (the image runner already redirects playwright's own artifacts to TMPDIR for this reason), but the suite wrote three things relative to cwd: the per-role storageState files, the failure-screenshot directory, and the html report. Reproduced in the pod: storageState raises EROFS, mkdir test-results raises ENOENT. Worse, the catch block that exists to capture a screenshot threw its own ENOENT while handling a failure, so the real login error was replaced by a filesystem error. That is why the run looked like a missing directory rather than whatever actually went wrong. Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort so it can never mask the underlying failure, and point playwright's reporter and outputDir at the same place so a bare `npx playwright test` works there too. fixtures/users.ts had its own copy of the five storageState filenames; it now re-exports the ones from constants so the paths have a single definition. Verified in the read-only pod: both writes fail before, both succeed after. 85 tests enumerate and tsc --noEmit is clean. Refs LIT-4821 * fix(e2e): create the ui artifact root before writing into it storageState() does not create missing parents, and nothing created ARTIFACT_DIR itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet therefore failed with ENOENT on the very first role's snapshot, before any UI test ran; the same class of failure the artifact-dir change was meant to remove, just moved one level up. Reproduced: writing admin.storageState.json into a missing directory raises ENOENT. My earlier pod verification masked this because the probe called mkdirSync itself, which the real code path never did. mkdir the root once at the top of globalSetup, before the login loop. recursive makes it idempotent, handles nested paths, and keeps the default "." a no-op. Playwright creates its own outputDir lazily, so globalSetup is the only place that needs this, and migration.serverRootPath.globalSetup delegates here so it is covered too. * test(e2e): skip the mid-conversation cache checks pending LIT-4873 A mid-conversation role="system" reminder invalidates the prompt cache on the vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder turn, same conversation shape throughout: direct to api.anthropic.com 7013 read cache preserved litellm -> anthropic/claude-opus-4-8 7013 read cache preserved litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed and the Vertex control with the same added assistant/user turns but no reminder reads 7013, so it is the reminder on the non-first-party paths and not the extra turns. Anthropic keeping the cache rules out provider behavior; litellm's first-party anthropic path keeping it rules out the shared Messages transform. That makes these assertions correct and the failure a real billing bug, so the tests are skipped rather than weakened; the bodies stay intact and must be restored unchanged with the fix. Registry rows are left in place, so the three mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps. Skips are decorators rather than a pytest.skip() inside the shared helper: a mid-function skip fires only after setup has already registered a real deployment via /model/new and left the rest of the body unreachable. Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are inferred from matching nightly failures and should be confirmed with the fix. Refs LIT-4821, LIT-4873	2026-07-27 19:20:57 -07:00
ryan-crabbe-berri	0fcaadf11c	test(e2e): move Admin UI Playwright suite to tests/e2e/ui (#34196) ... Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end suites live under tests/e2e. The suite stays in TypeScript and becomes a self-contained npm package with its own package.json, lockfile and tsconfig instead of leaning on the dashboard's toolchain; the dashboard drops its @playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs. CI paths follow the move: both CircleCI jobs (main e2e and the SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now install and run Playwright from tests/e2e/ui, with the node cache keyed on both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec edits keep skipping backend jobs. The suite's mock LLM fixture is excluded from the e2e basedpyright zero-error gate in pyrightconfig.json since it belongs to the TS suite, not the typed Python harness.	2026-07-22 19:43:10 +00:00