7 commits

Author	SHA1	Message	Date
ryan-crabbe-berri	ecf17513fb	refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits	2026-09-19 18:53:14 -07:00
ryan-crabbe-berri	fe480533e8	feat(proxy)!: refuse to start with an unset, empty, or publicly known master key ... The proxy used to boot with no master key (every request accepted without authentication) and with sk-1234, the key every example used. It now stops at startup, before it connects to the database, and prints how to fix it: where the bad key came from, a copy-pastable command that generates a secure key, and, when the public key is also encrypting a database, a link to the rotation guide general_settings.dangerously_allow_unsafe_proxy: true or LITELLM_DANGEROUSLY_ALLOW_UNSAFE_PROXY=true starts the proxy anyway, for local development. CI and test boots that rely on sk-1234 or on no key set it BREAKING CHANGE: deployments with no master key, an empty one, or sk-1234 no longer start until they set a real key or opt in to the override	2026-09-19 13:44:00 -07:00
Yuneng Jiang	b0e53bfbe4	fix(e2e/ui): fail the run when the Presidio fixture never comes up ... Both readiness loops broke out on success and fell through on timeout, so a mock Presidio server that failed to bind left the run going with nothing serving /analyze. The guardrail then errored at request time and the failure surfaced as an unrelated Playwright assertion in presidioUserStory.spec.ts rather than as the missing fixture it actually was. Fail the local runner with the port in the message, and add the matching wait step to both CircleCI UI jobs, which had no readiness check at all.	2026-09-06 03:13:09 -07:00
Yuneng Jiang	274a489c46	test(e2e/ui): automate the RC checklist's Presidio guardrail walk ... The guardrail section of the release checklist is done by hand every cut: create a Presidio guardrail through the wizard, send a sentence with PII from the playground, then open Logs and check the guardrail caught it. Nothing covered that path, so a break anywhere along it surfaced only when someone happened to repeat the steps. Adds a spec that walks it once and turns the eyeball checks into assertions. The strongest of them is the leak check: it reads the request back and fails if the stored prompt still carries the raw address or number, which is what the manual step is really looking for. The stack gains a Presidio stand-in that answers the two routes the guardrail calls, detecting a fixed regex set with a Luhn check on card numbers. Real Presidio's detection quality is Presidio's business, and pinning the UI lane to it would mean two heavy containers with spaCy models on every CI run for a test that is about LiteLLM's integration. The real analyzer stays covered in the Python lane. The stand-in returns the same entities at the same spans as the real one for the checklist's sentence, and driving the real guardrail against it produces the same record shape, so a test written against it is written against the product's real behavior. Making the stand-in return the text unmasked turns the spec red on the raw address reaching the spend log, so the leak assertion reads live data. run_e2e.sh and both CircleCI UI jobs start the stand-in alongside the mock LLM. Its port is overridable like the others so two checkouts can run at once.	2026-09-06 02:36:46 -07:00
Yuneng Jiang	6b493cb61e	test(guardrails): pin the presidio spend-log record and the UI's masked-entity persistence ... The presidio suite proved masking happened by reading the served answer, and the UI suite proved the wizard's "Select All & Mask" produced a row. Neither checked the thing an operator actually looks at afterwards: the audit trail. Adds an e2e test asserting the spend log carries the pre_call guardrail record for a masked request: status, provider, per-entity masked counts, and the detected-entity list the dashboard's guardrail panel renders its scores from. It keys off the x-litellm-applied-guardrails response header rather than masked text in the answer, because whether the model echoes the prompt back is a model decision, not a guardrail one. Both a mutation that stops writing the record and one that empties the entity list turn it red. Two records land on one log, pre_call and post_call, so the assertion selects on mode as well as name; picking by name alone could hand it the empty post_call record depending on write order. On the UI side, the Presidio wizard test now reads the stored guardrail back and asserts every persisted entity carries the MASK action. A row appearing in the table did not prove the entity selection survived the save, so a wizard that persisted an empty pii_entities_config would have passed. SpendLogRow gains a typed metadata field. guardrail_response is left as object because each provider writes its own shape there (presidio a list of entities, bedrock an assessment object, a failed run the exception string); a union narrow enough to be useful would fail to parse the others and break every suite that reads a spend log. The caller validates the shape it expects with a TypeAdapter. run_e2e.sh now pins PROXY_BASE_URL to the stack's own origin. The proxy builds its post-login redirect from that variable when it is set, so a value inherited from a developer's .env sent the browser off the relocated stack and the suite's login step timed out on every port but 4000.	2026-09-06 00:43:13 -07:00
yuneng-jiang	32a5259407	test(e2e-ui): verify UI mutations against the API instead of trusting the toast (#36632) ... * test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows These three pages carried no e2e coverage, so the manual QA checklist was the only thing standing behind them. Playground: sends a chat from the UI for both configured models, and for both virtual-key sources (the logged-in session, and a key pasted into the panel). This is the only spec that drives the dashboard's own LLM call path rather than an admin CRUD endpoint. Logs: a request the proxy actually served appears in the table, its drawer expands to the real request and response bodies, both copy to the clipboard, the Input card collapses, the JSON view exposes Request/Response, and the End User filter narrows the table to one customer. Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card toggles between table and chart, and the key opens its key-info panel. Router settings: the existing spec proved the UI can record a fallback; the new one proves the fallback is honoured, by pointing a model at an unreachable upstream and asserting the reply comes back anyway. It asserts the un-fallen-back call fails first, so a quietly-working primary cannot fake a pass. Supporting changes: - helpers/traffic.ts generates the traffic these pages render, rather than seeding rows no code produced. Its two wait helpers exist because the Logs and Usage pages read different stores: spend logs are flushed on a timer, and the Usage page reads a background rollup *and* fetches once on mount, so waiting on the DOM there can never converge. - helpers/playground.ts holds the playground controls, now shared with the fallback spec. Everything is scoped to the visible copy of the config panel, which is rendered twice for the docked and collapsed layouts. - run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a spec can be re-run against it without paying for a UI rebuild each iteration. Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped. * test(e2e-ui): cover listing and calling MCP tools Covers the two MCP manual-QA items the create-only spec cannot reach: opening a server's tool list, and calling a tool and seeing its result. Both need an MCP server that actually answers, so this points at DeepWiki's public MCP server -- Streamable HTTP, auth None, so there is no credential to hold and nothing to leak from a public repo. The call is made by the proxy, not the browser; nothing in the e2e chart restricts that egress. The external dependency is real and is left visible: an upstream outage turns these red rather than auto-skipping, because a spec that skips itself on connection trouble also skips when the proxy's MCP client is what broke. E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out. Not yet executed against a live stack. * test(e2e-ui): verify key mutations round-trip instead of trusting the toast The recurring customer report is a form that says "Saved!" and then either no-ops or clobbers an unrelated field. A toast-only assertion passes in both cases, and outside three specs that is all this suite checks. Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing, credentials and routerSettings already use: capture the outgoing request body, then read the resource back through the management API. Applies it to the keys spec: - create: the key is readable from /key/list and owns a team_id, rather than trusting a table row rendered from the create response the UI already held - update limits: TPM/RPM are on the wire AND persisted, and the key's models and team are unchanged -- bumping one field wiping another is the reported failure mode (PR #34452), not a hypothetical - delete: the key is gone from /key/list, not merely toasted as deleted - regenerate: the stored token actually changed /key/list shape is per KeyListResponseObject in litellm/proxy/_types.py. Not yet executed: ports 4000/8090 are held by a parallel run. * test(e2e-ui): let the local harness run on non-default ports Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/ 8090, so the second aborts on "port 4000 is in use" and the only way forward is to stop someone else's stack. PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the historical values so an unset environment behaves exactly as before -- CI, the CircleCI job and the chart's sidecar all keep working untouched. Two details that would otherwise make a relocated stack fail confusingly: - the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000 independently, so the run would build and boot correctly and then test whatever was on the default port. run_e2e.sh now derives it. - the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the proxy's own pod is the contract the e2e chart's sidecar is written against. * test(e2e-ui): cover MCP server edit and delete, verified via the API mcpServers.spec.ts only ever creates a server, and creation is the one MCP operation nobody has complained about. The reports are all on the other side: an alias rename that needs three or four saves to take, a delete that needs two attempts. Both produce a success toast on the failing attempt, so a toast-only assertion cannot tell them from working software. Rename asserts the new alias and the target server_id are on the PUT, then polls /v1/mcp/server until the stored alias matches -- one save has to be enough. Delete asserts the server is really gone from the list. Points at an unreachable URL: these exercise litellm's persistence, never the upstream, so a live MCP server would add a network dependency for nothing. mcpTools.spec.ts is where a real upstream is needed. Both pass against a local stack, as do the mcpTools specs from 5e189e9b1a. Neither reproduced the reported failures on this build -- they guard, they did not catch. * test(e2e-ui): verify team create, invite and delete against the API Three team mutations stopped at a toast, and one of those toasts is matched as loosely as /success/i -- almost any notification satisfied it. - create: the team is readable from /team/list and kept the models chosen in the modal, rather than trusting the UI's own "Team created" - invite: the invited address really appears in members_with_roles, which is the point of the flow - delete: the team is gone from /team/list. The existing assertion was that the row vanished, which is the client dropping it from local state and happens whether or not the delete reached the database. Shapes read off a live proxy: /team/list is a bare array; /team/info nests the record under team_info. All 6 tests pass locally. * test(e2e-ui): verify team-admin member and key mutations against the API The team-admin flows stopped at a success toast. A member add that lands on the wrong team, a remove that takes out the wrong row, and a key that comes back unscoped all produce the same toast as the working case, so the existing assertions could not tell them apart. Each mutation now pins what went on the wire and reads the result back: member add/remove assert team_id and the member identifier on the request, then poll /team/info's roster; the team key asserts team_id on /key/generate and reads /key/list back to confirm the key is owned by the admin's own team rather than orphaned. * test(e2e-ui): verify model add and limit edits against the stored deployment The Models specs checked the rendered result: the TPM/RPM edit asserted the new numbers were visible in view mode, and the two add flows asserted a row showed up in the table. Both render from state the UI already holds, so a save the backend dropped and a save it kept look the same. Each mutation now pins the request and reads the deployment back. The limits edit also asserts the fields it did not touch -- upstream model and team ownership -- are unchanged, because handleModelUpdate rebuilds and PATCHes the whole litellm_params blob, which is how an unrelated field gets clobbered by a save that reports success. The two add flows assert model_name, the routed model and custom_llm_provider on the wire and in storage; a deployment that loses its provider looks correct in the table and is unroutable. The Team-BYOK test is unchanged -- it is skipped without a license, so any change to it would be unverified. * test(e2e-ui): delete the MCP servers these specs create MCP servers outlive the test that made them, the MCP page contacts every server it lists, and most of the ones these specs create point at an unreachable host. They accumulate, and each one makes navigateToPage's networkidle wait a little slower to settle. Measured on a local stack: with eleven leaked servers the whole MCP suite failed on a 30s navigation timeout, including specs that leaked nothing. Deleting the leftovers made all five pass. With per-test cleanup added, a run from a clean slate leaves zero behind and takes 20s instead of 1m24s. mcpServers.spec.ts carried a note that no teardown was needed because the runner brings up a fresh database each time. That holds for CI and is why this went unnoticed; it does not hold for a local stack that is reused. * test(e2e-ui): say why the team-model setup call failed The setup that creates a team-scoped model asserted a bare `ok()`, so a failure read "expected true, received false" and pointed at the UI. The call is enterprise-gated -- creating a model with model_info.team_id returns 403 without LITELLM_LICENSE -- and that is invisible from the old message. It now carries the status and body, which names the cause immediately. The api_base also pointed at the mock's default port rather than the one the harness started; nothing in the test calls the model, but the two should not disagree. * test(e2e-ui): add a model through the UI and serve traffic with it Every existing Add Model test stops at "the row appears in the table", which a deployment that cannot serve a single request also does. The manual-QA item this replaces is the whole loop: fill the form, pass Test Connect, add it, confirm it works. The new test ends by calling the model it just created. That is the only assertion that rules out a dropped api_base, a mangled provider prefix, or a name the router never registers -- all of which look identical in the UI. No provider credential is involved. OpenAI-Compatible is the provider whose form exposes API Base, so the deployment points at the harness's own mock LLM. The mock speaks the OpenAI wire format, so Test Connect performs a real completion against a real endpoint and really succeeds. Also adds teardown for the deployment it creates. A local run throws its database away, but the deployed stack does not, and a leaked deployment shows up in every later Models table and /v2/model/info readback. Both new assertions were mutation-tested: pointing the traffic poll at a name that was never created fails the test, and the wire assertion fails when the typed name is not what reaches /model/new. * test(e2e-ui): print the proxy log when the proxy dies on its own In E2E_KEEP_ALIVE=1 mode the harness blocks until the proxy pid goes away, then printed a bare "Proxy exited." and fell straight into cleanup, which rm -f's the log. The proxy has now exited by itself twice, minutes after a run had finished, leaving nothing to look at. Both startup failure paths already tail -n 100 the log before giving up, so this was the one death that stayed silent Dump the same 100 lines before exiting. A normal Ctrl-C teardown still deletes the log and prints nothing, which is why INT and TERM now exit instead of running cleanup and falling back into the wait loop: under the single trap a SIGTERM deleted the log, resumed the loop, and would then report "tail: no such file", besides running cleanup twice * test(e2e-ui): split the log-drawer copy assertions off the expand test The copy assertions need `navigator.clipboard`, which the browser only exposes in a secure context. Locally the suite runs against http://127.0.0.1 and localhost is trustworthy, so it is there. In CI the run pod is pointed at a plain-HTTP cluster DNS name, where it is undefined -- and InputCard.handleCopy calls writeText unguarded, so the click throws before MessageManager.success and no toast ever renders. That failed all three attempts of litellm-e2e-ui build 10. Measured rather than inferred: on http://127.0.0.1:4100 isSecureContext/typeof navigator.clipboard are true/"object", and on a DNS name resolving to that same address they are false/"undefined", which reproduces the CI failure exactly. Splitting keeps the drawer-rendering coverage running everywhere and confines the skip to the part the browser has actually switched off. The copy assertions still run in full wherever the origin is trustworthy. The underlying product behaviour is left alone deliberately: any deployment served over plain HTTP on a hostname has a copy button that throws and gives no feedback, and that deserves its own fix rather than being papered over from a test. * test(e2e-ui): cut the added comments back to what the code cannot say itself Greptile flagged the helper commentary, and it was right: CLAUDE.md says not to write comments unless they explain very complex business logic, and much of what was added here narrated ordinary test setup and motivation instead. Trims 310 comment lines across the 14 files this branch touched. Kept only the notes that record something unrecoverable from the code: why the request listener is armed before the click, why a locator walks up the DOM, why an assertion exists beyond the toast. Pre-existing comments are left alone. No behaviour change. The only non-comment hunk is a prettier reformat.	2026-08-12 13:39:58 -07:00
ryan-crabbe-berri	0fcaadf11c	test(e2e): move Admin UI Playwright suite to tests/e2e/ui (#34196) ... Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end suites live under tests/e2e. The suite stays in TypeScript and becomes a self-contained npm package with its own package.json, lockfile and tsconfig instead of leaning on the dashboard's toolchain; the dashboard drops its @playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs. CI paths follow the move: both CircleCI jobs (main e2e and the SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now install and run Playwright from tests/e2e/ui, with the node cache keyed on both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec edits keep skipping backend jobs. The suite's mock LLM fixture is excluded from the e2e basedpyright zero-error gate in pyrightconfig.json since it belongs to the TS suite, not the typed Python harness.	2026-07-22 19:43:10 +00:00

Renamed from ui/litellm-dashboard/e2e_tests/run_e2e.sh (Browse further)