DataTable loses the comment that narrated its sticky header classes. The
table scrolling e2e spec now types every management API response it
reads, seeds rows through an immutable reduce instead of pushing into
arrays, and deletes what it seeded in each test's finally block instead
of draining a shared mutable list in afterEach.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
Virtual Keys, Teams, Request Logs and Tags now hand DataTable a bounded
flex chain and use fillHeight, so the app shell main stays the only page
scroller, the rows scroll under a pinned header and the pagination footer
sits at the bottom of the page. DataTable keeps the sticky header inside
its own scroller in maxBodyHeight mode too, which is what let the header
scroll away with the rows on Keys, Teams and Models. Model Hub, Vector
Stores and the team detail keys tab drop their 75vh boxes and flow with
the page scroller.
Adds an e2e spec that fails on the merge base for every one of those
pages and passes at this tip.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
PR #39604 renamed the Internal Users search box placeholder to "Search by email or ID…" but left searchUsers.spec.ts looking for the old "Search by email…" copy, so e2e_ui_testing has been red on litellm_internal_staging since it merged. Point the locator at the shipped placeholder
`POST /v1/agents/{id}/make_public` appended the agent id to
`litellm.public_agent_groups` and only then called `get_config()`, which
re-applies the DB's `litellm_settings` over the module globals and threw the
append away. The config it saved was therefore a no-op: the endpoint answered
200 with an empty `public_agent_groups`, the agent never reached
`GET /public/agent_hub`, and re-publishing never hit the "already public" 400.
Read the config first, derive the new list from the refreshed globals, save it,
then update the global
Also fixes the e2e model hub spec, which is flaky for a second reason: the
"Make Models Public" modal preselects the groups that are already public, so a
blind click on "Select All" cleared them and left "Next" disabled for the full
15s action timeout. Check the box instead of toggling it, and wait for "Next"
to be enabled before clicking
* fix(ui): paginate request logs by session groups server-side
The logs table server-paginated raw spend logs and then collapsed
multi-call sessions client-side, so a page could render 3 rows while
the footer claimed 25 and sessions straddled pages. Adds an opt-in
group_by_session param to /spend/logs/ui that pages and counts one
representative row per session (DISTINCT ON, newest non-MCP call),
keeps the bounded count contract, enriches whole-session llm/agent
composition counts, and deletes the client-side collapse pipeline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* feat(ui): add a 10 rows-per-page option and default request logs to it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): key session aggregates per api key in the logs enrichment
Grouped pagination splits a reused session id into one row per api key,
but the enrichment still aggregated by session_id alone, so both rows
showed combined spend and counts. The aggregate query now groups by
(session_id, api_key), the count folds into it (the separate group_by
query is deleted), and each row reads its own key's totals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): treat an empty api_key as a real session group value
The spend-log schema defaults api_key to an empty string; truthiness
guards in the enrichment treated it as missing, so keyless multi-call
sessions lost their count and spend. Only None means missing now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The default password policy from #39381 (12+ chars with upper, lower, number, and special) makes globalSetup's POST /user/update with password "test" fail with a 400, so every role login in the UI e2e suite has been failing since that merge. Seed the roles, the noteam@test.local login, and the second-admin spec with policy-compliant passwords and update the scrypt hash in seed.sql to match.
* test(e2e/ui): cover the Usage page activity tabs
Usage had one test, on Top Virtual Keys. The Key, Model and Endpoint Activity
tabs are the ones an admin reads to answer where the spend went, and none of
them was covered.
Also fixes waitForKeyInDailyActivity, which only read the first page of
/user/daily/activity. The route paginates, so once a run generates more keys
than one page holds, the helper spins for its full 120 seconds and then blames
the rollup for a key the rollup wrote correctly. The Usage page itself already
walks every page; the helper now matches it.
* test(e2e/ui): route the user-creation call through SERVER_ROOT_PATH
Review caught /user/new posting to the server root, which misses the proxy
when it is mounted under a prefix. traffic.ts already had the helper for
this; it is now exported so specs making their own management calls can use
it too.
Also drops the mutable accumulators from the daily-activity paging, which
the repo conventions ask for.
---------
Co-authored-by: Claude <noreply@anthropic.com>
The Teams tests covered creating, deleting and membership, but nothing on the
Settings tab, which is the form that posts the whole team back. That is the
shape behind the reports of a team losing its metadata or its model aliases
after an unrelated edit.
Each test creates its own team rather than editing a seeded one. The limits
test pins the models and members the edit had no business touching, and the
alias test calls the new alias with a team key instead of trusting the
readback, since an alias the router never resolves reads the same either way.
Co-authored-by: Claude <noreply@anthropic.com>
* test(e2e/ui): stop the suite failing on things that are not regressions
Five tests in the UI suite fail for reasons that have nothing to do with the
product being broken, which is enough to keep the whole leg red.
Two need a premium proxy and fail hard without one: Regenerate Key renders
disabled when the proxy is unlicensed, and /model/new refuses a team-scoped
deployment. Both now skip without LITELLM_LICENSE, the way three other tests
in this suite already do.
Three consumed a seeded fixture: Delete key, Delete a team and remove a member
each destroyed the row they needed, so the retries CI runs with were guaranteed
to fail and the suite could not run twice against one database. They now create
what they destroy.
Top Virtual Keys ranks by spend and every mock deployment costs $0, so which
keys make the list came down to how ties happened to sort. It now sends its
traffic through a priced deployment and earns its place.
* test(e2e/ui): clean up the fixtures these tests create
Review caught two leaks: the priced deployment the usage test registers and
the user the team-admin test adds both outlived the run, so repeated runs
grew shared state that later routing and rosters can see.
Also brings in the paginated daily-activity read. /user/daily/activity pages
its per-key breakdown and the helper only read the first page, so the usage
test spent its full timeout blaming the rollup for a key the rollup wrote.
* test(e2e/ui): read the licence from the proxy, not the runner
Review pointed out that checking LITELLM_LICENSE in the runner's environment
describes the wrong machine: Playwright can be pointed at a proxy configured
somewhere else, and then the skip either hides coverage or runs a premium
test against an unlicensed target.
The admin session JWT already carries the premium_user claim the dashboard
itself reads to enable these controls, so both skips now use that.
* test(e2e/ui): clean up fixtures on the failure path too
Review caught both cleanups sitting at the end of the test body, where a
failing assertion skips them, and both discarding the response so a refused
delete passed quietly. They move to afterEach and assert the delete landed.
The priced deployment matters most: left behind it keeps its custom pricing
and goes on changing what later runs route and what they cost.
* test(e2e/ui): wait for the priced deployment to become routable
The Top Virtual Keys test registered a priced deployment and sent the key's
traffic through it on the next line, so on the deployed stack it failed with
"no healthy deployments for e2e-usage-priced-...": /model/new had written
the row but the router had not picked it up yet.
Polls a ping until the deployment answers before the test sends the request
it measures, matching what the addModel spec already does for a model added
through the UI. A ping that fails writes no spend log, so the retries cannot
move the ranking this test asserts.
* test(e2e/ui): register fixtures for cleanup before the step that can fail
Review found both helpers handing their id back to the caller to record, with
a failure-prone call in between: the priced deployment was registered after
the routability wait, and the added user after /team/member_add. Either
failing left the resource in the shared database with nothing tracking it.
Both now take the teardown list and add themselves as soon as the resource
exists, so the afterEach removes it however the rest of setup goes.
* test(e2e/ui): resolve the priced deployment for teardown by name
Review pointed out the remaining gap: /model/new can persist the deployment
and still answer non-2xx, and the id was only recorded after the response was
asserted, so that path left it behind with its custom pricing.
The name is now claimed before the request and teardown looks it up in
/model/info, so a create that saved without answering 2xx is still removed and
one that never saved is simply not there.
* test(e2e/ui): claim the member id before creating the user
/user/new can persist the user and still answer non-2xx, and the id is chosen
by the test rather than returned by the proxy, so registering it before the
call is what closes the last create-failure path.
Teardown now skips an id whose user is not there, so claiming it up front
cannot fail a run where the create never landed.
* test(e2e/ui): wait out the router reload when resolving a deployment to delete
/model/info answers from the router, not from the database, and /model/new
catches and logs a failed in-request reload while still answering 2xx. A
deployment can therefore be persisted and absent from the listing until the
next reload, which is where teardown was giving up and leaking it.
Teardown now retries the lookup for a little over one
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS before treating the name as never
persisted, so the only names it skips are the ones that really are not there.
* test(e2e/ui): prove a stored credential survives a config reload before using it
The Test Connect assertion has been failing intermittently on the full-suite
runs. Artifacts from litellm-e2e-ui build 165 show the UI sending
litellm_credential_name and the proxy answering with the credential unapplied:
raw_request_api_base was https://api.openai.com/v1/ rather than the mock base
the credential carries, and the call died on an upstream 404 for the model. The
same credential had resolved on three probes eight seconds earlier.
The proxy's periodic credential refresh takes a database snapshot, prunes any
in-memory credential missing from it, then re-adds the snapshot. A credential
created while that is in flight gets pruned and stays gone until the next tick,
and load_credentials_from_list fails open onto the ambient key, so nothing in
the error names the credential.
The existing pre-check asked for three consecutive probe successes, but they
completed in under a second, so they could not span a refresh. Space them so
the run covers a whole interval, which is what proves the credential survived a
refresh and is therefore stable.
* test(e2e/ui): find a database-only deployment through the search listing
/model/info answers from the router, so a deployment that reached the database
while /model/new's in-request reload failed is invisible there, and waiting on
the next reload only helps if reconciliation eventually picks it up.
/v2/model/info?search= runs a bounded query against the model table and
deliberately returns rows the router does not hold, so it resolves those
deployments to the id /model/delete needs. Falling back to it removes the wait
as well: absent from both listings now means the deployment never persisted.
* test(e2e/ui): delete the temporary member without a lookup that can skip it
Teardown asked /user/info first and treated any non-2xx as absence, so a
transient failure on the lookup silently skipped the delete and left the user
behind, which is the leak the claimed id was meant to close.
/user/delete answers 404 for an id that is not there, so it can carry both
cases on its own: 404 means the create never persisted, and anything else that
is not 2xx now fails the teardown instead of passing quietly.
* test(e2e/ui): reach the database fallback when the router listing fails
Asserting on /model/info threw before the fallback could run, so a failure on
the router-backed listing aborted teardown and left the deployment persisted,
which is the leak the fallback was added to close.
The router listing is best-effort now: an unreadable response just falls
through to the search-backed one. That listing is the authoritative answer to
whether the deployment exists, so it is the one that has to be readable, and a
name missing from it is a create that never persisted.
---------
Co-authored-by: Claude <noreply@anthropic.com>
The Logs page had coverage for opening a request and for the End User filter,
but nothing for the filters an on-call engineer actually reaches for: whose
key made the request, and which requests failed.
Each test mints its own keys and asserts on request ids it generated itself,
so a filter that quietly does nothing fails on the other key's row still being
on screen rather than passing because our own row happens to be there.
Co-authored-by: Claude <noreply@anthropic.com>
* test(e2e/ui): cover creating, testing and deleting a guardrail
The Guardrails page had no browser coverage. The RC checklist covers it by
hand against a live Presidio, which is why it has always been skipped in CI.
These drive the LiteLLM content filter instead, which runs inside the proxy,
so the whole flow is exercised without a third-party moderation service. The
create test does not stop at the table row: it sends a prompt carrying the
keyword it just banned and asserts the gateway refuses it, then sends a clean
prompt through the same guardrail and asserts it is served.
* test(e2e/ui): delete the guardrails these tests create
Review caught the fixtures being left behind. Guardrails are database rows
that show up in the table and in the playground's list, so a run that leaves
them changes what the next run sees.
Also trims the comments that restated what the helpers already say.
* test(e2e/ui): fail the run when guardrail teardown does not delete
Review caught the afterEach discarding the DELETE response, so a failed
cleanup finished quietly and left the guardrail for the next run to trip on.
* test(e2e/ui): wait for a new guardrail to reach the request path
The wizard test drove one chat completion immediately after creating the
guardrail and required a 400. A trace from the deployed stack shows the
record is stored correctly (blocked_words, action BLOCK, block_on_violation)
and the call six seconds later is still served unguarded, so the first
request can land before the proxy picks the guardrail up.
Polls the same call to the same 400 instead, which keeps the assertion and
lets the refresh land. If it never blocks, this stays red, which is what we
want it to say.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test(e2e/ui): cover the Budgets page create, edit and delete flows
The Budgets page had no browser coverage at all, so an admin creating or
editing a spend cap through the UI was only exercised by hand at RC time.
Each test reads the budget back from /budget/list, a different route from
the one the table renders, so a row that only exists in the table's cache
does not pass. The edit test pins the rate limits an unrelated spend-cap
edit has no business touching.
* test(e2e/ui): trim comments that restate the test steps
Review flagged the explanatory comments as restating ordinary setup rather
than explaining anything. Keeps the two that carry the regression rationale
for an assertion and drops the rest.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Replace table tbody and data-slot locators with getByRole, restore prior
public MCP hub entries instead of clearing the whitelist on cleanup, seed
the public agent via the append-semantics per-agent route, and rework
mutable cleanup state into const-scoped try/finally blocks
UsersTable overrides DataTable's default noDataMessage with its own
EmptyState, so the row reads "No users found" rather than "No results".
Assert that, and pair it with the seeded user being absent so the check
cannot pass while the filter silently does nothing.
Two assertions were checking the wrong thing. The anchoring tests read
getByRole("listbox"), which resolves to SelectPrimitive.List; that sits at
full content height inside the popup that clips and scrolls it, so the box
overlapped the trigger even when nothing visible did. Measure the popup.
The SSO-ID search expected zero rows, but DataTable renders a "No results"
message row when a filter matches nothing, so the count is one. Assert the
empty state the user actually sees.
Adds Playwright coverage for the RC checklist items an audit marked
automatable today: Playground to Logs hand-off, public Agent/MCP hub
tabs, team models in the Playground dropdown via a team key, Add Model
with a stored credential, internal user team key creation, a second
admin account, team model deletion, and Presidio guardrail CRUD without
a live sidecar. Seeds e2e-team-keygen with the /key/generate member
permission so the internal user key flow avoids the team-list cache lag
The consolidated popup test only asserted the options never cover the
trigger, so opening above the trigger with room below it, the regression
PR #38554 fixed, would have passed. Split it back into a below-trigger
case and a cramped-viewport case. The header test accepted a single pixel
of vertical intersection; require the refresh control's centre to sit
within the tab row instead.
The migration smoke waited on `getByRole("button", { expanded: false })`
after clicking it. Playwright re-resolves that locator on every retry, so
once the clicked group flipped to expanded it matched the next collapsed
group instead, and the assertion could never pass. Count the remaining
collapsed groups and wait for that count to drop by one.
The UI e2e suite had a class of assertions that pin how the dashboard is
built rather than what it does, so an ordinary refactor turns them red
without any user-visible change.
Geometry. The auto-router template select had two tests made of pixel
arithmetic plus a data-side="bottom" check, which is Base UI's own
positioner signal. The regression they guard (#38554) is a popup opening
on top of the control that spawned it, so both cases collapse to one
invariant: the options never cover the trigger. It now runs at both
viewport heights and reads the popup as role=listbox. The models header
test compared the tabs and refresh centers within 2px, which a padding
change flips; it now asserts the two share a row.
Structure. The logs drawer test walked xpath=../../.. from a text node
and read collapsed state off chevron icon classes. SectionHeader now
renders a real disclosure button with aria-expanded, and its two copy
buttons carry distinct names instead of both being "Copy". Sidebar group
toggles expose aria-expanded too, so the migration spec can ask for a
collapsed group by state rather than by nesting depth.
Positional lookups. keyRow.locator("button").first(), row.locator("td")
.first() and getByTestId(grid).locator("div").first() all named a
position where they meant an action; they now name the control. Table
scoping moves from "table tbody" to role=row.
Timing. Nine waitForTimeout calls are gone. Every assertion that followed
them already retried to its own timeout, so the sleeps only slowed the
run down.
Both files under tests/users/ were wrapped in test.skip("...", () => {}),
which registers one skipped test and never runs the body, so the four
tests inside had never executed and were written against a UI that has
since changed (the search placeholder is "Search by email…", the ID
filters moved into a drawer, pagination is labelled "Go to previous
page"). Rewritten against the current surface: the suite goes from 104
collected tests to 107.
Left in place deliberately: the chip and dialog-footer data-slot
selectors, because the accessible names they work around live in
components/ui/, which is shadcn CLI-managed and not hand-edited.
The sidebar toggle sits absolutely positioned over the drawer's flex row. With
the sidebar expanded it lands on the sidebar header, but once collapsed it lands
on the drawer header, which is sticky at z-chrome (10). The named-z-scale
refactor moved the toggle from z-20 to z-raised (1), so from then on the header
painted over it and swallowed the click: collapse the trace list and there was
no way to bring it back.
Moves the toggle to z-floating (30) and folds the two mirrored buttons into one,
since they only ever differed by icon, label and handler.
Covered by a Playwright spec, which is the tier that can see the layering: the
button stays visible and enabled either way, so the pre-fix failure is a click
interception that jsdom cannot reproduce.
Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.
Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
The shared SelectContent wrapper defaulted alignItemWithTrigger to true,
which puts Base UI's positioner into item-aligned mode and places the
popup so the active item sits on top of the trigger. In that mode the
side and sideOffset the wrapper passes two lines above are ignored, and
the popup reports data-side="none".
The overlap only becomes visible once the items are tall enough to
matter, which is why the autorouter Template picker shows it clearly:
its options are three-line cards, so the popup covers both the select
box and its own label.
No call site in the dashboard asked for item-aligned mode. 21 of them
across 15 files already passed alignItemWithTrigger={false} by hand to
undo the default, and the remaining 127 inherited the bug. Flipping the
default makes side and sideOffset live, so collision handling works and
a select with no room below now flips above the trigger rather than
covering it. The 21 hand-written opt-outs are deleted as redundant.
litellm-e2e-ui 68 failed the test this PR was meant to stabilise: "fallback
never took effect", streak 4 of a required 5, 60s timeout. Requiring a
consecutive streak of 200s after the fallback is set was wrong. It asserts that
the fallback path succeeds five times running, which is a reliability claim the
test never intended to make, and the path is inherently retry-ish because the
broken primary is attempted first on every call. One intermittent non-200
resets the streak, so a mostly-working fallback never converges.
The two directions are not symmetric:
before the write proving NO replica serves it -> needs every replica
after the write proving the fallback serves it -> one success is the claim
So the control keeps a multi-sample window and the success assertion goes back
to polling for a first sighting, on the wider 60s budget rather than the
original 30s that expired on litellm-e2e-ui 63.
Also drops the two local rebinds Greptile flagged against the repo's
no-reassignment convention: the streak counter is gone with the helper it lived
in, and the cache-round loop is now a lazy generator consumed by next().
Two e2e tests fail on timing rather than on litellm behaviour. Measured over the
last ~35 litellm-e2e / litellm-e2e-ui runs:
routerSettings.spec.ts:254 9/35 runs (7 flaky-on-retry, 2 hard failures)
test_cost_headers_e2e.py 1/29 runs it appeared in
Router fallback control
-----------------------
The e2e stack runs replicaCount 2 with proxy_config_reload_interval_seconds 7,
and every request is routed independently, so an observation of the new config
only proves the replica that served it reloaded. patchRouterSettings returns as
soon as /config/update returns, and clearBrokenFallback never waits at all, so a
retry's one-shot control assertion could be answered by a sibling replica still
holding the previous attempt's fallback. That is exactly the observed pair of
errors: "fallback never took effect" on the first attempt and "broken primary
unexpectedly succeeded on its own" on the retry.
Both assertions now poll for a consecutive streak spanning more than one reload
cycle, mirroring the PROPAGATION_TIMEOUT / settle_propagation doctrine the Python
suite already applies in e2e_config.py.
Cost-header cache read
----------------------
The prime and measure calls fired back to back with no gap, and each retry threw
away the prefix it had just paid to prime in favour of a fresh one. OpenAI
publishes a primed prefix asynchronously and routes cache lookups by
prompt_cache_key, so the test was rerolling the least likely path to a hit.
Each round now pins a prompt_cache_key and re-reads the same primed prefix up to
CACHE_REREADS times before rotating, so a fresh prefix is spent only after the
primed one has genuinely failed to become readable.
No production code changes; prompt_cache_key is added to the e2e ChatBody model,
which serializes exclude_none and so is inert for every other caller.
* test(ui): repoint the e2e locators at the post-antd form controls
Nine Playwright specs went red after the tremor and antd removals, none of
them because the product broke. The specs selected on markup those libraries
owned: tremor's TextInput stamped data-testid="base-input", the antd toast
facade rendered .ant-notification, and the team member modal's email field was
an .ant-select. Removing the libraries deleted those hooks silently.
Repoint each onto a user-facing locator that survives the next migration:
getByLabel for the key name, the MCP tool argument and the two cache pricing
fields, getByRole("combobox") for the team member email search, and the toast
container for the add-model success message.
The pricing fields needed a source change to be reachable at all. antd's
Form.Item used to assign the field name as the input id and tie the label to
it; the react-hook-form rewrite renders FieldLabel with no htmlFor and lets
FormField generate an opaque control id, so both cache inputs lost their
accessible name and could only be told apart by placeholder, which they share.
Pin the id back to the field name and point the label at it.
* fix(ui): let FormField own the pricing field label instead of hand-rendering one
The previous commit gave the cache cost inputs an accessible name by rendering
a FieldLabel with htmlFor next to the FormField. FieldLabel forwards Label's
props, and Label only accepts children, so next build failed type checking.
FormField already renders a label wired to the control id it generates, and
FormField.test.tsx covers that association, so passing label through is both
type-safe and less markup. The read-only branch keeps its plain FieldLabel,
which has no control to point at.
The log details drawer moved off Ant Design in 03d2b16bc, so its section
header renders lucide ChevronUp/ChevronDown rather than antd's UpOutlined
and DownOutlined. The collapse test still waited on .anticon-up and
.anticon-down, which no longer exist anywhere under view_logs, so it
failed on every run and burned all three attempts identically.
Point the three assertions at .lucide-chevron-up and .lucide-chevron-down,
matching how the dashboard's other suites address lucide icons.
The playground, logs drawer and AI Hub modal moved off antd, so the specs
that reached for .ant-select, .ant-drawer-content, .ant-modal and
.ant-radio-button-wrapper no longer match anything and time out.
Address the same controls through their accessible role and name instead,
which holds across the component library swap and reads closer to what a
user does.
Rebuilds the thirteen form-free components under common_components on
the in-repo shadcn layer, so they inherit the dashboard's design tokens
instead of styling themselves through Ant Design and Tremor.
SearchSelect and the three dropdowns that wrap it now forward an
optional input id, so an antd Form.Item label still resolves to its
control. The e2e steps that reached into antd's Select and Modal
internals now go through the test id, role and data-slot.
Rebuilds public_model_hub, MakeSkillPublicForm, ModelSelect and the
guardrail LogViewer on the in-repo shadcn layer, so they inherit the
dashboard's design tokens instead of styling themselves through Ant
Design and Tremor.
Public prop signatures are unchanged, so no caller moves. The two
teams e2e steps that reached into antd's Select internals now drive
the combobox through its test id, role and data-slot instead.
* test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows
These three pages carried no e2e coverage, so the manual QA checklist was the
only thing standing behind them.
Playground: sends a chat from the UI for both configured models, and for both
virtual-key sources (the logged-in session, and a key pasted into the panel).
This is the only spec that drives the dashboard's own LLM call path rather than
an admin CRUD endpoint.
Logs: a request the proxy actually served appears in the table, its drawer
expands to the real request and response bodies, both copy to the clipboard,
the Input card collapses, the JSON view exposes Request/Response, and the End
User filter narrows the table to one customer.
Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card
toggles between table and chart, and the key opens its key-info panel.
Router settings: the existing spec proved the UI can record a fallback; the new
one proves the fallback is honoured, by pointing a model at an unreachable
upstream and asserting the reply comes back anyway. It asserts the un-fallen-back
call fails first, so a quietly-working primary cannot fake a pass.
Supporting changes:
- helpers/traffic.ts generates the traffic these pages render, rather than
seeding rows no code produced. Its two wait helpers exist because the Logs
and Usage pages read different stores: spend logs are flushed on a timer, and
the Usage page reads a background rollup *and* fetches once on mount, so
waiting on the DOM there can never converge.
- helpers/playground.ts holds the playground controls, now shared with the
fallback spec. Everything is scoped to the visible copy of the config panel,
which is rendered twice for the docked and collapsed layouts.
- run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a
spec can be re-run against it without paying for a UI rebuild each iteration.
Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped.
* test(e2e-ui): cover listing and calling MCP tools
Covers the two MCP manual-QA items the create-only spec cannot reach:
opening a server's tool list, and calling a tool and seeing its result.
Both need an MCP server that actually answers, so this points at DeepWiki's
public MCP server -- Streamable HTTP, auth None, so there is no credential to
hold and nothing to leak from a public repo. The call is made by the proxy,
not the browser; nothing in the e2e chart restricts that egress.
The external dependency is real and is left visible: an upstream outage turns
these red rather than auto-skipping, because a spec that skips itself on
connection trouble also skips when the proxy's MCP client is what broke.
E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out.
Not yet executed against a live stack.
* test(e2e-ui): verify key mutations round-trip instead of trusting the toast
The recurring customer report is a form that says "Saved!" and then either
no-ops or clobbers an unrelated field. A toast-only assertion passes in both
cases, and outside three specs that is all this suite checks.
Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing,
credentials and routerSettings already use: capture the outgoing request body,
then read the resource back through the management API.
Applies it to the keys spec:
- create: the key is readable from /key/list and owns a team_id, rather than
trusting a table row rendered from the create response the UI already held
- update limits: TPM/RPM are on the wire AND persisted, and the key's models
and team are unchanged -- bumping one field wiping another is the reported
failure mode (PR #34452), not a hypothetical
- delete: the key is gone from /key/list, not merely toasted as deleted
- regenerate: the stored token actually changed
/key/list shape is per KeyListResponseObject in litellm/proxy/_types.py.
Not yet executed: ports 4000/8090 are held by a parallel run.
* test(e2e-ui): let the local harness run on non-default ports
Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/
8090, so the second aborts on "port 4000 is in use" and the only way forward is
to stop someone else's stack.
PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the
historical values so an unset environment behaves exactly as before -- CI, the
CircleCI job and the chart's sidecar all keep working untouched.
Two details that would otherwise make a relocated stack fail confusingly:
- the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000
independently, so the run would build and boot correctly and then test
whatever was on the default port. run_e2e.sh now derives it.
- the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT
there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the
proxy's own pod is the contract the e2e chart's sidecar is written against.
* test(e2e-ui): cover MCP server edit and delete, verified via the API
mcpServers.spec.ts only ever creates a server, and creation is the one MCP
operation nobody has complained about. The reports are all on the other side:
an alias rename that needs three or four saves to take, a delete that needs two
attempts. Both produce a success toast on the failing attempt, so a toast-only
assertion cannot tell them from working software.
Rename asserts the new alias and the target server_id are on the PUT, then
polls /v1/mcp/server until the stored alias matches -- one save has to be
enough. Delete asserts the server is really gone from the list.
Points at an unreachable URL: these exercise litellm's persistence, never the
upstream, so a live MCP server would add a network dependency for nothing.
mcpTools.spec.ts is where a real upstream is needed.
Both pass against a local stack, as do the mcpTools specs from 5e189e9b1a.
Neither reproduced the reported failures on this build -- they guard, they did
not catch.
* test(e2e-ui): verify team create, invite and delete against the API
Three team mutations stopped at a toast, and one of those toasts is matched as
loosely as /success/i -- almost any notification satisfied it.
- create: the team is readable from /team/list and kept the models chosen in
the modal, rather than trusting the UI's own "Team created"
- invite: the invited address really appears in members_with_roles, which is
the point of the flow
- delete: the team is gone from /team/list. The existing assertion was that
the row vanished, which is the client dropping it from local state and
happens whether or not the delete reached the database.
Shapes read off a live proxy: /team/list is a bare array; /team/info nests the
record under team_info. All 6 tests pass locally.
* test(e2e-ui): verify team-admin member and key mutations against the API
The team-admin flows stopped at a success toast. A member add that lands on
the wrong team, a remove that takes out the wrong row, and a key that comes
back unscoped all produce the same toast as the working case, so the existing
assertions could not tell them apart.
Each mutation now pins what went on the wire and reads the result back:
member add/remove assert team_id and the member identifier on the request,
then poll /team/info's roster; the team key asserts team_id on /key/generate
and reads /key/list back to confirm the key is owned by the admin's own team
rather than orphaned.
* test(e2e-ui): verify model add and limit edits against the stored deployment
The Models specs checked the rendered result: the TPM/RPM edit asserted the
new numbers were visible in view mode, and the two add flows asserted a row
showed up in the table. Both render from state the UI already holds, so a save
the backend dropped and a save it kept look the same.
Each mutation now pins the request and reads the deployment back. The limits
edit also asserts the fields it did not touch -- upstream model and team
ownership -- are unchanged, because handleModelUpdate rebuilds and PATCHes the
whole litellm_params blob, which is how an unrelated field gets clobbered by a
save that reports success.
The two add flows assert model_name, the routed model and custom_llm_provider
on the wire and in storage; a deployment that loses its provider looks correct
in the table and is unroutable.
The Team-BYOK test is unchanged -- it is skipped without a license, so any
change to it would be unverified.
* test(e2e-ui): delete the MCP servers these specs create
MCP servers outlive the test that made them, the MCP page contacts every
server it lists, and most of the ones these specs create point at an
unreachable host. They accumulate, and each one makes navigateToPage's
networkidle wait a little slower to settle.
Measured on a local stack: with eleven leaked servers the whole MCP suite
failed on a 30s navigation timeout, including specs that leaked nothing.
Deleting the leftovers made all five pass. With per-test cleanup added, a
run from a clean slate leaves zero behind and takes 20s instead of 1m24s.
mcpServers.spec.ts carried a note that no teardown was needed because the
runner brings up a fresh database each time. That holds for CI and is why
this went unnoticed; it does not hold for a local stack that is reused.
* test(e2e-ui): say why the team-model setup call failed
The setup that creates a team-scoped model asserted a bare `ok()`, so a
failure read "expected true, received false" and pointed at the UI. The call
is enterprise-gated -- creating a model with model_info.team_id returns 403
without LITELLM_LICENSE -- and that is invisible from the old message. It now
carries the status and body, which names the cause immediately.
The api_base also pointed at the mock's default port rather than the one the
harness started; nothing in the test calls the model, but the two should not
disagree.
* test(e2e-ui): add a model through the UI and serve traffic with it
Every existing Add Model test stops at "the row appears in the table",
which a deployment that cannot serve a single request also does. The
manual-QA item this replaces is the whole loop: fill the form, pass Test
Connect, add it, confirm it works.
The new test ends by calling the model it just created. That is the only
assertion that rules out a dropped api_base, a mangled provider prefix,
or a name the router never registers -- all of which look identical in
the UI.
No provider credential is involved. OpenAI-Compatible is the provider
whose form exposes API Base, so the deployment points at the harness's
own mock LLM. The mock speaks the OpenAI wire format, so Test Connect
performs a real completion against a real endpoint and really succeeds.
Also adds teardown for the deployment it creates. A local run throws its
database away, but the deployed stack does not, and a leaked deployment
shows up in every later Models table and /v2/model/info readback.
Both new assertions were mutation-tested: pointing the traffic poll at a
name that was never created fails the test, and the wire assertion fails
when the typed name is not what reaches /model/new.
* test(e2e-ui): print the proxy log when the proxy dies on its own
In E2E_KEEP_ALIVE=1 mode the harness blocks until the proxy pid goes away, then
printed a bare "Proxy exited." and fell straight into cleanup, which rm -f's the
log. The proxy has now exited by itself twice, minutes after a run had finished,
leaving nothing to look at. Both startup failure paths already tail -n 100 the log
before giving up, so this was the one death that stayed silent
Dump the same 100 lines before exiting. A normal Ctrl-C teardown still deletes the
log and prints nothing, which is why INT and TERM now exit instead of running
cleanup and falling back into the wait loop: under the single trap a SIGTERM
deleted the log, resumed the loop, and would then report "tail: no such file",
besides running cleanup twice
* test(e2e-ui): split the log-drawer copy assertions off the expand test
The copy assertions need `navigator.clipboard`, which the browser only
exposes in a secure context. Locally the suite runs against
http://127.0.0.1 and localhost is trustworthy, so it is there. In CI the
run pod is pointed at a plain-HTTP cluster DNS name, where it is
undefined -- and InputCard.handleCopy calls writeText unguarded, so the
click throws before MessageManager.success and no toast ever renders.
That failed all three attempts of litellm-e2e-ui build 10.
Measured rather than inferred: on http://127.0.0.1:4100
isSecureContext/typeof navigator.clipboard are true/"object", and on a
DNS name resolving to that same address they are false/"undefined",
which reproduces the CI failure exactly.
Splitting keeps the drawer-rendering coverage running everywhere and
confines the skip to the part the browser has actually switched off. The
copy assertions still run in full wherever the origin is trustworthy.
The underlying product behaviour is left alone deliberately: any
deployment served over plain HTTP on a hostname has a copy button that
throws and gives no feedback, and that deserves its own fix rather than
being papered over from a test.
* test(e2e-ui): cut the added comments back to what the code cannot say itself
Greptile flagged the helper commentary, and it was right: CLAUDE.md says not to
write comments unless they explain very complex business logic, and much of what
was added here narrated ordinary test setup and motivation instead.
Trims 310 comment lines across the 14 files this branch touched. Kept only the
notes that record something unrecoverable from the code: why the request
listener is armed before the click, why a locator walks up the DOM, why an
assertion exists beyond the toast. Pre-existing comments are left alone.
No behaviour change. The only non-comment hunk is a prettier reformat.
Pure rename, no behavior change. create_mcp_server.tsx and its test move
to CreateMCPServer, the two importers and one stale e2e comment follow,
and the local/filename-pascal-case suppression drops now that the file
passes the rule on its own.
The rename is scoped to this one component rather than the whole
directory because three PRs are currently open against its snake_case
siblings; the rest can follow once those land.
A keyless internal user signing in to the Admin UI was redirected off the
post-login landing to /ui/connect, which renders nothing but the MCP apps panel,
so a plain gateway sign-in ended on an MCP OAuth surface the user never asked
for. The landing now renders the keys dashboard for every role. The key lookup
that existed only to make that routing decision goes with it, along with the
useKeys enabled flag it was the sole caller of and the role-hydration hold that
guarded its one-frame dashboard flash
The gateway DCR consent flow moves the other way. Its /authorize handed the
browser to /ui/chat/integrations, whose layout hard-blocks when enable_chat_ui
is off, which is the default, and client-side redirects to /ui/ without the
query string; that destroys the connect_flow handle and strands the MCP client
until the 600s flow cookie expires. It now lands on /ui/connect, which reads
connect_flow and connect_client, mounts the consent banner and puts the apps
panel in connect mode. /ui/chat/integrations keeps its connect-mode handling
this release so flows sealed before the deploy still finish
Resolves LIT-5104
Resolves LIT-4911
* test(e2e): let the ui suite run from a read-only cwd
The playwright suite never executed on stage. It died in globalSetup before a
single test ran, and the reported error was a red herring.
/app/e2e/ui is a read-only filesystem in the packaged e2e image (the image
runner already redirects playwright's own artifacts to TMPDIR for this reason),
but the suite wrote three things relative to cwd: the per-role storageState
files, the failure-screenshot directory, and the html report. Reproduced in the
pod: storageState raises EROFS, mkdir test-results raises ENOENT.
Worse, the catch block that exists to capture a screenshot threw its own ENOENT
while handling a failure, so the real login error was replaced by a filesystem
error. That is why the run looked like a missing directory rather than whatever
actually went wrong.
Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to
keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort
so it can never mask the underlying failure, and point playwright's reporter and
outputDir at the same place so a bare `npx playwright test` works there too.
fixtures/users.ts had its own copy of the five storageState filenames; it now
re-exports the ones from constants so the paths have a single definition.
Verified in the read-only pod: both writes fail before, both succeed after.
85 tests enumerate and tsc --noEmit is clean.
Refs LIT-4821
* fix(e2e): create the ui artifact root before writing into it
storageState() does not create missing parents, and nothing created ARTIFACT_DIR
itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet
therefore failed with ENOENT on the very first role's snapshot, before any UI
test ran; the same class of failure the artifact-dir change was meant to remove,
just moved one level up.
Reproduced: writing admin.storageState.json into a missing directory raises
ENOENT. My earlier pod verification masked this because the probe called
mkdirSync itself, which the real code path never did.
mkdir the root once at the top of globalSetup, before the login loop. recursive
makes it idempotent, handles nested paths, and keeps the default "." a no-op.
Playwright creates its own outputDir lazily, so globalSetup is the only place
that needs this, and migration.serverRootPath.globalSetup delegates here so it is
covered too.
* test(e2e): skip the mid-conversation cache checks pending LIT-4873
A mid-conversation role="system" reminder invalidates the prompt cache on the
vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder
turn, same conversation shape throughout:
direct to api.anthropic.com 7013 read cache preserved
litellm -> anthropic/claude-opus-4-8 7013 read cache preserved
litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed
and the Vertex control with the same added assistant/user turns but no reminder
reads 7013, so it is the reminder on the non-first-party paths and not the extra
turns. Anthropic keeping the cache rules out provider behavior; litellm's
first-party anthropic path keeping it rules out the shared Messages transform.
That makes these assertions correct and the failure a real billing bug, so the
tests are skipped rather than weakened; the bodies stay intact and must be
restored unchanged with the fix. Registry rows are left in place, so the three
mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps.
Skips are decorators rather than a pytest.skip() inside the shared helper: a
mid-function skip fires only after setup has already registered a real
deployment via /model/new and left the rest of the body unreachable.
Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are
inferred from matching nightly failures and should be confirmed with the fix.
Refs LIT-4821, LIT-4873
Both specs assert against UI that has since moved, so they fail on selectors
rather than on behavior.
The MCP discovery modal became a shadcn/Base UI dialog when mcp-servers
migrated off antd, so `.ant-modal` no longer matches it; locate it by its
dialog role instead. The create form below it is still an antd Modal and keeps
its existing locator.
The no-team internal user has no keys, and a keyless non-admin is now sent to
/ui/connect on the post-login landing, which has no sidebar. Wait for that
redirect to settle, then navigate to the keys page explicitly; the redirect is
gated on the ?login=success marker that the fresh navigation drops, so the
dashboard sticks and the rest of the test is unchanged.
* fix(proxy): restore atomic user upsert when adding team members
Parallel /team/new calls naming the same not-yet-existing member were
returning 500 "Unique constraint failed on the fields: (`user_id`)".
The upsert in add_new_member passed an empty update branch. Prisma only
compiles an upsert down to a single INSERT ... ON CONFLICT when that branch
writes something; with an empty one it emits SELECT-then-INSERT instead, so
concurrent requests all read "no such user" and all insert. Postgres
statement logs confirm it: the empty form logs BEGIN/SELECT/INSERT/COMMIT,
the non-empty form logs INSERT ... ON CONFLICT ("user_id") DO UPDATE SET.
Re-state user_id in the update branch as a no-op so the native upsert path
comes back. The teams append stays in the filtered update below it, so an
already-existing member still cannot pick up a duplicate team id.
tests/test_team.py::test_team_new failed 9 of 15 runs against a live proxy
before this and 0 of 15 after. The existing unit test asserted only that
upsert had been called on a mock, so it passed either way; it now pins the
shape of both branches and fails when the update branch goes back to empty.
* test: point the live codex tests at gpt-5.3-codex
OpenAI deprecated gpt-5.2-codex, so test_openai_codex and
test_openai_codex_stream started failing against the live API with
model_not_found. gpt-5.3-codex is the current codex model; both tests pass
on it. The remaining gpt-5.2-codex references in the suite are mocked
transformation tests and are unaffected.
* test(e2e): update models page specs for the shared DataTable
The DataTable migration in #34363 changed three things the models page
specs were pinned to, and five tests went red.
Row click no longer opens the detail view; the Model ID cell owns that
now, so both specs click its `model-id-<id>` test id instead of the row.
The search box placeholder switched from an ASCII "..." to a real
ellipsis, so the specs use getByPlaceholder with a substring instead of
an exact attribute match that punctuation can break again. The results
count moved from `models-results-count` ("Showing 1 - 50 of 137 results")
to the shared pagination's `pagination-range` ("Showing 1-50 of 137").
The Team-BYOK test also filtered rows on the team alias, which the Team
ID column has never rendered in either the old or the new table; it
filters on the team id now, which is what the column actually shows and
what the assertion's own comment intends.
Verified against a local proxy serving a fresh build with the seeded
e2e postgres and mock upstream: all five failing tests pass, and the
full suite is 82 passed / 4 skipped at CI parity (workers=1).
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
