mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
8 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yuneng Jiang
|
b1cc1e8dae
|
test(ui): assert navigation through accessible links and page content | ||
|
tin-berri
|
ff1f21aea9
|
fix(ui): paginate request logs by session groups server-side (#39257)
* fix(ui): paginate request logs by session groups server-side The logs table server-paginated raw spend logs and then collapsed multi-call sessions client-side, so a page could render 3 rows while the footer claimed 25 and sessions straddled pages. Adds an opt-in group_by_session param to /spend/logs/ui that pages and counts one representative row per session (DISTINCT ON, newest non-MCP call), keeps the bounded count contract, enriches whole-session llm/agent composition counts, and deletes the client-side collapse pipeline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve * feat(ui): add a 10 rows-per-page option and default request logs to it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve * fix(ui): key session aggregates per api key in the logs enrichment Grouped pagination splits a reused session id into one row per api key, but the enrichment still aggregated by session_id alone, so both rows showed combined spend and counts. The aggregate query now groups by (session_id, api_key), the count folds into it (the separate group_by query is deleted), and each row reads its own key's totals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve * fix(ui): treat an empty api_key as a real session group value The spend-log schema defaults api_key to an empty string; truthiness guards in the enrichment treated it as missing, so keyless multi-call sessions lost their count and spend. Only None means missing now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
978aa2816b
|
test(e2e/ui): stop the suite failing on things that are not regressions (#39063)
* test(e2e/ui): stop the suite failing on things that are not regressions Five tests in the UI suite fail for reasons that have nothing to do with the product being broken, which is enough to keep the whole leg red. Two need a premium proxy and fail hard without one: Regenerate Key renders disabled when the proxy is unlicensed, and /model/new refuses a team-scoped deployment. Both now skip without LITELLM_LICENSE, the way three other tests in this suite already do. Three consumed a seeded fixture: Delete key, Delete a team and remove a member each destroyed the row they needed, so the retries CI runs with were guaranteed to fail and the suite could not run twice against one database. They now create what they destroy. Top Virtual Keys ranks by spend and every mock deployment costs $0, so which keys make the list came down to how ties happened to sort. It now sends its traffic through a priced deployment and earns its place. * test(e2e/ui): clean up the fixtures these tests create Review caught two leaks: the priced deployment the usage test registers and the user the team-admin test adds both outlived the run, so repeated runs grew shared state that later routing and rosters can see. Also brings in the paginated daily-activity read. /user/daily/activity pages its per-key breakdown and the helper only read the first page, so the usage test spent its full timeout blaming the rollup for a key the rollup wrote. * test(e2e/ui): read the licence from the proxy, not the runner Review pointed out that checking LITELLM_LICENSE in the runner's environment describes the wrong machine: Playwright can be pointed at a proxy configured somewhere else, and then the skip either hides coverage or runs a premium test against an unlicensed target. The admin session JWT already carries the premium_user claim the dashboard itself reads to enable these controls, so both skips now use that. * test(e2e/ui): clean up fixtures on the failure path too Review caught both cleanups sitting at the end of the test body, where a failing assertion skips them, and both discarding the response so a refused delete passed quietly. They move to afterEach and assert the delete landed. The priced deployment matters most: left behind it keeps its custom pricing and goes on changing what later runs route and what they cost. * test(e2e/ui): wait for the priced deployment to become routable The Top Virtual Keys test registered a priced deployment and sent the key's traffic through it on the next line, so on the deployed stack it failed with "no healthy deployments for e2e-usage-priced-...": /model/new had written the row but the router had not picked it up yet. Polls a ping until the deployment answers before the test sends the request it measures, matching what the addModel spec already does for a model added through the UI. A ping that fails writes no spend log, so the retries cannot move the ranking this test asserts. * test(e2e/ui): register fixtures for cleanup before the step that can fail Review found both helpers handing their id back to the caller to record, with a failure-prone call in between: the priced deployment was registered after the routability wait, and the added user after /team/member_add. Either failing left the resource in the shared database with nothing tracking it. Both now take the teardown list and add themselves as soon as the resource exists, so the afterEach removes it however the rest of setup goes. * test(e2e/ui): resolve the priced deployment for teardown by name Review pointed out the remaining gap: /model/new can persist the deployment and still answer non-2xx, and the id was only recorded after the response was asserted, so that path left it behind with its custom pricing. The name is now claimed before the request and teardown looks it up in /model/info, so a create that saved without answering 2xx is still removed and one that never saved is simply not there. * test(e2e/ui): claim the member id before creating the user /user/new can persist the user and still answer non-2xx, and the id is chosen by the test rather than returned by the proxy, so registering it before the call is what closes the last create-failure path. Teardown now skips an id whose user is not there, so claiming it up front cannot fail a run where the create never landed. * test(e2e/ui): wait out the router reload when resolving a deployment to delete /model/info answers from the router, not from the database, and /model/new catches and logs a failed in-request reload while still answering 2xx. A deployment can therefore be persisted and absent from the listing until the next reload, which is where teardown was giving up and leaking it. Teardown now retries the lookup for a little over one PROXY_CONFIG_RELOAD_INTERVAL_SECONDS before treating the name as never persisted, so the only names it skips are the ones that really are not there. * test(e2e/ui): prove a stored credential survives a config reload before using it The Test Connect assertion has been failing intermittently on the full-suite runs. Artifacts from litellm-e2e-ui build 165 show the UI sending litellm_credential_name and the proxy answering with the credential unapplied: raw_request_api_base was https://api.openai.com/v1/ rather than the mock base the credential carries, and the call died on an upstream 404 for the model. The same credential had resolved on three probes eight seconds earlier. The proxy's periodic credential refresh takes a database snapshot, prunes any in-memory credential missing from it, then re-adds the snapshot. A credential created while that is in flight gets pruned and stays gone until the next tick, and load_credentials_from_list fails open onto the ambient key, so nothing in the error names the credential. The existing pre-check asked for three consecutive probe successes, but they completed in under a second, so they could not span a refresh. Space them so the run covers a whole interval, which is what proves the credential survived a refresh and is therefore stable. * test(e2e/ui): find a database-only deployment through the search listing /model/info answers from the router, so a deployment that reached the database while /model/new's in-request reload failed is invisible there, and waiting on the next reload only helps if reconciliation eventually picks it up. /v2/model/info?search= runs a bounded query against the model table and deliberately returns rows the router does not hold, so it resolves those deployments to the id /model/delete needs. Falling back to it removes the wait as well: absent from both listings now means the deployment never persisted. * test(e2e/ui): delete the temporary member without a lookup that can skip it Teardown asked /user/info first and treated any non-2xx as absence, so a transient failure on the lookup silently skipped the delete and left the user behind, which is the leak the claimed id was meant to close. /user/delete answers 404 for an id that is not there, so it can carry both cases on its own: 404 means the create never persisted, and anything else that is not 2xx now fails the teardown instead of passing quietly. * test(e2e/ui): reach the database fallback when the router listing fails Asserting on /model/info threw before the fallback could run, so a failure on the router-backed listing aborted teardown and left the deployment persisted, which is the leak the fallback was added to close. The router listing is best-effort now: an unreadable response just falls through to the search-backed one. That listing is the authoritative answer to whether the deployment exists, so it is the one that has to be readable, and a name missing from it is a create that never persisted. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yuneng Jiang
|
fdc259077e
|
test(e2e/ui): automate 8 manual QA checklist flows
Adds Playwright coverage for the RC checklist items an audit marked automatable today: Playground to Logs hand-off, public Agent/MCP hub tabs, team models in the Playground dropdown via a team key, Add Model with a stored credential, internal user team key creation, a second admin account, team model deletion, and Presidio guardrail CRUD without a live sidecar. Seeds e2e-team-keygen with the /key/generate member permission so the internal user key flow avoids the team-list cache lag |
||
|
ryan-crabbe-berri
|
0b374541bb
|
refactor(ui): migrate the last antd components off antd onto shadcn (#37569)
Converts the remaining dashboard components that still imported antd: admin panel, agents, MCP toolsets, policies, prompts, bulk user edit, create user, plugin settings, teams, add model, auto router, cloudzero export, BYOK credentials, credential modal, onboarding link, create key and routing groups. Primitives map onto the house shadcn set: Typography onto semantic tags, Select onto ui/select, SearchSelect or MultiSelect, Input onto ui/input, Tooltip onto SimpleTooltip, Card, Table, Tabs, Switch, Checkbox, Radio, Tag onto Badge, Divider onto Separator, Spin onto UiLoadingSpinner, Modal onto Dialog, message onto toast, and Space, Row, Col, Flex and Layout onto flex containers. |
||
|
Yuneng Jiang
|
edc1d65558
|
test(e2e/ui): address the migrated dashboard controls by role, not antd classes
The playground, logs drawer and AI Hub modal moved off antd, so the specs that reached for .ant-select, .ant-drawer-content, .ant-modal and .ant-radio-button-wrapper no longer match anything and time out. Address the same controls through their accessible role and name instead, which holds across the component library swap and reads closer to what a user does. |
||
|
yuneng-jiang
|
32a5259407
|
test(e2e-ui): verify UI mutations against the API instead of trusting the toast (#36632)
* test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows
These three pages carried no e2e coverage, so the manual QA checklist was the
only thing standing behind them.
Playground: sends a chat from the UI for both configured models, and for both
virtual-key sources (the logged-in session, and a key pasted into the panel).
This is the only spec that drives the dashboard's own LLM call path rather than
an admin CRUD endpoint.
Logs: a request the proxy actually served appears in the table, its drawer
expands to the real request and response bodies, both copy to the clipboard,
the Input card collapses, the JSON view exposes Request/Response, and the End
User filter narrows the table to one customer.
Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card
toggles between table and chart, and the key opens its key-info panel.
Router settings: the existing spec proved the UI can record a fallback; the new
one proves the fallback is honoured, by pointing a model at an unreachable
upstream and asserting the reply comes back anyway. It asserts the un-fallen-back
call fails first, so a quietly-working primary cannot fake a pass.
Supporting changes:
- helpers/traffic.ts generates the traffic these pages render, rather than
seeding rows no code produced. Its two wait helpers exist because the Logs
and Usage pages read different stores: spend logs are flushed on a timer, and
the Usage page reads a background rollup *and* fetches once on mount, so
waiting on the DOM there can never converge.
- helpers/playground.ts holds the playground controls, now shared with the
fallback spec. Everything is scoped to the visible copy of the config panel,
which is rendered twice for the docked and collapsed layouts.
- run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a
spec can be re-run against it without paying for a UI rebuild each iteration.
Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped.
* test(e2e-ui): cover listing and calling MCP tools
Covers the two MCP manual-QA items the create-only spec cannot reach:
opening a server's tool list, and calling a tool and seeing its result.
Both need an MCP server that actually answers, so this points at DeepWiki's
public MCP server -- Streamable HTTP, auth None, so there is no credential to
hold and nothing to leak from a public repo. The call is made by the proxy,
not the browser; nothing in the e2e chart restricts that egress.
The external dependency is real and is left visible: an upstream outage turns
these red rather than auto-skipping, because a spec that skips itself on
connection trouble also skips when the proxy's MCP client is what broke.
E2E_SKIP_EXTERNAL_MCP=1 is the explicit opt-out.
Not yet executed against a live stack.
* test(e2e-ui): verify key mutations round-trip instead of trusting the toast
The recurring customer report is a form that says "Saved!" and then either
no-ops or clobbers an unrelated field. A toast-only assertion passes in both
cases, and outside three specs that is all this suite checks.
Adds helpers/roundTrip.ts, factoring out the idiom clearCustomPricing,
credentials and routerSettings already use: capture the outgoing request body,
then read the resource back through the management API.
Applies it to the keys spec:
- create: the key is readable from /key/list and owns a team_id, rather than
trusting a table row rendered from the create response the UI already held
- update limits: TPM/RPM are on the wire AND persisted, and the key's models
and team are unchanged -- bumping one field wiping another is the reported
failure mode (PR #34452), not a hypothetical
- delete: the key is gone from /key/list, not merely toasted as deleted
- regenerate: the stored token actually changed
/key/list shape is per KeyListResponseObject in litellm/proxy/_types.py.
Not yet executed: ports 4000/8090 are held by a parallel run.
* test(e2e-ui): let the local harness run on non-default ports
Two checkouts cannot run run_e2e.sh at the same time: it hardcodes 4000/5432/
8090, so the second aborts on "port 4000 is in use" and the only way forward is
to stop someone else's stack.
PROXY_PORT / POSTGRES_PORT / MOCK_LLM_PORT now override those, defaulting to the
historical values so an unset environment behaves exactly as before -- CI, the
CircleCI job and the chart's sidecar all keep working untouched.
Two details that would otherwise make a relocated stack fail confusingly:
- the suite resolves its target from E2E_UI_BASE_URL, which defaults to :4000
independently, so the run would build and boot correctly and then test
whatever was on the default port. run_e2e.sh now derives it.
- the mock server binds its port in server.py, so moving it needs MOCK_LLM_PORT
there too. Its HOST stays loopback-only: 127.0.0.1:8090 from inside the
proxy's own pod is the contract the e2e chart's sidecar is written against.
* test(e2e-ui): cover MCP server edit and delete, verified via the API
mcpServers.spec.ts only ever creates a server, and creation is the one MCP
operation nobody has complained about. The reports are all on the other side:
an alias rename that needs three or four saves to take, a delete that needs two
attempts. Both produce a success toast on the failing attempt, so a toast-only
assertion cannot tell them from working software.
Rename asserts the new alias and the target server_id are on the PUT, then
polls /v1/mcp/server until the stored alias matches -- one save has to be
enough. Delete asserts the server is really gone from the list.
Points at an unreachable URL: these exercise litellm's persistence, never the
upstream, so a live MCP server would add a network dependency for nothing.
mcpTools.spec.ts is where a real upstream is needed.
Both pass against a local stack, as do the mcpTools specs from
|
||
|
ryan-crabbe-berri
|
0fcaadf11c
|
test(e2e): move Admin UI Playwright suite to tests/e2e/ui (#34196)
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end suites live under tests/e2e. The suite stays in TypeScript and becomes a self-contained npm package with its own package.json, lockfile and tsconfig instead of leaning on the dashboard's toolchain; the dashboard drops its @playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs. CI paths follow the move: both CircleCI jobs (main e2e and the SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now install and run Playwright from tests/e2e/ui, with the node cache keyed on both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec edits keep skipping backend jobs. The suite's mock LLM fixture is excluded from the e2e basedpyright zero-error gate in pyrightconfig.json since it belongs to the TS suite, not the typed Python harness. |