From 7675ba8717c920d9699ce8eda7a6666789e82dcd Mon Sep 17 00:00:00 2001 From: yuneng-jiang Date: Wed, 19 Aug 2026 13:40:14 -0700 Subject: [PATCH] test(ui): split the vitest suite into unit, component, integration and type projects (#37488) * test(ui): split the vitest suite into unit, component and integration tiers Every test file booted jsdom, including the ~1800 that assert pure functions and never render. They now run as a separate vitest project in the node environment, where the whole tier finishes in under four seconds. The tiers are vitest projects rather than a naming convention, so CI can run them as independent jobs. A .test.ts that renders React, a hook test being the usual case, is listed explicitly and stays in the jsdom tier. * test(ui): report per-test duration against a per-tier budget A timeout only catches a hung test, and it has to stay generous enough to survive a loaded runner, so it never reports the multi-second render tests that make CI fail the moment the box is busy. Budgets are separate and far tighter: 50ms unit, 1s component, 3s integration. The counts are laptop measurements, so the CI job is report-only for now. Flipping it to blocking is one line once CI has published its own numbers. * test(ui): run the tiers as separate CI jobs and stop clicking popups by text The old job ran every file in one process, so the single slowest file set the wall clock and a bigger box bought nothing. The tiers now run as separate jobs with the component tier sharded four ways. getByText and findByText match hidden nodes, so they resolve against a closed Base UI popup whose positioner still carries pointer-events: none, and the click lands or not depending on how far the open transition got. Two files failed this way, one three runs in five and one every run. Querying the option by role waits for it to be visible, and both are now stable. A lint rule keeps the pattern from coming back. * test(ui): give React Testing Library's async queries a CI-sized window findBy* and waitFor run on asyncUtilTimeout, which defaults to 1000ms and is independent of vitest's testTimeout. Raising the vitest timeout therefore did nothing for them: a query still gave up after one second while the test had 59 seconds of budget left, which is why a loaded runner produced 'Unable to find role=...' rather than a timeout. UserSearchModal is the worked example. The role query it makes resolves in 249ms on a laptop and blew past 1000ms on CI, failing the run at 1494ms. Five seconds keeps the same assertions and only widens the window a failing query waits before reporting; a passing query still resolves the moment the element appears. * test(ui): calibrate the tier budgets from real CI numbers and report by default The first CI run showed the laptop counts were badly off: component 176 local against 326 on CI, integration 87 against 128. The maxima now come from that run with headroom. continue-on-error still painted the check red, which is the opposite of the point, so the report-only decision moves into test-budgets.json as an explicit enforce flag. The job passes and prints the counts; flipping enforce to true makes it a gate. * docs(ui): drop the CLAUDE.md edits from the tier split Keeping this PR to the vitest, CI and test changes. * ci(ui): run every tier in one job instead of eight check rows Sharding bought nothing. Measured on the first run of this branch, the component tier unsharded finishes in 198s while the integration tier is floored at 384s by a single file, so integration was always the critical path and the four component shards only added rows. One job running every project comes in around 384s against the 426s the split jobs took. Eight rows named things like 'component (2)' also told a reviewer nothing, on a PR page that already carries forty checks. The job keeps the id ui-unit-tests because guard-internal-staging requires that exact context; renaming the jobs had silently stopped it reporting, which would have blocked every merge on a check that no longer existed. The workflow's display name becomes UI Tests since it runs more than unit tests. The tier split itself is untouched: it lives in the vitest projects config, so the unit tier still runs in node with no jsdom, and each tier keeps its own timeout and budget. * fix(ui): stop the type check from running the whole suite a second time test:types was 'vitest --run --typecheck.only'. Under test.projects that flag is ignored and the root-level typecheck block is not inherited, so the step collected each project's normal include and ran all 8464 runtime tests instead of type-checking. It took 542s on CI against 33s on the flat config it replaced, and the job then ran the same suite again in the next step. Typecheck now belongs to a project of its own, with an empty include so it contributes no runtime tests, and the CI job runs one vitest invocation for all four. The type tier adds about 3s to a full run and reports 'Type Errors: no errors' rather than a suite of tests. Verified it still catches things: breaking SortingState in DataTable.test-d.tsx fails with 'Type number is not assignable to type string' and exit 1, and restoring it passes. * test(ui): scope the split down to the vitest tier projects Removes everything from this branch that was not the tier split. The three lint rules brought 1381 lines of grandfathered suppressions in eslint-suppressions.json, which is 81% of the branch's added lines and debt nobody is going to pay down. The per-test duration budget does not scale as a CI step. Both are gone, along with the two query rewrites the no-click-by-text rule forced: those files pass 10/10 at this base, quiet and under load, so there was no failure behind them. The workflow is byte-identical to the base again. It already runs npm run test:types and then vitest related on pull requests, so PR cost is unchanged; the split only repoints test:types at the new project. That project is required, not optional: vitest silently ignores --typecheck.only under test.projects, so without it the type script collects the whole suite instead of the one typed file. Restores the base 60s testTimeout on the unit tier. The 5s cap was not part of the split and failed ChatShell.serverRootPath.test.ts, a 960ms test, under load. --- ui/litellm-dashboard/package.json | 5 +- .../multi_export_utils.test.ts | 2 + .../_components/createOAuthUiState.test.ts | 2 + .../EntityUsageExport/utils.test.ts | 2 + .../src/utils/cookieUtils.test.ts | 2 + .../src/utils/dataUtils.test.ts | 2 + .../src/utils/localStorageUtils.test.ts | 2 + .../src/utils/mcpTokenStore.test.ts | 2 + .../src/utils/returnUrlUtils.test.ts | 2 + ui/litellm-dashboard/tests/setup.unit.ts | 12 +++ ui/litellm-dashboard/vitest.config.ts | 101 +++++++++++++----- 11 files changed, 104 insertions(+), 30 deletions(-) create mode 100644 ui/litellm-dashboard/tests/setup.unit.ts diff --git a/ui/litellm-dashboard/package.json b/ui/litellm-dashboard/package.json index 9407ff25a1f..ac13a12620d 100644 --- a/ui/litellm-dashboard/package.json +++ b/ui/litellm-dashboard/package.json @@ -9,8 +9,11 @@ "start": "next start", "lint": "eslint .", "test": "vitest", + "test:unit": "vitest run --project unit", + "test:component": "vitest run --project component", + "test:integration": "vitest run --project integration", "test:dot": "vitest --reporter=dot", - "test:types": "vitest --run --typecheck.only", + "test:types": "vitest run --project types", "test:watch": "vitest -w", "test:coverage": "vitest run --coverage", "format": "prettier --write .", diff --git a/ui/litellm-dashboard/src/app/(dashboard)/cost-tracking/_components/pricing_calculator/multi_export_utils.test.ts b/ui/litellm-dashboard/src/app/(dashboard)/cost-tracking/_components/pricing_calculator/multi_export_utils.test.ts index 3c319e5ba18..25b4da869bb 100644 --- a/ui/litellm-dashboard/src/app/(dashboard)/cost-tracking/_components/pricing_calculator/multi_export_utils.test.ts +++ b/ui/litellm-dashboard/src/app/(dashboard)/cost-tracking/_components/pricing_calculator/multi_export_utils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import { exportMultiToPDF, exportMultiToCSV } from "./multi_export_utils"; import type { MultiModelResult } from "./types"; diff --git a/ui/litellm-dashboard/src/app/(dashboard)/mcp-servers/_components/createOAuthUiState.test.ts b/ui/litellm-dashboard/src/app/(dashboard)/mcp-servers/_components/createOAuthUiState.test.ts index e2e7814f0d7..94544ec6e2d 100644 --- a/ui/litellm-dashboard/src/app/(dashboard)/mcp-servers/_components/createOAuthUiState.test.ts +++ b/ui/litellm-dashboard/src/app/(dashboard)/mcp-servers/_components/createOAuthUiState.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { beforeEach, describe, expect, it, vi } from "vitest"; import { setSecureItem } from "@/utils/secureStorage"; import { CreateUiSnapshot, readCreateUiSnapshot, writeCreateUiSnapshot } from "./createOAuthUiState"; diff --git a/ui/litellm-dashboard/src/components/EntityUsageExport/utils.test.ts b/ui/litellm-dashboard/src/components/EntityUsageExport/utils.test.ts index d80294aa2e6..7ed014d43b2 100644 --- a/ui/litellm-dashboard/src/components/EntityUsageExport/utils.test.ts +++ b/ui/litellm-dashboard/src/components/EntityUsageExport/utils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import type { DateRangePickerValue } from "@/components/shared/date_picker_types"; import Papa from "papaparse"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; diff --git a/ui/litellm-dashboard/src/utils/cookieUtils.test.ts b/ui/litellm-dashboard/src/utils/cookieUtils.test.ts index cc4fe64e4c8..a38b96e74c2 100644 --- a/ui/litellm-dashboard/src/utils/cookieUtils.test.ts +++ b/ui/litellm-dashboard/src/utils/cookieUtils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { describe, it, expect, beforeEach, vi } from "vitest"; import { clearTokenCookies, getCookie, storeLoginToken } from "./cookieUtils"; import { getToken, setToken } from "./mcpTokenStore"; diff --git a/ui/litellm-dashboard/src/utils/dataUtils.test.ts b/ui/litellm-dashboard/src/utils/dataUtils.test.ts index d975a4a73f4..f696780bc0a 100644 --- a/ui/litellm-dashboard/src/utils/dataUtils.test.ts +++ b/ui/litellm-dashboard/src/utils/dataUtils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { describe, it, expect, beforeEach, vi, afterEach } from "vitest"; import { copyToClipboard, formatNumberWithCommas, getSpendString, updateExistingKeys } from "./dataUtils"; diff --git a/ui/litellm-dashboard/src/utils/localStorageUtils.test.ts b/ui/litellm-dashboard/src/utils/localStorageUtils.test.ts index 934c358cb74..2ef5e1d5b43 100644 --- a/ui/litellm-dashboard/src/utils/localStorageUtils.test.ts +++ b/ui/litellm-dashboard/src/utils/localStorageUtils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { describe, it, expect, vi, beforeEach } from "vitest"; import { emitLocalStorageChange, diff --git a/ui/litellm-dashboard/src/utils/mcpTokenStore.test.ts b/ui/litellm-dashboard/src/utils/mcpTokenStore.test.ts index 0d096677f0d..e384f41042a 100644 --- a/ui/litellm-dashboard/src/utils/mcpTokenStore.test.ts +++ b/ui/litellm-dashboard/src/utils/mcpTokenStore.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { clearAllMcpTokens, getToken, isTokenValid, removeToken, setToken } from "./mcpTokenStore"; diff --git a/ui/litellm-dashboard/src/utils/returnUrlUtils.test.ts b/ui/litellm-dashboard/src/utils/returnUrlUtils.test.ts index 0f3a8b4bf69..3d1e83ad236 100644 --- a/ui/litellm-dashboard/src/utils/returnUrlUtils.test.ts +++ b/ui/litellm-dashboard/src/utils/returnUrlUtils.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment jsdom + import { buildLoginUrlWithReturn, clearStoredReturnUrl, diff --git a/ui/litellm-dashboard/tests/setup.unit.ts b/ui/litellm-dashboard/tests/setup.unit.ts new file mode 100644 index 00000000000..4eb0c473989 --- /dev/null +++ b/ui/litellm-dashboard/tests/setup.unit.ts @@ -0,0 +1,12 @@ +import { vi } from "vitest"; + +vi.mock("@/lib/toast", () => ({ + toast: { + success: vi.fn(), + info: vi.fn(), + warning: vi.fn(), + error: vi.fn(), + fromError: vi.fn(), + dismiss: vi.fn(), + }, +})); diff --git a/ui/litellm-dashboard/vitest.config.ts b/ui/litellm-dashboard/vitest.config.ts index 7d8825909d8..ca2a25cba9f 100644 --- a/ui/litellm-dashboard/vitest.config.ts +++ b/ui/litellm-dashboard/vitest.config.ts @@ -12,17 +12,81 @@ const staticImageData: Plugin = { }, }; -const config: ViteUserConfig = { +const sharedViteConfig = { plugins: [staticImageData], + resolve: { alias: { "@": resolve(__dirname, "src") } }, + define: { "import.meta.vitest": "undefined" }, + esbuild: { jsx: "automatic", jsxImportSource: "react" } as const, +}; + +const TEST_TS_FILES_THAT_RENDER_REACT: readonly string[] = [ + "src/**/hooks/**/*.test.ts", + "src/**/cost-tracking/_components/**/use_*.test.ts", + "src/**/models-and-endpoints/detailNavigation.test.ts", + "src/**/models-and-endpoints/vertexCredentialsUpload.test.ts", + "src/components/chat/useChatHistory.test.ts", + "src/lib/forms/pickDirty.test.ts", +]; + +const jsdomTier = { + environment: "./tests/jsdomFetchEnv.ts", + setupFiles: ["tests/setupTests.ts"], + globals: true, + css: true, + testTimeout: 60_000, + hookTimeout: 30_000, +}; + +const config: ViteUserConfig = { + ...sharedViteConfig, test: { - environment: "./tests/jsdomFetchEnv.ts", - setupFiles: ["tests/setupTests.ts"], - globals: true, - css: true, // lets you import CSS/modules without extra mocks - testTimeout: 60000, - hookTimeout: 30000, + projects: [ + { + ...sharedViteConfig, + test: { + name: "unit", + environment: "node", + setupFiles: ["tests/setup.unit.ts"], + globals: true, + testTimeout: 60_000, + hookTimeout: 30_000, + include: ["src/**/*.test.ts", "tests/**/*.test.ts"], + exclude: ["node_modules/**", ...TEST_TS_FILES_THAT_RENDER_REACT], + }, + }, + { + ...sharedViteConfig, + test: { + ...jsdomTier, + name: "component", + include: ["src/**/*.test.tsx", "tests/**/*.test.tsx", ...TEST_TS_FILES_THAT_RENDER_REACT], + exclude: ["node_modules/**", "**/*.integration.test.tsx"], + }, + }, + { + ...sharedViteConfig, + test: { + ...jsdomTier, + name: "integration", + include: ["src/**/*.integration.test.tsx", "tests/**/*.integration.test.tsx"], + exclude: ["node_modules/**"], + }, + }, + { + ...sharedViteConfig, + test: { + name: "types", + include: [], + typecheck: { + enabled: true, + include: ["src/**/*.test-d.ts", "src/**/*.test-d.tsx"], + ignoreSourceErrors: true, + }, + }, + }, + ], silent: process.env.CI ? "passed-only" : false, - teardownTimeout: 60000, + teardownTimeout: 60_000, coverage: { provider: "v8", reporter: ["text", "lcov"], @@ -32,37 +96,16 @@ const config: ViteUserConfig = { "**/*.test.*", "**/*.test-d.*", "**/*.spec.*", - "tests/**", - "node_modules/**", ".next/**", "out/**", - "**/*.config.*", "postcss.config.*", "tailwind.config.*", "next.config.*", ], }, - exclude: ["node_modules/**"], - include: ["src/**/*.test.ts", "src/**/*.test.tsx", "tests/**/*.test.ts", "tests/**/*.test.tsx"], - typecheck: { - include: ["src/**/*.test-d.ts", "src/**/*.test-d.tsx"], - ignoreSourceErrors: true, - }, - }, - resolve: { - alias: { - "@": resolve(__dirname, "src"), - }, - }, - define: { - "import.meta.vitest": "undefined", - }, - esbuild: { - jsx: "automatic", - jsxImportSource: "react", }, };