mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
328e41b1f9
* test(e2e): let the ui suite run from a read-only cwd The playwright suite never executed on stage. It died in globalSetup before a single test ran, and the reported error was a red herring. /app/e2e/ui is a read-only filesystem in the packaged e2e image (the image runner already redirects playwright's own artifacts to TMPDIR for this reason), but the suite wrote three things relative to cwd: the per-role storageState files, the failure-screenshot directory, and the html report. Reproduced in the pod: storageState raises EROFS, mkdir test-results raises ENOENT. Worse, the catch block that exists to capture a screenshot threw its own ENOENT while handling a failure, so the real login error was replaced by a filesystem error. That is why the run looked like a missing directory rather than whatever actually went wrong. Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort so it can never mask the underlying failure, and point playwright's reporter and outputDir at the same place so a bare `npx playwright test` works there too. fixtures/users.ts had its own copy of the five storageState filenames; it now re-exports the ones from constants so the paths have a single definition. Verified in the read-only pod: both writes fail before, both succeed after. 85 tests enumerate and tsc --noEmit is clean. Refs LIT-4821 * fix(e2e): create the ui artifact root before writing into it storageState() does not create missing parents, and nothing created ARTIFACT_DIR itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet therefore failed with ENOENT on the very first role's snapshot, before any UI test ran; the same class of failure the artifact-dir change was meant to remove, just moved one level up. Reproduced: writing admin.storageState.json into a missing directory raises ENOENT. My earlier pod verification masked this because the probe called mkdirSync itself, which the real code path never did. mkdir the root once at the top of globalSetup, before the login loop. recursive makes it idempotent, handles nested paths, and keeps the default "." a no-op. Playwright creates its own outputDir lazily, so globalSetup is the only place that needs this, and migration.serverRootPath.globalSetup delegates here so it is covered too. * test(e2e): skip the mid-conversation cache checks pending LIT-4873 A mid-conversation role="system" reminder invalidates the prompt cache on the vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder turn, same conversation shape throughout: direct to api.anthropic.com 7013 read cache preserved litellm -> anthropic/claude-opus-4-8 7013 read cache preserved litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed and the Vertex control with the same added assistant/user turns but no reminder reads 7013, so it is the reminder on the non-first-party paths and not the extra turns. Anthropic keeping the cache rules out provider behavior; litellm's first-party anthropic path keeping it rules out the shared Messages transform. That makes these assertions correct and the failure a real billing bug, so the tests are skipped rather than weakened; the bodies stay intact and must be restored unchanged with the fix. Registry rows are left in place, so the three mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps. Skips are decorators rather than a pytest.skip() inside the shared helper: a mid-function skip fires only after setup has already registered a real deployment via /model/new and left the rest of the body unreachable. Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are inferred from matching nightly failures and should be confirmed with the fix. Refs LIT-4821, LIT-4873
48 lines
2.4 KiB
TypeScript
48 lines
2.4 KiB
TypeScript
import * as path from "path";
|
|
|
|
export const UI_BASE_URL = (
|
|
process.env.E2E_UI_BASE_URL ||
|
|
process.env.LITELLM_PROXY_URL ||
|
|
"http://localhost:4000"
|
|
).replace(/\/+$/, "");
|
|
|
|
// Directory every artifact this suite writes must land in: storage states,
|
|
// failure screenshots, playwright output. A local `./run_e2e.sh` run has a
|
|
// writable cwd, so it keeps the historical behavior of writing beside the
|
|
// suite. In the packaged e2e image the suite ships on a read-only filesystem
|
|
// (/app/e2e/ui), so bare relative paths raise EROFS/ENOENT and the run dies in
|
|
// globalSetup before a single test executes. Point E2E_UI_ARTIFACT_DIR at a
|
|
// writable path (the image runner already exports TMPDIR) to relocate them.
|
|
export const ARTIFACT_DIR = process.env.E2E_UI_ARTIFACT_DIR || ".";
|
|
|
|
const storagePath = (name: string): string => path.join(ARTIFACT_DIR, name);
|
|
|
|
// Storage state paths for each role
|
|
export const ADMIN_STORAGE_PATH = storagePath("admin.storageState.json");
|
|
export const ADMIN_VIEWER_STORAGE_PATH = storagePath("adminViewer.storageState.json");
|
|
export const INTERNAL_USER_STORAGE_PATH = storagePath("internalUser.storageState.json");
|
|
export const INTERNAL_VIEWER_STORAGE_PATH = storagePath("internalViewer.storageState.json");
|
|
export const TEAM_ADMIN_STORAGE_PATH = storagePath("teamAdmin.storageState.json");
|
|
|
|
// Seeded user identities (match seed.sql)
|
|
export const E2E_PROXY_ADMIN_USER_ID = "e2e-proxy-admin";
|
|
export const E2E_PROXY_ADMIN_EMAIL = "admin@test.local";
|
|
export const E2E_INTERNAL_USER_ID = "e2e-internal-user";
|
|
export const E2E_INTERNAL_USER_EMAIL = "internal@test.local";
|
|
|
|
// Key aliases for seeded test keys (match seed.sql)
|
|
export const E2E_UPDATE_LIMITS_KEY_ALIAS = "e2eUpdateLimitsKey";
|
|
export const E2E_DELETE_KEY_ALIAS = "e2eDeleteKey";
|
|
export const E2E_REGENERATE_KEY_ALIAS = "e2eRegenerateKey";
|
|
export const E2E_INTERNAL_USER_KEY_ALIAS = "e2eInternalUserKey";
|
|
export const E2E_VIEWER_KEY_ALIAS = "e2eViewerKey";
|
|
|
|
// Team identifiers (match seed.sql)
|
|
export const E2E_TEAM_CRUD_ID = "e2e-team-crud";
|
|
export const E2E_TEAM_CRUD_ALIAS = "E2E Team CRUD";
|
|
export const E2E_TEAM_DELETE_ID = "e2e-team-delete";
|
|
export const E2E_TEAM_DELETE_ALIAS = "E2E Team Delete";
|
|
export const E2E_TEAM_ORG_ID = "e2e-team-org";
|
|
export const E2E_TEAM_ORG_ALIAS = "E2E Team In Org";
|
|
export const E2E_TEAM_NO_ADMIN_ID = "e2e-team-no-admin";
|
|
export const E2E_TEAM_NO_ADMIN_ALIAS = "E2E Team No Admin";
|