mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
328e41b1f9
* test(e2e): let the ui suite run from a read-only cwd The playwright suite never executed on stage. It died in globalSetup before a single test ran, and the reported error was a red herring. /app/e2e/ui is a read-only filesystem in the packaged e2e image (the image runner already redirects playwright's own artifacts to TMPDIR for this reason), but the suite wrote three things relative to cwd: the per-role storageState files, the failure-screenshot directory, and the html report. Reproduced in the pod: storageState raises EROFS, mkdir test-results raises ENOENT. Worse, the catch block that exists to capture a screenshot threw its own ENOENT while handling a failure, so the real login error was replaced by a filesystem error. That is why the run looked like a missing directory rather than whatever actually went wrong. Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort so it can never mask the underlying failure, and point playwright's reporter and outputDir at the same place so a bare `npx playwright test` works there too. fixtures/users.ts had its own copy of the five storageState filenames; it now re-exports the ones from constants so the paths have a single definition. Verified in the read-only pod: both writes fail before, both succeed after. 85 tests enumerate and tsc --noEmit is clean. Refs LIT-4821 * fix(e2e): create the ui artifact root before writing into it storageState() does not create missing parents, and nothing created ARTIFACT_DIR itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet therefore failed with ENOENT on the very first role's snapshot, before any UI test ran; the same class of failure the artifact-dir change was meant to remove, just moved one level up. Reproduced: writing admin.storageState.json into a missing directory raises ENOENT. My earlier pod verification masked this because the probe called mkdirSync itself, which the real code path never did. mkdir the root once at the top of globalSetup, before the login loop. recursive makes it idempotent, handles nested paths, and keeps the default "." a no-op. Playwright creates its own outputDir lazily, so globalSetup is the only place that needs this, and migration.serverRootPath.globalSetup delegates here so it is covered too. * test(e2e): skip the mid-conversation cache checks pending LIT-4873 A mid-conversation role="system" reminder invalidates the prompt cache on the vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder turn, same conversation shape throughout: direct to api.anthropic.com 7013 read cache preserved litellm -> anthropic/claude-opus-4-8 7013 read cache preserved litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed and the Vertex control with the same added assistant/user turns but no reminder reads 7013, so it is the reminder on the non-first-party paths and not the extra turns. Anthropic keeping the cache rules out provider behavior; litellm's first-party anthropic path keeping it rules out the shared Messages transform. That makes these assertions correct and the failure a real billing bug, so the tests are skipped rather than weakened; the bodies stay intact and must be restored unchanged with the fix. Registry rows are left in place, so the three mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps. Skips are decorators rather than a pytest.skip() inside the shared helper: a mid-function skip fires only after setup has already registered a real deployment via /model/new and left the rest of the body unreachable. Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are inferred from matching nightly failures and should be confirmed with the fix. Refs LIT-4821, LIT-4873 |
||
|---|---|---|
| .. | ||
| fixtures | ||
| helpers | ||
| tests | ||
| constants.ts | ||
| globalSetup.ts | ||
| migration.serverRootPath.config.ts | ||
| migration.serverRootPath.globalSetup.ts | ||
| package-lock.json | ||
| package.json | ||
| playwright.config.ts | ||
| run_e2e.sh | ||
| tsconfig.json | ||