The absence of a CI signal is indistinguishable from a passing one, and
that shape has now produced several independent holes: whole test
directories no job runs, and shipped images no job builds. Nothing was
watching for either, so each was found by accident.
assert_ci_coverage.py enumerates every test_*.py under tests/ and every
Dockerfile in the repo, then credits only what the workflows and the
CircleCI config actually invoke. Paths filters and lint steps that merely
name a directory do not count as coverage, because crediting a mention is
the same mistake one level up. Anything neither invoked nor listed in
.github/ci-coverage-allowlist.yml with a written reason fails the job.
The guard found 275 uncovered test files and 6 unbuilt Dockerfiles. Fixed
here: the root Dockerfile, the primary published image, now gets an
image-scan leg that builds it and runs the offline migration check against
it, and 8 tests/test_litellm subdirectories join the shards that already
enumerate their siblings. Everything else is allowlisted per file, so a
new file cannot inherit an exemption, and the remaining decisions are
tracked rather than invisible.
The job reports but is not in branch protection, so it does not block
merges; promoting it is a separate change once the allowlist has survived
contact with a few pull requests.
heal_incomplete_nodeenv_cache() stats a $HOME-derived path with a bare
Path.is_dir(). pathlib only swallows ENOENT-shaped errnos, so a cache
directory the process cannot search raises PermissionError instead of
answering False. Images bake that cache under the build user's home, whose
mode is 0700, so a container started under any other uid dies there before
the Prisma CLI is ever invoked, and the migration never runs.
Tolerate OSError while inspecting the cache, matching the guard
nodeenv_cache_dir() already carries, so an unreachable cache means there is
nothing to heal rather than a crash. This restores the never-raises
contract ensure_prisma_toolchain() documents.
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
* fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid
The non_root image baked the prisma CLI and engines under /app/.cache and used
the CLI's default (library) engine mode. Prisma stopped baking the library
engine, so `prisma migrate deploy` fell back to downloading it at startup,
which needs network egress and a writable cache. Under an arbitrary non-root
uid (OpenShift restricted-v2), an air-gapped network, or a readOnlyRootFilesystem,
that download fails and the proxy starts on an empty schema while every DB
endpoint returns 500. The migration entrypoint exits 0 on that failure, so a
default-uid `docker run` with network never surfaced it
Bake to /opt/prisma, a fixed world-readable path no cache mount shadows, and
pin PRISMA_CLI_PATH plus PRISMA_CLI_QUERY_ENGINE_TYPE=binary so the baked binary
engine is used directly, matching Dockerfile and Dockerfile.database. A
build-time guard asserts the binary query engine is present, so a future prisma
change that stops baking it fails the image build instead of silently degrading
migrations
Adds docker/test_offline_migration.sh, run from image-scan, which migrates a
fresh Postgres with no egress as a non-root uid and asserts the schema was
created, the case a default-uid `docker run` with network cannot catch
* test(docker): move the offline migration check into a gated pytest and stop pinning XDG_CACHE_HOME at the read-only bake
The offline migration check lived in docker/ as a shell script. It now lives in
tests/proxy_migration_tests/ as a pytest gated on LITELLM_IMAGE, matching the
sibling schema-migration test gated on DATABASE_URL, and image-scan invokes it
with pytest instead of bash. It also asserts the migration entrypoint's exit
code alongside the table count, so a crash or a container-startup failure fails
loudly rather than only surfacing as a low table count
Runtime XDG_CACHE_HOME pointed at /opt/prisma/.cache, which is baked a+rX with
no write, so any XDG-aware library writing a cache at runtime would be denied
for every uid. Leave it unset so it falls back to $HOME/.cache (/app/.cache,
created here and owned by the runtime uid), matching Dockerfile and
Dockerfile.database which never pin XDG at runtime. A second test guards against
a future edit pointing a cache or home var back at the read-only bake
grype defaults match.python.using-cpes to false, so PyPI packages are
matched only against the GitHub Advisory Database. When a CVE is
published to NVD but its GHSA has not propagated to the global advisory
database, the scan reports clean even though grype's own database
already carries the NVD record with the correct version ranges.
The pypdf CVEs (CVE-2026-59935 / 59936 / 59937 / 59938, analyzed in NVD
since 2026-07-08) are the case that exposed this; their GHSA IDs are
still repo-level and return 404 from the global advisory API, so the
ecosystem matcher has nothing to match on.
Enabling CPE matching for Python closes that gap. Measured against a
v1.91.1 build the finding count goes from 28 to 38; the additions are
mostly actionable, and the few cross-product CPE collisions cannot fail
the build because --only-fixed drops the ones carrying no fix version
and the remainder land below the --fail-on high threshold.
* ci(image-scan): add Grype image scan for OS + library CVEs
Builds each of the 6 Dockerfiles via a matrix and scans the resulting image
with Grype (pinned v0.114.0, sha256 verified), failing on fixable HIGH or
CRITICAL across both OS/apk and language packages. This catches the layer
osv-scan is structurally blind to (Wolfi/apk OS packages and vendored deps
like prisma's node engine), which is the structural reason the openssl CVE
slipped past CI and a customer's image scanner flagged it.
Skipped on fork PRs so an outside contributor cannot run arbitrary code on
our hosted runner via a malicious Dockerfile RUN line. The same pattern is
used by guard-fork-dependencies.yml.
Grype runs as a pinned binary with a verified checksum, so there is no
mutable-tag GitHub Action in the dependency chain and no vendor credentials
in the scan job. The job uses read-only contents permissions and an empty
top-level permissions block.
* ci(image-scan): scan only Dockerfile.non_root (rootless target)
All Dockerfile variants share the same wolfi base and apk set today, so a single scan of Dockerfile.non_root gives the same OS-layer coverage at one-sixth the build cost. Dockerfile.non_root is the rootless variant we ship (USER 65534), so the scan tracks the image customers actually run. Matrix-scan if the variants ever diverge.
* ci: retrigger checks (proxy_pass_through_endpoint_tests flaked on prior run)
