DataTable accepted any mix of its 40-odd props and rejected the incoherent
combinations at runtime, from a validator that threw during the first render.
A caller only found out it had wired server sorting without a `sorting` prop
when the page blew up in front of them.
Split the public prop type into mode-keyed unions instead, so the compiler
rejects those combinations at the call site. `validateDataTableConfig` and
`DataTableConfigError` go away; the component body reads an unchanged flat
`DataTableResolvedProps`, which every union member is assignable to, so there
is no narrowing inside it.
All 44 existing call sites typecheck against the new union unchanged, which
`next build` covers. That build only typechecks the app module graph, so the
prop type itself needed a gate of its own: `npm run test:types` runs vitest's
typecheck mode over `*.test-d.tsx`, and the unit workflow now runs it. The
four guards deleted from `DataTable.test.tsx` come back there as compile-time
assertions, and loosening the union back to the flat shape fails all five.
GitHub expressions have no arithmetic operators, so
`${{ inputs.timeout-minutes + inputs.setup-timeout-minutes }}` was not a value
but a startup failure. The proxy-db workflow died before creating any job on
both prior commits, which posts no check run at all: the entire suite stopped
running while the PR's checks stayed green.
Pass the job backstop in as `job-timeout-minutes` instead of computing it, and
size it as the test budget plus the 30 minutes of setup ceilings plus 5 minutes
of runner overhead the job clock charges but no step owns.
check_workflow_startup_safety.py makes this class of mistake visible before
merge, since CI cannot report it: it rejects arithmetic inside an expression
and checks every caller of the reusable workflow keeps a job budget large
enough that the deadline cannot preempt pytest inside its own budget.
The summed job deadline alone did not protect the test budget. Setup that
overran its allowance still ate into pytest's window, which is the same
failure this change set out to remove, just with more headroom.
Every step before pytest now carries its own ceiling, and their sum is the
`setup-timeout-minutes` default. Setup can no longer overrun into the test
budget without failing its own step first, and a slow setup step now reports
as a red step naming itself rather than a cancelled shard whose tests passed.
Model the workflow YAML the guard reads with Pydantic instead of bare dicts,
so the shapes it depends on are validated once at the boundary. A workflow
that does not parse is now reported as a finding rather than a traceback.
`prisma generate` runs `npm install prisma@<version>` whenever the
prisma-client-py binary cache directory has no CLI entrypoint, pulling ~85 MB
of query and schema engines over the network. Every workflow pointed
PRISMA_BINARY_CACHE_DIR at `${{ runner.temp }}/prisma-cache`, which GitHub
wipes and recreates per job, so that cache was empty on every job of every
run and the download was never avoidable.
The download is normally a few seconds and occasionally minutes. On one
proxy-db run it took 5m18s on a single shard against 3.8s on its eleven
siblings, which pushed the job past its 15 minute timeout and cancelled a
shard whose tests were at 99% and all passing.
Leave PRISMA_BINARY_CACHE_DIR unset so the binaries land in the
prisma-client-py default, which is already keyed by prisma and engine
version, and restore both that path and the @prisma/engines staging cache
through a shared composite action.
Job timeouts also counted setup against the test budget. `timeout-minutes`
now bounds the pytest step, with a separate allowance for checkout,
dependency install, and client generation, so slow setup shows up as a slow
job instead of a cancelled test run.
check_prisma_binary_cache.py guards all three invariants: no workflow
reintroduces the override, every job that generates the client restores the
cache, and the version the action greps out of uv.lock still resolves.
Adds a User Flow section right below the TLDR so every PR describes the same end user doing the same task before and after the change, plus comment instructions and a worked example so contributors can write it without any local tooling.
PR #36166 added tests/test_litellm/proxy/credential_endpoints/test_endpoints.py
but no CI job invokes it, so the CI Coverage guard failed on
litellm_internal_staging. Add the directory to the proxy-endpoints
job's test-path list so pytest actually runs the new tests and the
coverage assertion is satisfied.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
prisma resolves its prisma-client-py generator through a plain /bin/sh PATH
lookup, never through the interpreter that ran prisma generate, so the gate's
generate step landed the client in whatever venv the caller had on PATH: the
owned env never received one, every gate run regenerated, the caller's venv
was mutated instead, and any invocation without a venv on PATH (the rewritten
publisher workflow) failed outright
The generate now runs with the target interpreter's bin directory pinned to
the front of the child PATH. The prisma schema joins the environment
fingerprint so clientless counts recorded before this commit can never be
compared against clientful ones, a cold provision announces itself on stderr
instead of sitting silent for two minutes, and the CI gate step reuses the
job's prisma binary cache
The gate previously measured whatever environment the caller happened to
have. Locally that is the fat bootstrap venv (--extra proxy pulls in
fastapi-sso, whose type info flips a reportUnnecessaryIsInstance
diagnostic in ui_sso.py), while CI's publisher venv only has the
proxy-dev and e2e-dev groups, so identical trees measured 866 locally vs
865 in CI and every local gate run breached by a phantom +1
scripts/type_check_gate.py now provisions .venv-typecheck itself: a
frozen uv sync of the canonical proxy-dev and e2e-dev groups, the
interpreter pinned to pyrightconfig.json's pythonVersion, plus the
generated Prisma client. Every measurement pass is pinned to that env
with --pythonpath, because basedpyright auto-detects a .venv in the
project root and that auto-detection beats both PATH order and
VIRTUAL_ENV, so the CLI flag is the only pin that actually works. The
dependency-group set is folded into the environment fingerprint, so
artifacts or caches recorded under a different group set never match
and the gate falls back to computing base counts locally instead of
comparing mismatched environments
The publisher workflow drops its own install and prisma steps and lets
the script build the measurement env, and the node heap for the
full-tree pass drops from 12GB to 8GB (peak RSS measured at 5.4GB)
The absence of a CI signal is indistinguishable from a passing one, and
that shape has now produced several independent holes: whole test
directories no job runs, and shipped images no job builds. Nothing was
watching for either, so each was found by accident.
assert_ci_coverage.py enumerates every test_*.py under tests/ and every
Dockerfile in the repo, then credits only what the workflows and the
CircleCI config actually invoke. Paths filters and lint steps that merely
name a directory do not count as coverage, because crediting a mention is
the same mistake one level up. Anything neither invoked nor listed in
.github/ci-coverage-allowlist.yml with a written reason fails the job.
The guard found 275 uncovered test files and 6 unbuilt Dockerfiles. Fixed
here: the root Dockerfile, the primary published image, now gets an
image-scan leg that builds it and runs the offline migration check against
it, and 8 tests/test_litellm subdirectories join the shards that already
enumerate their siblings. Everything else is allowlisted per file, so a
new file cannot inherit an exemption, and the remaining decisions are
tracked rather than invisible.
The job reports but is not in branch protection, so it does not block
merges; promoting it is a separate change once the allowlist has survived
contact with a few pull requests.
* fix(ci): run every helm test suite, not just the first one per file
helm-unittest gained support for multiple suites in one test file in
v0.5.0; CI and the Makefile both pinned v0.4.4, the last release that
decodes a single YAML document per file. Any suite after a `---`
separator was parsed away and its assertions never ran, while the
summary still reported a clean pass.
Upgrading the pin to v0.8.2, the newest release that installs under the
pinned helm 3.11.1, brings the litellm-helm chart from 11 suites / 90
tests to 14 suites / 93 tests with no change to any test file. All the
recovered tests pass.
The run step now compares the number of declared `suite:` documents
against the number of suites the runner reports, so the same class of
silent skip fails the job loudly instead of passing quietly. The
Makefile target upgrades a stale local plugin instead of swallowing the
"already installed" error and leaving the developer on an old version.
* ci: install helm-unittest from a pinned, checksum-verified artifact
`helm plugin install <git url>` clones the plugin repo and executes its
install hook, which downloads the release tarball itself. The old
integrity step then checked the cloned repo's HEAD, which happens after
the hook has already run and never covers the binary that was actually
downloaded.
The plugin now comes from a full pinned release URL, verified against
the SHA-256 the project publishes in its helm-unittest-checksum.sha
sidecar, before anything is unpacked or run. Nothing remote executes
ahead of the check, and a re-published release asset fails the job
instead of installing silently.
* fix(ci): make every remaining CI checkout shallow
PR #35982 only covered the lint and budget-ratchet jobs, so secret-scan
kept spending minutes fetching every branch inside its 5 minute timeout
and PRs kept getting cancelled. The UI lint and UI unit jobs carried the
same fetch-depth 0 checkout
secret-scan now checks out at depth 1, runs the hardcoded-secret pytest
without building the project environment, and lets the ggshield step
deepen history itself when a key is configured. UI lint resolves the
merge base through the API instead of local history. UI unit tests
compute the changed files the same way and feed them to vitest related,
because vitest --changed does a three-dot diff that silently selects
zero tests on a shallow clone
The daily branch creation workflows also did full checkouts, then
failed every run since persist-credentials: false left git push with no
credentials. They now create the ref through the GitHub API without a
checkout at all
* fix(ci): feed deleted UI files into vitest related selection
vitest --changed fed git's full change list to the related filter,
deletions included, so a deletion-only dashboard PR still selected the
tests importing the removed files. Keep that behavior by dropping the
diff filter and existence guard; vitest resolves nonexistent paths fine
and --passWithNoTests covers the nothing-related case
heal_incomplete_nodeenv_cache() stats a $HOME-derived path with a bare
Path.is_dir(). pathlib only swallows ENOENT-shaped errnos, so a cache
directory the process cannot search raises PermissionError instead of
answering False. Images bake that cache under the build user's home, whose
mode is 0700, so a container started under any other uid dies there before
the Prisma CLI is ever invoked, and the migration never runs.
Tolerate OSError while inspecting the cache, matching the guard
nodeenv_cache_dir() already carries, so an unreachable cache means there is
nothing to heal rather than a crash. This restores the never-raises
contract ensure_prisma_toolchain() documents.
The 12 GB NODE_OPTIONS setting lived only in the Makefile export and the
CI env line, so any hand-run gate pipeline forgot it and node OOMed at
the ~4 GB default after 80 seconds, with || true feeding the gate empty
output. The gate now spawns basedpyright itself for both the head and
base passes, appends the heap flag last so it wins node's last-flag-wins
resolution while preserving other caller flags, and fails loudly on
crash exit codes instead of reading them as zero errors.
* chore(build): move the Admin UI toolchain to Node 24
Node 18 and Node 20 both reached end of life (2025-04-30 and 2026-04-30), and
the release images along with every CI lane were still building on them. Node 24
is the current LTS through 2028-04-30, so this moves the four UI build images,
the CircleCI lanes, and the four GitHub Actions workflows onto it
Node 24 also ships npm 11.17, which is the first line that implements the
min-release-age setting this repo already carries in its .npmrc files. On npm 10
the key is parsed and discarded, so the release-age gate has had no effect
regardless of its value. Tightening the dashboard's engines range and turning on
engine-strict makes an unsupported npm fail loudly rather than skip the gate
quietly, and a new step in the UI build workflow probes an impossible cooldown
so an inert setting cannot pass unnoticed again
Node 24's bundled undici tightened its brand check on RequestInit.signal, which
rejects the AbortSignal jsdom installs and broke the two cases in
src/lib/http/api.test.ts that rebase a request onto a runtime base url. Under
jsdom the Request global comes from Node while AbortSignal comes from jsdom;
tests/jsdomFetchEnv.ts delegates to the jsdom environment and then restores
Node's native AbortController and AbortSignal so both come from one realm.
Upgrading jsdom does not address this, as jsdom still does not own Request
The workflows now read ui/litellm-dashboard/.nvmrc instead of repeating a
literal, so the Node version has a single source of truth, and ui/Dockerfile is
pinned by digest to match the other three build images. The lockfile changes are
npm 11 normalising the engines range and dropping optional peer entries it no
longer records
* fix(build): point every Admin UI build script at .nvmrc
The enterprise Docker path was left on Node 18. docker/build_admin_ui.sh runs
only when enterprise/enterprise_ui/enterprise_colors.json is present, which it
never is in the OSS tree, so neither CI nor a default image build reaches it;
it pinned nvm to v18.17.0 and then built the dashboard, which now requires Node
24, so a customized enterprise image would have failed EBADENGINE
All three UI build scripts now resolve the version from
ui/litellm-dashboard/.nvmrc rather than carrying their own pin, so the Node
version has a single home across Docker, CI, and local builds. build_ui.sh was
on v20 and build_ui_custom_path.sh on v18.17.0
Also drops the dependency-cooldown probe from the UI build workflow. The
engines floor plus engine-strict already fails an unsupported npm loudly at
install time, so the probe was redundant, and treating any nonzero exit from a
live registry call as proof of enforcement made it unsound besides
The migrations image ran `prisma migrate deploy` against a bake anchored in
$HOME with no node in the runtime stage, so prisma-client-py fell through to
nodeenv and tried to download a Node runtime on first start. In an
egress-restricted cluster that fails outright, and under an arbitrary uid the
uid-specific cache path is unreadable, so the job never applies a migration.
Move the bake to /opt/prisma with world-readable modes, install node in the
runtime stage, and pin PRISMA_BINARY_CACHE_DIR / PRISMA_CLI_PATH /
PRISMA_OFFLINE_MODE so the migration entrypoint runs the cached CLI directly.
This is the same treatment the root, non_root and database images already
carry.
Resolves LIT-4727
The unit workflows trigger on pull_request only, so a commit that
actually lands on a gated branch ends up with no unit-test check runs at
all. The commit status API reports success for those commits, which a
release gate reads as "nothing failed" rather than "never tested". PR
checks also only ever ran against the merge preview, not the commit that
landed, so two branches that are each green can still land broken
together
Add a push trigger on the two gated branches to the twelve unit
workflows and to the code-quality workflow, so every landed commit gets
check runs addressable by its SHA
test-linting.yml is deliberately left on pull_request only; six of its
steps gate on a diff against github.event.pull_request.base.sha, which
is empty outside a pull request, and a "what did this branch add" check
has no meaning on a merge commit
Also key the concurrency group on github.sha and restrict
cancel-in-progress to pull_request. The previous group was stable across
pushes to a branch, so consecutive merges would cancel the in-flight run
for the earlier commit and leave that SHA without a result, which is the
same blind spot this change is meant to close
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
basedpyright's inference load now exceeds node's ~4GB default heap cap on
ubuntu-latest once the Any hotspots carry real types; the node process died
with a JS heap OOM, emitted nothing, and the gate refused the vacuous run.
12GB leaves headroom on the 16GB runner.
`UI Lint / frontend-lint` collected its file list from
`"$BASE_SHA"...HEAD`, where `BASE_SHA` is the base branch tip captured when
the PR was opened and `HEAD` is the merge of the PR into the *current* base
tip that actions/checkout leaves behind. The three-dot merge base of those two
is `BASE_SHA` itself, so the diff spans every base-branch commit landed since
the PR was opened.
Any PR opened before an eslint violation landed on the base branch therefore
fails on files it never touched. PR #34192 changes two Python files and no UI
file at all, and the job still linted 283 dashboard files and failed on three
`no-restricted-imports` antd errors from unrelated commits.
Diffing the PR head against its own merge base gives exactly the files the PR
changed, whether the checkout leaves HEAD on a merge commit or on the head
commit.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
* fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid
The non_root image baked the prisma CLI and engines under /app/.cache and used
the CLI's default (library) engine mode. Prisma stopped baking the library
engine, so `prisma migrate deploy` fell back to downloading it at startup,
which needs network egress and a writable cache. Under an arbitrary non-root
uid (OpenShift restricted-v2), an air-gapped network, or a readOnlyRootFilesystem,
that download fails and the proxy starts on an empty schema while every DB
endpoint returns 500. The migration entrypoint exits 0 on that failure, so a
default-uid `docker run` with network never surfaced it
Bake to /opt/prisma, a fixed world-readable path no cache mount shadows, and
pin PRISMA_CLI_PATH plus PRISMA_CLI_QUERY_ENGINE_TYPE=binary so the baked binary
engine is used directly, matching Dockerfile and Dockerfile.database. A
build-time guard asserts the binary query engine is present, so a future prisma
change that stops baking it fails the image build instead of silently degrading
migrations
Adds docker/test_offline_migration.sh, run from image-scan, which migrates a
fresh Postgres with no egress as a non-root uid and asserts the schema was
created, the case a default-uid `docker run` with network cannot catch
* test(docker): move the offline migration check into a gated pytest and stop pinning XDG_CACHE_HOME at the read-only bake
The offline migration check lived in docker/ as a shell script. It now lives in
tests/proxy_migration_tests/ as a pytest gated on LITELLM_IMAGE, matching the
sibling schema-migration test gated on DATABASE_URL, and image-scan invokes it
with pytest instead of bash. It also asserts the migration entrypoint's exit
code alongside the table count, so a crash or a container-startup failure fails
loudly rather than only surfacing as a low table count
Runtime XDG_CACHE_HOME pointed at /opt/prisma/.cache, which is baked a+rX with
no write, so any XDG-aware library writing a cache at runtime would be denied
for every uid. Leave it unset so it falls back to $HOME/.cache (/app/.cache,
created here and owned by the runtime uid), matching Dockerfile and
Dockerfile.database which never pin XDG at runtime. A second test guards against
a future edit pointing a cache or home var back at the read-only bake
The UI vitest suite is CPU-bound; move it to a 16-core larger runner and raise vitest fork concurrency from 4 to 14 (leaving headroom for the coordinator, jsdom, and the OS) so the full suite and PR-scoped runs finish faster.
The SSO and Email Server settings pages read only stored config, so a gateway
configured entirely through environment variables rendered every field blank
even though both features were live. Rather than add per-endpoint env fallback,
resolve each setting through one typed config object.
A FieldDescriptor names, for one setting, where it lives in the stored row
(db_key), which process env var carries it (env_var), whether it is a secret,
and its effective default. A pure resolve_fields reconciles a descriptor table
against the stored row and the process environment with a fixed precedence and
reports per-field provenance (db, env, default, or unset). The SSO descriptor
table single-sources the field-to-env mapping that the read and write paths
previously duplicated, so they can no longer drift.
get_sso_settings and the /get/config/callbacks alerting block read through the
resolver instead of their own inline fallbacks. get_sso_settings no longer
decrypts stored values into os.environ; decryption happens once inside the
resolver via the pure helper, so a GET stops mutating the process environment.
The SSO response carries provenance so the UI can distinguish an env-sourced
value from a stored one, and secrets are masked at the endpoint (the resolver
returns them unmasked so the login path could consume them). os.environ remains
the runtime carrier; the SSO login and mail-send paths are unchanged.
The settings pages also submit only fields an admin actually edited, so a
rendered mask or env-sourced value is never written back over a working
secret, and generic_scope is a real SSO form field. Omitting a field from
/update/sso_settings clears it, which provider switching relies on; the deeper
write-path concern that behaviour points at is tracked in LIT-4498.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
