The unit workflows trigger on pull_request only, so a commit that
actually lands on a gated branch ends up with no unit-test check runs at
all. The commit status API reports success for those commits, which a
release gate reads as "nothing failed" rather than "never tested". PR
checks also only ever ran against the merge preview, not the commit that
landed, so two branches that are each green can still land broken
together
Add a push trigger on the two gated branches to the twelve unit
workflows and to the code-quality workflow, so every landed commit gets
check runs addressable by its SHA
test-linting.yml is deliberately left on pull_request only; six of its
steps gate on a diff against github.event.pull_request.base.sha, which
is empty outside a pull request, and a "what did this branch add" check
has no meaning on a merge commit
Also key the concurrency group on github.sha and restrict
cancel-in-progress to pull_request. The previous group was stable across
pushes to a branch, so consecutive merges would cancel the in-flight run
for the earlier commit and leave that SHA without a result, which is the
same blind spot this change is meant to close
The Headroom guardrail sent every message to /v1/compress, including the
system prompt and the user's current instruction. On an agentic /v1/messages
request the live turn is the largest compressible blob, so it came back as a
hash marker; the model then called headroom_retrieve and got its own
instruction returned in a tool_result block, which reads as data it fetched
rather than a request to act on, so it described the content instead of doing
the work.
litellm already owns the policy for what a compressor may never rewrite:
get_protected_indices covers the system rows, the last user row and the last
assistant row, and compress() expands it over whole tool exchanges. Headroom
now consults it (promoted from a private name and given tests) and expands it
the same way, so the trailing tool result cannot come back as a marker
standing in for the result of the call the model just made. Protected rows are
withheld from the payload rather than pinned afterwards, so their tokens are
not reported as savings that are never applied; the write-back discards a
compressed system prompt outright, so that saving never existed. The cost is
that a query-aware service no longer sees the newest user message.
A response whose row count differs from what was sent can no longer be
interleaved with the withheld rows, so it goes through the configured fail
policy instead of being adopted. Fail-open now returns the caller's own inputs
object: translation handlers detect a rewrite by identity, so a rebuilt copy
sent an unchanged request through the Anthropic write-back for nothing.
That write-back rebuilt the request with one anthropic_messages_pt call, which
merges every run of consecutive user/tool rows, so a tool_result turn and the
user turn after it arrived fused. Converting a row at a time would separate
them but breaks tool pairing: with modify_params on, an assistant row whose
results are converted separately reads as an orphaned tool call and the
sanitizer answers it with a synthetic "tool execution skipped" result while
dropping the real one. Conversion is now grouped by tool_call_id ownership,
which satisfies both, and the same grouping decides which rows headroom
protects, so the two agree by construction.
The CCR follow-up also dropped any text the model wrote alongside its tool
call, and echoed tool calls it had no results for. Both are fixed by reusing
compresr's extraction helper, now shared instead of duplicated.
Resolves LIT-5018
basedpyright's inference load now exceeds node's ~4GB default heap cap on
ubuntu-latest once the Any hotspots carry real types; the node process died
with a JS heap OOM, emitted nothing, and the gate refused the vacuous run.
12GB leaves headroom on the 16GB runner.
`UI Lint / frontend-lint` collected its file list from
`"$BASE_SHA"...HEAD`, where `BASE_SHA` is the base branch tip captured when
the PR was opened and `HEAD` is the merge of the PR into the *current* base
tip that actions/checkout leaves behind. The three-dot merge base of those two
is `BASE_SHA` itself, so the diff spans every base-branch commit landed since
the PR was opened.
Any PR opened before an eslint violation landed on the base branch therefore
fails on files it never touched. PR #34192 changes two Python files and no UI
file at all, and the job still linted 283 dashboard files and failed on three
`no-restricted-imports` antd errors from unrelated commits.
Diffing the PR head against its own merge base gives exactly the files the PR
changed, whether the checkout leaves HEAD on a merge commit or on the head
commit.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
* fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid
The non_root image baked the prisma CLI and engines under /app/.cache and used
the CLI's default (library) engine mode. Prisma stopped baking the library
engine, so `prisma migrate deploy` fell back to downloading it at startup,
which needs network egress and a writable cache. Under an arbitrary non-root
uid (OpenShift restricted-v2), an air-gapped network, or a readOnlyRootFilesystem,
that download fails and the proxy starts on an empty schema while every DB
endpoint returns 500. The migration entrypoint exits 0 on that failure, so a
default-uid `docker run` with network never surfaced it
Bake to /opt/prisma, a fixed world-readable path no cache mount shadows, and
pin PRISMA_CLI_PATH plus PRISMA_CLI_QUERY_ENGINE_TYPE=binary so the baked binary
engine is used directly, matching Dockerfile and Dockerfile.database. A
build-time guard asserts the binary query engine is present, so a future prisma
change that stops baking it fails the image build instead of silently degrading
migrations
Adds docker/test_offline_migration.sh, run from image-scan, which migrates a
fresh Postgres with no egress as a non-root uid and asserts the schema was
created, the case a default-uid `docker run` with network cannot catch
* test(docker): move the offline migration check into a gated pytest and stop pinning XDG_CACHE_HOME at the read-only bake
The offline migration check lived in docker/ as a shell script. It now lives in
tests/proxy_migration_tests/ as a pytest gated on LITELLM_IMAGE, matching the
sibling schema-migration test gated on DATABASE_URL, and image-scan invokes it
with pytest instead of bash. It also asserts the migration entrypoint's exit
code alongside the table count, so a crash or a container-startup failure fails
loudly rather than only surfacing as a low table count
Runtime XDG_CACHE_HOME pointed at /opt/prisma/.cache, which is baked a+rX with
no write, so any XDG-aware library writing a cache at runtime would be denied
for every uid. Leave it unset so it falls back to $HOME/.cache (/app/.cache,
created here and owned by the runtime uid), matching Dockerfile and
Dockerfile.database which never pin XDG at runtime. A second test guards against
a future edit pointing a cache or home var back at the read-only bake
The UI vitest suite is CPU-bound; move it to a 16-core larger runner and raise vitest fork concurrency from 4 to 14 (leaving headroom for the coordinator, jsdom, and the OS) so the full suite and PR-scoped runs finish faster.
The SSO and Email Server settings pages read only stored config, so a gateway
configured entirely through environment variables rendered every field blank
even though both features were live. Rather than add per-endpoint env fallback,
resolve each setting through one typed config object.
A FieldDescriptor names, for one setting, where it lives in the stored row
(db_key), which process env var carries it (env_var), whether it is a secret,
and its effective default. A pure resolve_fields reconciles a descriptor table
against the stored row and the process environment with a fixed precedence and
reports per-field provenance (db, env, default, or unset). The SSO descriptor
table single-sources the field-to-env mapping that the read and write paths
previously duplicated, so they can no longer drift.
get_sso_settings and the /get/config/callbacks alerting block read through the
resolver instead of their own inline fallbacks. get_sso_settings no longer
decrypts stored values into os.environ; decryption happens once inside the
resolver via the pure helper, so a GET stops mutating the process environment.
The SSO response carries provenance so the UI can distinguish an env-sourced
value from a stored one, and secrets are masked at the endpoint (the resolver
returns them unmasked so the login path could consume them). os.environ remains
the runtime carrier; the SSO login and mail-send paths are unchanged.
The settings pages also submit only fields an admin actually edited, so a
rendered mask or env-sourced value is never written back over a working
secret, and generic_scope is a real SSO form field. Omitting a field from
/update/sso_settings clears it, which provider switching relies on; the deeper
write-path concern that behaviour points at is tracked in LIT-4498.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
Adds get_external to e2e_http.py for absolute third-party GETs (no proxy base url or auth, same Result classification) and rewires fetch_agent_card through it, dropping the urllib.request escape hatch. Creates tests/code_coverage_tests/check_e2e_no_raw_requests.py, the checker tests/e2e/CLAUDE.md already referenced, and wires it into the code-quality workflow so raw HTTP client imports outside the transport fail CI; pre-existing uses (root conftest liveness probe, claude_code version resolver) are grandfathered and exception-type-only imports stay allowed.
* test(ui): run vitest unit tests in GitHub Actions and fix stale key-info tests
The dashboard's vitest suite only ran on CircleCI; GitHub Actions covered the
UI build, lint and api-types sync but never the unit tests. Add a UI Unit Tests
workflow that runs the suite, sharded across a matrix so the wall-clock is not
bound by a single 4-core runner.
Porting it surfaced 17 pre-existing failures. Adding the block/unblock key
action moved Delete Key and Reset Spend into a "More key actions" dropdown and
introduced a React Query hook; KeyInfoHeader's own test was updated but the two
KeyInfoView test files were not. Reach those actions through the dropdown and
stub the new hook the way the neighbouring hook is already stubbed.
The same refactor had quietly hollowed out assertions that still passed:
"should not show Reset Spend button for regular key owner" queried for a button
role that no longer exists, so it held green regardless of the permission
check. Those now open the menu and assert on the menu item, which fails when
canResetSpend is forced true.
Also add the missing cost-optimization page description; page_utils guards that
every navigable page carries one.
* ci(ui): scope PR runs to changed tests, run the full suite on staging
Running the whole vitest suite on every pull request costs about five minutes,
and none of it is recoverable through parallelism: vitest schedules by file and
create_mcp_server.test.tsx alone accounts for 252s of the 255s total, so shards
and extra cores cannot get under that floor. Measured on this branch, css:false,
pool=threads and isolate=false all landed within noise of the baseline.
Scope pull requests to tests reachable from the diff instead, which takes 11s
here, and keep a full run on pushes to litellm_internal_staging so nothing rots
behind a gap in the module graph. Backend-only pull requests match no test files
and exit zero; --passWithNoTests states that rather than leaning on it being the
current default. The checkout needs full history for --changed to resolve the
base commit.
grype defaults match.python.using-cpes to false, so PyPI packages are
matched only against the GitHub Advisory Database. When a CVE is
published to NVD but its GHSA has not propagated to the global advisory
database, the scan reports clean even though grype's own database
already carries the NVD record with the correct version ranges.
The pypdf CVEs (CVE-2026-59935 / 59936 / 59937 / 59938, analyzed in NVD
since 2026-07-08) are the case that exposed this; their GHSA IDs are
still repo-level and return 404 from the global advisory API, so the
ecosystem matcher has nothing to match on.
Enabling CPE matching for Python closes that gap. Measured against a
v1.91.1 build the finding count goes from 28 to 38; the additions are
mostly actionable, and the few cross-product CPE collisions cannot fail
the build because --only-fixed drops the ones carrying no fix version
and the remainder land below the --fail-on high threshold.
Removes the scheduled workflow that cut litellm_oss_daily_YYYY_MM_DD
branches and the guardrails workflow that only ran on them. The secret
scan and ruff checks that workflow duplicated already run on PRs to
litellm_internal_staging via test-linting.yml, so no coverage is lost.
Retargets contributor-facing messaging in CONTRIBUTING.md, CLAUDE.md,
and the guard-main-branch error output at litellm_internal_staging.
The eslint-metrics.json snapshot duplicated the violation counts already
enforced by eslint-budgets.json. Keeping it current added a CI drift check,
a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics
script, none of which caught anything the budget gate did not, yet all of
which failed noisily whenever the snapshot went stale. This drops the file
and that machinery while leaving eslint-budgets.json as the actual ratchet
gate
The guard-main-branch error messages and the contributor docs still
pointed people at litellm_oss_staging. Redirect them to the current
daily OSS branch (litellm_oss_daily_YYYY_MM_DD), a fresh one of which
is cut each weekday, so contributors should target the most recent
The lint job lived in test-litellm-ui-build.yml (workflow name "UI Build
Check") next to the build job, so its check surfaced as the misleading
"UI Build Check / frontend-lint" even though it does prettier, eslint, lint
budgets, and knip, not building. Split it into test-litellm-ui-lint.yml
(name "UI Lint") so the check reads "UI Lint / frontend-lint". The build
workflow keeps only build-ui; the lint job (including the knip step) moves
over unchanged.
Note for whoever manages branch protection: this renames the lint required-
check context from "UI Build Check / frontend-lint" to
"UI Lint / frontend-lint"; update the required-check entry so PRs don't strand.
knip was producing garbage locally and was never wired into CI, so nobody
trusted it. Two structural problems: it silently degrades when deps are
missing (a partial worktree install flagged all 436 test files as unused),
and its config had blind spots that surfaced as false positives.
Fixes so a knip run means something:
- Register every playwright config (serverRootPath + migration variants), not
just the main one. serverRootPath.config.ts is invoked via --config in
test_server_root_path.yml, which knip can't see; it was falsely flagged as
an unused file
- Treat src/components/ui/** as entry points. These are shadcn design-system
primitives, intentionally part of the palette before every one is consumed;
knip was flagging not-yet-used ones (e.g. select.tsx) as dead files and
their sub-exports as unused. Marking the directory as the design-system
surface is the correct fix, not deleting components someone is about to use
- Declare @ant-design/icons as a direct dependency. It was imported in ~198
files but only resolved via antd hoisting, so every one showed up as an
"unlisted dependency"
- Add an explicit vitest plugin block so test-file classification no longer
rides on auto-detection
- Stage severities via rules: gate the now-clean categories (files,
dependencies, unlisted, unresolved) as errors and keep exports/types/
duplicates as warnings, so CI enforces what's at zero today while the
remaining findings ratchet down in follow-ups
- Run npm run knip in the frontend-lint CI job, which installs with npm ci so
it never sees a partial tree
knip now exits 0 with the gated categories clean
The in-workflow regex list was hard to maintain and, because it runs on pull_request, could be modified by the same PR it inspects. Path gating for the OSS daily branches now lives in repository branch protection settings, so this workflow keeps only the OSS-safe checks: the hardcoded-secret test and ruff
* ci: skip unit test workflows when only docs or ui files change
Mirror the CircleCI backend path filter (.circleci/scripts/classify_changes.sh)
in the GitHub Actions unit test workflows by adding paths-ignore for ui/**,
docs/**, *.md and *.mdx to every test-unit-*.yml pull_request trigger
* ci: drop docs/** from unit test paths-ignore since the folder no longer exists
* feat(terraform): vendor terraform-provider-litellm as source of truth with endpoint drift CI
* fix(terraform): address review feedback on vendored provider
Replace deprecated io/ioutil with io. Remove the unused org/team CRUD
client methods so the endpoint audit only tracks live call sites
(54 -> 46). Redact request/response logs by parsing the JSON and
recursively masking sensitive fields, which fixes the nested-object
leak in the old credential_values regex, with a regex fallback for
non-JSON payloads; covered by new unit tests. Docs: stop showing
api_key inside vector store litellm_params and document that Sensitive
attributes still persist in plaintext state, recommending
litellm_credential_name and an encrypted state backend.
* fix(terraform): stop persisting server-returned litellm_params into vector store state
The vector store Read wrote litellm_params straight back from the API
response into state. The proxy redacts secrets in those responses, so
the readback overwrote user config with redaction sentinels and caused
perpetual diffs, and against a server that returns raw values it would
persist secrets into a non-Sensitive attribute. Read now preserves the
config value like the credential and model resources do, litellm_params
is marked Sensitive, and a regression test pins that a server-returned
api_key never lands in state
* fix(terraform): send role on team member update and stop persisting server env into MCP state
The team member update payload omitted role, and the proxy leaves role
unchanged when the field is absent, so a role downgrade reported as
applied by Terraform never took effect on the proxy. The update now
always sends the configured role (the attribute is Required).
The MCP server resource wrote env straight back from API responses
into a non-Sensitive attribute, pulling admin-visible secrets into
state and, for sanitized responses, blanking user config. Read now
preserves the config value, env is marked Sensitive, and the docs warn
against passing secrets via args. Regression tests cover both fixes
and fail against the previous behavior.
The componentized chart at helm/litellm had no way to mount extra
volumes into its deployments, so custom callback or SSO handler code
could not be mounted the way the docs describe for the monolithic
chart. Adds per-component volumes and volumeMounts values for gateway,
backend, and ui, merged with the existing gateway-config volume, plus
a helm-unittest suite for the chart wired into the helm unit test
workflow
Resolves LIT-4209
* fix(release): create tag before release and set make_latest post-publish
The Create Release workflow failed for every stable maintenance release
while pre-releases succeeded. Two independent bugs were behind that.
createRelease was minting the tag from target_commitish, and that path
returns "Resource not accessible by integration" (403) to the Actions
token, or 404 to a user token, for certain commits (cli/cli#9773). The
stable-line tips tripped it; the dev/rc commits happened not to. Create
the tag up front with git.createRef and drop target_commitish so the
release attaches to the existing tag instead of minting one. A 422 from
createRef (tag already exists) is tolerated so re-runs are idempotent.
make_latest is silently ignored during the draft-to-published
transition (cli/cli#8201), so a backport that published would seize the
repo "latest" badge from a newer line. Publish first, then set
make_latest in a separate call, and only for non-prereleases.
Each operation here is already runtime-proven: git.createRef under the
workflow token by prior release-branch jobs, the no-target createRelease
and non-prerelease publish and separate make_latest PATCH by a manual
1.89.5 cut.
* fix(release): pin tag_name on publish so the draft binding can't reset
Pre-creating the tag means the draft is edited while a tag ref already exists, and a draft PATCH that omits tag_name can reset it to the untagged placeholder. Send tag_name explicitly on both updateRelease calls so publish always attaches to the intended tag.
* fix(release): fail loudly when the tag exists at a different commit
The createRef 422 swallow kept re-runs idempotent but also masked a tag that already exists at the wrong SHA, which would publish the release against the wrong commit silently. On 422, compare the existing tag ref to the intended commit and error on a mismatch, keeping idempotency only for a genuine same-SHA re-run.
* ci(codspeed): pin benchmark runner to ubuntu-24.04
ubuntu-latest resolves to different runner images between the BASE
(main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different
runtime environments detected' and emits false-positive regressions
(e.g. a -25.2% swing on test_completion_multi_turn in #31684, an MCP
auth fix with no LLM code changes). Pinning the runner to a fixed
image keeps BASE and HEAD on the same hardware so 1 ms swings on a
~3 ms benchmark stop blocking unrelated PRs.
Fixes#31738
* ci(codspeed): stop running benchmarks on litellm_internal_staging
The CodSpeed check flip-flops on internal staging and on PRs targeting
it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the
next) because the comparison flags "different runtime environments" and
the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings
the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR
helps the head side, but the internal_staging base is still recorded on
the old unpinned runner, so comparisons keep flapping until the pin
merges and the base is re-baselined.
Until that settles, the red X's on internal staging make the OSS
project look unhealthy and confuse contributors, so drop the
litellm_internal_staging push and pull_request triggers and keep
CodSpeed running on main only.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Guard the per-request CPU cost of the chat completion, MCP tool and A2A
message transforms against regressions on every commit. All benchmarks are
pure in-process work with no network I/O so they stay deterministic under
CodSpeed's simulation mode, and they import under the base dependency set the
benchmark job installs.
Inference covers the full SDK overhead via mock_response (simple, multi-turn,
tools, streaming) plus convert_to_model_response_object as a deterministic
anchor. MCP covers the client-side tool translation and the proxy server-side
tool-name prefix round-trip. A2A covers the client request/response transforms
and the proxy server-ingress message conversion.
Adds the mcp and a2a-sdk packages to the benchmark run since those transform
modules need them, and broadens the workflow triggers to litellm_internal_staging
so the internal branch flow is benchmarked too.
* test(batches): add 1:1 test file scaffold for batches component paths
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add harness test for create batch endpoint
* Add retrieve endpoint harness tests
* Add list endpoint harness tests
* Add cancel endpoint harness tests
* Add cancel endpoint harness tests
* Add test for litellm/batches/main.py
* Add test for litellm/tests/test_litellm/batches/test_batch_utils.py
* Add handler and transformation tests for all providers
* Fix: run batches tests in cicd
* fix(tests): remove azure/__init__.py that shadowed azure namespace package
Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to
insert tests/test_litellm/llms/ into sys.path[0], making our empty
azure/ dir shadow the real azure-identity namespace package. Any test
that patched azure.identity.* would then fail with AttributeError.
* style(tests): apply ruff format to test_batch_utils.py
Base migrated the formatter from black to ruff format (#31317); reformat the
batches scaffold test file to match.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Keep only video test files and CI workflow entries; drop unrelated
production code and non-video test changes from this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
