mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
986bdb5889
|
build(deps): bump the github-actions group across 1 directory with 21 updates
Bumps the github-actions group with 21 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4.2.2` | `7.0.1` | | [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` | | [actions/cache](https://github.com/actions/cache) | `4.3.0` | `6.1.0` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.1` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4.2.1` | `8.0.1` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.4` | `7.1.1` | | [actions/setup-node](https://github.com/actions/setup-node) | `4.4.0` | `7.0.0` | | [github/codeql-action/init](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [github/codeql-action/analyze](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.12.1` | `5.2.1` | | [actions/github-script](https://github.com/actions/github-script) | `7.0.1` | `9.0.0` | | [openai/codex-action](https://github.com/openai/codex-action) | `1.9` | `1.12` | | [azure/setup-helm](https://github.com/azure/setup-helm) | `4.3.1` | `5.0.1` | | [ossf/scorecard-action](https://github.com/ossf/scorecard-action) | `2.4.1` | `2.4.4` | | [actions/stale](https://github.com/actions/stale) | `8.0.0` | `11.0.0` | | [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) | `6.2.0` | `6.3.0` | | [taiki-e/install-action](https://github.com/taiki-e/install-action) | `2.87.8` | `2.87.18` | | [hashicorp/setup-terraform](https://github.com/hashicorp/setup-terraform) | `3.1.2` | `4.0.1` | | [actions/setup-go](https://github.com/actions/setup-go) | `6.2.0` | `7.0.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.6` | `0.6.4` | Updates `actions/checkout` from 4.2.2 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.2.2...3d3c42e5aac5ba805825da76410c181273ba90b1) Updates `actions/setup-python` from 5.6.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits]( |
||
|
Yuneng Jiang
|
ac1eb1029a |
fix(ci): name the file mutmut actually writes partial results to
The step-timeout comment claimed mutmut streams each mutant's result into mutants/mutmut-stats.json. It does not. That file holds the pre-run test timings and coverage map (tests_by_mangled_function_name, duration_by_test, stats_time) written once by save_stats() before mutation starts. Per-mutant results live in mutants/<source path>.meta. Verified against mutmut 3.5.0: SourceFileMutationData.register_result() calls save() after every single result, and export-cicd-stats walks those .meta files to build mutmut-cicd-stats.json. So the reason the step deadline exists is still right, an interrupted run keeps the mutants it already scored, but the comment pointed at the wrong file. Also upload the .meta files, since they are the partial results the comment relies on and the artifact could not otherwise show them. |
||
|
Yuneng Jiang
|
ae63786cfb
|
fix(ci): let the mutation workflow find covered lines so it generates mutants
mutmut's gather_coverage() looks each source file's covered lines up by absolute path, but [tool.coverage.run] sets relative_files = true, so every lookup misses. With mutate_only_covered_lines = true that leaves no line eligible for mutation, and the run ends on "Stopping early, because we could not find any test case for any mutant" after spending 26 minutes collecting coverage. The last four dispatches all died that way. Point COVERAGE_RCFILE at a small rc file for mutation runs only, so the coverage instance mutmut builds stores absolute paths. Scoped to one module locally this takes the run from 0 mutants to 8 generated and 8 killed. Also give the mutmut step a deadline inside the job's own. mutmut records each mutant's verdict to mutants/mutmut-stats.json as it finishes, so a run that outlasts its budget still scores what it got through, but a cancelled job skips the report and upload steps and publishes nothing. That is how the two runs before these four ended. Ignore mutants/ and .venv-mutmut, which a local run leaves behind untracked. |
||
|
yuneng-jiang
|
afec9b8ab9
|
perf(ci): cache the Rust build the unit shards compile from scratch (#37795)
* perf(ci): cache the Rust build the unit shards compile from scratch Every unit shard installs the workspace, and the root package builds through maturin, so each of the eleven jobs compiles litellm-rust/crates/python-bridge in release mode before a single test runs. That step measured 2m40s a shard on 2026-08-21, which is more wall clock than the entire unit tier spends running tests, and none of it was cached: the uv cache covers wheels it downloads, not wheels it builds, and a path dependency whose source moves every commit can never hit that cache anyway. A composite action now exports CARGO_TARGET_DIR to a fixed workspace path and caches it alongside the Cargo registry, keyed on Cargo.lock. Cargo rebuilds only what changed, so a warm job pays for the bridge crate rather than its whole dependency graph. Measured locally, that is 34s cold against 8s warm, including after a Python-only or Rust-only edit. The absolute path matters: uv builds the wheel from its own working directory, so a relative target directory lands the artifacts where nothing can find them again. * perf(ci): cache the Rust build in the other four workflows that sync the workspace code-quality, mcp, documentation and the schema.d.ts check each install the workspace and so each compile the bridge from scratch, measured at 138s, 177s, 163s and 154s on 2026-08-21. The lint job pays the same and is left to #37783, which already owns that file's setup section. * perf(ci): cache the Rust build in the lint job too * fix(ci): cache cargo's own target directory instead of redirecting it uv builds the wheel in place, so cargo already writes to litellm-rust/target, which test-rust.yml has cached all along. Redirecting CARGO_TARGET_DIR bought nothing and cost a GITHUB_ENV write that zizmor rejects as a code-execution path. * chore(ci): raise the job backstop for the added setup step The cargo cache is a fifth bounded setup step, so the base's setup ceiling goes 30m to 35m and every job budget follows: 55 to 60, and proxy-server's 95 to 100. check_workflow_startup_safety enforces exactly this sum, and failed on the first push without it. * perf(ci): cache the Rust build in the four remaining workflows that sync Six workflows were wired; ten install the workspace. The four left out still compile the pyo3 bridge from scratch. test-terraform-provider.yml is the one that matters per PR: its endpoint-drift job triggers on any change under litellm/proxy/**. The other three are a scheduled load check, a manual mutation run, and the staging-push counts publisher, whose gate syncs the project inside scripts/type_check_gate.py rather than in a workflow step, so nothing in the file names the build it pays for. |
||
|
mateo-berri
|
6a83a84f31
|
ci: cache Prisma CLI and engine binaries, split test timeout from setup
`prisma generate` runs `npm install prisma@<version>` whenever the
prisma-client-py binary cache directory has no CLI entrypoint, pulling ~85 MB
of query and schema engines over the network. Every workflow pointed
PRISMA_BINARY_CACHE_DIR at `${{ runner.temp }}/prisma-cache`, which GitHub
wipes and recreates per job, so that cache was empty on every job of every
run and the download was never avoidable.
The download is normally a few seconds and occasionally minutes. On one
proxy-db run it took 5m18s on a single shard against 3.8s on its eleven
siblings, which pushed the job past its 15 minute timeout and cancelled a
shard whose tests were at 99% and all passing.
Leave PRISMA_BINARY_CACHE_DIR unset so the binaries land in the
prisma-client-py default, which is already keyed by prisma and engine
version, and restore both that path and the @prisma/engines staging cache
through a shared composite action.
Job timeouts also counted setup against the test budget. `timeout-minutes`
now bounds the pytest step, with a separate allowance for checkout,
dependency install, and client generation, so slow setup shows up as a slow
job instead of a cancelled test run.
check_prisma_binary_cache.py guards all three invariants: no workflow
reintroduces the override, every job that generates the client restores the
cache, and the version the action greps out of uv.lock still resolves.
|
||
|
Yassin Kortam
|
35dc982692
|
feat(proxy): add SAML 2.0 SSO for the admin UI (#31429)
litellm already supports Google, Microsoft and generic OIDC SSO through fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an unused enum value. This adds real SAML 2.0 single sign-on for the admin UI. A new SAMLAuthHandler validates signed assertions with the OneLogin python3-saml toolkit and maps them onto a CustomOpenID, then reuses the shared post-login path every other provider goes through, so provisioning, role/team mapping and the UI session JWT are unchanged. Both SP-initiated and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are bound to the browser that started them via an HttpOnly state cookie plus a cached AuthnRequest id, and the ACS rejects any response whose InResponseTo doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected by a consumed-assertion guard whose lifetime tracks each assertion's NotOnOrAfter, and both the replay guard and the login-state binding go through the proxy's shared in-memory + Redis cache for multi-instance deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the free-SSO-user Enterprise gate after the assertion is validated, so an unvalidated POST can no longer drive the billable-user count query. SAML is configurable from the admin UI SSO settings (IdP metadata URL or inline XML, SP entity ID, and an allow-unsolicited toggle), which persists the SAML_* environment variables the handler reads, exactly like the Google, Microsoft and generic OIDC providers. python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels bundle the native libraries so no system packages are required, and the import is guarded so the proxy still starts without the package with the SAML routes returning a clear 501. Resolves LIT-4016 |
||
|
mateo-berri
|
bb1b3dc937 | fix(ci): retry setup-uv installs to survive transient manifest fetch failures | ||
|
ishaan-berri
|
a2d04ccdbb
|
ci: harden cargo fetches during maturin builds (#31348) | ||
|
ryan-crabbe-berri
|
be84d5cd7d
|
ci: add manually-triggered mutation testing workflow (#27576)
* ci: add manually-triggered mutation testing smoke workflow Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut against a single source/test pair (router_settings_endpoints) to validate the tooling end-to-end before scaling. The workflow reinstalls litellm non-editable so the mutants/ sandbox is not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so the trampolined sandbox copy wins over site-packages. mutmut itself is pulled in via uv run --with so it does not appear in uv.lock or affect the shared dev environment. Includes a temporary push: trigger scoped to this branch so we can iterate before the workflow file lands on the default branch — to be removed before merging (workflow_dispatch only requires the file on the default branch to surface the manual trigger button). * ci(mutation): disable rerun and xdist plugins for mutmut runs mutmut's in-process pytest.main() call hits `INTERNALERROR: no option named 'filtered_exceptions'` from pytest-retry's pytest_configure hook. Reruns are also wrong for mutation testing — a "failed" mutant test that gets retried would mask which mutants are killed vs. survive. Disable retry, rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut]. * ci(mutation): uninstall pytest-retry before mutmut runs `-p no:retry` (and similar names) didn't match pytest-retry's entry-point name, so the plugin still loaded and crashed during mutmut's "Running clean tests" phase. Uninstalling the package is surgical and doesn't depend on guessing the entry-point name. * ci(mutation): emit per-survivor diffs to run-page summary + artifact The previous artifact only contained `mutmut results` text (which in mutmut 3.x lists survivor names but not the actual mutations). Adds: - `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the killed/survived/total scoreboard. - `mutmut show <name>` per surviving mutant to capture each mutation as a unified diff. - A `mutmut-report.md` that combines summary + run-progress tail + per-survivor diffs, written to both the artifact and $GITHUB_STEP_SUMMARY (visible on the run page, no download needed). - Corrected artifact paths: stats files live under mutants/, not the project root. - The trampolined source file from the sandbox so survivors can be inspected even outside `mutmut show`. * ci(mutation): document intended manual weekly cadence in trigger comment * ci(mutation): generate ACH-style report with embedded function bodies Replaces the inline bash markdown generation with a Python script that: - Groups survivors by function (one section per function, function body shown once per section, surviving mutants nested as subsections) - Embeds each enclosing function's source via Python AST (so the agent has full context, not just a 3-line `mutmut show` diff) - Inlines the existing test file(s) listed in [tool.mutmut].tests_dir - Writes an ACH-style task description at the bottom following the prompt template from arXiv 2501.12862 Output goes to mutation-report.md (artifact) and the head of the file is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility. * fix(mutation report): correctly parse function names with leading underscores mutmut's mutant-name prefix is x_ (single underscore), so a function named _foo produces mutants x__foo__mutmut_N. The previous regex \.x__(.+)__mutmut_ ate the function's leading underscore as part of the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are preserved in the captured function name; verified for normal, leading- underscore, and dunder-method names. * feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters For each surviving mutant, parse the mutmut sandbox trampoline file and render the mutated function as it appears in the source — with the differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments, matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1). Renames the function header back to its original name so the agent sees the function as it would appear in the file. Falls back to the unified diff if the trampoline lookup fails. Handles replace, insert, and delete diff ops; uses difflib's SequenceMatcher to find the differing line ranges. The unified diff is preserved in a collapsible <details> block as secondary context. * ci(mutation): scope to whole management_endpoints folder, drop temp push trigger Final scope before merge: - paths_to_mutate / tests_dir broadened from one file to the entire management_endpoints source/test folders - Trigger is now `workflow_dispatch` only — the temporary push: block used during workflow iteration is removed - timeout-minutes bumped from 60 to 350 (just under the GH-hosted job cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can take a few hours - Artifact path for the trampoline files glob-expanded to cover all files under mutants/litellm/proxy/management_endpoints/ * fix(mutation report): warn when multiple functions in a file share a name Addresses the Greptile review concern: ast.walk's first-match-wins behavior could embed the wrong function body when a file defines the same name in multiple places (e.g., a module-level helper and a class method). mutmut's mutant identifier does not carry class context, so we can't always determine which definition was mutated. find_function_in_file now returns the start line of every matching definition; render() surfaces a "Note: N functions named X" warning in the report when there is more than one match. The first match is still embedded as the body — the warning tells the reader to verify manually instead of silently using the wrong context. Smoke-tested against the existing artifact: single-match files render unchanged. * Fix mutation report anchors * Fix mutation report TOC anchors --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |