Commit graph

876 commits

Author SHA1 Message Date
yujonglee
728d0953af
ci: simplify Rust checks and remove wheel PR comments (#39975)
* ci: limit Rust workflows to Rust directory changes

* ci: run Rust checks when their workflow changes

* ci: report Rust wheels only for successful Rust changes

* ci: keep Rust wheel reports in the workflow summary

* ci: group Rust lint and validation jobs

* ci: keep Rust job names distinct from required lint and test checks

* ci: drop the unused Python setup from the Rust lint job

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-07 09:00:58 -07:00
yujonglee
b9f5cd6036
ci: run unit tests on Python 3.12 (#39989) 2026-09-05 23:56:09 -07:00
mateo-berri
116f88b023 fix(e2e-changed): keep the gate off suites the stack cannot run
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.

tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.

Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
2026-09-05 21:03:50 -07:00
mateo-berri
a9e918577b ci(e2e): run the access_control canary on harness changes and name failed tests
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
2026-09-05 18:46:51 -07:00
mateo-berri
5a06845db1 fix(ci): mask only credential-length values in the e2e-changed log
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
2026-09-05 16:33:15 -07:00
mateo-berri
babc97f562 chore: merge litellm_internal_staging into litellm_/e2e-test-performance-7d53be 2026-09-05 16:15:11 -07:00
mateo-berri
65d8bbb8ac fix(e2e): wait for every gateway before using a new model and keep the network rerun
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
2026-09-05 16:10:40 -07:00
cursor[bot]
8544faec91
fix(ci): grant pull_requests write for release wheel reporter (#39922)
* fix(ci): grant pull_requests write for release wheel reporter

The reporter posts a PR comment via github.rest.issues.createComment.
GitHub requires both issues=write and pull_requests=write to comment on
a PR issue, as returned in x-accepted-github-permissions. The workflow
had pull-requests: read, so the POST failed with 403 'Resource not
accessible by integration'. Bumping to pull-requests: write fixes the
create path; the read-only pulls.get call still works. Same-repo scope
is preserved by the existing head_repository.full_name check.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>

* fix(ci): scope release wheel reporter permissions to pull requests

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
Co-authored-by: Yujong Lee <yujong@berri.ai>
2026-09-05 13:04:31 -07:00
devin-ai-integration[bot]
b290dd410e
feat(terraform/gcp): dependencies-only mode and bring-your-own-network for GKE (#39695)
* feat(terraform/gcp): add dependencies-only mode

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(terraform/gcp): review fixes for dependencies-only mode

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-05 12:22:52 -07:00
Yuneng Jiang
5051e6d44a
ci(e2e): include execution gate checks in code quality 2026-09-05 12:08:11 -07:00
Yuneng Jiang
0f59b6fb7a
ci(e2e): refine changed-test selection and runner lifecycle 2026-09-05 12:03:42 -07:00
Yuneng Jiang
431579dc16
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/mongodb-vector-store-e4ff63
# Conflicts:
#	.github/workflows/_test-unit-base.yml
#	litellm/litellm_core_utils/sensitive_data_masker.py
#	tests/test_litellm/litellm_core_utils/test_sensitive_data_masker.py
#	uv.lock
2026-09-04 14:00:10 -07:00
Yujong Lee
fae3d224eb Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_python_version_ci
# Conflicts:
#	basedpyright-code-budget.json
#	tests/sdk_function_trace/profiler.py
#	tests/sdk_function_trace/test_profiler.py
2026-09-04 09:01:13 -07:00
yujonglee
ee08c36fc0
refactor(tests): restructure rust python harness around strategy definitions (#39628)
* wip

* refactor(tests): move sdk function tracing into rust python harness

* dead code

* fix: handle harness keyboard interrupts

* refactor(tests): deduplicate rust python harness helpers

* fix(harness): expose validated strategy choices

* wip

* refactor(harness): let strategies own parity reports

* docs(harness): update strategy structure

* refactor(harness): localize strategy report views

* wip

* fix(harness): satisfy mapping runner type checks

* fix(harness): clarify trace parity output

* wip

* fix(harness): clarify unit mapping report

* fix(harness): finalize trace parity contracts

* refactor(harness): structure parity contracts

* feat: derive unit test mapping from traces

* feat(harness): map rstest test families

* feat(ocr): port Azure document intelligence tests

* feat(harness): enforce complete unit mappings

* feat(ocr): add reducto core transforms

* feat(harness): classify host-only unit tests

* fix(ocr): complete Rust provider plumbing

* fix(harness): reuse OCR parity workers
2026-09-03 21:15:01 -07:00
yujonglee
8fc0663198
fix(ci): pin setup-uv to v10.0.1 (#39111) 2026-09-03 20:44:24 -07:00
devin-ai-integration[bot]
aec083cdac
feat(proxy): per-worker admission control that rejects excess requests with 503 (#39352)
* feat(proxy): reject excess per-worker requests with 503

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): drop redundant suppressions in admission middleware

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): exempt the /metrics/ redirect target from admission control

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: allowlist live Granian saturation benchmark

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): regenerate dashboard API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): normalize root_path for admission exemptions, validate settings, inject state

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): cover prometheus metric factory, lifespan scope, and prefix lookalike paths

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): queue behind pending waiters, cache admission settings parsing, log invalid limits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): simplify invalid admission settings handling

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-03 18:19:04 -07:00
Mateo Wang
3a00e8020a
Merge pull request #39464 from BerriAI/litellm_containers_error_passthrough_pagination
fix(containers): pass upstream error status through and forward list pagination params
2026-09-03 14:36:44 -07:00
Mateo Wang
5e5cc54b28
Merge pull request #39496 from BerriAI/litellm_ui_build_check_image_boundary
ci(ui): run the UI build check through the image's ui-builder stage
2026-09-03 14:35:56 -07:00
Mateo Wang
c99149ac0f
Merge pull request #39493 from BerriAI/litellm_ci_rust_server_feature
ci(rust): build and test the ai-gateway server feature
2026-09-03 14:35:53 -07:00
yujonglee
2c30fe16b0
Merge pull request #38765 from BerriAI/litellm_ocr_sdk_parity_tests
test(harness): add OCR parity with migration strategy runners
2026-09-03 10:16:35 -07:00
mateo-berri
f986fc52f0 ci(rust): lint every gateway feature and keep one checks runbook
Clippy never links, so python-config's pyo3/auto-initialize needs no
libpython and the gateway clippy step can cover every feature at once.
The test step stays on --features server because cargo test does link
and this job installs no Python.

The check list existed in three places that had already drifted apart;
CLAUDE.md is now the only copy and the other two point at it.
2026-09-02 22:16:40 -07:00
mateo-berri
06e60e08d2 ci(ui): run the UI build check through the image's ui-builder stage
The build-ui check compiled the dashboard from a full checkout, so any
import reaching above ui/litellm-dashboard/ resolved there and only broke
inside the images, where the stage copies the dashboard tree alone.
Building the stage itself puts the check on the same file boundary the
shipped images use.
2026-09-02 22:09:55 -07:00
mateo-berri
cf958c0e6f ci(rust): build and test the ai-gateway server feature
litellm-ai-gateway's server feature is off by default and nothing in the workspace turns it on, so the workspace clippy and test steps never compiled src/auth, src/routes, src/state, src/realtime or the gateway binary. 43 tests ran instead of 57.

Adds the two steps CLAUDE.md already documents as the local gate, and fixes the three collapsible_if violations that had accumulated behind the flag.
2026-09-02 21:56:23 -07:00
mateo-berri
cd296814be Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_containers_error_passthrough_pagination 2026-09-02 21:26:40 -07:00
mateo-berri
425c8d37fc fix(containers): page upstream until a non-admin container list fills its limit
Forwarding limit to OpenAI made the ownership filter cut the page down after
the fact, so a key that owned an older container got an empty first page and
its cursor never moved. Non-admin lists now walk upstream pages of 100 until
they have enough owned containers (or five pages), trim to the requested
limit, and report first_id, last_id and has_more off what the caller keeps.

Also assigns tests/test_litellm/proxy/container_endpoints to a CI shard.
2026-09-02 21:26:34 -07:00
mateo-berri
85961201e7 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_python_version_ci 2026-09-02 18:30:13 -07:00
devin-ai-integration[bot]
92edcb90db
fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448)
* ci: guard against Python 3.10-incompatible typing imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: address Python 3.10 typing guard review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ci): honor version-guard direction and scan litellm-proxy-extras in py310 typing check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-02 18:27:19 -07:00
mateo-berri
748075be4f Merge origin/litellm_internal_staging into litellm_python_version_ci 2026-09-02 18:21:48 -07:00
Yuneng Jiang
a3ee6b2566
ci(e2e): fail a pass whose every collected test was skipped 2026-09-02 17:50:35 -07:00
Yuneng Jiang
1af9c229fa
ci(e2e): leave the managed-files opt-in file to its own lane instead of failing on an empty collection 2026-09-02 17:29:56 -07:00
Yuneng Jiang
1c4e46f17e
ci(e2e): reload the stack config every 7s so the harness propagation budget holds, and only mask credential-named values 2026-09-02 17:15:36 -07:00
yujonglee
62e318de8e
fix(python-bridge): harden sync and async route boundaries (#39332) 2026-09-02 16:26:35 -07:00
yujonglee
198906495f
refactor(python-bridge): split routes and add shared function tracing (#39031)
* refactor(python-bridge): split non-streaming bridge modules

* refactor(python-bridge): bring shared function tracing into route layer

* feat(dev): list Python route functions and call sites

* feat(dev): list Rust route functions and call sites

* docs(dev): record OCR parity gaps across Python and Rust

* feat(dev): list executed SDK calls with runtime tracing

* feat(dev): report Python vs Rust SDK pipeline steps in one CLI

* feat(dev): side-by-side pipeline step report in compare CLI

* fix(dev): drop invalid Final annotations in compare cell loop

* feat(dev): blue python-only and yellow rust-only steps in compare CLI

* feat(dev): vertical layout with section spacing in compare CLI

* fix(dev): validate SDK trace stages across sync and async routes

* refactor(rust): align SDK route call structure with Python

* refactor(python-bridge): share sync and async route call wrappers

* refactor(dev): split compare CLI into fixtures, runtime, and report modules

* fix(ci): run SDK trace tests and satisfy test lint
2026-09-02 16:26:35 -07:00
Yuneng Jiang
c62643f0cf
ci(e2e): drop trailing newlines from fetched secret values before writing the env 2026-09-02 15:39:21 -07:00
Yuneng Jiang
743f94f82a
ci(e2e): fail when nothing collects and ignore lanes with their own checks in the smoke trigger 2026-09-02 14:59:15 -07:00
Yuneng Jiang
b348ed7f09
ci(e2e): only tail stack logs when the stack actually booted 2026-09-02 14:56:52 -07:00
Yuneng Jiang
4a646dd9a0
ci(e2e): run a PR's changed e2e tests three times behind a human-approved environment
Adds a required-check candidate that selects the tests/e2e test files a PR added or
modified, boots a stage-mirror stack on the runner (migrations, backend, two gateway
processes behind nginx, Postgres, Jaeger, TLS cluster Valkey), and runs those files
three times with retries off. The run job sits behind the e2e-changed GitHub
environment, so a reviewer approves each run before the OIDC token that reads the
provider keys from AWS Secrets Manager exists. Supersedes #34981
2026-09-02 14:53:40 -07:00
Yujong Lee
77d6aedf0a fix: address cross-version CI failures 2026-09-02 14:17:19 -07:00
Yujong Lee
8010f267e2 ci: keep Python compatibility coverage in unit workflow 2026-09-02 13:42:22 -07:00
Yujong Lee
0313dcea61 ci: test Python 3.10 through 3.14 compatibility 2026-09-02 13:38:28 -07:00
Yuneng Jiang
3d0223b661 ci: install the mongodb extra for the unit test shard that runs the provider
tests/test_litellm/llms/mongodb imports pymongo's exception classes to check the
error translation against the real hierarchy, and the shard that runs it
(tests/test_litellm/llms, per test-unit.yml) synced --extra google, proxy,
semantic-router and saml but not mongodb, so 24 of 109 tests would have errored
with ModuleNotFoundError on the first CI run. CircleCI hid this because it syncs
--all-groups --all-extras.

  uv export --frozen ... --extra saml                  -> no pymongo
  uv export --frozen ... --extra saml --extra mongodb  -> pymongo==4.17.0

Also close the two gaps a mutation run found in the suite: nothing asserted that
a short request timeout shortens server selection as well as connect, and the
existing code 13 case carried "not authorized", which the message markers match
too, so it could not tell whether the code was still being checked. 28 of 28
mutants now die.
2026-09-02 12:51:19 -07:00
Yujong Lee
90eadac409
test(build): keep wheel checks outside package 2026-09-02 12:16:25 -07:00
Yujong Lee
ce0c85ea69
refactor(rust): colocate native wheel contract checks 2026-09-02 12:16:25 -07:00
Yujong Lee
cbb8a1784d
chore(ci): extract setup-uv pin 2026-09-02 12:16:25 -07:00
Yujong Lee
38150dfc2c
fix(ci): pin workflow toolchain dependencies 2026-09-02 12:16:25 -07:00
Yujong Lee
ef9a207ed5
fix(ci): harden release wheel reporting 2026-09-02 12:16:25 -07:00
Yujong Lee
e6a317e079
fix(ci): enforce release wheel metadata contract 2026-09-02 12:16:25 -07:00
Yujong Lee
9dc9cd325c
fix(ci): isolate release wheel reporting permissions 2026-09-02 12:16:25 -07:00
Yujong Lee
2f362cfec2
fix(ci): preserve release wheel contract parity 2026-09-02 12:16:25 -07:00
Yujong Lee
25987cb961
test(build): validate release wheel contracts 2026-09-02 12:16:25 -07:00