mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-02 07:13:02 +00:00
62b8ee78de
* test: add a manually dispatched integration trial on larger runners Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one * test: run the integration trial on push to this branch * test: give integration trial workers a local redis-server * test: run the providers trial on a 64-core runner with 32 workers The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4 * test: run every integration group in the trial on sized runners * test: add a scan that refuses to upload integration results holding the license * test: drop the member-name pass the archive byte scan already covers * test: use a secret whose base64 and base64url encodings differ * ci: run integration groups on run-ci pull requests and scheduled main runs * ci: scope the integration license to a protected environment
53 lines
3.2 KiB
Markdown
53 lines
3.2 KiB
Markdown
# tests/integration
|
|
|
|
Real proxy, Postgres, Redis, scripted upstream. `README.md` has shards and CI wiring
|
|
|
|
## What good looks like
|
|
|
|
The root example is from here. The spend row lands async: poll, never sleep
|
|
|
|
```python
|
|
rows = eventually(
|
|
lambda: read_rows('SELECT spend FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (body["id"],)),
|
|
lambda values: len(values) == 1,
|
|
seconds=70,
|
|
)
|
|
assert float(rows[0]["spend"]) == pytest.approx(10 * 0.001 + 5 * 0.0001 + 7 * 0.002 + 4 * 0.002)
|
|
```
|
|
|
|
`sleep(3)` fails on a slow runner and taxes every fast one. Assert the outbound body in the upstream
|
|
handler. A leaked field is invisible from the response. `monkeypatch.setenv` is fine. Patching our own
|
|
function in a full stack is not
|
|
|
|
## Where it goes
|
|
|
|
By the domain a user would name: `pricing`, `spend`, `routing`, `mcp`. A file only needs to live in a
|
|
directory that a `GROUPS` entry in `run.py` selects. There is no manifest and no `covers` marker on new
|
|
tests. A product bug the test exposes is `pytest.skip("BUG: <symptom>")` at the top of the body, not a
|
|
fix in the test and not a deletion. Needs no network, proxy, DB or Redis: `tests/unit`
|
|
|
|
CircleCI's integration groups run every file here except the OCI and sandbox tests, which no CI runs, and
|
|
the files in `GITHUB_FILES` in `run.py`. `test-integration.yml` runs the same groups through the same
|
|
`run_integration.sh` on GitHub Actions, on main every three hours from 8am to 8pm Pacific Monday to Saturday,
|
|
and on a pull request when someone adds the `run-ci` label. Neither runs every pull request, so a new file here
|
|
first runs on main after its PR merges, or earlier if the PR gets `run-ci`. Wanting a file to run before merge
|
|
is no reason to change CI, and a new
|
|
dedicated workflow for it is the same mistake as a shared shard. Write the file, leave `GITHUB_FILES` and
|
|
every workflow alone, and tell the user it first runs on main. `GITHUB_FILES` is only for a file that
|
|
needs what CircleCI's groups cannot give it, like the images `image-scan.yml` builds or the plain pytest
|
|
run CircleCI's `roi-database` job gives `test_roi_observed.py`. Add a file there only in that case,
|
|
together with its dedicated job, and list a new dedicated GitHub workflow with the directory it owns in
|
|
`DEDICATED_INTEGRATION_WORKFLOWS` in `.github/scripts/assert_ci_coverage.py`. The `ci-coverage` check
|
|
fails a file here that a GitHub workflow runs unless it is in `GITHUB_FILES` and that workflow owns its
|
|
directory (#46780)
|
|
|
|
## Fixtures that boot a proxy
|
|
|
|
Scope them to the module or class, or pin a session one to a single worker with an `xdist_group` mark
|
|
as `_llm_gateway` in `mcp/test_mcp_llm_endpoints.py` does, and request them as arguments, never through
|
|
`request.getfixturevalue`. The harness puts tests that request the same set of module or class fixtures
|
|
on one xdist worker, so the proxy they share boots there once and stops when that worker moves to another
|
|
file. A test that requests one more of those fixtures than its neighbors gets its own group and can boot
|
|
the proxy again on another worker, so keep that set the same across the tests that share a proxy. An
|
|
unpinned session fixture boots a copy on every worker that runs one of its tests and keeps them all until
|
|
the node ends, and those copies ran CircleCI integration-extensions nodes out of memory
|