mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-02 07:13:02 +00:00
03f7015f5f
* test: add a manually dispatched integration trial on larger runners
Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers
run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one
* test: run the integration trial on push to this branch
* test: give integration trial workers a local redis-server
* test: run the providers trial on a 64-core runner with 32 workers
The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4
* test: run every integration group in the trial on sized runners
* test: add a scan that refuses to upload integration results holding the license
* test: drop the member-name pass the archive byte scan already covers
* test: use a secret whose base64 and base64url encodings differ
* ci: run integration groups on run-ci pull requests and scheduled main runs
* ci: scope the integration license to a protected environment
Backport note: tests/integration/AGENTS.md keeps rc/1.108.0's text, since main's copy of that paragraph also carries later changes rc does not have
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| _support | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.