scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-09 22:31:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 133
litellm/tests/load_tests
History
Exact
Cursor Agent ae9fc0341f perf: add orjson import fallbacks + benchmark suite for fast-litellm integration 
Fixes bare 'import orjson' in three hot-path modules that broke
non-proxy users (orjson is an optional dependency). Each module
now uses try/except with a stdlib json fallback:

- litellm/llms/openai_like/chat/handler.py
- litellm/llms/openai_like/chat/transformation.py
- litellm/llms/custom_httpx/llm_http_handler.py

Adds comprehensive benchmarking tools inspired by neul-labs/fast-litellm:

- benchmark_perf_integration.py: micro-benchmarks for each optimized
  hot path (JSON serialization, URL parsing, deployment lookup,
  spend-log sanitization, Prometheus label caching, routing, etc.)
- benchmark_sdk_hotpath.py: end-to-end SDK throughput measurement
  against a local mock server
- loadtest_config_nodb.yaml: proxy config without database dependency

Measured speedups (per-component, micro-benchmark):
  JSON serialization:    13.7x (orjson vs stdlib json)
  JSON deserialization:   3.7x
  httpx URL parsing:    202.2x (LRU-cached vs raw parse)
  Deployment lookup:     10.6x (O(1) index vs O(n) scan)
  Prometheus labels:     30.5x (cached model_dump)
  Simple shuffle:         5.8x (lazy logging)
  safe_json_dumps:        5.7x (orjson final step)
  Overall cumulative:    18.7x

Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
2026-03-10 18:11:37 +00:00
..
benchmark_perf_integration.py perf: add orjson import fallbacks + benchmark suite for fast-litellm integration 2026-03-10 18:11:37 +00:00
benchmark_sdk_hotpath.py perf: add orjson import fallbacks + benchmark suite for fast-litellm integration 2026-03-10 18:11:37 +00:00
compare_perf_results.py System performance bottlenecks (#23075) 2026-03-07 19:24:36 -08:00
compare_results.py feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
loadtest_config.yaml feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
loadtest_config_nodb.yaml perf: add orjson import fallbacks + benchmark suite for fast-litellm integration 2026-03-10 18:11:37 +00:00
loadtest_config_perf.yaml System performance bottlenecks (#23075) 2026-03-07 19:24:36 -08:00
loadtest_config_sidecar.yaml feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
locustfile.py feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
locustfile_perf.py System performance bottlenecks (#23075) 2026-03-07 19:24:36 -08:00
memory_leak_utils.py Add memory leak detection tests with CI integration (#18881) 2026-01-09 17:36:10 -08:00
mock_openai_server.py feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
run_loadtest.sh feat: add Rust sidecar binary for high-performance HTTP forwarding 2026-03-07 19:04:10 +00:00
run_perf_comparison.sh System performance bottlenecks (#23075) 2026-03-07 19:24:36 -08:00
test_datadog_load_test.py test: update tests to new deployment model (#10142) 2025-04-18 14:22:12 -07:00
test_langsmith_load_test.py fix langsmith load tests 2024-09-11 21:19:03 -07:00
test_linear_memory_growth.py Add memory leak detection tests with CI integration (#18881) 2026-01-09 17:36:10 -08:00
test_memory_usage.py (perf) Fix memory leak on /completions route (#8551) 2025-02-14 18:58:16 -08:00
test_otel_load_test.py test: update tests to new deployment model (#10142) 2025-04-18 14:22:12 -07:00
test_vertex_embeddings_load_test.py test: test_embedding_performance 2025-05-14 21:31:07 -07:00
test_vertex_load_tests.py vertex testing use pathrise-convert-1606954137718 2025-01-05 14:00:17 -08:00
vertex_key.json ci/cd update vertex acct 2025-01-05 13:43:32 -08:00