mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-28 06:23:07 +00:00
d93353bcb6
Adds litellm-rust/loadtest: a mock LLM (OpenAI chat, Anthropic messages, Bedrock invoke with event-stream framing) that takes its timing from the prompt, an open-loop load generator with workloads covering 10k to 1M token contexts, 6 minute non-streaming calls and 30 minute streams, a config and virtual key generator, a summarizer and a /proc sampler, plus Terraform for an AWS rig with load generators, gateway and mocks on separate machines The gateway sets TCP_NODELAY on accepted sockets, removing a ~40 ms stall per streamed response, and its Docker build takes LTO and CODEGEN_UNITS build args for faster iteration builds |
||
|---|---|---|
| .. | ||
| key-churn.json | ||
| large-context.json | ||
| long-streams.json | ||
| realistic-ramp.json | ||
| smoke.json | ||
| soak.json | ||