mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
8c42d8b97b
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary. Chunking alone only makes the stall shorter, so the async paths now count in a worker thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other requests while a count is in flight. The /utils/token_counter endpoint awaits the new atoken_counter, and the router's async deployment selection counts off-loop and hands the result to _pre_call_checks instead of making it count inline. The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or silently report zero tokens, and an arbitrarily large one restored the quadratic cost this exists to remove. Out-of-range and unparseable values warn and fall back to 1024. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|---|---|---|
| .. | ||
| .coverage_baseline | ||
| __init__.py | ||
| _coverage_check.py | ||
| _pin_check.py | ||
| conftest.py | ||
| test_background_health.py | ||
| test_exception_handlers.py | ||
| test_harness_smoke.py | ||
| test_lifecycle.py | ||
| test_openapi_customization.py | ||
| test_proxy_config.py | ||
| test_routes_anthropic_beta.py | ||
| test_routes_assistants.py | ||
| test_routes_audio.py | ||
| test_routes_chat_completions.py | ||
| test_routes_completions.py | ||
| test_routes_config.py | ||
| test_routes_embeddings.py | ||
| test_routes_invitation.py | ||
| test_routes_login_sso.py | ||
| test_routes_misc.py | ||
| test_routes_model_cost_map.py | ||
| test_routes_model_info.py | ||
| test_routes_model_metrics.py | ||
| test_routes_models.py | ||
| test_routes_moderations.py | ||
| test_routes_onboarding.py | ||
| test_routes_queue.py | ||
| test_routes_threads.py | ||
| test_routes_utils.py | ||
| test_spend_counters.py | ||
| test_streaming_helpers.py | ||
| test_team_model_name_translation.py | ||