mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
a6494e6fe3
- Introduce `_CallbackCapabilities` dataclass and `ProxyLogging._callback_capabilities()` static method that inspects `litellm.callbacks` once and caches capability flags keyed on (list length, member ids); invalidates automatically when the callback list mutates without per-request iteration overhead - Replace O(n) `litellm.callbacks` walks in `async_pre_call_hook`, `during_call_hook`, `async_post_call_streaming_iterator_hook`, `async_post_call_streaming_hook`, and `post_call_response_headers_hook` with fast-path exits when no relevant callbacks are registered - Add `needs_iterator_wrap()` and `needs_per_chunk_streaming_hook()` instance methods to decouple iterator-level wrapping from per-chunk hook execution; avoids `get_response_string` materialization per chunk when no guardrail or chunk-hook callback is active - Introduce `_fast_serialize_simple_model_response_stream()` using `orjson` for common single-choice text streaming chunks, bypassing the full Pydantic serializer; falls back to `model_dump_json` for tool calls, logprobs, usage, and provider-specific fields - Add early-return in `_restamp_streaming_chunk_model` when downstream model already matches the requested model, avoiding unnecessary string comparisons on every chunk - Fix stale zero-cost cache bug in `_is_model_cost_zero`: move the per-router `_zero_cost_cache` dict onto the `Router` instance and clear it in `_invalidate_model_group_info_cache` so in-place pricing updates via `upsert_deployment` immediately resume budget enforcement - Add `scripts/benchmark_chat_completions_perf.py`: standalone async benchmarking tool with a mock OpenAI provider, LiteLLM proxy process management, non-streaming RPS, streaming TTFT, and full-stream latency measurements with repeat/median run support - Add comprehensive unit tests covering capability detection, cache invalidation, fast-path correctness, zero-cost cache regression, and the no-callback streaming fast path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|---|---|---|
| .. | ||
| test_admin_viewer_handler_access.py | ||
| test_auth_checks.py | ||
| test_auth_exception_handler.py | ||
| test_auth_hot_path_network_requests.py | ||
| test_auth_utils.py | ||
| test_banned_params_extra_body.py | ||
| test_cli_auth.py | ||
| test_custom_auth_end_user_budget.py | ||
| test_handle_jwt.py | ||
| test_info_routes.py | ||
| test_litellm_license.py | ||
| test_login_utils.py | ||
| test_mcp_ip_filtering.py | ||
| test_model_checks.py | ||
| test_model_checks_fallbacks.py | ||
| test_multi_budget_windows.py | ||
| test_oauth2_proxy_hook.py | ||
| test_object_permission_loading.py | ||
| test_onboarding.py | ||
| test_organization_budget_enforcement.py | ||
| test_password_hashing.py | ||
| test_route_checks.py | ||
| test_router_override_fallback_auth.py | ||
| test_team_member_budget.py | ||
| test_unmapped_model_budget_enforcement.py | ||
| test_user_api_key_auth.py | ||