mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
1c8761111f
* feat(proxy/utils.py): track pre-call hooks in OTEL some pre call hooks can cause latency in high traffic - make sure this is tracked * fix(router.py): move redis call on deployment_callback_on_success to pipeline operation reduces p99 latency by half when redis is enabled * fix(parallel_request_limiter_v3.py): only run check if any item has rate limits set Prevents unnecessary latency added by rate limit checks * test: add unit tests * Latency Improvements: only track tpm/rpm usage when set on deployment+ LLM Caching - use an in-memory cache to reduce redis calls + OTEL - track time spent on LLM caching (#13472) * fix(router.py): only track usage for deployments with tpm/rpm set ensures additional latency avoided for non-tpm/rpm models * fix(caching_handler.py): log time spent on request get cache to OTEL enables easy debugging of call latency * fix(caching_handler.py): use dual cache object for in-memory caching + trace redis call within caching handler * fix(caching_handler.py): working in-memory cache for redis calls ensures dual cache works when redis cache setup for llm calls makes calls quicker by only checking redis when in-memory cache missed for llm api call * test: remove redundant test * test: add unit tests |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| anthropic_endpoints | ||
| auth | ||
| client | ||
| common_utils | ||
| db | ||
| experimental/mcp_server | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| spend_tracking | ||
| test_configs | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| test_caching_routes.py | ||
| test_common_request_processing.py | ||
| test_custom_proxy.py | ||
| test_fastapi_offline_routes.py | ||
| test_health_check_functions.py | ||
| test_litellm_pre_call_utils.py | ||
| test_proxy_cli.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_route_llm_request.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||