mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
0ed261b34e
* fix(gemini): prevent negative text_tokens with explicit caching (#18750) ## Problem When using Gemini with explicit caching (especially with images), text_tokens would become negative (e.g., -3327) due to incorrectly subtracting total cached_tokens from modality-specific text_tokens. ## Root Cause The old code did: ```python text_tokens = text_tokens - cached_tokens # 737 - 4064 = -3327 ``` This was wrong because: - cached_tokens includes ALL modalities (text + image + audio + video) - text_tokens only contains text - Subtracting total from specific caused negative values ## Solution Parse cacheTokensDetails to get per-modality cached token breakdown: ```python if "cacheTokensDetails" in usage_metadata: cached_text_tokens = parse from cacheTokensDetails["TEXT"] text_tokens = text_tokens - cached_text_tokens # Correct! ``` Now we subtract cached tokens per modality, preventing negatives. ## Changes - Parse cacheTokensDetails field from Gemini response - Calculate non-cached tokens per modality (text, image, audio) - Remove incorrect global cached_tokens subtraction - Add tests for explicit caching and implicit/no caching scenarios ## Testing - Added test_gemini_cache_tokens_details_no_negative_values - Added test_gemini_without_cache_tokens_details - All existing Gemini caching tests pass Fixes #18750 * feat: add cache_read_input_tokens to Usage object Addresses reviewer feedback to include cached tokens at the top level of the Usage object. This aligns with how Anthropic provider handles cached tokens and ensures they are visible in the final usage response. * fix: add cacheTokensDetails field to UsageMetadata TypedDict Fixes mypy error where cacheTokensDetails was being accessed but not defined in the UsageMetadata TypedDict type definition. |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| code_coverage_tests | ||
| documentation_tests | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| README.MD | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_passthrough_endpoints.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.