mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
0ec2d95506
`test_gemini_chat_returns_content_and_logs_cost` asks gemini-2.5-flash to
"reply with the single word pong" under `max_tokens=32`, and has been seen
returning no content at all:
completion_tokens=29, reasoning_tokens=29, content=None
gemini-2.5-flash defaults to dynamic thinking, and `max_tokens` maps to
`maxOutputTokens`, which on the 2.5 family counts thinking tokens as well as
visible output. So the model is free to spend the entire budget on thoughts and
emit nothing, which is exactly what the usage above shows.
Raising the limit alone does not fix this. Dynamic thinking on 2.5 Flash is
documented up to 24576 tokens, so no budget small enough to be reasonable for a
one-word smoke test is safe. The fix is to take thinking out of the picture:
`reasoning_effort="none"` maps to `thinkingConfig.thinkingBudget=0` for the 2.5
family, so the whole limit is available to visible output. Verified against this
checkout:
get_optional_params(model="gemini-2.5-flash", custom_llm_provider="gemini",
max_tokens=32)
-> {'max_output_tokens': 32} # no thinkingConfig at all
get_optional_params(model="gemini-2.5-flash", custom_llm_provider="gemini",
max_tokens=64, reasoning_effort="none")
-> {'max_output_tokens': 64,
'thinkingConfig': {'thinkingBudget': 0, 'includeThoughts': False}}
This mirrors what the OpenAI tool tests in this same file already do with
gpt-5.6 for the same failure mode. `max_tokens` goes to 64 for headroom; with
thinking disabled that is ample for a one-word answer.
Neither `covers` claim changes: the call still exercises the gemini chat
translation path and still produces a costed SpendLogs row.
|
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.