scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 84
litellm/tests
History
Cursor Agent 0220609ab6 Add validation and fallback protection for model cost map 
This prevents upstream GitHub changes from breaking LLM calls by:

1. Runtime validation in get_model_cost_map.py:
   - Validates JSON structure before accepting remote data
   - Checks for required fields (litellm_provider)
   - Validates mode values against allowed list
   - Validates numeric cost fields
   - Falls back to local backup if validation fails
   - Logs warnings when falling back

2. Enhanced CI validation (.github/workflows/test-model-map.yaml):
   - JSON syntax validation (fast fail)
   - Comprehensive schema validation via Python script
   - Validates both main file and backup file
   - Checks backup freshness

3. New validation script (scripts/validate_model_cost_map.py):
   - Validates all entries have required fields
   - Checks mode values against allowed modes
   - Validates numeric fields are actually numeric
   - Sanity checks for suspiciously high costs
   - Detailed error reporting

4. Unit tests for validation logic

The key improvement: if GitHub returns valid JSON but semantically
broken data (missing fields, wrong types), we now detect this and
fall back to the local backup instead of using the bad data.

Customers can also set LITELLM_LOCAL_MODEL_COST_MAP=True to completely
disable remote fetching for maximum stability.

Co-authored-by: ishaan <ishaan@berri.ai>
2026-01-30 21:26:58 +00:00
..
agent_tests Fix CI: Revert security scan changes and add GitGuardian ignore rules (#18358) 2025-12-22 17:03:53 -08:00
audio_tests [Feat] New provider TTS - Add AWS polly API for TTS (#18326) 2025-12-22 18:19:34 +05:30
basic_proxy_startup_tests Fix: remove merge markdown (#17586) 2025-12-06 05:38:16 -08:00
batches_tests Add cost tacking and usage info in call_type=aretrieve_batch 2026-01-29 15:27:41 +05:30
code_coverage_tests [Release Day] - Fixed CI/CD issues & changed processes (#19902) 2026-01-28 17:57:24 -08:00
documentation_tests fix: test_env_keys 2026-01-20 18:37:56 +05:30
enterprise Add tests for user level permissions on file and batch access 2026-01-29 12:29:10 +05:30
guardrails_tests [Release Day] - Fixed CI/CD issues & changed processes (#19902) 2026-01-28 17:57:24 -08:00
image_gen_tests Fix: imagegeneration@006 has been deprecated 2026-01-22 18:24:59 +05:30
litellm Fix stream_chunk_builder to preserve images from streaming chunks (#19654) 2026-01-28 21:31:06 -08:00
litellm-proxy-extras fix: resolve 'does not exist' migration errors as applied in setup_database (#19281) 2026-01-26 22:11:36 -08:00
litellm_core_utils fix(bedrock): deduplicate tool calls in assistant history (#15178) (#19324) 2026-01-19 10:56:49 -08:00
litellm_utils_tests feat: hashicorp vault rotate support 2026-01-23 17:32:55 +09:00
llm_responses_api_testing Fix: Responses API logging error for StopIteration 2026-01-23 18:42:33 +05:30
llm_translation Add web search tests 2026-01-30 14:56:11 +05:30
load_tests Add memory leak detection tests with CI integration (#18881) 2026-01-09 17:36:10 -08:00
local_testing Add disable flahg for anthropic gemini cache translation 2026-01-30 14:58:10 +05:30
logging_callback_tests fix: add openinference span kinds to arize phoenix 2026-01-23 16:32:49 -05:00
mcp_tests Merge branch 'main' into litellm_oss_staging_01_26_2026 2026-01-27 17:00:58 +05:30
multi_instance_e2e_tests fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
ocr_tests OCR test fixes 2026-01-11 08:00:31 -08:00
old_proxy_tests/tests remove prompt caching headers as the support has been removed 2026-01-02 11:08:35 +05:30
openai_endpoints_tests test_e2e_batches_files 2025-11-27 09:14:43 -08:00
otel_tests User metrics for promethus 2026-01-07 15:05:09 -08:00
pass_through_tests test_anthropic_messages_to_wildcard_model 2026-01-07 15:03:06 +05:30
pass_through_unit_tests TestBedrockInvokeToolSearch 2026-01-24 15:36:30 -08:00
proxy_admin_ui_tests Make test_get_users_key_count deterministic by creating dedicated test user (#19795) 2026-01-26 10:13:15 -08:00
proxy_e2e_anthropic_messages_tests [Feat] LiteLLM x Claude Agent SDK Integration (#20035) 2026-01-29 17:48:38 -08:00
proxy_security_tests test fix 2025-10-04 10:57:02 -07:00
proxy_unit_tests Add tests for native_background_mode feature 2026-01-27 16:48:22 -08:00
router_unit_tests Add test for Router.get_valid_args, fix router code coverage encoding (#19797) 2026-01-26 10:14:58 -08:00
scim_tests
search_tests skip brave tests 2026-01-22 10:50:23 -08:00
spend_tracking_tests increasing time for spend tracking 2026-01-20 16:31:25 -08:00
store_model_in_db_tests fix: test mock 2026-01-02 17:38:52 +09:00
test_litellm Add validation and fallback protection for model cost map 2026-01-30 21:26:58 +00:00
unified_google_tests Revert "[Infra] Changing Google Tests to use Gemini 3 Flash Preview" 2026-01-20 17:32:10 -08:00
vector_store_tests [Feat] RAG API - Add s3_vectors as provider on /vector_store/search API + UI for creating + PDF support for /rag/ingest (#19895) 2026-01-27 16:30:59 -08:00
windows_tests
__init__.py [Feat] Add github co-pilot as a new LLM API provider (#12325) 2025-07-04 13:12:16 -07:00
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
README.MD [Feat] MCP Gateway Fine-grained Tools Addition (#15153) 2025-10-03 10:16:29 -07:00
test_budget_management.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_callbacks_on_proxy.py Fix CI: Revert security scan changes and add GitGuardian ignore rules (#18358) 2025-12-22 17:03:53 -08:00
test_config.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_debug_warning.py
test_default_encoding_non_root.py fix: resolve Read-only file system error in non-root images (#19449) 2026-01-20 19:00:52 -08:00
test_end_users.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_entrypoint.py
test_fallbacks.py Ollama Chat - parse tool calls on streaming (#11171) 2025-05-27 16:14:49 -07:00
test_gpt5_azure_temperature_support.py Fix: Azure GPT-5 incorrectly routed to O-series config (temperature parameter unsupported) (#16246) 2025-11-07 10:24:27 -08:00
test_health.py
test_keys.py Adding retries to flaky tests 2026-01-22 15:21:44 -08:00
test_litellm_proxy_responses_config.py Add native Responses API support for litellm_proxy provider (#15347) 2025-10-08 18:31:26 -07:00
test_logging.conf
test_models.py test_add_model_run_health 2025-09-27 10:59:25 -07:00
test_openai_endpoints.py test fix claude-sonnet-4-5-20250929 2025-10-28 19:05:13 -07:00
test_organizations.py Adding retries to flaky tests 2026-01-22 15:21:44 -08:00
test_otel_thread_leak.py Fix thread leak in OpenTelemetry dynamic header path (#19946) 2026-01-28 10:35:37 -08:00
test_passthrough_endpoints.py fix: update authorization header to use 'Bearer' instead of 'bearer' 2025-09-21 10:44:47 +00:00
test_presidio_latency.py fix(presidio): reuse HTTP connections to prevent OOMs (#19964) 2026-01-28 16:08:53 -08:00
test_proxy_server_non_root.py deactivating non root tests 2026-01-23 22:55:36 -08:00
test_ratelimit.py test_async_rate_limit 2025-11-15 10:34:24 -08:00
test_resource_cleanup.py Fix: Properly close aiohttp client sessions to prevent resource leaks (#12251) 2025-07-09 09:25:17 -07:00
test_service_logger_otel.py fix(langfuse_otel): ignore service logs and fix callback shadowing (#19298) 2026-01-19 05:53:47 -08:00
test_spend_logs.py Fix CI: Revert security scan changes and add GitGuardian ignore rules (#18358) 2025-12-22 17:03:53 -08:00
test_team.py test_team_update_sc_2 2026-01-24 13:17:32 -08:00
test_team_logging.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_team_members.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_users.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.