scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-09 03:18:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 109
litellm/tests
History
mateo-berri 7d999a1586 fix(cognition): price swe-1.7 at the standard tier, add swe-1.7-lightning 
The cost map shipped cognition/swe-1.7 at $2.50 in / $12.50 out per million with
$1.00 cache reads. Those are the Lightning numbers. Cognition's own model list at
https://docs.devin.ai/desktop/models has uid swe-1-7 at $0.50 / $2.50 with $0.20
cache reads, and uid swe-1-7-lightning at $2.50 / $12.50 with $1.00 cache reads,
so every swe-1.7 call has been costed at 5x since the entry landed.

swe-1.7 now carries the standard rates and the Lightning tier gets its own entry,
in both cost map copies. The source field on both moves to the desktop models page,
which is the one that lists both tiers.
2026-08-20 19:41:12 -07:00
..
agent_tests test: repair stale CircleCI contracts 2026-08-08 12:19:29 -07:00
audio_tests test: remove tests that never execute 2026-08-12 10:45:38 -07:00
base_sdk_tests test(cli): cover the keyless token record and keep keyring to the cli extra 2026-08-20 03:45:00 -07:00
basic_proxy_startup_tests
batches_tests test: build redaction and batch limiter fixtures the way production does (#37416) 2026-08-18 19:37:50 -07:00
benchmarks
code_coverage_tests test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
documentation_tests ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
e2e Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_5870_passthrough_e2e_pins_wt 2026-08-20 17:33:35 -07:00
enterprise feat(ui): support project input and output TPM limits (#37676) 2026-08-20 17:43:52 -07:00
guardrails_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
image_gen_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
integration
litellm/proxy/_experimental/mcp_server test: merge three stranded twins into the files that shadow them (#37600) 2026-08-20 14:25:23 -07:00
litellm-proxy-extras test: remove tests that never execute 2026-08-12 10:45:38 -07:00
litellm_utils_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
llm_responses_api_testing test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
llm_translation test: point the live together_ai suites at a model together still serves (#37746) 2026-08-20 18:53:27 -07:00
load_tests
local_testing test: point the live together_ai suites at a model together still serves (#37746) 2026-08-20 18:53:27 -07:00
logging_callback_tests test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
mcp_tests test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
multi_instance_e2e_tests
ocr_tests test(ocr): update Azure DI supported-params assertion for req_format 2026-08-18 19:21:46 -07:00
openai_endpoints_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
otel_tests
pass_through_tests
pass_through_unit_tests test: allow protocol-constrained pass-through routes to declare fewer methods (#37415) 2026-08-18 19:38:05 -07:00
proxy_admin_ui_tests fix(access groups): sync assigned_team_ids from the team write paths (#36825) 2026-08-14 04:45:36 +00:00
proxy_behavior feat(proxy): add /team/daily/activity/aggregated and switch the Usage team tab to it (#36562) 2026-08-18 11:29:57 -07:00
proxy_e2e_anthropic_messages_tests ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
proxy_migration_tests test(docker): gate the componentized gateway and backend images on an arbitrary-uid offline boot (#36136) 2026-08-07 09:57:37 -07:00
proxy_security_tests
proxy_unit_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
router_unit_tests feat(complexity_router): custom classifier plugins via classifier_type 'custom' (#37249) 2026-08-18 14:09:19 -07:00
search_tests Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr32448_tinyfish_headers 2026-08-18 13:03:33 -07:00
spend_tracking_tests
store_model_in_db_tests test: refresh three suites that drifted from the code they cover 2026-08-15 15:37:09 -07:00
test_litellm fix(cognition): price swe-1.7 at the standard tier, add swe-1.7-lightning 2026-08-20 19:41:12 -07:00
unified_google_tests
vector_store_tests test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
windows_tests
__init__.py
_fake_openai_endpoint_server.py test(ci): serve /moderations from the canned OpenAI mock (#37739) 2026-08-20 16:58:38 -07:00
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_vcr_conftest_common.py
_vcr_redis_persister.py
_wait_helpers.py test: replace blind sleeps with deadline waits in callback and caching tests (#37660) 2026-08-20 18:48:43 +00:00
_ws_vcr.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_service_logger_otel.py
test_spend_logs.py
test_team.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_team_logging.py
test_team_members.py
test_users.py

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.