scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 178
litellm/tests
History
mateo-berri 7b8cc0319e fix(proxy-extras): only spend a migrate-deploy attempt when a pass made no progress 
The v2 migration resolver gave `prisma migrate deploy` four attempts, and
every recovery path ended in a bare `continue`, so each one burned an attempt.
A database first brought up with `--use_prisma_db_push` has a full schema and
no migrations ledger, so the baseline spent attempt one and the first three
migrations whose objects already existed spent the rest. The proxy then exited
before binding its port, and that database could never be moved onto the
resolver.

The retry budget now counts only attempts that got nowhere. Creating the
baseline, and each migration newly marked applied, leaves the budget alone, so
a push-created database works through its pre-existing objects one pass at a
time. Timeouts, deadlock rollbacks, advisory-lock waits, and a repeat of a
recovery that already ran still spend an attempt, so a run that stops making
progress gives up exactly as before.
2026-09-02 23:56:09 -07:00
..
agent_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
audio_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
base_sdk_tests
basic_proxy_startup_tests
batches_tests merge(litellm_internal_staging): reconcile batch observability with per-line resilience 2026-08-24 19:08:40 -04:00
benchmarks
code_coverage_tests fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448) 2026-09-02 18:27:19 -07:00
documentation_tests Merge remote-tracking branch 'origin/main' into litellm_bedrock_messages_disconnect_billing 2026-08-31 08:58:37 -07:00
e2e fix(ui): paginate request logs by session groups server-side (#39257) 2026-09-02 17:54:17 -07:00
enterprise fix(prometheus): bound requested_model label cardinality on client failure paths 2026-09-01 10:34:58 -07:00
guardrails_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
image_gen_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
integration test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
litellm-proxy-extras fix(proxy-extras): only spend a migrate-deploy attempt when a pass made no progress 2026-09-02 23:56:09 -07:00
litellm_utils_tests test(aiohttp): pin NO_PROXY so proxy env cannot hijack the refused-port probe 2026-08-28 12:45:46 -07:00
llm_responses_api_testing fix(guardrails): deliver modify_response block as valid SSE on streaming chat and Responses 2026-08-31 16:01:39 -07:00
llm_translation test: exempt MockTransport request-shape embedding tests from VCR replay 2026-09-01 13:29:12 -07:00
load_tests Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_ruff_dead_test_code 2026-08-24 09:46:56 -07:00
local_testing Merge pull request #39185 from BerriAI/litellm_fix_embedding_encoding_format_suite_break 2026-09-01 13:52:51 -07:00
logging_callback_tests test(vector-store): accept embedding_executor in the Bedrock KB hook fake handler 2026-09-02 19:22:12 -07:00
mcp_tests fix(mcp): follow tools/list pagination from upstream servers (#39172) 2026-09-01 14:34:14 -07:00
multi_instance_e2e_tests test: say whether a match= pattern is a regex or a literal (ruff RUF043) 2026-08-21 16:25:33 -07:00
ocr_tests fix(vertex): avoid duplicate DeepSeek OCR model namespace 2026-09-01 14:39:15 -07:00
openai_endpoints_tests test(responses): expect the 404 OpenAI now returns for an unknown model 2026-09-02 18:08:45 -07:00
otel_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
pass_through_tests fix(anthropic_messages): dispatch deferred spend logging when the client disconnects mid-relay 2026-08-31 10:17:27 -07:00
pass_through_unit_tests test(websearch): carry a reasoned test-quality suppression on the router patch 2026-08-31 22:32:05 -07:00
proxy_admin_ui_tests test(access-groups): give each xdist worker its own fixture ids 2026-08-28 09:35:12 -07:00
proxy_behavior fix(proxy): reset a stuck team member's budget (#37971) 2026-08-25 09:50:09 -07:00
proxy_e2e_anthropic_messages_tests ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
proxy_migration_tests test(proxy-extras): use the modern optional annotation in the deploy budget test 2026-09-02 12:36:29 -07:00
proxy_security_tests
proxy_unit_tests fix(proxy): share per-model budget counters across replicas through the spend counter cache (#39375) 2026-09-02 12:40:59 -07:00
router_unit_tests feat(router): arm safeguard-refusal fallback on generic chains when no content-policy list exists (#39274) 2026-09-02 18:31:25 -07:00
rust-python-harness test: add mistral ocr transformation parity coverage (#39482) 2026-09-02 20:40:00 -07:00
sdk_function_trace refactor(python-bridge): split routes and add shared function tracing (#39031) 2026-09-02 16:26:35 -07:00
search_tests fix(search): harden bing_grounding auth, result cap, status, and cost 2026-08-24 12:26:07 -07:00
spend_tracking_tests
store_model_in_db_tests test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_litellm fix(mcp): never exchange the LiteLLM virtual key as the upstream subject token (#39446) 2026-09-02 20:32:43 -07:00
unified_google_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
vector_store_tests fix(vector-store): carry request metadata into the Router executor built from the router kwarg 2026-09-02 16:53:05 -07:00
windows_tests test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
__init__.py
_fake_openai_endpoint_server.py test(ci): serve /moderations from the canned OpenAI mock (#37739) 2026-08-20 16:58:38 -07:00
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_vcr_conftest_common.py
_vcr_redis_persister.py
_wait_helpers.py test: replace blind sleeps with deadline waits in callback and caching tests (#37660) 2026-08-20 18:48:43 +00:00
_ws_vcr.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
test_fallbacks.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py test: fix staging CI regressions from #38182, #38144, #38265, #37962, and #37969 2026-08-25 23:01:20 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_openai_endpoints.py test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_resource_cleanup.py
test_rust_python_harness.py feat(rust-python-harness): wire existing e2e SDK tests into the matrix (#39463) 2026-09-03 02:50:08 +00:00
test_service_logger_otel.py
test_spend_logs.py
test_team.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_team_logging.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00
test_team_members.py test: reject assertions on a caught error inside except (ruff PT017) 2026-08-21 13:35:08 -07:00
test_users.py test: enforce F811 so a duplicate definition cannot silently replace the first 2026-08-21 12:06:19 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.