mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
b5028b81c6
The coverage number is documented as static, a property of the source tree, but
the same tree reported 312, 314, or 315 covered cells depending on the machine.
Markers were harvested from session.items in pytest_collection_finish, which
runs after every pytest_collection_modifyitems hook, so anything the runner
dropped vanished from the numerator: the weekly load test is deselected unless
E2E_WEEKLY_ANOMALY is set, and the MCP OAuth module sits behind
pytest.importorskip("mcp") / importorskip("playwright.async_api").
Markers are now read straight off the source with ast, so a cell counts when a
test declaring it exists, whatever the runner does with it. The collect-only
pytest pass stays for the two things the source text cannot give: markers built
at import time (pytest.mark.covers(*fn(...)) inside a pytest.param) and the
nodeids that failed to import, whose cells are genuinely unknowable and still
drive --fail-on-collection-errors. Files that cannot be parsed are reported the
same way instead of being dropped.
The TypeScript Playwright suite emits no pytest markers, so the two
surface: ui mgmt rows were structurally uncoverable. It now declares its cells
in tests/e2e/ui/coverage.yaml, which the collector unions in; an id there that
is not in the registry surfaces as an orphan marker exactly like a mistyped
pytest marker. Each row also names the spec and the Playwright test title behind
it, and both are resolved against the tree on every run, so renaming, deleting
or commenting out that test drops the cell out of the numerator and fails
--strict instead of leaving it counted forever.
Title matching is per declaration, never per file. Comments are stripped before
titles are read, by a string-aware pass that keeps literals intact so a "//"
inside a title is never mistaken for a comment opener. An interpolated title is
matched against its literal segments, so it is checked as far as it can be, and
a dynamic title elsewhere in the spec never exempts a row naming a literal one.
A title with no literal text, or one assembled from variables, backs no row, so
an unmatchable shape fails loudly rather than waving a whole file through. Only
mgmt.key.update.happy_path is claimed, proven by the "Update key TPM and RPM
limits" spec. mgmt.key.generate.happy_path is left unclaimed and documented:
that row scopes itself to SSO-driven key gen and every role in the suite logs in
with username/password.
Pruned llm.embeddings.anthropic.basic.nonstream.works. Anthropic ships no
embeddings API, the anthropic handler's embedding() is a pass stub, no anthropic
dispatch exists in embedding(), and no anthropic row in
model_prices_and_context_window.json carries mode: embedding, so the row could
never pass.
Also corrected the suite-folder list in tests/e2e/CLAUDE.md: gateway/,
embeddings/, security/, and a top-level realtime/ do not exist (realtime lives
under llm_translation/), and guardrails/ was undocumented.
Same tree now reports 316/430 with and without E2E_WEEKLY_ANOMALY and with or
without mcp/playwright installed.
|
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.