mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
978aa2816b
* test(e2e/ui): stop the suite failing on things that are not regressions Five tests in the UI suite fail for reasons that have nothing to do with the product being broken, which is enough to keep the whole leg red. Two need a premium proxy and fail hard without one: Regenerate Key renders disabled when the proxy is unlicensed, and /model/new refuses a team-scoped deployment. Both now skip without LITELLM_LICENSE, the way three other tests in this suite already do. Three consumed a seeded fixture: Delete key, Delete a team and remove a member each destroyed the row they needed, so the retries CI runs with were guaranteed to fail and the suite could not run twice against one database. They now create what they destroy. Top Virtual Keys ranks by spend and every mock deployment costs $0, so which keys make the list came down to how ties happened to sort. It now sends its traffic through a priced deployment and earns its place. * test(e2e/ui): clean up the fixtures these tests create Review caught two leaks: the priced deployment the usage test registers and the user the team-admin test adds both outlived the run, so repeated runs grew shared state that later routing and rosters can see. Also brings in the paginated daily-activity read. /user/daily/activity pages its per-key breakdown and the helper only read the first page, so the usage test spent its full timeout blaming the rollup for a key the rollup wrote. * test(e2e/ui): read the licence from the proxy, not the runner Review pointed out that checking LITELLM_LICENSE in the runner's environment describes the wrong machine: Playwright can be pointed at a proxy configured somewhere else, and then the skip either hides coverage or runs a premium test against an unlicensed target. The admin session JWT already carries the premium_user claim the dashboard itself reads to enable these controls, so both skips now use that. * test(e2e/ui): clean up fixtures on the failure path too Review caught both cleanups sitting at the end of the test body, where a failing assertion skips them, and both discarding the response so a refused delete passed quietly. They move to afterEach and assert the delete landed. The priced deployment matters most: left behind it keeps its custom pricing and goes on changing what later runs route and what they cost. * test(e2e/ui): wait for the priced deployment to become routable The Top Virtual Keys test registered a priced deployment and sent the key's traffic through it on the next line, so on the deployed stack it failed with "no healthy deployments for e2e-usage-priced-...": /model/new had written the row but the router had not picked it up yet. Polls a ping until the deployment answers before the test sends the request it measures, matching what the addModel spec already does for a model added through the UI. A ping that fails writes no spend log, so the retries cannot move the ranking this test asserts. * test(e2e/ui): register fixtures for cleanup before the step that can fail Review found both helpers handing their id back to the caller to record, with a failure-prone call in between: the priced deployment was registered after the routability wait, and the added user after /team/member_add. Either failing left the resource in the shared database with nothing tracking it. Both now take the teardown list and add themselves as soon as the resource exists, so the afterEach removes it however the rest of setup goes. * test(e2e/ui): resolve the priced deployment for teardown by name Review pointed out the remaining gap: /model/new can persist the deployment and still answer non-2xx, and the id was only recorded after the response was asserted, so that path left it behind with its custom pricing. The name is now claimed before the request and teardown looks it up in /model/info, so a create that saved without answering 2xx is still removed and one that never saved is simply not there. * test(e2e/ui): claim the member id before creating the user /user/new can persist the user and still answer non-2xx, and the id is chosen by the test rather than returned by the proxy, so registering it before the call is what closes the last create-failure path. Teardown now skips an id whose user is not there, so claiming it up front cannot fail a run where the create never landed. * test(e2e/ui): wait out the router reload when resolving a deployment to delete /model/info answers from the router, not from the database, and /model/new catches and logs a failed in-request reload while still answering 2xx. A deployment can therefore be persisted and absent from the listing until the next reload, which is where teardown was giving up and leaking it. Teardown now retries the lookup for a little over one PROXY_CONFIG_RELOAD_INTERVAL_SECONDS before treating the name as never persisted, so the only names it skips are the ones that really are not there. * test(e2e/ui): prove a stored credential survives a config reload before using it The Test Connect assertion has been failing intermittently on the full-suite runs. Artifacts from litellm-e2e-ui build 165 show the UI sending litellm_credential_name and the proxy answering with the credential unapplied: raw_request_api_base was https://api.openai.com/v1/ rather than the mock base the credential carries, and the call died on an upstream 404 for the model. The same credential had resolved on three probes eight seconds earlier. The proxy's periodic credential refresh takes a database snapshot, prunes any in-memory credential missing from it, then re-adds the snapshot. A credential created while that is in flight gets pruned and stays gone until the next tick, and load_credentials_from_list fails open onto the ambient key, so nothing in the error names the credential. The existing pre-check asked for three consecutive probe successes, but they completed in under a second, so they could not span a refresh. Space them so the run covers a whole interval, which is what proves the credential survived a refresh and is therefore stable. * test(e2e/ui): find a database-only deployment through the search listing /model/info answers from the router, so a deployment that reached the database while /model/new's in-request reload failed is invisible there, and waiting on the next reload only helps if reconciliation eventually picks it up. /v2/model/info?search= runs a bounded query against the model table and deliberately returns rows the router does not hold, so it resolves those deployments to the id /model/delete needs. Falling back to it removes the wait as well: absent from both listings now means the deployment never persisted. * test(e2e/ui): delete the temporary member without a lookup that can skip it Teardown asked /user/info first and treated any non-2xx as absence, so a transient failure on the lookup silently skipped the delete and left the user behind, which is the leak the claimed id was meant to close. /user/delete answers 404 for an id that is not there, so it can carry both cases on its own: 404 means the create never persisted, and anything else that is not 2xx now fails the teardown instead of passing quietly. * test(e2e/ui): reach the database fallback when the router listing fails Asserting on /model/info threw before the fallback could run, so a failure on the router-backed listing aborted teardown and left the deployment persisted, which is the leak the fallback was added to close. The router listing is best-effort now: an unreadable response just falls through to the search-backed one. That listing is the authoritative answer to whether the deployment exists, so it is the one that has to be readable, and a name missing from it is a create that never persisted. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.