mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
948e5755eb
* test(e2e): cover presidio post_call, tool_permission, and weave logging cells Five registry cells in Logging & Guardrails had no covering test. Each one now has a live scenario read back from the real destination: - guardrail.presidio.post_call.masks: an output-scoped Presidio guardrail anonymizes the PII the model repeats back. The prompt also asks for the address's local part, which Presidio does not mask, so one response proves the model saw the raw address (no pre-call masking) while the address itself comes back as <EMAIL_ADDRESS> - guardrail.tool_permission.pre_call.blocks / .allows: an allow-list of one tool. A request declaring an unlisted tool is rejected 400 naming it; a request declaring the permitted tool is served and carries x-litellm-applied-guardrails, so the allow half cannot pass by the guardrail never running - logging.niche_integrations.success.logs_spend / .failure.logs_spend: a key-scoped weave_otel callback delivers to the real Weave project, read back through Weave's query API. Success asserts exactly one call whose llm.response.cost equals the x-litellm-response-cost header; failure asserts one ERROR-status call naming the provider exception and carrying no cost Logging & Guardrails coverage goes 24/59 to 29/59. No registry rows are added. * test(e2e): make the tool-permission allow case deterministic and scope the Weave read-back Review follow-ups on the coverage PR. - the allow scenario forced the outcome to depend on whether the model felt like calling an optional tool, and checked for the tool name as a substring of the whole body, which a prose mention would satisfy. It now sends tool_choice="required" and asserts the parsed response carries exactly one tool call, for the permitted tool - the Weave read-back queried the newest 200 calls of a shared project and filtered client-side, so busy traffic could push the target out of the window and read as a delivery failure. The query now scopes server-side to the litellm_request op and to calls started after the request, and pages through the window with offset - the reader builds its results as tuples instead of accumulating into lists Also unblocks the lint gate: `basedpyright tests/e2e` runs only on PRs that touch tests/e2e, and it has been failing on staging for three FakeItem arguments in test_junit_properties.py. The stand-in now goes through one typed adapter that says why, so the gate is green without touching junit_properties.py itself. * test(e2e): scope the presidio post_call guardrail to email and phone Running the suite three times in a row caught a real flake: Presidio's broader recognizers sometimes claim the email's local part as an NRP entity, so the answer came back as `<NRP>\n<EMAIL_ADDRESS>\n<PHONE_NUMBER>` and the assertion that the raw local part survives failed. That token is what tells output masking apart from input masking, so it has to survive. The post_call guardrail now registers pii_entities_config for EMAIL_ADDRESS and PHONE_NUMBER only, which is also the narrower thing the scenario means. Verified against the exact marker that failed, plus two others. * test(e2e): mark weave logging cells stage red * test(e2e): use per-test stage red skips for the weave logging cells |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| rust-python-harness | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_rust_python_harness.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.