mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-11 03:38:38 +00:00
0223383d94
* fix(e2e): make the datadog read-back find what DataDog actually indexes Live verification of the merged #33604 against real DataDog (us5) exposed three read-back defects that the local-sink tests could never see; all three fixes are verified against the real API: - Marker search: DataDog consumes the shipped JSON message into the event's attributes and leaves the indexed message EMPTY, so the full-text '"marker"' query matched nothing and every test failed with zero events. The query is now '*:*marker*', which scans all attributes (the marker sits in messages.content); verified to return exactly the event for the call. - Rate limit: the Logs Search API budget is 2 requests per 10s org-wide (x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s sat exactly at the limit and the reader hard-failed on the first 429. Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs off and retries up to 5 times; only non-429 failures stay hard fails. - Envelope status: DataDog re-derives the indexed event status from the parsed payload's status attribute ('success') and normalizes it to its OK severity, so the assertion expects 'ok', not the shipped 'info'. Live run: chat_completions and responses pass every assertion including the exact response-cost cross-check; messages red-pins the LIT-4447 duplicate for real (one call -> two sync-sweep copies + one async batch copy, same request id, confirmed in proxy debug logs). The duplicate is race-dependent, so the pin flickers until #33589 lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): datadog log delivery for streamed chat, messages, and responses Rewritten from the dd-sink version (original #33566) to judge delivery on what real DataDog ingested, matching the merged #33604 conversion: the dd_logs reader searches events back through the Logs Search API and the assertions validate the indexed envelope (source:litellm tag, ok status) and the StandardLoggingPayload fields under the event's attributes. Each streamed test drives one STREAMED call per route, asserts the stream actually streamed (event-stream content type, >0 chunks, no upstream error event), then pins exactly one DataDog event whose payload records stream=true, the aggregated token count, and a response_cost equal to the /spend/logs row for the call - a stream's headers ship before its cost exists, so the spend row is the cross-check anchor, and the spend row and DataDog event must also agree on total_tokens. Coverage registry: adds logging.datadog.stream.exports_metric exercised on chat_completions, messages, and responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update test_datadog_log_e2e.py --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.