mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
8bb8628ab5
* feat(otel): record the GenAI duration metric on failed requests `_record_metrics` ran only from `async_log_success_event`, so `gen_ai.client.operation.duration` counted only the requests that worked. Latency read off it during an incident was the latency of the surviving traffic, and with no error dimension anywhere there was no way to build a failure-rate panel or a success/failure split per model. A failed call now records the same duration histogram, tagged with the semconv `error.type` (the mapped provider exception's class name, bounded by construction; the message stays on the span). Success attributes are untouched, so an existing query can still isolate the old series with `error_type=""`. The other five instruments describe a completed generation and are skipped rather than filled with a fabricated zero: litellm hands the failure callback no `response_obj`, so there is no usage to split and no completion-token count, and it zeroes `response_cost` on failure. A proxy-gate rejection (auth / rate limit) records nothing, for the same reason it gets no span; no upstream call happened. `error.type` is stamped after the cardinality filter, like `gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot strip the discriminator and silently merge failures into the success series. Resolves LIT-4955 * fix(otel): bound the failure metric's attribute set The failure datapoint reused the success path's full attribute set, which carries client-supplied fields (`metadata.requester_metadata`, `metadata.spend_logs_metadata`, the end-user id taken from the request's `user` field) and per-request ones (the `hidden_params` blob holding the provider's response headers). A failed request needs no provider spend, so nothing rate-limits a caller who puts a unique value in a field they control and mints one histogram series per request. A failure now carries a bounded allowlist: the operation enum, provider, request model, framework, the key/alias/team/org/user identifiers, and `error.type`. Every entry is a fixed enum or an operator-provisioned identifier, so the failure series count is bounded by the deployment's own key, team and user count while the labels still answer which team on which model is failing and how. The user email is left out as PII duplicating the user id already on the series. The operator's `otel.attributes` filter layers on top, so it narrows the allowlist further and never widens it. * fix(otel): cap metric attributes so series count does not grow with traffic (#35166) `GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object onto every metric datapoint as one label value. That object is per-request by construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`, `usage_object` and the provider's `additional_headers` rate-limit counters all move on every call. A unique label value is a new time series, and all six GenAI instruments share those attributes, so one request minted up to six series that would never be written to again That is the steady-state behavior of the feature rather than an abuse case, and it is wrong twice over. Hosted backends bill on series count, so recommending metrics be enabled would have meant a bill proportional to traffic. And a histogram whose every datapoint sits in its own series cannot be aggregated, so the dashboards would have looked populated while answering nothing Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces the failure-only allowlist so the two paths cannot drift. The cap runs before the operator's `otel.attributes` filter, so an operator can narrow it and never widen it back to an unbounded label. Client-supplied and per-request metadata (`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`, `requester_ip_address`) is metric-ineligible and stays on the span, which already carries it and where cardinality is free. `hidden_params` survives as a label but carries only `model_id` and `api_base`, which are bounded by the router's own deployment list and are the part a per-deployment panel reads Four tests fail against the previous behavior, the load-bearing one being that two requests differing only in per-request fields must land in one series rather than two |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.