mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
715e9bb08d
* fix(prometheus): count provider-side 401s in LLM failure and deployment metrics The invalid virtual key skip no longer applies to an error the LLM provider returned while the deployment was called with its own credential, so a 401 for a bad deployment credential increments litellm_llm_api_failed_requests_metric, litellm_deployment_failure_responses and litellm_deployment_total_requests and marks litellm_deployment_state as a partial outage again. Proxy-gate rejections and a failing credential that belongs to the caller (a rejected per-user credential, an api_key sent with the request, a per-user GitHub Copilot session) keep the skip * fix(prometheus): keep a caller's own Anthropic OAuth token failure out of the deployment metrics A provider 401 on an Anthropic OAuth token the caller forwarded through the proxy is the caller's credential failing, so it stays out of the deployment failure and outage series like a caller-supplied api_key. Adds integration cells for an Anthropic deployment's own key, the forwarded caller OAuth token and a direct SDK call that brings its own key * test(prometheus): pin how a caller's own key or Authorization header on a deployment is counted An SDK Router call that passes its own api_key is counted against the one-off deployment the router builds for that key, so the operator deployment keeps only its own failure. A provider 401 on an Authorization header the caller sends through the proxy counts against the deployment, as on v1.80.12 * fix(prometheus): keep caller-sent provider credentials and token-exchange 401s out of deployment metrics A caller can send its own provider credential in the request body, such as azure_ad_token, azure_scope, the Azure Entra app fields, watsonx token, gigachat_access_token, OCI keys, AWS keys or Vertex credentials, at the top level or inside extra_body. A provider 401 on that credential is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. A token-exchange deployment trades the caller's identity token on every call, so its 401s stay out as well, matching the router cooldown. * test(prometheus): Final locals, precise helper types and one for per comprehension in the new test helpers * test(prometheus): pin that a provider 401 at a caller-sent api_base counts against its one-off deployment * fix(prometheus): keep a provider 401 on caller-sent headers out of deployment metrics A caller can override the deployment's credential with headers it sends in the request body: headers or provider_specific_header, or an Authorization, api-key, x-api-key, x-goog-api-key or Ocp-Apim-Subscription-Key entry in extra_headers. A provider 401 on such a request is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. * fix(prometheus): skip a provider 401 only when the caller's headers carry a credential The proxy now records headers, extra_headers or provider_specific_header in credential_fields only when they name a provider credential header (Authorization, api-key, x-api-key, x-goog-api-key, Ocp-Apim-Subscription-Key), the same way api_key is recorded. A caller that sends a plain header such as a trace id still has a provider 401 on the deployment's own key counted against the deployment. * fix(prometheus): stamp credential_fields after the proxy forwards the caller's headers A provider credential header the proxy forwards from the caller's HTTP request (forward_llm_provider_auth_headers + forward_client_headers_to_llm_api) lands in data["headers"] after proxy_server_request is built, so the stamp missed it and a 401 on that caller credential was counted against the deployment. Compute the stamp after the header forwarding steps and pin both forwarded cases. * fix(prometheus): skip a provider 401 when the caller sends any header or an OpenAI organization The proxy now records headers, extra_headers or provider_specific_header in credential_fields whenever the caller sends or the proxy forwards any header there, and records organization when the caller sends one in the body or as a forwarded OpenAI-Organization header. A forwarded header can be a gateway key such as x-portkey-api-key or an OpenAI-Project id, and a wrong organization gets a 401 from OpenAI, so these 401s stay out of the deployment metrics as on main instead of marking a healthy deployment as a partial outage. * fix(prometheus): ignore the user information the proxy adds when stamping caller headers With add_user_information_to_llm_headers on and header forwarding enabled, the proxy adds x-litellm-user_api_key_* headers to the forwarded headers. Those carry the proxy's own key and user details, not the caller's, so they no longer make a request look caller-scoped, and a provider 401 on the deployment's own key on such a request is counted. * fix(prometheus): skip a 401 raised before the provider is called or at the caller's own endpoint A deployment with no key of its own fails with LiteLLM's local "Missing Anthropic API Key" 401 before any provider call, and a caller who sends aws_bedrock_runtime_endpoint (or another provider endpoint field) can get a 401 from their own gateway. Both were counted against the shared deployment and marked it as a partial outage, where main skipped every 401. Count a 401 only once the provider was called (api_call_start_time is set), and treat caller-sent provider endpoint fields like caller credential fields. * fix(prometheus): count the deployment's own 401 on requests that carry only anthropic-version or anthropic-beta The Anthropic SDK sends anthropic-version on every request, and the proxy copies anthropic-version and anthropic-beta into provider_specific_header for Anthropic, Bedrock and Vertex. Both were treated as caller headers, so a 401 on an Anthropic deployment's own key from any Anthropic SDK client stayed out of the deployment metrics. These two headers only pick the API version and beta features. Anthropic answers a bad anthropic-version or an unknown anthropic-beta with a 400, never a 401, so they cannot turn the deployment's own key into a 401 and no longer mark the request as carrying a caller header. * test(prometheus): read and write the owned proxy's own database in the expired-key and MCP cells Since #47213 an owned proxy boots on a private database, so the expired-key update and the MCP guardrail row read now go through the gateway's database_url instead of the worker's. * fix(prometheus): keep credential_fields as main stamps it and give Prometheus its own caller_credential_fields credential_fields tells the Rust OCR bridge which stripped body fields the caller sent. Repurposing it for the Prometheus caller-credential check changed its value (organization and extra_headers added, anthropic-version-only headers dropped). Restore main's stamp unchanged and stamp the Prometheus classification under caller_credential_fields, which only Prometheus reads. * fix(prometheus): skip a provider 401 on a stored credential the caller picks by litellm_credential_name A request body litellm_credential_name swaps the deployment's stored credential for one the caller chose, so a 401 on it is the caller's, not the deployment's. Treat it like the other caller credential fields so it no longer marks a healthy shared deployment as a partial outage. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _azure_storage_support.py | ||
| _logging_only_scope_support.py | ||
| _openinference_support.py | ||
| _s3_v2_support.py | ||
| _ternary_support.py | ||
| azure_dispatch_support.py | ||
| conftest.py | ||
| logging_worker_drain_canary.py | ||
| test_akto_logging_only.py | ||
| test_apply_guardrail_endpoint.py | ||
| test_arize_otel_v2_openinference_chaos.py | ||
| test_arize_otel_v2_openinference_family.py | ||
| test_arize_otel_v2_openinference_modes.py | ||
| test_arize_otel_v2_openinference_sad_edge.py | ||
| test_arize_otel_v2_openinference_spans.py | ||
| test_azure_content_safety_audit.py | ||
| test_azure_content_safety_dispatch.py | ||
| test_azure_content_safety_dispatch_resilience.py | ||
| test_azure_content_safety_endpoints.py | ||
| test_azure_storage_chaos.py | ||
| test_azure_storage_client_ttl.py | ||
| test_azure_storage_file_names.py | ||
| test_batch_line_items_cold_storage.py | ||
| test_bedrock_converse_guardrail_rewrites.py | ||
| test_bedrock_converse_stream_post_call_guardrail.py | ||
| test_bedrock_error_request_id.py | ||
| test_bedrock_guardrail_attachments.py | ||
| test_bedrock_invoke_stream_post_call_guardrail.py | ||
| test_cache_hit_guardrail_metrics.py | ||
| test_cache_hit_guardrail_metrics_chaos.py | ||
| test_callback_delivery.py | ||
| test_content_filter_logging_only.py | ||
| test_custom_logger_hook_counts.py | ||
| test_custom_logger_hook_counts_chaos.py | ||
| test_custom_logger_hook_counts_modes.py | ||
| test_custom_logger_hook_counts_post_call_block.py | ||
| test_custom_logger_hook_counts_vendors.py | ||
| test_custom_logger_redaction.py | ||
| test_datadog_logging_delivery.py | ||
| test_file_usage_counter_span.py | ||
| test_generic_guardrail_api_wire.py | ||
| test_generic_guardrail_num_retries.py | ||
| test_grayswan_wire.py | ||
| test_grayswan_wire_chaos.py | ||
| test_guardrail_attachment.py | ||
| test_guardrail_effects.py | ||
| test_guardrail_load_balancing.py | ||
| test_guardrail_stream_scope.py | ||
| test_guardrail_stream_scope_chaos.py | ||
| test_guardrail_stream_scope_matrix.py | ||
| test_guardrail_timeout_all_providers.py | ||
| test_langfuse_delivery.py | ||
| test_langsmith_logging_delivery.py | ||
| test_langtrace_delivery.py | ||
| test_logging_only_scope_chaos.py | ||
| test_logging_only_scope_config.py | ||
| test_logging_only_scope_runtime.py | ||
| test_logging_worker_drain.py | ||
| test_otel_cache_hit_cost.py | ||
| test_otel_conversation_id.py | ||
| test_otel_excluded_services.py | ||
| test_otel_excluded_services_matrix.py | ||
| test_otel_team_capture_message_content.py | ||
| test_otel_team_sampling_across_pods.py | ||
| test_otel_text_completion_choices.py | ||
| test_otel_trace_response_header.py | ||
| test_otel_v1_request_trace.py | ||
| test_passthrough_guardrail_scan_text.py | ||
| test_passthrough_upstream_error_chaos.py | ||
| test_passthrough_upstream_error_visibility.py | ||
| test_presidio_entity_masking.py | ||
| test_presidio_overlapping_spans.py | ||
| test_presidio_streaming_output.py | ||
| test_prometheus_deployment_failure_api_provider.py | ||
| test_prometheus_project_alias_label.py | ||
| test_prometheus_provider_auth_failure.py | ||
| test_prometheus_provider_auth_failure_chaos.py | ||
| test_prometheus_request_metrics.py | ||
| test_prometheus_series_cap.py | ||
| test_prometheus_series_cap_chaos.py | ||
| test_prometheus_shared_custom_labels.py | ||
| test_realtime_guardrail_translation.py | ||
| test_s3_v2_flush_surfaces.py | ||
| test_s3_v2_partition_granularity.py | ||
| test_s3_v2_streaming_failure_dedupe.py | ||
| test_s3_v2_upload_fanout.py | ||
| test_signoz_delivery.py | ||
| test_straiker_v3_platform.py | ||
| test_stream_error_alert_api_base.py | ||
| test_ternary_focus_export.py | ||
| test_wingback_guardrail.py | ||
| test_xecguard_wire.py | ||