scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214
litellm/tests/integration/observability
History
Exact
quietjules 534a60d28d
fix(proxy): name the deployment in stream error alerts on chat, completions, responses and messages (#51034) 
* fix(proxy): name the deployment in stream error alerts on every provider and route

The llm_exceptions alert for a cut stream on Anthropic chat, /v1/completions,
native /v1/responses and native /v1/messages had no API Base, model_group or
deployment line. The shared exception mapper now attaches its debug info at its
single exit, mid-stream fallback errors inherit it from the error they wrap,
and the native relays add it to the same error object they re-raise.

* fix(proxy): keep the prompt out of attached alert debug info and name the deployment on more stream exits

* fix(proxy): name the deployment on stream errors that arrive already mapped

exception_type returned errors that were already litellm types before it attached the debug info, so a finish_reason=abort chunk or the max streaming duration timeout still sent an alert with no deployment. It now builds the info first and attaches the prompt-free copy on that early return too, and native Responses streams attach it on their generic error exit

Also adds tests for the dropped `Messages: None` line and the mapping-crash exit

* fix(proxy): read the router's deployment from litellm_metadata on Messages and Responses errors

/v1/messages and /v1/responses keep the router's model_group, deployment and api_base in litellm_metadata, so their errors before the stream named only the model in alerts. The debug info now reads litellm_metadata first, falls back to metadata, and takes API Base from it when the call passes none

get_first_chars_messages returns an empty string again for input that is not a mapping, as it did before this branch

* fix(proxy): merge both metadata buckets when error alerts name the deployment

A pre_call guardrail adds litellm_metadata to a chat request while the router keeps model_group and deployment in metadata, and generic routes such as moderations keep the key alias and team in metadata and the deployment in litellm_metadata. Reading only one bucket dropped lines the merge base showed. The debug info now merges both, and litellm_metadata wins when both carry the same key, so a caller's own metadata cannot replace the router's values

* fix(proxy): keep a reused error's alert details on the request that raised it

An exception instance raised for more than one request, such as an SDK mock_response exception or a handler that raises one shared error, kept the first request's Key Name, model_group and deployment in every later alert

The attached details now record the call id that set them. A later call with a different id replaces them, and a layer that re-maps the error without the request keeps them

* fix(proxy): drop alert details from an error object raised by more than one request

One exception object raised by two concurrent requests (a shared mock_response error, or a
handler that raises one error object for every request) let a request's alert show the other
request's Key Name and deployment, since the later attach replaced the earlier one. An error
object tagged by one call id and attached again under another now drops its details for good,
so no alert names the wrong request. One request re-mapping its own error, with or without a
call id, keeps the lines it already has

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-11 22:50:23 +00:00
..
_azure_storage_support.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
_langfuse_otel_metadata_support.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
_logging_only_scope_support.py test(integration): fix four flaky integration tests (#49554) 2026-11-02 09:23:17 -08:00
_openinference_support.py feat(proxy): let a key's logging entry reference a proxy credential (#50572) 2026-11-09 16:02:46 -08:00
_s3_v2_support.py fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
_ternary_support.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
azure_dispatch_support.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
conftest.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
logging_worker_drain_canary.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_akto_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_anthropic_overloaded_failure_signals.py fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes (#49993) 2026-11-04 16:37:55 -08:00
test_apply_guardrail_endpoint.py fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) 2026-10-30 06:58:05 -07:00
test_arize_key_credential_reference.py feat(proxy): let a key's logging entry reference a proxy credential (#50572) 2026-11-09 16:02:46 -08:00
test_arize_otel_v2_openinference_chaos.py test(observability): reserve sink port throughout outage (#50100) 2026-11-04 21:07:21 +00:00
test_arize_otel_v2_openinference_family.py test(integration): repair tests that merged changes left stale (#50782) 2026-11-10 01:08:35 +00:00
test_arize_otel_v2_openinference_modes.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_arize_otel_v2_openinference_sad_edge.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_arize_otel_v2_openinference_spans.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_azure_content_safety_audit.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_content_safety_dispatch.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py test(integration): budget proxy restart tests from calibrated boot time (#49761) 2026-11-07 15:37:18 -08:00
test_azure_content_safety_endpoints.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_chaos.py test(integration): budget proxy restart tests from calibrated boot time (#49761) 2026-11-07 15:37:18 -08:00
test_azure_storage_client_ttl.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_file_names.py fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099) 2026-10-01 23:23:37 -07:00
test_batch_line_items_cold_storage.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_bedrock_converse_guardrail_rewrites.py fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) 2026-10-17 20:38:14 -07:00
test_bedrock_converse_stream_post_call_guardrail.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_bedrock_error_request_id.py fix(responses): preserve provider request IDs in streaming failure logs (#45453) 2026-10-16 00:29:11 -07:00
test_bedrock_guardrail_attachments.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_bedrock_invoke_stream_post_call_guardrail.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_cache_hit_guardrail_metrics.py test(integration): repair release qualification timing and labels (#49737) 2026-11-02 18:02:28 -08:00
test_cache_hit_guardrail_metrics_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_callback_delivery.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_content_filter_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_custom_logger_hook_counts.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_chaos.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_modes.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_post_call_block.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_vendors.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_redaction.py fix(logging)!: redact Responses instructions and callback output (#43315) 2026-10-20 20:38:17 +00:00
test_datadog_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_generic_guardrail_api_wire.py fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338) 2026-10-16 14:52:35 -07:00
test_generic_guardrail_num_retries.py feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) 2026-10-17 12:22:41 -07:00
test_grayswan_wire.py fix(grayswan): send request conversation and tool calls to post-call monitor (#43770) 2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_guardrail_effects.py fix(proxy)!: preserve Responses SSE event framing (#47980) 2026-11-06 20:17:57 -08:00
test_guardrail_load_balancing.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_guardrail_pipeline_applied_header.py fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails (#49264) 2026-10-30 17:00:44 -07:00
test_guardrail_stream_scope.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_stream_scope_chaos.py test(integration): remove BUG skips for 3 tests that pass on main (#47364) 2026-10-19 21:39:35 -07:00
test_guardrail_stream_scope_matrix.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_timeout_all_providers.py feat(guardrails): add Wingback guardrail integration (#46136) 2026-10-12 22:18:10 -07:00
test_langfuse_delivery.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_langfuse_otel_console_output.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
test_langfuse_otel_legacy_destinations.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_metadata_chaos.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_langfuse_otel_metadata_clients.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_langfuse_otel_metadata_destinations.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_identity.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_tenant_only.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_trace_fields.py fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705) 2026-10-27 10:34:55 -07:00
test_langsmith_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_chaos.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_logging_only_scope_config.py test(integration): repair release qualification timing and labels (#49737) 2026-11-02 18:02:28 -08:00
test_logging_only_scope_runtime.py test(integration): fix four flaky integration tests (#49554) 2026-11-02 09:23:17 -08:00
test_logging_worker_drain.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_otel_auto_router_model.py fix(otel): name the routed model on auto-router LLM spans (#50498) 2026-11-07 15:52:00 -08:00
test_otel_cache_hit_cost.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_otel_conversation_id.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_otel_destination_metadata_keys.py feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813) 2026-11-11 13:57:41 -08:00
test_otel_excluded_services.py test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) 2026-10-13 09:55:11 -07:00
test_otel_excluded_services_matrix.py test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846) 2026-10-15 22:35:23 -07:00
test_otel_team_capture_message_content.py test(integration): wait for the stalled-tenant root span and accept a bare POST name (#49406) 2026-10-31 12:24:33 -07:00
test_otel_team_request_metadata.py fix(otel): seed request metadata baggage for team-only OTel v2 destinations (#50373) 2026-11-06 14:47:48 -08:00
test_otel_team_sampling_across_pods.py fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468) 2026-10-26 11:44:30 -07:00
test_otel_text_completion_choices.py fix(otel): keep text completion choice fields beside the synthesized message (#42537) 2026-09-22 14:24:32 -07:00
test_otel_trace_response_header.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_otel_v1_request_trace.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_ovalix_guardrail.py feat(ovalix)!: scan files, tool calls and tool results (#49309) 2026-10-31 17:01:12 -07:00
test_panw_airs_block_masked_response.py feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block (#46219) 2026-10-29 10:45:00 -07:00
test_passthrough_guardrail_scan_text.py fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
test_passthrough_operation_name.py fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) 2026-10-28 10:48:34 -07:00
test_passthrough_upstream_error_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_upstream_error_visibility.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_presidio_entity_masking.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_presidio_masking_endpoints.py test(observability): match the full card number instead of a 4111 substring that timestamps can contain (#50612) 2026-11-08 06:01:49 +00:00
test_presidio_overlapping_spans.py fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) 2026-10-17 11:21:03 -07:00
test_presidio_streaming_output.py fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) 2026-10-17 16:48:37 +00:00
test_prometheus_deployment_failure_api_provider.py fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384) 2026-10-19 13:38:26 -07:00
test_prometheus_project_alias_label.py fix(prometheus): skip custom metadata labels a metric already carries (#46267) 2026-10-13 18:20:52 -07:00
test_prometheus_provider_auth_failure.py test(integration): repair stale contract fixtures (#49700) 2026-11-02 11:23:11 -08:00
test_prometheus_provider_auth_failure_chaos.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_request_metrics.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py test(integration): fix the baseline accounting and series cap flakes (#51096) 2026-11-11 12:36:02 -08:00
test_prometheus_series_cap_chaos.py feat(docker)!: one shipped image and one entrypoint beside the Dockerfile (#43332) 2026-11-10 18:32:16 -08:00
test_prometheus_shared_custom_labels.py test(integration): wait on the right signal in five flaky integration contracts (#46737) 2026-10-15 13:48:51 -07:00
test_realtime_guardrail_translation.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py test(integration): look up owned cache keys and clean up sweep models (#47255) 2026-10-18 10:27:53 +00:00
test_s3_v2_partition_granularity.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_s3_v2_streaming_failure_dedupe.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_signoz_delivery.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_straiker_v3_platform.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_stream_error_alert_api_base.py fix(proxy): name the deployment in stream error alerts on chat, completions, responses and messages (#51034) 2026-11-11 22:50:23 +00:00
test_ternary_focus_export.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
test_weave_key_scoped_delivery.py test: move Sail, Weave and Presidio e2e tests to integration tests (#49533) 2026-10-31 18:31:01 -07:00
test_wingback_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_xecguard_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00