scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-01 07:03:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 128
litellm/tests/integration/observability
History
yuneng-jiang fbd126e518
test: fix integration tests that fail on test order, races and intended changes (#49310) 
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
2026-10-30 13:25:47 -07:00
..
_azure_storage_support.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
_langfuse_otel_metadata_support.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
_logging_only_scope_support.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
_openinference_support.py fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) 2026-10-21 02:26:05 +00:00
_s3_v2_support.py fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
_ternary_support.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
azure_dispatch_support.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
conftest.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
logging_worker_drain_canary.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_akto_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_apply_guardrail_endpoint.py fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) 2026-10-30 06:58:05 -07:00
test_arize_otel_v2_openinference_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_arize_otel_v2_openinference_family.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_arize_otel_v2_openinference_modes.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_arize_otel_v2_openinference_sad_edge.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_arize_otel_v2_openinference_spans.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_azure_content_safety_audit.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_content_safety_dispatch.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_azure_content_safety_endpoints.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_azure_storage_client_ttl.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_file_names.py fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099) 2026-10-01 23:23:37 -07:00
test_batch_line_items_cold_storage.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_bedrock_converse_guardrail_rewrites.py fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) 2026-10-17 20:38:14 -07:00
test_bedrock_converse_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_bedrock_error_request_id.py fix(responses): preserve provider request IDs in streaming failure logs (#45453) 2026-10-16 00:29:11 -07:00
test_bedrock_guardrail_attachments.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_bedrock_invoke_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_callback_delivery.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_content_filter_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_custom_logger_hook_counts.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_chaos.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_modes.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_post_call_block.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_vendors.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_redaction.py fix(logging)!: redact Responses instructions and callback output (#43315) 2026-10-20 20:38:17 +00:00
test_datadog_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_generic_guardrail_api_wire.py fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338) 2026-10-16 14:52:35 -07:00
test_generic_guardrail_num_retries.py feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) 2026-10-17 12:22:41 -07:00
test_grayswan_wire.py fix(grayswan): send request conversation and tool calls to post-call monitor (#43770) 2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_guardrail_effects.py test: fix integration tests that fail on test order, races and intended changes (#49310) 2026-10-30 13:25:47 -07:00
test_guardrail_load_balancing.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_guardrail_stream_scope.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_stream_scope_chaos.py test(integration): remove BUG skips for 3 tests that pass on main (#47364) 2026-10-19 21:39:35 -07:00
test_guardrail_stream_scope_matrix.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_timeout_all_providers.py feat(guardrails): add Wingback guardrail integration (#46136) 2026-10-12 22:18:10 -07:00
test_langfuse_delivery.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_langfuse_otel_console_output.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
test_langfuse_otel_legacy_destinations.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_metadata_chaos.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_clients.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_destinations.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_identity.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_tenant_only.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_trace_fields.py fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705) 2026-10-27 10:34:55 -07:00
test_langsmith_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_chaos.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_config.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_logging_only_scope_runtime.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_logging_worker_drain.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_otel_cache_hit_cost.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_otel_conversation_id.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
test_otel_excluded_services.py test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) 2026-10-13 09:55:11 -07:00
test_otel_excluded_services_matrix.py test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846) 2026-10-15 22:35:23 -07:00
test_otel_team_capture_message_content.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_otel_team_sampling_across_pods.py fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468) 2026-10-26 11:44:30 -07:00
test_otel_text_completion_choices.py fix(otel): keep text completion choice fields beside the synthesized message (#42537) 2026-09-22 14:24:32 -07:00
test_otel_trace_response_header.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_otel_v1_request_trace.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_panw_airs_block_masked_response.py feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block (#46219) 2026-10-29 10:45:00 -07:00
test_passthrough_guardrail_scan_text.py fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
test_passthrough_operation_name.py fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) 2026-10-28 10:48:34 -07:00
test_passthrough_upstream_error_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_upstream_error_visibility.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_presidio_entity_masking.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_presidio_overlapping_spans.py fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) 2026-10-17 11:21:03 -07:00
test_presidio_streaming_output.py fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) 2026-10-17 16:48:37 +00:00
test_prometheus_deployment_failure_api_provider.py fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384) 2026-10-19 13:38:26 -07:00
test_prometheus_project_alias_label.py fix(prometheus): skip custom metadata labels a metric already carries (#46267) 2026-10-13 18:20:52 -07:00
test_prometheus_provider_auth_failure.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_provider_auth_failure_chaos.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_request_metrics.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_prometheus_series_cap_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_prometheus_shared_custom_labels.py test(integration): wait on the right signal in five flaky integration contracts (#46737) 2026-10-15 13:48:51 -07:00
test_realtime_guardrail_translation.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py test(integration): look up owned cache keys and clean up sweep models (#47255) 2026-10-18 10:27:53 +00:00
test_s3_v2_partition_granularity.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_s3_v2_streaming_failure_dedupe.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_signoz_delivery.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
test_straiker_v3_platform.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_stream_error_alert_api_base.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_ternary_focus_export.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
test_wingback_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_xecguard_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00