yuneng-jiang 21ab2cb87b test: update stale integration and guardrail e2e tests to intended behavior (#48665) ... * test(integration): update stale expectations to intended provider behavior Each expectation moved because a merged PR changed behavior on purpose. - chatgpt string input is wrapped as one user message (#41522) - Gemini 2.5 tool media sits beside function_response (#48428) - sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645) - a body base_model is stripped without allow_client_pricing_override (#47639) - ai_decide answers without usage return usage null (#48261) - azure_ai forwards mapped client anthropic-beta values (#48033) - the OpenCode scripted upstream mints a fresh id per request * test(e2e-ui): type the guardrail name before deleting it GuardrailDeleteModal requires the exact name before Delete enables since #45893. All three guardrail delete flows now go through one helper. * test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces, so the gateway's traceresponse now ends in -03. The producer copies the server span's own trace flags, so the tests now accept any two-digit flags with the sampled bit set instead of pinning -01 #48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an object" on purpose, so the [1] body case is renamed non_object_body and expects that 400 * test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test The outage test waited for "Exception while exporting Span batch" in the proxy log, which is OpenTelemetry's wording and no longer appears with 1.45. The test keeps asserting what the gateway guarantees, callers unaffected during the outage, a live gateway, and every span after recovery delivered once with no outage span duplicated * test(guardrails): count the Responses disconnect spend rows #47611 added Since #47611 a streamed /v1/responses disconnect writes its own spend row, so the concurrent disconnect test saw 20 rows where it counted 10 by model group. It now reads each row's call type and expects ten acompletion rows with the outage verdict and ten aresponses rows with success. /v1/messages disconnects still write no row on main, so they are left out of the count rather than pinned as absent The responses-httpx case of the single disconnect test no longer needs its LIT-8603 skip and passes * test(observability): size K4's timeout from the harness boot budget K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to 58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of 30 runs hit pytest-timeout. The cap and the restart gate wait now come from readiness_seconds and graceful_stop_seconds, like other multi-boot tests		2026-10-28 01:11:25 +00:00
..
_azure_storage_support.py	fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914)	2026-09-30 18:17:13 -07:00
_logging_only_scope_support.py	test(integration): stop GET /__observations from draining the upstream log (#47256)	2026-10-18 09:58:05 +00:00
_openinference_support.py	fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)	2026-10-21 02:26:05 +00:00
_s3_v2_support.py	fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022)	2026-09-26 14:58:28 -07:00
_ternary_support.py	feat(focus): add ternary destination for FOCUS export (#46106)	2026-10-12 22:52:40 -07:00
azure_dispatch_support.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
conftest.py	fix(ci): isolate integration workers and repair test fixtures (#45863)	2026-10-11 01:56:02 -07:00
logging_worker_drain_canary.py	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)	2026-10-11 00:17:21 -07:00
test_akto_logging_only.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_apply_guardrail_endpoint.py	fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)	2026-10-14 08:56:05 -07:00
test_arize_otel_v2_openinference_chaos.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_arize_otel_v2_openinference_family.py	fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508)	2026-10-20 11:46:17 -07:00
test_arize_otel_v2_openinference_modes.py	fix(ci): isolate integration workers and repair test fixtures (#45863)	2026-10-11 01:56:02 -07:00
test_arize_otel_v2_openinference_sad_edge.py	fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698)	2026-10-08 15:53:07 -07:00
test_arize_otel_v2_openinference_spans.py	fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508)	2026-10-20 11:46:17 -07:00
test_azure_content_safety_audit.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_azure_content_safety_dispatch.py	fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067)	2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py	test(integration): hand every harness server a port from one per-worker block (#46761)	2026-10-15 16:56:40 -07:00
test_azure_content_safety_endpoints.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_azure_storage_chaos.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_azure_storage_client_ttl.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_azure_storage_file_names.py	fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099)	2026-10-01 23:23:37 -07:00
test_batch_line_items_cold_storage.py	test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)	2026-10-16 22:45:29 +00:00
test_bedrock_converse_guardrail_rewrites.py	fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874)	2026-10-17 20:38:14 -07:00
test_bedrock_converse_stream_post_call_guardrail.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_bedrock_error_request_id.py	fix(responses): preserve provider request IDs in streaming failure logs (#45453)	2026-10-16 00:29:11 -07:00
test_bedrock_guardrail_attachments.py	fix(ci): isolate integration workers and repair test fixtures (#45863)	2026-10-11 01:56:02 -07:00
test_bedrock_invoke_stream_post_call_guardrail.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics_chaos.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_callback_delivery.py	test(integration): repair seven stale integration tests on main (#47600)	2026-10-20 11:21:04 -07:00
test_content_filter_logging_only.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_custom_logger_hook_counts.py	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)	2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_chaos.py	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)	2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_modes.py	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)	2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_post_call_block.py	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)	2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_vendors.py	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)	2026-10-20 15:19:14 -07:00
test_custom_logger_redaction.py	fix(logging)!: redact Responses instructions and callback output (#43315)	2026-10-20 20:38:17 +00:00
test_datadog_logging_delivery.py	test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)	2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py	feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632)	2026-10-06 03:06:36 +00:00
test_generic_guardrail_api_wire.py	fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338)	2026-10-16 14:52:35 -07:00
test_generic_guardrail_num_retries.py	feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965)	2026-10-17 12:22:41 -07:00
test_grayswan_wire.py	fix(grayswan): send request conversation and tool calls to post-call monitor (#43770)	2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py	test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)	2026-10-08 15:55:57 -07:00
test_guardrail_effects.py	test: update stale integration and guardrail e2e tests to intended behavior (#48665)	2026-10-28 01:11:25 +00:00
test_guardrail_load_balancing.py	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)	2026-10-11 00:17:21 -07:00
test_guardrail_stream_scope.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_guardrail_stream_scope_chaos.py	test(integration): remove BUG skips for 3 tests that pass on main (#47364)	2026-10-19 21:39:35 -07:00
test_guardrail_stream_scope_matrix.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_guardrail_timeout_all_providers.py	feat(guardrails): add Wingback guardrail integration (#46136)	2026-10-12 22:18:10 -07:00
test_langfuse_delivery.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_langfuse_otel_trace_fields.py	fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705)	2026-10-27 10:34:55 -07:00
test_langsmith_logging_delivery.py	test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)	2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py	test: update stale integration and guardrail e2e tests to intended behavior (#48665)	2026-10-28 01:11:25 +00:00
test_logging_only_scope_chaos.py	test: update stale integration and guardrail e2e tests to intended behavior (#48665)	2026-10-28 01:11:25 +00:00
test_logging_only_scope_config.py	test(integration): stop GET /__observations from draining the upstream log (#47256)	2026-10-18 09:58:05 +00:00
test_logging_only_scope_runtime.py	test(integration): stop GET /__observations from draining the upstream log (#47256)	2026-10-18 09:58:05 +00:00
test_logging_worker_drain.py	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)	2026-10-11 00:17:21 -07:00
test_otel_cache_hit_cost.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_otel_conversation_id.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_otel_excluded_services.py	test(integration): isolate shared proxy state behind six order-dependent flakes (#46172)	2026-10-13 09:55:11 -07:00
test_otel_excluded_services_matrix.py	test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846)	2026-10-15 22:35:23 -07:00
test_otel_team_capture_message_content.py	feat(otel)!: let a team capture_message_content override the global setting for its OTel v2 destination (#44264)	2026-10-10 13:52:32 -07:00
test_otel_team_sampling_across_pods.py	fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468)	2026-10-26 11:44:30 -07:00
test_otel_text_completion_choices.py	fix(otel): keep text completion choice fields beside the synthesized message (#42537)	2026-09-22 14:24:32 -07:00
test_otel_trace_response_header.py	test: update stale integration and guardrail e2e tests to intended behavior (#48665)	2026-10-28 01:11:25 +00:00
test_otel_v1_request_trace.py	fix(caching): key response cache by router model group in litellm_metadata (#44542)	2026-10-05 17:06:21 +00:00
test_passthrough_guardrail_scan_text.py	fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276)	2026-10-26 12:07:58 -07:00
test_passthrough_upstream_error_chaos.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_passthrough_upstream_error_visibility.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_presidio_entity_masking.py	test(integration): stop GET /__observations from draining the upstream log (#47256)	2026-10-18 09:58:05 +00:00
test_presidio_overlapping_spans.py	fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024)	2026-10-17 11:21:03 -07:00
test_presidio_streaming_output.py	fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096)	2026-10-17 16:48:37 +00:00
test_prometheus_deployment_failure_api_provider.py	fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384)	2026-10-19 13:38:26 -07:00
test_prometheus_project_alias_label.py	fix(prometheus): skip custom metadata labels a metric already carries (#46267)	2026-10-13 18:20:52 -07:00
test_prometheus_provider_auth_failure.py	fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668)	2026-10-26 17:57:06 -07:00
test_prometheus_provider_auth_failure_chaos.py	fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668)	2026-10-26 17:57:06 -07:00
test_prometheus_request_metrics.py	test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)	2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py	feat(prometheus): cap series per metric for every labeled metric (#44420)	2026-10-06 01:28:44 +00:00
test_prometheus_series_cap_chaos.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_prometheus_shared_custom_labels.py	test(integration): wait on the right signal in five flaky integration contracts (#46737)	2026-10-15 13:48:51 -07:00
test_realtime_guardrail_translation.py	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)	2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py	test(integration): look up owned cache keys and clean up sweep models (#47255)	2026-10-18 10:27:53 +00:00
test_s3_v2_partition_granularity.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_s3_v2_streaming_failure_dedupe.py	test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037)	2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py	test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)	2026-10-16 22:45:29 +00:00
test_signoz_delivery.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_straiker_v3_platform.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_stream_error_alert_api_base.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_ternary_focus_export.py	feat(focus): add ternary destination for FOCUS export (#46106)	2026-10-12 22:52:40 -07:00
test_wingback_guardrail.py	test(integration): give each owned proxy a private database and Redis by default (#47213)	2026-10-18 03:37:41 +00:00
test_xecguard_wire.py	refactor(types): replace Any with proven types in 7 files (#43704)	2026-09-29 06:12:58 -07:00