litellm/tests/integration/observability
devin-ai-integration[bot] 02feb67c20
test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037)
* test(integration): build worker stacks concurrently and record proxy boot phases

The controller built every xdist worker's database, Redis, upstream and gateway
proxy one after another before any test ran, about 95 s per extensions shard,
and since every multi-worker suite is now isolated the same serial wait hits
mcp, security and cost too. The stacks now start on a thread pool bounded by
the CPU count. Each stack reads its endpoints and results directory from an
explicit environment instead of a patched os.environ, every stack that started
is registered for cleanup, and a failure in one stops the others before the
error is raised

Owned proxy startup.json now records the wall-clock spawn time, the worker
count and when the Prisma CLI child ran, when the port started accepting and
when the first worker answered readiness, and exit.json records when the stop
started and ended, so the boot time can be split and overlapping gateway boots
can be seen in the artifacts

* test(integration): boot one uvicorn worker where a test does not assert cross-worker behaviour

A 2-worker owned proxy takes about twice as long to become ready as a 1-worker
one, because the uvicorn parent imports the app and each spawned worker imports
it again. The passthrough upstream error visibility cases, the single-request
S3 dedupe cases and the single-request spend log payload cases send one or two
requests and assert on the response, the proxy log and the spend row, none of
which depends on a second worker, so they now boot the default single worker

Cases that do rely on two workers keep them, the concurrent S3 burst through a
sink outage, previous_response_id replay, response cache hits, the concurrent
spend bursts and upstream stop and recovery

* test(observability): settle OTel excluded-services traces by a quiet period instead of a full window

_trace_spans slept through its whole 15 or 30 s window on every call even when
the trace had long stopped growing. It now polls and returns once the trace
has spans and has not gained one for five tenant export intervals, and the
module pins the tenant batch exporter's schedule delay to 1 s so that bound is
set by the test rather than by the SDK default. Absence checks still run only
after the trace stopped growing, and against a build that leaks postgres spans
to the tenant they still fail

These cases also boot one uvicorn worker now. None of them asserts anything
about a second worker

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
2026-10-12 09:48:42 -07:00
..
_azure_storage_support.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
_logging_only_scope_support.py feat(guardrails): split logging_only_scope direction from input-failure policy (#45709) 2026-10-10 01:46:39 -07:00
_openinference_support.py fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698) 2026-10-08 15:53:07 -07:00
_s3_v2_support.py fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
azure_dispatch_support.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
conftest.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
logging_worker_drain_canary.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_akto_logging_only.py feat(guardrails): support logging_only mode for the Akto guardrail (#45461) 2026-10-08 17:28:08 -07:00
test_arize_otel_v2_openinference_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_arize_otel_v2_openinference_family.py fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698) 2026-10-08 15:53:07 -07:00
test_arize_otel_v2_openinference_modes.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_arize_otel_v2_openinference_sad_edge.py fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698) 2026-10-08 15:53:07 -07:00
test_arize_otel_v2_openinference_spans.py fix(proxy): classify /v1/messages pass-through streams as Anthropic on any host (#45602) 2026-10-09 11:57:03 -07:00
test_azure_content_safety_audit.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_azure_content_safety_dispatch.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_azure_content_safety_endpoints.py fix(guardrails): scan Responses API input in Azure Prompt Shield (#43786) 2026-09-30 22:48:20 -07:00
test_azure_storage_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_azure_storage_client_ttl.py fix(azure_storage): keep the DataLakeServiceClient alive until its TTL elapses (#43082) 2026-09-30 18:50:15 -07:00
test_azure_storage_file_names.py fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099) 2026-10-01 23:23:37 -07:00
test_bedrock_error_request_id.py test(integration): cover customer reported cache key, cache_control, bedrock request id, responses schema, scim and tag budget contracts (#42785) 2026-09-23 13:19:38 -07:00
test_bedrock_guardrail_attachments.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_cache_hit_guardrail_metrics.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_cache_hit_guardrail_metrics_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_callback_delivery.py test(integration): regression tests for August cost tracking and budgeting bugs (#42622) 2026-09-23 04:13:03 +00:00
test_datadog_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_grayswan_wire.py fix(grayswan): send request conversation and tool calls to post-call monitor (#43770) 2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_guardrail_effects.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_load_balancing.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_guardrail_stream_scope.py feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801) 2026-10-08 13:46:53 -07:00
test_guardrail_stream_scope_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_stream_scope_matrix.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_guardrail_timeout_all_providers.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_langfuse_delivery.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_langsmith_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_logging_only_scope_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_logging_only_scope_config.py feat(guardrails): split logging_only_scope direction from input-failure policy (#45709) 2026-10-10 01:46:39 -07:00
test_logging_only_scope_runtime.py feat(guardrails): split logging_only_scope direction from input-failure policy (#45709) 2026-10-10 01:46:39 -07:00
test_logging_worker_drain.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_otel_conversation_id.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_otel_excluded_services.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_otel_excluded_services_matrix.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_otel_team_capture_message_content.py feat(otel)!: let a team capture_message_content override the global setting for its OTel v2 destination (#44264) 2026-10-10 13:52:32 -07:00
test_otel_text_completion_choices.py fix(otel): keep text completion choice fields beside the synthesized message (#42537) 2026-09-22 14:24:32 -07:00
test_otel_v1_request_trace.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_passthrough_upstream_error_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_passthrough_upstream_error_visibility.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_presidio_entity_masking.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_presidio_streaming_output.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_prometheus_request_metrics.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_prometheus_series_cap_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_realtime_guardrail_translation.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
test_s3_v2_partition_granularity.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_s3_v2_streaming_failure_dedupe.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_signoz_delivery.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_straiker_v3_platform.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_xecguard_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00