scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-31 06:56:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 245
litellm/tests/integration/observability
History
rj-cairn 42d2909210
test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 
* test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module

Session scope booted one 2-worker proxy per xdist worker and kept it until the
worker's session ended, so a CircleCI node holding two of these files piled up
proxies until the VM stopped responding. Module scope lets the harness group
each file's rig tests onto one worker and tears the proxies down after them.

* docs(tests): keep proxy fixtures module scoped in integration tests

* docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 03:28:39 -07:00
..
_azure_storage_support.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
_langfuse_otel_metadata_support.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
_logging_only_scope_support.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
_openinference_support.py fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) 2026-10-21 02:26:05 +00:00
_s3_v2_support.py
_ternary_support.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
azure_dispatch_support.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
conftest.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
logging_worker_drain_canary.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_akto_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_apply_guardrail_endpoint.py fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359) 2026-10-14 08:56:05 -07:00
test_arize_otel_v2_openinference_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_arize_otel_v2_openinference_family.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_arize_otel_v2_openinference_modes.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_arize_otel_v2_openinference_sad_edge.py fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698) 2026-10-08 15:53:07 -07:00
test_arize_otel_v2_openinference_spans.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_azure_content_safety_audit.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_content_safety_dispatch.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_azure_content_safety_endpoints.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_azure_storage_client_ttl.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_file_names.py fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099) 2026-10-01 23:23:37 -07:00
test_batch_line_items_cold_storage.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_bedrock_converse_guardrail_rewrites.py fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) 2026-10-17 20:38:14 -07:00
test_bedrock_converse_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_bedrock_error_request_id.py fix(responses): preserve provider request IDs in streaming failure logs (#45453) 2026-10-16 00:29:11 -07:00
test_bedrock_guardrail_attachments.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_bedrock_invoke_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_callback_delivery.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_content_filter_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_custom_logger_hook_counts.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_chaos.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_modes.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_post_call_block.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_vendors.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_redaction.py fix(logging)!: redact Responses instructions and callback output (#43315) 2026-10-20 20:38:17 +00:00
test_datadog_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_generic_guardrail_api_wire.py fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338) 2026-10-16 14:52:35 -07:00
test_generic_guardrail_num_retries.py feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) 2026-10-17 12:22:41 -07:00
test_grayswan_wire.py fix(grayswan): send request conversation and tool calls to post-call monitor (#43770) 2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_guardrail_effects.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_guardrail_load_balancing.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_guardrail_stream_scope.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_stream_scope_chaos.py test(integration): remove BUG skips for 3 tests that pass on main (#47364) 2026-10-19 21:39:35 -07:00
test_guardrail_stream_scope_matrix.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_timeout_all_providers.py feat(guardrails): add Wingback guardrail integration (#46136) 2026-10-12 22:18:10 -07:00
test_langfuse_delivery.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_langfuse_otel_metadata_chaos.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_clients.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_destinations.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_identity.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_trace_fields.py fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705) 2026-10-27 10:34:55 -07:00
test_langsmith_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_chaos.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_config.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_logging_only_scope_runtime.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_logging_worker_drain.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_otel_cache_hit_cost.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_otel_conversation_id.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
test_otel_excluded_services.py test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) 2026-10-13 09:55:11 -07:00
test_otel_excluded_services_matrix.py test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846) 2026-10-15 22:35:23 -07:00
test_otel_team_capture_message_content.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_otel_team_sampling_across_pods.py fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468) 2026-10-26 11:44:30 -07:00
test_otel_text_completion_choices.py
test_otel_trace_response_header.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_otel_v1_request_trace.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_passthrough_guardrail_scan_text.py fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
test_passthrough_operation_name.py fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) 2026-10-28 10:48:34 -07:00
test_passthrough_upstream_error_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_upstream_error_visibility.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_presidio_entity_masking.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
test_presidio_overlapping_spans.py fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) 2026-10-17 11:21:03 -07:00
test_presidio_streaming_output.py fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) 2026-10-17 16:48:37 +00:00
test_prometheus_deployment_failure_api_provider.py fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384) 2026-10-19 13:38:26 -07:00
test_prometheus_project_alias_label.py fix(prometheus): skip custom metadata labels a metric already carries (#46267) 2026-10-13 18:20:52 -07:00
test_prometheus_provider_auth_failure.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_provider_auth_failure_chaos.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_request_metrics.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_prometheus_series_cap_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_prometheus_shared_custom_labels.py test(integration): wait on the right signal in five flaky integration contracts (#46737) 2026-10-15 13:48:51 -07:00
test_realtime_guardrail_translation.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py test(integration): look up owned cache keys and clean up sweep models (#47255) 2026-10-18 10:27:53 +00:00
test_s3_v2_partition_granularity.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_s3_v2_streaming_failure_dedupe.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_signoz_delivery.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
test_straiker_v3_platform.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_stream_error_alert_api_base.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_ternary_focus_export.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
test_wingback_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_xecguard_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00