scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214
litellm/tests/integration/observability
History
Exact
yuneng-jiang 71c1a2aff6
test(integration): repair tests that merged changes left stale (#50782) 
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import.

The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so.

The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again.

The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths.

The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys.

The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change.
2026-11-10 01:08:35 +00:00
..
_azure_storage_support.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
_langfuse_otel_metadata_support.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
_logging_only_scope_support.py test(integration): fix four flaky integration tests (#49554) 2026-11-02 09:23:17 -08:00
_openinference_support.py feat(proxy): let a key's logging entry reference a proxy credential (#50572) 2026-11-09 16:02:46 -08:00
_s3_v2_support.py fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
_ternary_support.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
azure_dispatch_support.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
conftest.py test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) 2026-10-29 03:28:39 -07:00
logging_worker_drain_canary.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_akto_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_anthropic_overloaded_failure_signals.py fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes (#49993) 2026-11-04 16:37:55 -08:00
test_apply_guardrail_endpoint.py fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) 2026-10-30 06:58:05 -07:00
test_arize_key_credential_reference.py feat(proxy): let a key's logging entry reference a proxy credential (#50572) 2026-11-09 16:02:46 -08:00
test_arize_otel_v2_openinference_chaos.py test(observability): reserve sink port throughout outage (#50100) 2026-11-04 21:07:21 +00:00
test_arize_otel_v2_openinference_family.py test(integration): repair tests that merged changes left stale (#50782) 2026-11-10 01:08:35 +00:00
test_arize_otel_v2_openinference_modes.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_arize_otel_v2_openinference_sad_edge.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_arize_otel_v2_openinference_spans.py fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508) 2026-10-20 11:46:17 -07:00
test_azure_content_safety_audit.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_content_safety_dispatch.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_azure_content_safety_dispatch_resilience.py test(integration): budget proxy restart tests from calibrated boot time (#49761) 2026-11-07 15:37:18 -08:00
test_azure_content_safety_endpoints.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_chaos.py test(integration): budget proxy restart tests from calibrated boot time (#49761) 2026-11-07 15:37:18 -08:00
test_azure_storage_client_ttl.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_azure_storage_file_names.py fix(azure_storage): keep client call ids from sharing one Data Lake file (#44099) 2026-10-01 23:23:37 -07:00
test_batch_line_items_cold_storage.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_bedrock_converse_guardrail_rewrites.py fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874) 2026-10-17 20:38:14 -07:00
test_bedrock_converse_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_bedrock_error_request_id.py fix(responses): preserve provider request IDs in streaming failure logs (#45453) 2026-10-16 00:29:11 -07:00
test_bedrock_guardrail_attachments.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
test_bedrock_invoke_stream_post_call_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_cache_hit_guardrail_metrics.py test(integration): repair release qualification timing and labels (#49737) 2026-11-02 18:02:28 -08:00
test_cache_hit_guardrail_metrics_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_callback_delivery.py test(integration): repair seven stale integration tests on main (#47600) 2026-10-20 11:21:04 -07:00
test_content_filter_logging_only.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_custom_logger_hook_counts.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_chaos.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_modes.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_post_call_block.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_hook_counts_vendors.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
test_custom_logger_redaction.py fix(logging)!: redact Responses instructions and callback output (#43315) 2026-10-20 20:38:17 +00:00
test_datadog_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_file_usage_counter_span.py feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632) 2026-10-06 03:06:36 +00:00
test_generic_guardrail_api_wire.py fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338) 2026-10-16 14:52:35 -07:00
test_generic_guardrail_num_retries.py feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) 2026-10-17 12:22:41 -07:00
test_grayswan_wire.py fix(grayswan): send request conversation and tool calls to post-call monitor (#43770) 2026-09-30 17:52:53 -07:00
test_grayswan_wire_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_guardrail_attachment.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_guardrail_effects.py fix(proxy)!: preserve Responses SSE event framing (#47980) 2026-11-06 20:17:57 -08:00
test_guardrail_load_balancing.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_guardrail_pipeline_applied_header.py fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails (#49264) 2026-10-30 17:00:44 -07:00
test_guardrail_stream_scope.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_stream_scope_chaos.py test(integration): remove BUG skips for 3 tests that pass on main (#47364) 2026-10-19 21:39:35 -07:00
test_guardrail_stream_scope_matrix.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_guardrail_timeout_all_providers.py feat(guardrails): add Wingback guardrail integration (#46136) 2026-10-12 22:18:10 -07:00
test_langfuse_delivery.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_langfuse_otel_console_output.py fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723) 2026-10-29 12:20:07 -07:00
test_langfuse_otel_legacy_destinations.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_metadata_chaos.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_langfuse_otel_metadata_clients.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_langfuse_otel_metadata_destinations.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_metadata_identity.py fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697) 2026-10-27 18:54:10 -07:00
test_langfuse_otel_tenant_only.py fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718) 2026-10-29 12:19:08 -07:00
test_langfuse_otel_trace_fields.py fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705) 2026-10-27 10:34:55 -07:00
test_langsmith_logging_delivery.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_langtrace_delivery.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_logging_only_scope_chaos.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_logging_only_scope_config.py test(integration): repair release qualification timing and labels (#49737) 2026-11-02 18:02:28 -08:00
test_logging_only_scope_runtime.py test(integration): fix four flaky integration tests (#49554) 2026-11-02 09:23:17 -08:00
test_logging_worker_drain.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_otel_auto_router_model.py fix(otel): name the routed model on auto-router LLM spans (#50498) 2026-11-07 15:52:00 -08:00
test_otel_cache_hit_cost.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_otel_conversation_id.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_otel_excluded_services.py test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) 2026-10-13 09:55:11 -07:00
test_otel_excluded_services_matrix.py test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846) 2026-10-15 22:35:23 -07:00
test_otel_team_capture_message_content.py test(integration): wait for the stalled-tenant root span and accept a bare POST name (#49406) 2026-10-31 12:24:33 -07:00
test_otel_team_request_metadata.py fix(otel): seed request metadata baggage for team-only OTel v2 destinations (#50373) 2026-11-06 14:47:48 -08:00
test_otel_team_sampling_across_pods.py fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468) 2026-10-26 11:44:30 -07:00
test_otel_text_completion_choices.py fix(otel): keep text completion choice fields beside the synthesized message (#42537) 2026-09-22 14:24:32 -07:00
test_otel_trace_response_header.py test: update stale integration and guardrail e2e tests to intended behavior (#48665) 2026-10-28 01:11:25 +00:00
test_otel_v1_request_trace.py fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
test_ovalix_guardrail.py feat(ovalix)!: scan files, tool calls and tool results (#49309) 2026-10-31 17:01:12 -07:00
test_panw_airs_block_masked_response.py feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block (#46219) 2026-10-29 10:45:00 -07:00
test_passthrough_guardrail_scan_text.py fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) 2026-10-26 12:07:58 -07:00
test_passthrough_operation_name.py fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) 2026-10-28 10:48:34 -07:00
test_passthrough_upstream_error_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_passthrough_upstream_error_visibility.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_presidio_entity_masking.py fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540) 2026-10-29 18:20:52 -07:00
test_presidio_masking_endpoints.py test(observability): match the full card number instead of a 4111 substring that timestamps can contain (#50612) 2026-11-08 06:01:49 +00:00
test_presidio_overlapping_spans.py fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) 2026-10-17 11:21:03 -07:00
test_presidio_streaming_output.py fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) 2026-10-17 16:48:37 +00:00
test_prometheus_deployment_failure_api_provider.py fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384) 2026-10-19 13:38:26 -07:00
test_prometheus_project_alias_label.py fix(prometheus): skip custom metadata labels a metric already carries (#46267) 2026-10-13 18:20:52 -07:00
test_prometheus_provider_auth_failure.py test(integration): repair stale contract fixtures (#49700) 2026-11-02 11:23:11 -08:00
test_prometheus_provider_auth_failure_chaos.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
test_prometheus_request_metrics.py test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
test_prometheus_series_cap.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_prometheus_series_cap_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_prometheus_shared_custom_labels.py test(integration): wait on the right signal in five flaky integration contracts (#46737) 2026-10-15 13:48:51 -07:00
test_realtime_guardrail_translation.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_s3_v2_flush_surfaces.py test(integration): look up owned cache keys and clean up sweep models (#47255) 2026-10-18 10:27:53 +00:00
test_s3_v2_partition_granularity.py test(integration): size restart-mid-burst timeouts from the proxy boot budget (#49376) 2026-10-31 10:17:34 -07:00
test_s3_v2_streaming_failure_dedupe.py test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) 2026-10-12 09:48:42 -07:00
test_s3_v2_upload_fanout.py test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) 2026-10-16 22:45:29 +00:00
test_signoz_delivery.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_straiker_v3_platform.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_stream_error_alert_api_base.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_ternary_focus_export.py feat(focus): add ternary destination for FOCUS export (#46106) 2026-10-12 22:52:40 -07:00
test_weave_key_scoped_delivery.py test: move Sail, Weave and Presidio e2e tests to integration tests (#49533) 2026-10-31 18:31:01 -07:00
test_wingback_guardrail.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_xecguard_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00