mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-17 04:34:25 +00:00
0dac61a555
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open GuardrailToolParam required a type key, so Gemini native tools (function_declarations, googleSearch) failed validation inside the hook and either returned a 500 or skipped the scan when fail_on_error was false. The type key is now optional and left out of the forwarded tool when absent, so an echoing guardrail server cannot write type: null into the provider request. structured_messages was re-validated against the chat completion message types, which silently emptied tool_result, tool_use, image and document content blocks to content: [] and dropped undeclared keys before the guardrail saw them. The request model now forwards rows as opaque mappings, which also retires the write-back shim that restored the original rows for echoed-back entries. The fail-open critical log line now carries error_type so a timeout, a refused connection and an upstream HTTP status can be told apart without the traceback. * fix(httpx): retry aiohttp server disconnects the way httpx does aiohttp raises ServerDisconnectedError when a kept-alive connection is closed by the peer before a response arrives, which is what a load balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces. The transport mapped it to httpx.ReadError, which the handler's connection-error retry does not cover, so the request failed instead of being resent on a fresh connection. It now maps to httpx.RemoteProtocolError, the error httpx itself raises for the same condition, so the existing retry fires. * test(httpx): build the stub server reply in one bytes literal * fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable The connection-error retry closed its fresh client before a streamed body was read, so an aiohttp disconnect now billed a second upstream request and still failed. It also skipped masking on a status error from the resend, resent PATCH and DELETE as POST, dropped multipart files and ignored the handler's ssl_verify. The retry now resends the request the first attempt built, on a client with the handler's TLS settings, closes that client when a streamed body is closed, and masks a status error the same way the first attempt does. * fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type * fix(guardrails): resend a generic guardrail check cut before any response A load balancer that closes an idle keepalive connection made the guardrail POST fail with "Server disconnected", because #46336 only resends POST on ConnectError. AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to False, that gives a RemoteProtocolError the same single fresh-client resend PUT and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a guardrail check is a pure verdict over the request. Every other POST keeps the #46336 rule The fail-open log line now puts error_type at the end so the pre-PR text stays contiguous for existing alert matchers test_rows_reach_the_guardrail_verbatim now reads the body off a real local guardrail server instead of mock call args * fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration A ServerDisconnectedError raised while reading a response body now stays httpx.ReadError, as it was before this PR. Only a disconnect before any response maps to RemoteProtocolError, which is the case the resend rule is about. Stream readers that end on ReadError, like the SAP stream iterator, keep their old behavior The loopback-server disconnect, TLS and guardrail resend tests move to tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is in-process only. Unit keeps in-process MockTransport checks for the idempotent POST resend, masking on the resend, streamed and buffered client closing, the guardrail resend, and the guardrail receiving rows verbatim * fix(guardrails): resend a reset guardrail check and keep typed echoes on texts An idempotent POST now resends once on any network error after connect (ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load balancer that answers a stale keepalive with a TCP reset rides it out too. That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes back to ReadError. This stops PUT and DELETE from stacking litellm's resend on top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil Guard from replaying its analyze POST on a disconnect Every resend now runs inside the method's own error mapping, so a timeout on the resend is a litellm.Timeout like the first attempt and a status error is masked once by the caller A guardrail server that parses rows with its own message types and echoes them back without the keys those types lack (annotations, tool_calls[].index) is read as an echo again, so its texts redaction reaches the LLM. A row only counts as an echo when no value changed and every key the chat completion types declare is kept The wire tests for the PUT/DELETE resend, its masking and its TLS settings now run on the httpx transport, where litellm's resend is the only retry, and the POST and guardrail tests cover both transports and both cut styles * test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head * fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement * refactor(guardrails): move the dropped-connection guardrail resend out of this PR The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line * test(guardrails): pick the closed guardrail port from the integration port allocator Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port * feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _azure_storage_support.py | ||
| _logging_only_scope_support.py | ||
| _openinference_support.py | ||
| _s3_v2_support.py | ||
| _ternary_support.py | ||
| azure_dispatch_support.py | ||
| conftest.py | ||
| logging_worker_drain_canary.py | ||
| test_akto_logging_only.py | ||
| test_apply_guardrail_endpoint.py | ||
| test_arize_otel_v2_openinference_chaos.py | ||
| test_arize_otel_v2_openinference_family.py | ||
| test_arize_otel_v2_openinference_modes.py | ||
| test_arize_otel_v2_openinference_sad_edge.py | ||
| test_arize_otel_v2_openinference_spans.py | ||
| test_azure_content_safety_audit.py | ||
| test_azure_content_safety_dispatch.py | ||
| test_azure_content_safety_dispatch_resilience.py | ||
| test_azure_content_safety_endpoints.py | ||
| test_azure_storage_chaos.py | ||
| test_azure_storage_client_ttl.py | ||
| test_azure_storage_file_names.py | ||
| test_batch_line_items_cold_storage.py | ||
| test_bedrock_error_request_id.py | ||
| test_bedrock_guardrail_attachments.py | ||
| test_cache_hit_guardrail_metrics.py | ||
| test_cache_hit_guardrail_metrics_chaos.py | ||
| test_callback_delivery.py | ||
| test_content_filter_logging_only.py | ||
| test_custom_logger_redaction.py | ||
| test_datadog_logging_delivery.py | ||
| test_file_usage_counter_span.py | ||
| test_generic_guardrail_api_wire.py | ||
| test_grayswan_wire.py | ||
| test_grayswan_wire_chaos.py | ||
| test_guardrail_attachment.py | ||
| test_guardrail_effects.py | ||
| test_guardrail_load_balancing.py | ||
| test_guardrail_stream_scope.py | ||
| test_guardrail_stream_scope_chaos.py | ||
| test_guardrail_stream_scope_matrix.py | ||
| test_guardrail_timeout_all_providers.py | ||
| test_langfuse_delivery.py | ||
| test_langsmith_logging_delivery.py | ||
| test_langtrace_delivery.py | ||
| test_logging_only_scope_chaos.py | ||
| test_logging_only_scope_config.py | ||
| test_logging_only_scope_runtime.py | ||
| test_logging_worker_drain.py | ||
| test_otel_conversation_id.py | ||
| test_otel_excluded_services.py | ||
| test_otel_excluded_services_matrix.py | ||
| test_otel_team_capture_message_content.py | ||
| test_otel_text_completion_choices.py | ||
| test_otel_trace_response_header.py | ||
| test_otel_v1_request_trace.py | ||
| test_passthrough_upstream_error_chaos.py | ||
| test_passthrough_upstream_error_visibility.py | ||
| test_presidio_entity_masking.py | ||
| test_presidio_streaming_output.py | ||
| test_prometheus_project_alias_label.py | ||
| test_prometheus_request_metrics.py | ||
| test_prometheus_series_cap.py | ||
| test_prometheus_series_cap_chaos.py | ||
| test_prometheus_shared_custom_labels.py | ||
| test_realtime_guardrail_translation.py | ||
| test_s3_v2_flush_surfaces.py | ||
| test_s3_v2_partition_granularity.py | ||
| test_s3_v2_streaming_failure_dedupe.py | ||
| test_s3_v2_upload_fanout.py | ||
| test_signoz_delivery.py | ||
| test_straiker_v3_platform.py | ||
| test_ternary_focus_export.py | ||
| test_wingback_guardrail.py | ||
| test_xecguard_wire.py | ||