mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
68a8b12ab3
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent |
||
|---|---|---|
| .. | ||
| _experimental | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| container_endpoints | ||
| credential_endpoints | ||
| db | ||
| decisions_endpoints | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| example_config_yaml | ||
| experimental | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| lens | ||
| list_api | ||
| logging_endpoints | ||
| management | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| ocr_endpoints | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| rerank_endpoints | ||
| response_api_endpoints | ||
| response_polling | ||
| roi_calculator | ||
| search_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| telemetry | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test__lazy_features.py | ||
| test__types.py | ||
| test_admin_mcp.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_aproxy_startup.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_body_snapshot_callback_params.py | ||
| test_budget_reservation.py | ||
| test_bug_report_config.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_claude_code_marketplace.py | ||
| test_collector.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_conftest.py | ||
| test_cors_config.py | ||
| test_credential_slot_registry.py | ||
| test_custom_callback_input.py | ||
| test_custom_logger_s3_gcs.py | ||
| test_custom_proxy.py | ||
| test_custom_tokenizer_bug.py | ||
| test_db_schema_changes.py | ||
| test_deprecated_key_grace_period.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_get_favicon.py | ||
| test_get_image.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_init_litellm_callbacks.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_deprecations_endpoint.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_aliases.py | ||
| test_model_list_callback_filter.py | ||
| test_model_list_discoverable.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_moyai_endpoints.py | ||
| test_native_compaction.py | ||
| test_openai_ws_passthrough_routes.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pointfive_dashboard_config.py | ||
| test_pointfive_ui_callback.py | ||
| test_pricing_field_strip.py | ||
| test_prisma_client_backoff_retry.py | ||
| test_prisma_engine_watchdog.py | ||
| test_prisma_migration.py | ||
| test_prometheus_cleanup.py | ||
| test_prometheus_metrics_server.py | ||
| test_prompt_test_endpoint.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_config_unit_test.py | ||
| test_proxy_custom_auth.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_reject_logging.py | ||
| test_proxy_server.py | ||
| test_proxy_server_custom_logger.py | ||
| test_proxy_server_endpoints_and_startup.py | ||
| test_proxy_server_exception_mapping.py | ||
| test_proxy_setting_guardrails.py | ||
| test_proxy_token_counter.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_proxy_utils_model_creation_and_error_logging.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_reducto_ocr_route.py | ||
| test_response_model_sanitization.py | ||
| test_response_polling_pre_call_checks.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_route_priority.py | ||
| test_sensitive_route_auth.py | ||
| test_server_root_path.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_tracing_endpoints.py | ||
| test_ui_path_detection.py | ||
| test_unit_test_proxy_hooks.py | ||
| test_update_llm_router_resilience.py | ||
| test_update_spend.py | ||
| test_zero_cost_model_budget_bypass.py | ||
| test_zerobus_dashboard_config.py | ||
| vertex_key.json | ||