mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
3a31331435
* fix(proxy): run pre-call guardrails on batch input file uploads POST /v1/files with purpose=batch was the only route in files_endpoints that never reached pre_call_hook, so guardrails did not see batch content at all and records reached the provider unscanned. Stream the uploaded JSONL a record at a time and run each record's body through the existing pre_call_hook dispatch under the call type its url maps to, so guardrail resolution, key and team config, and the per-endpoint translations are reused rather than reimplemented. The hook gains a guardrails_only mode for this, since the same callback loop also drives rate limiters, budget hooks, prompt templates and hanging-request alerting, none of which should fire once per record. A guardrail that blocks raises its own exception, which propagates untouched so its status code survives. A record a guardrail would rewrite, a record that cannot be parsed, and a record whose url cannot be scanned all reject the upload, since silently skipping any of them is the bypass this is meant to close. Per-record redaction lands separately. The scan only runs when a guardrail that actually runs pre_call, or a guardrail pipeline, is configured, so deployments without one are byte for byte unchanged. * fix(proxy): compare the dict a batch guardrail returns, not the one it was given async_pre_call_hook may return a replacement dict instead of mutating its input, and process_pre_call_hook_response then makes that replacement the request. The scan only inspected the dict it passed in, so a guardrail that redacts by returning a copy was treated as a no-op and its record uploaded unchanged. * fix(proxy): treat a missing batch body key as different from a null one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): document the guardrails_only mode on pre_call_hook * fix(proxy): resolve a batch record's scan type from its body when the url is unfamiliar The scanner only accepted five exact urls, but callers write that field by hand and the provider transformers are far more permissive: bedrock treats any non-empty url as chat and vertex strips query strings and trailing slashes. Uploads that work today would have started failing the moment a pre-call guardrail was configured. Normalize the url before lookup and fall back to the body shape when it is unfamiliar, so a record we can still read is a record we still scan. Only a body with no messages, prompt or input is now refused, and the error says so instead of listing urls that were never the whole set. Also pins the default side of the guardrails_only gate: the hanging-request alert and prompt templating are asserted to still fire when the flag is absent. * refactor(proxy): drop batch guardrail checks the upload validation already makes check_batch_file_upload now runs first and rejects a line that does not parse, a line that is not an object, and a line missing custom_id, method, url or body, so the guardrail scan can rely on all four. Its own parse handling was unreachable through the endpoint and is gone, along with the tests for it. What is left is the case that validation does not cover, a body whose value is not an object, since it only checks that the key is present. * fix(proxy): resolve a batch record's call type from the url path, not the whole url A record naming its route in full, which is how callers actually write batch files, matched no known route, so it fell through to the body shape. A Responses record carries `input`, and that reads as an embedding, so the record was scanned as the wrong call type and any guardrail scoped to chat or Responses skipped it while the upload was accepted. Chat records survived only because their body shape happens to map back to the same call type. The url is now reduced to its path before matching. Guardrails that pick their policy from a request header, such as noma choosing an application id, saw no headers at all during the scan and fell back to a default, so a batch record could be evaluated under a different policy than the same content sent online. The sanitized headers the proxy already stores in request metadata now travel with the scan. Also drops the bare `dict` annotation, the unreachable non-dict branch on the guardrail chain's own return, and the type alias that was missing its `TypeAlias`, which together were failing the lint gate. * fix(proxy): give each batch record its own copy of the scan metadata The narrowed metadata was handed to every record as a shallow copy, so `headers` and `tags` stayed shared with the upload request and with the other records in the same window. A guardrail that writes into one of those in place, which several do to record their own bookkeeping, would have its write show up in every record scanned after it and in the request itself. The narrowing already removed the values that cannot be copied, so each record now gets a deep copy. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| credential_endpoints | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| ocr_endpoints | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_claude_code_marketplace.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_conftest.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_init_litellm_callbacks.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_deprecations_endpoint.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_openai_ws_passthrough_routes.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pricing_field_strip.py | ||
| test_prisma_engine_watchdog.py | ||
| test_prometheus_cleanup.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||