mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-19 00:01:29 +00:00
2a55d23731
* fix(proxy): merge model-level guardrails before pre_call_hook DB/UI-assigned guardrails (litellm_params.guardrails) only fire on post_call paths today: _check_and_merge_model_level_guardrails is called in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but never before pre_call_hook in common_request_processing.py:963. PR #23774 fixed the non-streaming post_call case; pre_call was left broken. At the pre_call site, add_litellm_data_to_request strips client-supplied metadata.model_info (pricing spoofing guard) and route_request hasn't run yet, so model_info.id is unavailable. Extend the helper to fall back to llm_router.get_deployment_by_model_group_name(model_alias) when model_id is missing — that uses the O(1) model-name index already maintained by the router. Closes #29652 * fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload Addresses veria-ai Medium finding + proxy-infra CI failure on this PR. ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format was omitting the field, so a key with mcp_rpm_limit could exceed it via the MCP path. Reads from kwargs.get("mcp_rate_limit_server_name") to match how pre_call_tool_check resolves the alias-then-server-name fallback before invoking hooks. * fix(proxy): union guardrails across group deployments on alias fallback Addresses second veria-ai Medium on #29654: the alias fallback called get_deployment_by_model_group_name(), which returns ONE deployment. A guardrail set on a non-first deployment would silently not run on pre_call when the model_id is missing. Switch to get_model_list(model_name=...) and take the UNION of litellm_params.guardrails across all matching deployments (with dedup). Trade-off documented in the comment: pre_call cannot know which deployment route_request will select, so the conservative choice is to apply any guardrail set on any eligible deployment. Updated test stubs to use get_model_list. Added 3 new tests covering union, dedup, and the all-empty case. * test(model_level_guardrails): align integration test with get_model_list union API * fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint Addresses 3rd veria-ai Medium on #29654: add_litellm_data_to_request preserves client-supplied metadata.model_info when the caller's key/team has allow_client_pricing_override. The pre_call merge previously trusted that id, so a caller could spoof an unguarded model_info.id while requesting a guarded alias and bypass guardrails. New `trust_client_model_info: bool` param on the helper. The pre_call call site passes False; post_call paths (existing) keep True. Also fixes the ruff failure on the union loop: pulled the .get() into a local + isinstance(list) check before iterating, so mypy stops complaining about `object` not being iterable. 2 new regression tests covering spoof-and-bypass + default-trust behavior. * fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance Addresses two more reviewer findings on #29654: veria-ai Medium: route_request resolves team-scoped public model names with metadata.user_api_key_team_id. The pre_call alias fallback called get_model_list(model_name=...) without the team_id, so team-scoped deployments were invisible and their pre_call guardrails silently skipped. Now reads team_id from metadata or litellm_metadata and passes it to get_model_list. greptile P1: the isinstance(deployment_guardrails, list) guard added for mypy narrowing silently dropped bare-string guardrail values that the existing post_call path used to truthy-accept. Restored by wrapping a scalar string into a one-element list on both paths. 4 new tests: team_id passthrough (metadata + litellm_metadata), scalar on post_call, scalar on alias-union. 36/36 tests pass. * style: black formatting on _check_and_merge_model_level_guardrails team_id assignment * chore: ruff format * fix(lint): remove unused noqa PLR0915 directive RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic because PLR0915 is not in this repo's enabled ruff rule set (lint.extend-select in ruff.toml), so the directive suppresses nothing and fails the lint job. * refactor(proxy): hoist guardrail-merge import to module top The pre_call guardrail-merge helper was imported inside common_processing_pre_call_logic with a # noqa: PLC0415, which the type-discipline gate counts as an unexplained suppression (LIT003). The inline import's cyclic-import justification does not hold: this module already imports from litellm.proxy.utils at top level, and utils.py does not import common_request_processing at module load. Fold the helper into the existing top-level import and drop the inline import, clearing the suppression instead of budgeting for it. --------- Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pricing_field_strip.py | ||
| test_prometheus_cleanup.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||