mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-19 00:01:29 +00:00
5677bc237c
* fix: resolve unified_file_id to real storage_url before dispatching batch create litellm.create_batch() against a Vertex AI-backed model crashes with an opaque error when the input file was uploaded as a LiteLLM-managed 'unified file' (multi-model file upload). The base64-encoded unified_file_id token is a LiteLLM-internal identifier, not a real provider-side file reference, but the batches_endpoints create_batch handler forwards it unchanged to llm_router.acreate_batch() / litellm.acreate_batch() for the unified_file_id branch. Provider-specific code that expects a real file location (e.g. Vertex AI's batch transformation, which parses a 'publishers/' segment out of the GCS URI) then fails on the opaque token. Resolve the unified_file_id to its real backend location (LiteLLM_ManagedFileTable.storage_url) before dispatch, mirroring the same lookup already used by the files retrieve/download endpoints for managed files. Falls back to the previous (unchanged) behavior if no managed-file record exists. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(proxy/batches): null-guard await on find_first for sync MagicMock test harnesses * fix(proxy/batches): enforce ownership and correct lookup key when resolving managed input_file_id The adopted resolution queried LiteLLM_ManagedFileTable with the decoded litellm_proxy string, but the unified_file_id column stores the raw base64 file id (see schema.prisma and the enterprise managed-files hook), so the lookup never matched in production and silently fell back to the opaque id. Query with the raw id instead and lock the key with a regression test. Move the resolution above the dispatch branches so the load-balanced router path receives the resolved storage_url too, enforce managed-file ownership with the same can_access_resource semantics the files retrieve and download endpoints use (404 on denial), and downgrade database failures to a logged fallback instead of aborting batch creation. Unresolved ids still dispatch unchanged because the managed-files deployment hook can map them via model_file_id_mapping * fix(proxy/batches): fail closed when the managed file ownership lookup errors A lookup exception previously fell back to dispatching the original unified id with the ownership gate unexecuted; the managed-files deployment hook maps unified ids from cache without re-checking ownership, so a database outage let a caller dispatch another tenant's file. Raise a clear 503 instead and lock the behavior with a regression test. No-database and no-row cases still fall back unchanged * test(proxy/batches): default harness prisma_client to None The batch routing harness left proxy_server.prisma_client at its module global, which a sibling test in the same shard can leave as a MagicMock. The unified-file rows that do not opt into managed-file resolution then entered the resolver and awaited a non-awaitable mock, surfacing as a 503. Patch prisma_client to None by default so those rows stay a no-op; resolution tests still override it explicitly * fix(proxy/batches): keep unified resolution in its own branch and fail closed on missing row Cursor flagged that hoisting the storage_url substitution above the load-balanced dispatch branch broke two things on that path: the model_file_id_mapping deployment filter keys on the original unified id, and the response returned the internal storage_url instead of the unified id. Move the resolution back inside the unified branch and exclude unified ids from the load-balanced branch so a managed file always takes the resolving path (which restores input_file_id and the unified_file_id hidden param on the response), and a load-balanced batch keeps the original id for deployment filtering. Also fail closed with a 404 when a unified id has no managed-file row while a database is present: the id cannot be ownership-verified, and dispatching it would both bypass the gate and hit the Vertex publishers-segment IndexError. Owned rows without a storage_url (legacy) still dispatch the original id * fix(proxy/batches): do not divert unified files off the load-balanced branch Excluding unified ids from the load-balanced branch (and not unified_file_id) regressed a path that works on the base revision: a multi-model managed file dispatched with an explicit router model under load balancing was routed into the unified branch, which raises a 400 for anything other than exactly one target model. Verified live against base (200, managed-files deployment hook remaps the unified id per model) versus the guarded branch (400 Expected 1 model, got 2). Restore the original three-condition load-balanced branch so that path keeps working unchanged. Unified-file storage_url resolution and the ownership 404 still apply on the non-load-balanced unified branch, which is the common managed-batch flow; the load-balanced managed path retains its existing behavior and its pre-existing enterprise-hook ownership gap, unchanged from base * refactor(proxy/batches): scope managed-file handling to resolution, drop ownership check Narrow this PR to its one problem: resolving a managed unified input_file_id to its backend storage_url so provider batch handlers (Vertex parses a publishers/ segment) receive a real location instead of the opaque token, and failing closed with a 404 when the token has no backing row so it is never dispatched into the provider crash. Remove the cross-tenant ownership check (can_access_resource) added earlier. Batch-create had no ownership enforcement before this PR, and the gap spans every managed-file call type, so it belongs in the enterprise managed-files pre-call hook (its acreate_batch branch) where files, batches and fine-tuning are covered uniformly, not partially in this one endpoint. Filed as a follow-up. This also removes the load-balanced-path ownership inconsistency the bots flagged, since there is no ownership branch to skip. Drop the inline comments flagged against the no-comments rule; behavior is documented in the helper docstring and the test docstrings * fix(proxy/batches): fail closed with 503 when the managed-file lookup errors A lookup exception previously fell back to dispatching the unresolved unified token, which defeats the fail-closed guarantee: the token still reaches the provider and can hit the same publishers-segment IndexError the resolution prevents. Treat a lookup error like the missing-row case and fail closed, but with a retryable 503 since the condition is transient. No-database and no-storage_url rows still fall back unchanged --------- Co-authored-by: htourinho-clgx <htourinho@cotality.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pricing_field_strip.py | ||
| test_prometheus_cleanup.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||