mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-26 01:12:21 +00:00
020e5dee9b
* feat(anthropic): placement policy for mid-conversation system messages Pure functions over the OpenAI-format message list: split off the leading system run, keep later system messages as role=system at a placement Anthropic accepts on models flagged supports_mid_conversation_system (after a user turn, before an assistant turn or the end, never adjacent), and convert them to user turns in place elsewhere, keeping tool_result first in a merged user turn. * fix(anthropic): keep mid-conversation system out of the chat completions system prompt translate_system_message hoisted every role=system message, at any index, into the top-level system block. On a conversation carrying a mid-session reminder that rewrites the cached prefix, so the provider re-bills the whole history at cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages; the chat completions path, shared by first-party Anthropic, Vertex, Azure AI and Bedrock Invoke, still hoisted. Only the leading system run becomes the system prompt now. Later system messages go through the placement policy, and anthropic_messages_pt emits a system message instead of rejecting the role. The caller's message list is no longer mutated. Tests pin the two-turn prefix invariant across all four chat configs and both flag states. * refactor(anthropic): single-source the converted system note The /v1/messages pass-through and the chat completions path must prefix a converted system turn with the same operator note. * test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions Same priming and assertions as the /v1/messages cases, through /v1/chat/completions with OpenAI-format messages, for first-party Anthropic and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model. The reminder sits between the assistant turn and the next user turn, the shape OpenAI-style agent frameworks send, which is the placement the chat path has to translate. * test(anthropic): cover the cache_control rebuild shapes and type the test helpers Codecov flagged the 5m ttl branch and the empty-system path of the wire builder; both now have a test. Greptile asked for full typing on the new test helpers. * refactor(anthropic): read the mid-conversation flag through a public supports_ helper supports_mid_conversation_system joins the other supports_* helpers in litellm.utils, so the chat transformation stops importing the private _supports_factory. * chore(typing): declare the mid-conversation type aliases with TypeAlias The Final sweep tightened LIT010, which exempts TypeAlias declarations but counts a bare alias assignment as an unannotated binding. * fix(anthropic): let add_code_execution_tool take the pass-through message union The translator now emits role=system inside messages for models that accept it, so anthropic_messages_pt returns the pass-through union. add_code_execution_tool still declared the narrower user/assistant union while only ever reading content, so upstream's strip_advisor_blocks_from_messages call in between made the mismatch visible to the type checker. * fix(bedrock): keep mid-conversation system messages in place on converse path * fix: ruff format + multi tool_result order + regression test * fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865 * fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget * test(bedrock): cover mid-conversation system conversion branches - non-dict guard in _opens_with_tool_result - in-place conversion without tool context - str/list cache_control preservation in mid-conversation path - drop unreachable non-system guard in hoisted loop * Place type-discipline suppressions on the lines the gate scans * Narrow hoisted loop to system role so basedpyright sees the right TypedDict * fix(anthropic): place mid-conversation system runs by their neighbours only A run after an assistant turn now slides behind the user turn that immediately follows it, and a run that ends the array or precedes an assistant turn becomes a user turn in place. No later message can move an earlier run, so a client that replays the conversation with more turns appended sends a byte-identical prefix and preserved thinking blocks keep their binding * refactor(bedrock): share the converted system note with the anthropic module Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy of the same text, and the reordering helpers lose their comments * test: pin the replayed request prefix across preserved-thinking turns One test per audited feature, through the real entrypoint: the chat transformations for anthropic, bedrock invoke, vertex and converse, the modify_params dummy tool result, dotprompt with unchanged variables, and Presidio masking against an in-process fake. Each serializes system, tools and the earlier messages of turn N and N+1 and asserts they match. The e2e mid-conversation system test imports its content blocks from models.py again and is marked provider_live * fix(anthropic): move mid-conversation system placement into prompt_templates The prompt factory imported the placement helper from the Anthropic provider package, whose common_utils reads a factory constant at import time, so loading the factory first raised ImportError. The module now sits next to anthropic_messages_pt and every consumer imports core utils A user turn with content [] or None puts no block on the wire, so a system run anchored to it landed first in messages or behind an assistant turn. Such a run now converts in place; empty strings and empty text blocks still anchor because the factory fills them with a placeholder * fix(anthropic): anchor system messages only on user turns that reach the wire * fix(bedrock): type the converse system-message helpers over the message TypedDicts * fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing A history that replays the previous turn as the litellm.Message object was invisible to the Converse system-message helpers, so a mid-conversation system stayed between a tool call and its result or reached Converse as role: system. The helpers now read fields through the shared message_field and parts_of accessors and drop the local role predicate. Flagged placement anchored a system run on any assistant follower, but anthropic_messages_pt drops an assistant turn that puts no block on the wire (content None, an empty list, an unsigned thinking part), so the system landed directly before the next user turn, which Anthropic rejects. Such a run now converts in place. An empty or whitespace text turn still anchors, since the converter pads it with a placeholder. * fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement An assistant turn whose only blocks carry Responses API encrypted reasoning is dropped by anthropic_messages_pt, so a mid-conversation system run anchored before it landed directly before the next user turn. The unsignable-thinking predicate now lives in common_utils and both the factory and the placement policy consult it. * fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon as its content list carries an inline thinking or redacted_thinking part, so a turn whose inline part is unsigned puts nothing on the wire even when the separate block is signed. The placement policy now mirrors that rule. --------- Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com> Co-authored-by: ege-arhan <egearhany@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| container_endpoints | ||
| credential_endpoints | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| list_api | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| ocr_endpoints | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| rerank_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test__types.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_bug_report_config.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_claude_code_marketplace.py | ||
| test_collector.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_conftest.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_init_litellm_callbacks.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_deprecations_endpoint.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_aliases.py | ||
| test_model_list_callback_filter.py | ||
| test_model_list_discoverable.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_native_compaction.py | ||
| test_openai_ws_passthrough_routes.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pointfive_dashboard_config.py | ||
| test_pointfive_ui_callback.py | ||
| test_pricing_field_strip.py | ||
| test_prisma_engine_watchdog.py | ||
| test_prisma_migration.py | ||
| test_prometheus_cleanup.py | ||
| test_prometheus_metrics_server.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_route_priority.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||