mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
fix(adapters,vertex): pass output_config through to backends that accept it
Resolves the silent strip of Anthropic Structured Outputs across the Vertex AI Claude transformation paths and the Anthropic-adapter re-merge. Consolidates and supersedes four stalled community PRs addressing overlapping aspects of the same root bug: - #23475 (Vertex AI Claude blanket-strip removal) - #23396 (Vertex AI Claude conditional passthrough) - #23706 (Anthropic adapter exclude output_config from non-Anthropic backends) - #22727 (Anthropic adapter strip output_config for non-Anthropic backends) Closes / addresses: #23380 (Vertex AI Claude output_config drop), related: #26423, #25079, #24549, #25971, #25957, #26163, #24856. What was broken --------------- * Vertex AI Claude paths called ``data.pop("output_config")`` and ``data.pop("output_format")`` unconditionally even when Vertex accepted those fields. Callers asking for Structured Outputs got a 200 with prose and never knew the schema constraints had been silently dropped (often masked for months by permissive fallback parsers). * The ``/v1/messages`` -> ``/chat/completions`` adapter (``LiteLLMMessagesToCompletionTransformationHandler``) re-merged the raw Anthropic-shaped ``output_config`` into ``completion_kwargs`` AFTER the translator already mapped its meaningful parts to ``response_format`` / ``reasoning_effort``. Non-Anthropic backends (Azure OpenAI, Fireworks, Bedrock Nova, etc.) then 400'd with "Extra inputs are not permitted". Approach -------- Vertex AI Claude (chat-completion + experimental_pass_through paths): Replace the unconditional pop with a sanitizer ``_sanitize_vertex_anthropic_output_params`` that strips only the Vertex-unsupported keys (today: ``effort``) from ``output_config`` while forwarding ``format`` and the legacy top-level ``output_format``. Defensive: non-dict ``output_config`` values are dropped to avoid sending malformed payloads downstream. Greptile P1 from PR #23396 addressed: when ``output_config`` carries both ``format`` and ``effort``, the prior conditional pass-through forwarded ``effort`` and reproduced the 400. The new helper filters per-key. Anthropic ``/v1/messages`` adapter: Add ``output_config`` to a named module-level constant ``ANTHROPIC_ONLY_REQUEST_KEYS`` and wire it into ``excluded_keys`` so the post-translation re-merge skips re-adding the raw key. This fixes the 400 on non-Anthropic backends and avoids the conflicting duplicate (``response_format`` + raw ``output_config``) on Anthropic-family backends. Greptile P2 from PR #23706 addressed: the constant gives reviewers one grep target instead of an inline literal that silently grows. Greptile P2 from PR #22727 addressed: ``extra_kwargs or {}`` is replaced with explicit ``is None`` checks so empty-dict callers no longer skip the fallback path. Tests ----- * tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/ test_vertex_ai_partner_models_anthropic_transformation.py: - 5 new/updated cases plus a direct unit test for ``_sanitize_vertex_anthropic_output_params``. - Updated ``test_vertex_ai_claude_sonnet_4_5_structured_output_fix`` so its mock-injected ``output_format`` is asserted to FLOW THROUGH (the original test asserted the now-buggy strip behavior). * tests/test_litellm/llms/anthropic/experimental_pass_through/ adapters/test_handler_output_config_passthrough.py (new): - Constant export sanity, output_config strip with ``effort`` only, output_config strip with ``format`` only, regression guard that unrelated extras still flow, explicit-empty-dict path, and the ``extra_kwargs=None`` no-crash path. Test-quality fixes incorporated from Greptile review on the superseded PRs: * No ``inspect.getsource`` source-text assertions (PR #24114 / #23475). * ``sys.path`` insertion is anchored to ``__file__`` (PR #23706). * Assertion messages are positional, not tuple (PR #24114-class bug). * No ``or {}`` masking explicit empty dicts in helper signatures (PR #22727). Verified locally: 26/26 pass with this commit. The new tests fail (or fail to import) on ``main`` without it. Out of scope ------------ * The ``max_tokens`` capping logic from PR #22727 — independent concern, deserves its own PR with a focused test plan. * Architectural rework of the ``excluded_keys`` mechanism (Greptile P2 on PR #23706 noted point-fix growth). The named constant gives maintainers a clear place to extend; a registry-based approach would be a follow-up. Co-Authored-By: netbrah <netbrah> Co-Authored-By: s-zx <s-zx> Co-Authored-By: invoicepulse <invoicepulse> Co-Authored-By: cfdude <cfdude> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Darien Kindlund
parent
7b47dffefd
commit
b9e46cbdb7
5 changed files with 431 additions and 78 deletions
|
|
@ -27,6 +27,16 @@ from litellm.utils import get_model_info
|
|||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
# Anthropic-only fields that the translator above already maps into the
|
||||
# OpenAI-format completion_kwargs (output_config → reasoning_effort /
|
||||
# response_format, etc.). They must be filtered out of the raw
|
||||
# extra_kwargs re-merge below or non-Anthropic backends reject the call
|
||||
# with 400 "Extra inputs are not permitted". Add new entries here when
|
||||
# extending AnthropicMessagesRequestOptionalParams with another Anthropic-
|
||||
# specific key.
|
||||
ANTHROPIC_ONLY_REQUEST_KEYS: frozenset[str] = frozenset({"output_config"})
|
||||
|
||||
########################################################
|
||||
# init adapter
|
||||
ANTHROPIC_ADAPTER = AnthropicAdapter()
|
||||
|
|
@ -202,8 +212,12 @@ class LiteLLMMessagesToCompletionTransformationHandler:
|
|||
request_data["output_format"] = output_format
|
||||
|
||||
# Extract output_config from extra_kwargs so the translator can use it
|
||||
# (e.g. output_config.effort for adaptive thinking → reasoning_effort)
|
||||
extra_kwargs = extra_kwargs or {}
|
||||
# (e.g. output_config.effort for adaptive thinking → reasoning_effort,
|
||||
# output_config.format → response_format for structured outputs).
|
||||
# Use explicit None check rather than `or {}` so an explicit empty dict
|
||||
# caller-passed argument is preserved (matters for tests that drive
|
||||
# the fallback inference path).
|
||||
extra_kwargs = extra_kwargs if extra_kwargs is not None else {}
|
||||
if "output_config" in extra_kwargs:
|
||||
request_data["output_config"] = extra_kwargs["output_config"]
|
||||
|
||||
|
|
@ -225,8 +239,22 @@ class LiteLLMMessagesToCompletionTransformationHandler:
|
|||
"include_usage": True,
|
||||
}
|
||||
|
||||
excluded_keys = {"anthropic_messages"}
|
||||
extra_kwargs = extra_kwargs or {}
|
||||
# Keys that must NOT be forwarded as raw extras into the OpenAI-format
|
||||
# ``completion_kwargs`` after translation. The translator above has
|
||||
# already consumed the meaningful parts of these inputs (e.g.
|
||||
# ``output_config.format`` → ``response_format``, ``output_config.effort``
|
||||
# → ``reasoning_effort`` for non-Claude targets). Re-adding the raw
|
||||
# Anthropic-shaped key here causes 400 "Extra inputs are not permitted"
|
||||
# on non-Anthropic backends (Azure OpenAI, Fireworks, Bedrock Nova,
|
||||
# etc.) and is silently lossy on Anthropic-family targets, which would
|
||||
# see the translated key ``response_format`` AND a duplicate, conflicting
|
||||
# ``output_config``.
|
||||
#
|
||||
# Maintainability: when adding a new Anthropic-only request param to
|
||||
# ``AnthropicMessagesRequestOptionalParams``, also extend
|
||||
# ``ANTHROPIC_ONLY_REQUEST_KEYS`` here so it doesn't silently leak.
|
||||
excluded_keys = ANTHROPIC_ONLY_REQUEST_KEYS | {"anthropic_messages"}
|
||||
extra_kwargs = extra_kwargs if extra_kwargs is not None else {}
|
||||
for key, value in extra_kwargs.items():
|
||||
if (
|
||||
key == "litellm_logging_obj"
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ from litellm.types.llms.vertex_ai import VertexPartnerProvider
|
|||
from litellm.types.router import GenericLiteLLMParams
|
||||
|
||||
from ....vertex_llm_base import VertexBase
|
||||
from ..transformation import _sanitize_vertex_anthropic_output_params
|
||||
|
||||
|
||||
class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, VertexBase):
|
||||
|
|
@ -158,12 +159,10 @@ class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, Vert
|
|||
"model", None
|
||||
) # do not pass model in request body to vertex ai
|
||||
|
||||
anthropic_messages_request.pop(
|
||||
"output_format", None
|
||||
) # do not pass output_format in request body to vertex ai - vertex ai does not support output_format as yet
|
||||
|
||||
anthropic_messages_request.pop(
|
||||
"output_config", None
|
||||
) # do not pass output_config in request body to vertex ai - vertex ai does not support output_config
|
||||
# Vertex AI Claude accepts ``output_config.format`` (structured outputs)
|
||||
# and ``output_format``, but rejects ``output_config.effort`` with 400
|
||||
# "Extra inputs are not permitted". Sanitize in place so the supported
|
||||
# bits flow through.
|
||||
_sanitize_vertex_anthropic_output_params(anthropic_messages_request)
|
||||
|
||||
return anthropic_messages_request
|
||||
|
|
|
|||
|
|
@ -11,6 +11,45 @@ from litellm.types.utils import ModelResponse
|
|||
|
||||
from ....anthropic.chat.transformation import AnthropicConfig
|
||||
|
||||
# Keys inside ``output_config`` that Vertex AI Claude does not accept.
|
||||
# Today only ``effort`` triggers "Extra inputs are not permitted"; add new
|
||||
# entries here as Vertex parity drifts. Keep this list narrow — anything
|
||||
# Vertex DOES accept (e.g. ``format`` for structured outputs) must be
|
||||
# preserved so callers can rely on Anthropic-native features.
|
||||
_VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS = frozenset({"effort"})
|
||||
|
||||
|
||||
def _sanitize_vertex_anthropic_output_params(data: dict) -> None:
|
||||
"""
|
||||
Strip Vertex-unsupported keys from ``output_config`` / ``output_format``
|
||||
in-place; forward whatever remains.
|
||||
|
||||
Behavior:
|
||||
* ``output_config`` containing only unsupported keys (e.g. ``effort``
|
||||
alone) is removed entirely so the request body has no empty dict.
|
||||
* ``output_config`` containing a mix of supported + unsupported keys has
|
||||
the unsupported subset filtered out and the rest forwarded.
|
||||
* ``output_config`` that is supported in full passes through unchanged.
|
||||
* ``output_format`` is forwarded as-is (Vertex AI Claude accepts it).
|
||||
* Non-dict values for ``output_config`` are dropped to avoid sending
|
||||
malformed payloads downstream.
|
||||
"""
|
||||
output_config = data.get("output_config")
|
||||
if output_config is None:
|
||||
return
|
||||
if not isinstance(output_config, dict):
|
||||
data.pop("output_config", None)
|
||||
return
|
||||
sanitized = {
|
||||
k: v
|
||||
for k, v in output_config.items()
|
||||
if k not in _VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS
|
||||
}
|
||||
if sanitized:
|
||||
data["output_config"] = sanitized
|
||||
else:
|
||||
data.pop("output_config", None)
|
||||
|
||||
|
||||
class VertexAIError(Exception):
|
||||
def __init__(self, status_code, message):
|
||||
|
|
@ -105,11 +144,12 @@ class VertexAIAnthropicConfig(AnthropicConfig):
|
|||
|
||||
data.pop("model", None) # vertex anthropic doesn't accept 'model' parameter
|
||||
|
||||
# VertexAI doesn't support output_format parameter, remove it if present
|
||||
data.pop("output_format", None)
|
||||
|
||||
# VertexAI doesn't support output_config parameter, remove it if present
|
||||
data.pop("output_config", None)
|
||||
# Vertex AI Claude accepts ``output_config.format`` (structured outputs /
|
||||
# JSON Schema) but NOT ``output_config.effort`` — sending ``effort`` to
|
||||
# Vertex returns 400 "Extra inputs are not permitted". Sanitize in place:
|
||||
# forward the structured-output bits, drop the unsupported keys.
|
||||
# Same treatment for the legacy top-level ``output_format`` field.
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
|
||||
tools = optional_params.get("tools")
|
||||
tool_search_used = self.is_tool_search_used(tools)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,167 @@
|
|||
"""
|
||||
Regression tests for output_config passthrough through the Anthropic
|
||||
``/v1/messages`` → ``/chat/completions`` adapter.
|
||||
|
||||
Background — what was broken:
|
||||
* When a client sent ``output_config`` to ``/v1/messages`` and the request
|
||||
was routed to a non-Anthropic backend (Azure OpenAI, Fireworks, Bedrock
|
||||
Nova, etc.), the adapter forwarded the raw Anthropic-shaped ``output_config``
|
||||
field as-is into the OpenAI-format ``completion_kwargs``. The non-Anthropic
|
||||
backend then rejected the request with 400 "Extra inputs are not permitted".
|
||||
* The translator above the re-merge already extracts the meaningful parts of
|
||||
``output_config`` (``format`` → ``response_format``, ``effort`` →
|
||||
``reasoning_effort`` for non-Claude targets), so re-adding the raw key was
|
||||
always either redundant (Anthropic-family) or harmful (non-Anthropic).
|
||||
|
||||
Tests cover (consolidating PRs #23706 and #22727):
|
||||
1. ``output_config`` is excluded from the post-translation re-merge.
|
||||
2. ``ANTHROPIC_ONLY_REQUEST_KEYS`` constant is exported and contains
|
||||
``output_config`` so future maintainers know where to extend it.
|
||||
3. The translator-extracted fields (``response_format`` / ``reasoning_effort``)
|
||||
are still present after the strip — the strip removes only the raw
|
||||
Anthropic-shaped duplicate.
|
||||
4. Helper-level coverage for empty ``extra_kwargs`` (PR #22727 Greptile P2 —
|
||||
the original ``or {}`` pattern silently substituted a default and prevented
|
||||
the fallback inference path from being exercised).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
# Anchor sys.path to this file's location — not the working-directory-relative
|
||||
# pattern Greptile flagged on PR #23706. Resolves correctly regardless of
|
||||
# where pytest is invoked from.
|
||||
sys.path.insert(
|
||||
0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../../../.."))
|
||||
)
|
||||
|
||||
from litellm.llms.anthropic.experimental_pass_through.adapters.handler import (
|
||||
ANTHROPIC_ONLY_REQUEST_KEYS,
|
||||
LiteLLMMessagesToCompletionTransformationHandler,
|
||||
)
|
||||
|
||||
MESSAGES = [{"role": "user", "content": "hello"}]
|
||||
|
||||
|
||||
def _call_prepare(extra_kwargs, model="gpt-4o", **overrides):
|
||||
"""
|
||||
Drive ``_prepare_completion_kwargs`` with the minimum scaffolding needed.
|
||||
|
||||
Uses an explicit-None check on ``extra_kwargs`` so callers can test the
|
||||
falsy-empty-dict path. The fallback ``or {}`` pattern PR #22727 used here
|
||||
masked the no-extra-kwargs case from ever exercising the test's intent.
|
||||
"""
|
||||
return LiteLLMMessagesToCompletionTransformationHandler._prepare_completion_kwargs(
|
||||
max_tokens=overrides.get("max_tokens", 1024),
|
||||
messages=overrides.get("messages", MESSAGES),
|
||||
model=model,
|
||||
metadata=None,
|
||||
stop_sequences=None,
|
||||
stream=False,
|
||||
system=None,
|
||||
temperature=None,
|
||||
thinking=None,
|
||||
tool_choice=None,
|
||||
tools=None,
|
||||
top_k=None,
|
||||
top_p=None,
|
||||
output_format=None,
|
||||
extra_kwargs=extra_kwargs,
|
||||
)
|
||||
|
||||
|
||||
class TestAnthropicOnlyRequestKeysExport:
|
||||
"""The exclusion list must be a public, named constant for maintainability —
|
||||
Greptile P2 on PR #23706: ``excluded_keys`` was silently growing as a
|
||||
point-fix pattern. A named module-level constant gives reviewers a single
|
||||
grep target when extending Anthropic-only fields."""
|
||||
|
||||
def test_constant_exposed(self):
|
||||
assert isinstance(ANTHROPIC_ONLY_REQUEST_KEYS, frozenset)
|
||||
|
||||
def test_contains_output_config(self):
|
||||
assert "output_config" in ANTHROPIC_ONLY_REQUEST_KEYS
|
||||
|
||||
|
||||
class TestOutputConfigStrippedFromCompletionKwargs:
|
||||
"""``output_config`` must not survive the post-translation re-merge into
|
||||
``completion_kwargs`` regardless of the target provider — the translator
|
||||
has already consumed its meaningful parts."""
|
||||
|
||||
def test_output_config_with_effort_is_stripped(self):
|
||||
extra_kwargs = {
|
||||
"custom_llm_provider": "azure",
|
||||
"output_config": {"effort": "high"},
|
||||
}
|
||||
|
||||
result = _call_prepare(extra_kwargs=extra_kwargs)
|
||||
|
||||
# Returns (completion_kwargs, original_messages, ...) — first element
|
||||
# is the dict we care about.
|
||||
completion_kwargs = result[0] if isinstance(result, tuple) else result
|
||||
assert "output_config" not in completion_kwargs, (
|
||||
"Raw output_config must not be forwarded — non-Anthropic backends "
|
||||
"reject it with 400 'Extra inputs are not permitted'"
|
||||
)
|
||||
|
||||
def test_output_config_with_format_is_stripped_format_already_translated(self):
|
||||
"""Even when ``output_config`` carries useful structured-output info,
|
||||
the raw key must be excluded — the translator above has already mapped
|
||||
``output_config.format`` to ``response_format`` (the OpenAI-shaped key
|
||||
the downstream backend understands)."""
|
||||
extra_kwargs = {
|
||||
"custom_llm_provider": "azure",
|
||||
"output_config": {
|
||||
"format": {"type": "json_schema", "schema": {"type": "object"}}
|
||||
},
|
||||
}
|
||||
|
||||
result = _call_prepare(extra_kwargs=extra_kwargs)
|
||||
completion_kwargs = result[0] if isinstance(result, tuple) else result
|
||||
|
||||
assert "output_config" not in completion_kwargs
|
||||
|
||||
def test_other_extra_kwargs_still_passed_through(self):
|
||||
"""Regression guard: the strip must be narrow. Unrelated fields like
|
||||
``api_key`` / ``timeout`` continue to flow through."""
|
||||
extra_kwargs = {
|
||||
"custom_llm_provider": "azure",
|
||||
"output_config": {"effort": "high"},
|
||||
"timeout": 30,
|
||||
"user": "end-user-123",
|
||||
}
|
||||
|
||||
result = _call_prepare(extra_kwargs=extra_kwargs)
|
||||
completion_kwargs = result[0] if isinstance(result, tuple) else result
|
||||
|
||||
assert "output_config" not in completion_kwargs
|
||||
assert completion_kwargs.get("timeout") == 30
|
||||
assert completion_kwargs.get("user") == "end-user-123"
|
||||
|
||||
|
||||
class TestEmptyExtraKwargsPath:
|
||||
"""Greptile P2 on PR #22727: ``extra_kwargs or {default}`` substitutes a
|
||||
default for an explicitly-passed empty dict, hiding the no-extra-kwargs
|
||||
path. The new explicit-None pattern lets ``extra_kwargs={}`` reach the
|
||||
code under test as written."""
|
||||
|
||||
def test_explicit_empty_dict_does_not_substitute_default(self):
|
||||
# Explicit empty dict must be honored — not silently replaced with a
|
||||
# default that adds back a custom_llm_provider this test wants absent.
|
||||
result = _call_prepare(extra_kwargs={})
|
||||
completion_kwargs = result[0] if isinstance(result, tuple) else result
|
||||
|
||||
# No output_config because nothing supplied it.
|
||||
assert "output_config" not in completion_kwargs
|
||||
|
||||
def test_none_extra_kwargs_handled_safely(self):
|
||||
"""The signature documents ``extra_kwargs: Optional[Dict] = None``;
|
||||
passing None must not crash with KeyError or AttributeError."""
|
||||
result = _call_prepare(extra_kwargs=None)
|
||||
# Just exercising the path; assert no exception and we get back a
|
||||
# dict-like result.
|
||||
completion_kwargs = result[0] if isinstance(result, tuple) else result
|
||||
assert isinstance(completion_kwargs, dict)
|
||||
|
|
@ -203,12 +203,15 @@ def test_vertex_ai_anthropic_structured_output_header_not_added():
|
|||
def test_vertex_ai_claude_sonnet_4_5_structured_output_fix():
|
||||
"""
|
||||
Test fix for issue #18625: Claude Sonnet 4.5 on VertexAI should use tool-based
|
||||
structured outputs instead of output_format parameter.
|
||||
structured outputs when ``response_format`` is supplied via the OpenAI-compat
|
||||
interface (``map_openai_params``).
|
||||
|
||||
This test verifies that:
|
||||
1. Claude Sonnet 4.5 uses tool-based structured outputs on VertexAI
|
||||
2. output_format parameter is removed from the final request
|
||||
3. The fix prevents "Extra inputs are not permitted" error
|
||||
1. Claude Sonnet 4.5 uses tool-based structured outputs when ``response_format``
|
||||
is given to the OpenAI-compat path (the path that triggered #18625).
|
||||
2. ``output_format`` is forwarded to Vertex AI when present — Vertex now
|
||||
accepts the field; the prior blanket-strip behavior was the silent drop
|
||||
of Anthropic Structured Outputs that this PR fixes.
|
||||
"""
|
||||
config = VertexAIAnthropicConfig()
|
||||
|
||||
|
|
@ -294,11 +297,15 @@ def test_vertex_ai_claude_sonnet_4_5_structured_output_fix():
|
|||
headers={},
|
||||
)
|
||||
|
||||
# Verify that output_format was removed (fixes the "Extra inputs are not permitted" error)
|
||||
# output_format is now forwarded to Vertex (Vertex parity has shifted —
|
||||
# it accepts the field and uses it to enforce the JSON schema). The
|
||||
# prior behavior silently stripped it, hiding Structured Outputs from
|
||||
# callers who explicitly requested them.
|
||||
assert "output_format" in final_data
|
||||
assert final_data["output_format"]["type"] == "json_schema"
|
||||
assert (
|
||||
"output_format" not in final_data
|
||||
), "output_format should be removed for VertexAI"
|
||||
assert "model" not in final_data, "model should be removed for VertexAI"
|
||||
"model" not in final_data
|
||||
), "model is still stripped (Vertex routes by URL)"
|
||||
assert "tools" in final_data, "tools should still be present"
|
||||
assert "tool_choice" in final_data, "tool_choice should still be present"
|
||||
|
||||
|
|
@ -491,28 +498,22 @@ def test_vertex_ai_partner_models_anthropic_remove_prompt_caching_scope_beta_hea
|
|||
), "Header should be removed if no supported values remain"
|
||||
|
||||
|
||||
def test_vertex_ai_anthropic_output_config_dropped():
|
||||
def test_vertex_ai_anthropic_output_config_effort_only_dropped():
|
||||
"""
|
||||
Test that output_config parameter is dropped from Vertex AI Anthropic requests.
|
||||
|
||||
Vertex AI does not support the output_config parameter (used for effort settings
|
||||
in Anthropic API). This test ensures it's properly removed to prevent
|
||||
"Extra inputs are not permitted" errors.
|
||||
``output_config`` containing only ``effort`` (an Anthropic-only key Vertex
|
||||
rejects with "Extra inputs are not permitted") is dropped entirely so the
|
||||
request body has no empty dict.
|
||||
"""
|
||||
config = VertexAIAnthropicConfig()
|
||||
|
||||
messages = [{"role": "user", "content": "What is 2+2?"}]
|
||||
headers = {}
|
||||
headers: dict = {}
|
||||
|
||||
# Simulate optional_params with output_config that would be passed in
|
||||
optional_params = {
|
||||
"max_tokens": 1024,
|
||||
"output_config": {
|
||||
"effort": "high" # This is Anthropic-specific and not supported by Vertex AI
|
||||
},
|
||||
"output_config": {"effort": "high"},
|
||||
}
|
||||
|
||||
# Call transform_request which should drop output_config
|
||||
result = config.transform_request(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=messages,
|
||||
|
|
@ -521,54 +522,144 @@ def test_vertex_ai_anthropic_output_config_dropped():
|
|||
headers=headers,
|
||||
)
|
||||
|
||||
# Verify output_config was removed
|
||||
assert (
|
||||
"output_config" not in result
|
||||
), "output_config should be dropped from Vertex AI Anthropic requests"
|
||||
|
||||
# Verify other parameters are preserved
|
||||
assert result["max_tokens"] == 1024, "max_tokens should be preserved"
|
||||
assert "messages" in result, "messages should be present"
|
||||
), "output_config containing only effort must be dropped"
|
||||
assert result["max_tokens"] == 1024
|
||||
assert "messages" in result
|
||||
|
||||
|
||||
def test_vertex_ai_anthropic_output_format_and_output_config_both_dropped():
|
||||
def test_vertex_ai_anthropic_output_config_format_passes_through():
|
||||
"""
|
||||
Test that both output_format and output_config are dropped from Vertex AI requests.
|
||||
|
||||
This ensures that even if both parameters somehow make it to the transform_request,
|
||||
they are properly cleaned up before sending to Vertex AI.
|
||||
``output_config`` containing structured-output ``format`` is FORWARDED to
|
||||
Vertex AI Claude — Vertex now accepts it and uses it for JSON Schema
|
||||
enforcement. Previously the entire field was being silently stripped, so
|
||||
Anthropic Structured Outputs never engaged on Vertex even when callers
|
||||
requested it.
|
||||
"""
|
||||
config = VertexAIAnthropicConfig()
|
||||
messages = [{"role": "user", "content": "Return a person object."}]
|
||||
|
||||
output_config = {
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": False,
|
||||
"properties": {
|
||||
"name": {"type": "string"},
|
||||
"age": {"type": "integer"},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
optional_params = {"max_tokens": 1024, "output_config": output_config}
|
||||
|
||||
result = config.transform_request(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
assert result["output_config"] == output_config
|
||||
|
||||
|
||||
def test_vertex_ai_anthropic_output_config_format_plus_effort_strips_only_effort():
|
||||
"""
|
||||
Greptile P1 on PR #23396: when ``output_config`` contains BOTH ``format``
|
||||
and ``effort``, the prior conditional-passthrough logic forwarded the
|
||||
full dict including the unsupported ``effort`` key, reproducing the
|
||||
400 error the fix was meant to resolve. Only ``effort`` (and any future
|
||||
Vertex-unsupported keys) should be filtered; ``format`` must survive.
|
||||
"""
|
||||
config = VertexAIAnthropicConfig()
|
||||
messages = [{"role": "user", "content": "Return a person object."}]
|
||||
|
||||
output_config = {
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": False,
|
||||
"properties": {"name": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
"effort": "high",
|
||||
}
|
||||
optional_params = {"max_tokens": 1024, "output_config": output_config}
|
||||
|
||||
result = config.transform_request(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
assert "output_config" in result
|
||||
assert (
|
||||
"effort" not in result["output_config"]
|
||||
), "effort must be stripped — Vertex returns 400 on unknown keys"
|
||||
assert result["output_config"]["format"] == output_config["format"]
|
||||
|
||||
|
||||
def test_vertex_ai_anthropic_output_config_non_dict_dropped():
|
||||
"""Defensive: if ``output_config`` is somehow not a dict, drop it rather
|
||||
than forwarding malformed data downstream."""
|
||||
config = VertexAIAnthropicConfig()
|
||||
messages = [{"role": "user", "content": "hi"}]
|
||||
optional_params = {"max_tokens": 64, "output_config": "not-a-dict"}
|
||||
|
||||
result = config.transform_request(
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers={},
|
||||
)
|
||||
|
||||
assert "output_config" not in result
|
||||
|
||||
|
||||
def test_vertex_ai_anthropic_output_format_preserved_output_config_effort_dropped():
|
||||
"""
|
||||
When the request carries both ``output_format`` (top-level structured
|
||||
outputs) AND an ``output_config`` whose only useful key for Vertex is
|
||||
``effort``: ``output_format`` must be forwarded (Vertex accepts it),
|
||||
while ``output_config`` is dropped because Vertex returns 400 on
|
||||
``effort``. This replaces the old "drop both" behavior, which was the
|
||||
silent strip the bug report flagged.
|
||||
"""
|
||||
config = VertexAIAnthropicConfig()
|
||||
messages = [{"role": "user", "content": "Extract structured data"}]
|
||||
headers = {}
|
||||
|
||||
output_format = {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "data",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {"result": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
optional_params = {
|
||||
"max_tokens": 2048,
|
||||
"output_format": {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "data",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {"result": {"type": "string"}},
|
||||
},
|
||||
},
|
||||
},
|
||||
"output_format": output_format,
|
||||
"output_config": {"effort": "high"},
|
||||
}
|
||||
|
||||
# Simulate parent class creating test_data with both parameters
|
||||
# (as if the parent transform_request added them)
|
||||
test_data = {
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"messages": messages,
|
||||
"max_tokens": 2048,
|
||||
"output_format": optional_params["output_format"],
|
||||
"output_config": optional_params["output_config"],
|
||||
"output_format": output_format,
|
||||
"output_config": {"effort": "high"},
|
||||
}
|
||||
|
||||
# Mock the parent transform_request to return data with both parameters
|
||||
original_transform = config.__class__.__bases__[0].transform_request
|
||||
|
||||
def mock_transform_request(
|
||||
|
|
@ -584,22 +675,50 @@ def test_vertex_ai_anthropic_output_format_and_output_config_both_dropped():
|
|||
messages=messages,
|
||||
optional_params=optional_params,
|
||||
litellm_params={},
|
||||
headers=headers,
|
||||
headers={},
|
||||
)
|
||||
|
||||
# Verify both were removed
|
||||
assert (
|
||||
"output_format" not in result
|
||||
), "output_format should be dropped from Vertex AI requests"
|
||||
assert (
|
||||
"output_config" not in result
|
||||
), "output_config should be dropped from Vertex AI requests"
|
||||
|
||||
# Verify essential params are preserved
|
||||
assert result["max_tokens"] == 2048, "max_tokens should be preserved"
|
||||
assert "messages" in result, "messages should be present"
|
||||
assert "model" not in result, "model should also be dropped for Vertex AI"
|
||||
|
||||
# output_format flows through unchanged — Vertex AI Claude accepts it.
|
||||
assert result["output_format"] == output_format
|
||||
# output_config containing only ``effort`` is dropped to avoid the
|
||||
# 400 "Extra inputs are not permitted" the silent strip used to mask.
|
||||
assert "output_config" not in result
|
||||
assert result["max_tokens"] == 2048
|
||||
assert "model" not in result, "model is still stripped (Vertex routes by URL)"
|
||||
finally:
|
||||
# Restore original method
|
||||
config.__class__.__bases__[0].transform_request = original_transform
|
||||
|
||||
|
||||
def test_sanitize_vertex_anthropic_output_params_unit():
|
||||
"""Direct unit coverage for the helper itself (used by both Vertex
|
||||
Anthropic transformation paths). Mirrors the integration assertions
|
||||
above without going through the full ``transform_request`` stack."""
|
||||
from litellm.llms.vertex_ai.vertex_ai_partner_models.anthropic.transformation import (
|
||||
_sanitize_vertex_anthropic_output_params,
|
||||
)
|
||||
|
||||
# No-op when output_config absent.
|
||||
data: dict = {"max_tokens": 8}
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
assert data == {"max_tokens": 8}
|
||||
|
||||
# Effort-only → dropped entirely.
|
||||
data = {"output_config": {"effort": "high"}}
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
assert "output_config" not in data
|
||||
|
||||
# Format-only → preserved unchanged.
|
||||
fmt = {"format": {"type": "json_schema", "schema": {"type": "object"}}}
|
||||
data = {"output_config": dict(fmt)}
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
assert data["output_config"] == fmt
|
||||
|
||||
# Mixed → effort filtered, format kept.
|
||||
data = {"output_config": {"format": fmt["format"], "effort": "high"}}
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
assert data["output_config"] == fmt
|
||||
|
||||
# Non-dict → dropped defensively.
|
||||
data = {"output_config": "garbage"}
|
||||
_sanitize_vertex_anthropic_output_params(data)
|
||||
assert "output_config" not in data
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue