Resolves the silent strip of Anthropic Structured Outputs across the
Vertex AI Claude transformation paths and the Anthropic-adapter
re-merge. Consolidates and supersedes four stalled community PRs
addressing overlapping aspects of the same root bug:
- #23475 (Vertex AI Claude blanket-strip removal)
- #23396 (Vertex AI Claude conditional passthrough)
- #23706 (Anthropic adapter exclude output_config from non-Anthropic
backends)
- #22727 (Anthropic adapter strip output_config for non-Anthropic
backends)
Closes / addresses: #23380 (Vertex AI Claude output_config drop),
related: #26423, #25079, #24549, #25971, #25957, #26163, #24856.
What was broken
---------------
* Vertex AI Claude paths called ``data.pop("output_config")`` and
``data.pop("output_format")`` unconditionally even when Vertex
accepted those fields. Callers asking for Structured Outputs got a
200 with prose and never knew the schema constraints had been
silently dropped (often masked for months by permissive fallback
parsers).
* The ``/v1/messages`` -> ``/chat/completions`` adapter
(``LiteLLMMessagesToCompletionTransformationHandler``) re-merged the
raw Anthropic-shaped ``output_config`` into ``completion_kwargs``
AFTER the translator already mapped its meaningful parts to
``response_format`` / ``reasoning_effort``. Non-Anthropic backends
(Azure OpenAI, Fireworks, Bedrock Nova, etc.) then 400'd with
"Extra inputs are not permitted".
Approach
--------
Vertex AI Claude (chat-completion + experimental_pass_through paths):
Replace the unconditional pop with a sanitizer
``_sanitize_vertex_anthropic_output_params`` that strips only the
Vertex-unsupported keys (today: ``effort``) from ``output_config``
while forwarding ``format`` and the legacy top-level
``output_format``. Defensive: non-dict ``output_config`` values are
dropped to avoid sending malformed payloads downstream.
Greptile P1 from PR #23396 addressed: when ``output_config`` carries
both ``format`` and ``effort``, the prior conditional pass-through
forwarded ``effort`` and reproduced the 400. The new helper filters
per-key.
Anthropic ``/v1/messages`` adapter:
Add ``output_config`` to a named module-level constant
``ANTHROPIC_ONLY_REQUEST_KEYS`` and wire it into ``excluded_keys`` so
the post-translation re-merge skips re-adding the raw key. This
fixes the 400 on non-Anthropic backends and avoids the conflicting
duplicate (``response_format`` + raw ``output_config``) on
Anthropic-family backends.
Greptile P2 from PR #23706 addressed: the constant gives reviewers
one grep target instead of an inline literal that silently grows.
Greptile P2 from PR #22727 addressed: ``extra_kwargs or {}`` is
replaced with explicit ``is None`` checks so empty-dict callers no
longer skip the fallback path.
Tests
-----
* tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/
test_vertex_ai_partner_models_anthropic_transformation.py:
- 5 new/updated cases plus a direct unit test for
``_sanitize_vertex_anthropic_output_params``.
- Updated ``test_vertex_ai_claude_sonnet_4_5_structured_output_fix``
so its mock-injected ``output_format`` is asserted to FLOW THROUGH
(the original test asserted the now-buggy strip behavior).
* tests/test_litellm/llms/anthropic/experimental_pass_through/
adapters/test_handler_output_config_passthrough.py (new):
- Constant export sanity, output_config strip with ``effort`` only,
output_config strip with ``format`` only, regression guard that
unrelated extras still flow, explicit-empty-dict path, and the
``extra_kwargs=None`` no-crash path.
Test-quality fixes incorporated from Greptile review on the
superseded PRs:
* No ``inspect.getsource`` source-text assertions (PR #24114 / #23475).
* ``sys.path`` insertion is anchored to ``__file__`` (PR #23706).
* Assertion messages are positional, not tuple (PR #24114-class bug).
* No ``or {}`` masking explicit empty dicts in helper signatures
(PR #22727).
Verified locally: 26/26 pass with this commit. The new tests
fail (or fail to import) on ``main`` without it.
Out of scope
------------
* The ``max_tokens`` capping logic from PR #22727 — independent
concern, deserves its own PR with a focused test plan.
* Architectural rework of the ``excluded_keys`` mechanism (Greptile
P2 on PR #23706 noted point-fix growth). The named constant gives
maintainers a clear place to extend; a registry-based approach
would be a follow-up.
Co-Authored-By: netbrah <netbrah>
Co-Authored-By: s-zx <s-zx>
Co-Authored-By: invoicepulse <invoicepulse>
Co-Authored-By: cfdude <cfdude>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Waiting to run
Members tab column reads this field; dropping it from the type in the
previous revert broke the type check without affecting the reverted
render logic.
Temporary MCP OAuth sessions were kept in process-local memory, so on
multi-instance/LB proxy deployments a session created on instance A could
not be found when the follow-up /server/oauth/{server_id}/... request
landed on instance B.
Persist temporary session records to Redis (encrypted with the existing
proxy encryption helpers) as a best-effort L2 cache alongside the current
in-memory L1. Convert get_cached_temporary_mcp_server to async and await
it from the authorize/token/register OAuth endpoints.
Made-with: Cursor
Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not
{geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen.
common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building.
Proxy pass-through duplicates the same branching in a local get_vertex_base_url
(with trailing slashes) to avoid importing from common_utils there; live
WebSocket passthrough uses the same multi-region host logic for wss://.
Tests cover us/eu for the common_utils helper.
Made-with: Cursor
Relative labels ("today", "in 2 days", "on May 12, 2026") mixed three
shapes in one column, breaking scannability. Always render MMM D, YYYY
for consistency and easier at-a-glance comparison across members.
The create-branch job in create-release.yml calls the reusable
create-release-branch.yml workflow, which requires contents: write.
The top-level permissions: {} blocks the inherited default, and only
the release job overrode it, so the nested call failed with:
The nested job 'create-branch' is requesting 'contents: write',
but is only allowed 'contents: none'.
Add the permission at the calling job level so the reusable
workflow is granted what it needs.
The previous `while not self._queue.empty(): await self._queue.join()`
pattern skipped the join entirely when the worker had already dequeued a
task but not yet called task_done(). asyncio.Queue.join() tracks
_unfinished_tasks (incremented by put, decremented by task_done), not
queue depth, so it already handles that case on its own.
The mocked async_increment_cache_pipeline is invoked from Router's
deployment_callback_on_success, registered as an async success callback.
Those callbacks are enqueued to GLOBAL_LOGGING_WORKER and run on a
background task, so the mock may not have been called yet when the test
asserts on it. Flush the worker before asserting.
Previous commit from greptile-apps added a new `when: always` teardown
step without removing the prior `name:`-only step, leaving a `- run`
block with no `command:` — CircleCI config validation rejects that.
Collapse back to a single teardown step that runs on success and
failure.
Two independent deflakes:
1. test_ui_view_spend_logs_unauthorized (unit) was returning 400 instead
of 401/403 when earlier tests in the file left proxy-auth globals
(prisma_client, master_key, user_custom_auth, general_settings,
user_api_key_cache) in a state that let invalid tokens pass auth and
fall through to the endpoint's own start_date/end_date validation.
Add an autouse fixture that pins those globals to their import-time
defaults for every test in the file. Harden the assertion to include
response body so future flakes are diagnosable.
2. test_basic_spend_accuracy (CI job proxy_spend_accuracy_tests) depends
on the Redis transaction buffer flushing spend to Postgres. The buffer
uses a single global pod-lock key (cronjob_lock:db_spend_update_job)
and a single global buffer list key. Pointing the proxy at the shared
remote Redis means concurrent CI pipelines contend for the same lock
and can drain each other's buffer into the wrong database. Add a
start_redis reusable command that boots a per-job redis:7-alpine
container (digest-pinned), and switch proxy_spend_accuracy_tests to
REDIS_HOST=host.docker.internal:6379 so lock and buffer state are
isolated per CI run.
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
* feat: add gpt-5.5 to model cost map
Add gpt-5.5 entry with pricing from OpenAI flagship page:
input $5/1M, cached input $0.50/1M, output $30/1M, 272K context.
* test: add gpt-5.5 coverage for model cost map and gpt-5 routing
- Add gpt-5.5 to GPT5_MODELS parametrized list so both OpenAIGPT5Config
and AzureOpenAIGPT5Config routing tests cover the new model.
- Add test_generic_cost_per_token_gpt55 verifying the new entry's
cost-map values ($5/$0.50/$30 per 1M) and that generic_cost_per_token
returns the expected prompt/completion costs.
The periodic budget-window reset job filtered keys/teams with
`where={"budget_limits": {"not": None}}`. The prisma-client-python
library does not support null-filtering on `Json?` columns (no
DbNull/JsonNull sentinel — upstream issue #714). The client drops the
`None` value during serialization and the engine rejects the query with
`MissingRequiredValueError: where.budget_limits.not: A value is
required but not set`, so neither the key nor team reset path runs.
Switch those two `find_many` calls to `query_raw` with
`WHERE budget_limits IS NOT NULL`, selecting only the PK and the
`budget_limits` column. Writes still go through the ORM. Add unit tests
covering the expired/unexpired paths for keys and teams, string-encoded
JSON payloads, empty payloads, error isolation between the two paths,
and a regression guard asserting the query still uses `IS NOT NULL`.
Extracts release branch creation into a separate reusable workflow
(create-release-branch.yml) that can be triggered independently via
workflow_dispatch or called from other workflows via workflow_call.
create-release.yml now dispatches it as a dependent job after the
release publishes, keeping both workflows decoupled.
token_counter.py: the previous size-limit raises were inside except Exception: pass,
so they were silently swallowed. The post-read raise was worse — img_data was already
assigned the full body before the raise, so the oversized value was used downstream.
Restructured to only assign img_data when the body is within bounds.
vertex_ai/common_utils.py and llm_passthrough_endpoints.py: the is-not-None guard
skipped validation for None, falling through to produce "https://None-aiplatform..."
Added explicit None check that raises before the regex guard.
The rvm/install script sources scripts/functions/installer using
paths relative to the caller's working directory (not $0), so
invoking /tmp/rvm/install from /home/circleci/project fails with
'No such file or directory'. Switch to (cd /tmp/rvm && ./install).
Align the Ruby, Node.js, and npm install path with the rest of the
config. Three separate upstream installers were being invoked via
\`curl ... | bash\` or unlocked \`npm install\`:
- RVM's \`get.rvm.io/stable\` installer (mutable upstream script).
Replace with a shallow git clone of the rvm/rvm repo at tag 1.29.12
and verify HEAD matches the published commit SHA before running the
local \`./install\` script. Same pattern already used for the
helm-unittest plugin in .github/workflows/helm_unit_test.yml.
- NodeSource's \`deb.nodesource.com/setup_18.x\` piped into sudo bash.
Replace with a direct download of the Node.js 18.20.8 linux-x64
tarball from nodejs.org, verified against the published
SHASUMS256.txt digest before extraction.
- \`npm install @google-cloud/vertexai @google/generative-ai\` and
\`--save-dev jest\` resolved fresh from the npm registry on every
run. Add \`tests/pass_through_tests/package.json\` with pinned
direct-dep versions and commit the generated package-lock.json, then
switch CI to \`npm ci\` (exact lockfile install, fails on drift).
Also scopes the Ruby+JS test runners to \`tests/pass_through_tests/\`
so they pick up the committed package.json rather than writing
node_modules at repo root.
Stragglers from the 2026-04-21 Python 3.12 standardization:
- .github/workflows/check_duplicate_issues.yml (was 3.11)
- .github/workflows/llm-translation-testing.yml (was 3.11)
- .github/workflows/scan_duplicate_issues.yml (was 3.13)
- .circleci proxy_build_from_pip_tests (was 3.13)
The only intentional non-3.12 CI job is installing_litellm_on_python_3_13,
which exists as an explicit "latest supported Python" smoke matrix.
The Windows uv install step was piping a remote install.ps1 into
Invoke-Expression without any integrity check, while the Linux
install steps (install_uv command, line 89) download to a file,
verify SHA-256 against a hardcoded digest, and only then execute.
Bring the Windows path to the same pattern.
Also hardcode the kubectl v1.31.4 checksum in helm_chart_testing
instead of fetching kubectl.sha256 from the same origin as the
binary — if dl.k8s.io were ever to serve a tampered pair, a
co-hosted checksum provides no additional integrity.
The same branch filter block appeared 46 times in the workflow
declaration:
filters:
branches:
only:
- main
- /litellm_.*/
And the same pinned Python docker image appeared 29 times in jobs:
- image: cimg/python:3.12@sha256:9c796c...
auth:
username: ${DOCKERHUB_USERNAME}
password: ${DOCKERHUB_PASSWORD}
Replace with YAML anchors declared at first use:
- `&main_branches` on using_litellm_on_windows's filters block;
all other job entries reference it as `filters: *main_branches`.
- `&python312_image` on local_testing_part1's first docker image
entry; all other jobs reference `- *python312_image`, including
the multi-image jobs (auth_ui_unit_tests,
installing_litellm_on_python_v2_migration_resolver) which keep
their postgres sidecar entry inline afterwards.
Net result: one place to change when the image digest rolls or
the branch-filter convention changes. No behavior change — YAML
anchor resolution produces identical config at parse time.
Also adds Docker Hub auth block to upload-coverage (previously
pulled anonymously). No functional difference for a public
image, but avoids Docker Hub rate limits now that we reuse the
same entry.
Clean out copy-paste debug and workaround lines that serve no
purpose:
- `pwd && ls` echoes at the top of 30 "Run tests" steps (CCI
already logs working_directory on every step).
- "Show git commit hash" in local_testing_part1/part2 and
langfuse_logging_unit_tests (CCI shows the SHA in every job
header).
- "Verify Docker is available" stubs in 6 machine-executor
jobs (machine executors always have Docker).
- `sudo systemctl restart docker` in proxy_store_model_in_db_tests
(one-off workaround; not used anywhere else).
- Duplicated Black formatting step in local_testing_part1 and
local_testing_part2 — Black runs in the lint job, no reason to
run it again here.
- Second back-to-back `helm test litellm --logs` invocation in
helm_chart_testing (one call is enough).
No behavior change — these are all log-only or no-op steps.
The original check `"gpt-5-chat" not in model` already correctly
classifies all current gpt-5 variants (including gpt-5.3-chat and
gpt-5.1-chat, which do NOT contain the substring "gpt-5-chat"). This
change replaces it with an explicit `startswith("gpt-5-chat")` prefix
test on the provider-prefix-stripped model name.
The new check is functionally equivalent for all existing model names
but makes the classification boundary unambiguous and forward-safe:
future model names that might contain "gpt-5-chat" as an interior
substring won't accidentally be excluded from the GPT-5 reasoning path.
Also moves the new regression test from tests/ root to
tests/test_litellm/llms/openai/ so it is included in `make test-unit`.
