mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-07 07:58:09 +00:00
36c64ce85d
* feat(bedrock): enable native structured output by default for selected Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too. * fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action A role allowed only bedrock:InvokeModel got a 403 on schema streams once these models went native, since native streams call converse-stream while the JSON tool path buffered through converse. The retry now sends that request again as a buffered JSON tool stream, the shape the base sent. The wire test moves its neither-key and opt-in cells to Opus 4.7, its Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native. The batch wire test now checks a thinking schema record on both sides of the contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5 and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8 cells assumed those models were opt-in. * fix(bedrock): move the Claude 5 native output default to a key released proxies do not read Released proxies load the cost map from main and have none of the native output retries, so setting supports_native_structured_output on the Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide or min/max schemas into 400s on v1.108.1. Those rows keep supports_native_structured_output false and gain bedrock_native_structured_output_by_default true, which get_model_info reports as supports_native_structured_output, so this version behaves as before while released versions keep the JSON tool. * fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it * fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key A registered supports_bedrock_native_structured_output false now beats the native output default, and batch uploads read the deployment's model_info, so an opted-out deployment keeps the JSON tool on every record. The Opus 5 and Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default, so released proxies on main's cost map keep the JSON tool on them. The 64 property batch cutoff now applies only to rows with that key, so models that were already native keep their batch output shape. * fix(proxy): report a deployment's Bedrock native output opt-out on /model/info /model/info merged the deployment's model_info over the backend row, so a deployment that set only supports_bedrock_native_structured_output: false still showed the backend's derived supports_native_structured_output: true. The generic flag is now derived from the merged info unless the deployment set it itself, through the same helper get_model_info uses. * fix(bedrock): keep native output when a denied schema stream is resent buffered A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output. * fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend * fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model * fix(bedrock): answer a buffered invoke stream from the non-streaming reply * fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected * fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main. /model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _support | ||
| audio_tests | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.