* feat(cost-map): move -latest model aliases into the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the finalized local cost map in the moonshot alias test and assert alias pricing by behavior
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: compare alias cost map rows to their targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): give each -latest alias its own alias_of row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 91e6cf2c27)
* fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit a54c42218a)
* feat(cost-map): add -latest alias rows for the tier 1 and 2 hosting providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-map): derive hosted alias targets from the cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): skip alias rows when checking cache-write rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(xai): use billed image cost and resolution prices
* test(xai): mark image generation tests as a package
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): add Kimi K3.1 rows inherited from Kimi K3
* fix(bedrock): sync published Kimi K3.1 Highspeed prices
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): jais-30b-chat price was per 1K tokens, store it per token
* fix(watsonx): jais-13b-chat price was per 1K tokens, store it per token
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): enable native structured output by default for selected Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool
A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too.
* fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action
A role allowed only bedrock:InvokeModel got a 403 on schema streams once these
models went native, since native streams call converse-stream while the JSON
tool path buffered through converse. The retry now sends that request again
as a buffered JSON tool stream, the shape the base sent.
The wire test moves its neither-key and opt-in cells to Opus 4.7, its
Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the
bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native.
The batch wire test now checks a thinking schema record on both sides of the
contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5
and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8
cells assumed those models were opt-in.
* fix(bedrock): move the Claude 5 native output default to a key released proxies do not read
Released proxies load the cost map from main and have none of the native
output retries, so setting supports_native_structured_output on the Opus
5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide
or min/max schemas into 400s on v1.108.1.
Those rows keep supports_native_structured_output false and gain
bedrock_native_structured_output_by_default true, which get_model_info
reports as supports_native_structured_output, so this version behaves as
before while released versions keep the JSON tool.
* fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default
The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it
* fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key
A registered supports_bedrock_native_structured_output false now beats the
native output default, and batch uploads read the deployment's model_info, so
an opted-out deployment keeps the JSON tool on every record. The Opus 5 and
Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default,
so released proxies on main's cost map keep the JSON tool on them. The 64
property batch cutoff now applies only to rows with that key, so models that
were already native keep their batch output shape.
* fix(proxy): report a deployment's Bedrock native output opt-out on /model/info
/model/info merged the deployment's model_info over the backend row, so a
deployment that set only supports_bedrock_native_structured_output: false
still showed the backend's derived supports_native_structured_output: true.
The generic flag is now derived from the merged info unless the deployment
set it itself, through the same helper get_model_info uses.
* fix(bedrock): keep native output when a denied schema stream is resent buffered
A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output.
* fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend
* fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment
The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model
* fix(bedrock): answer a buffered invoke stream from the non-streaming reply
* fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected
* fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info
A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main.
/model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): forward the per-turn-control beta to Bedrock Invoke
Claude Code sends `anthropic-beta: per-turn-control-2026-07-01` together
with a mid-conversation `role: system` message carrying a message-level
`output_config` (for example an effort change). The beta header config
mapped the beta to null for `bedrock`, so LiteLLM dropped it while
keeping the message in place on models flagged
`supports_mid_conversation_system`, and Bedrock Invoke rejected the turn
with "messages.N.output_config: Extra inputs are not permitted".
Direct Bedrock InvokeModel probes with a synthetic request show Claude
Opus 5.5 accepts that message shape only with this beta, and that Claude
Haiku 4.5, Sonnet 4.6, Opus 4.8, Opus 5, and Sonnet 5 accept the beta on
a plain request, so map it to itself for `bedrock`. `bedrock_converse`
is unchanged.
Generated with AI
Co-Authored-By: Claude Code
* test(anthropic): expect the per-turn-control beta to reach Bedrock Invoke
Generated with AI
Co-Authored-By: Claude Code
* test(bedrock): drop the duplicate per-turn-control filter test and cite the live check
* fix(rust): allow the per-turn-control beta on bedrock in the Rust beta table
The Rust table must match litellm/anthropic_beta_headers_config.json cell for cell, and the JSON now maps the beta for bedrock. Also type the new Bedrock Invoke test.
* fix(bedrock): forward per-turn-control on Bedrock Invoke only where the cost map flags the model
Bedrock InvokeModel accepts per-turn-control-2026-07-01 on Claude 4.5 and newer and answers 400 invalid beta flag
on Claude 3 Haiku. Released proxies load the beta map from main, so forwarding it in the bedrock row would turn
Claude 3 Haiku requests on every running release from 200 into 400. Keep that row null and forward the client's
beta from the Invoke transformation when a new claude-per-turn-control cost map rule sets supports_per_turn_control.
* fix(bedrock): keep the per-turn-control beta through the messages handler filter on Bedrock Invoke
The /v1/messages handler filtered client betas against the beta map before the
Bedrock Invoke config could add per-turn-control back for models the cost map
flags, so the beta never reached Bedrock. Provider configs now name betas their
model accepts beyond the beta map, and the handler filter keeps those
* fix(bedrock): keep a deployment's per-turn-control flag on that deployment
supports_per_turn_control is now deployment-scoped, so a deployment's opt-in or opt-out no longer changes the beta forward for other deployments of the same Bedrock model. The forward reads the deployment's own model_info first, then the model map.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): enable native structured output for Opus 5 models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep JSON-tool streaming coverage on unchanged route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): rolling registry audit, add gpt-6-cyber, Kimi K3.1 Pro rows, Jina rerankers and capability flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): declare gpt-6-cyber PDF input support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add the flex prices over 200k tokens to gemini-3.1-pro-preview and gemini-2.5-pro
* chore(cost-map): absorb GMI catalog rows and EmpirioLabs page prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add grounding prices to the gemini 3.8 live and 3.1 flash live rows
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): stop forcing the response_format json tool when Claude will think
A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.
AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.
The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.
* fix(bedrock): route native output and thinking token counts by whether Claude will think
Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.
* fix(anthropic): tell Claude to answer through the unforced response_format tool
Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output
* fix(anthropic): keep open-object schemas on the json tool when Claude will think
Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main
* fix(anthropic): hold streamed prose until the JSON tool decides the answer
In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.
* test(anthropic): build the stream test events without nested comprehensions
* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only
* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model
* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced
Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content
* fix(anthropic): drop the thinking False forcing case that completion never reaches
litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default
* test(bedrock): collect the Converse json_schema stream chunks in one assignment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost_map): fill missing Claude capability fields from the family baseline
Mapped Claude entries on Anthropic, Azure Foundry, Bedrock and Vertex now get
supports_parallel_function_calling and supports_system_messages from the
claude-family-baseline rule when the entry leaves them out. Values set on an
entry still win
* test(cost_map): keep the exact fill-missing assertion for Claude thinking rules
Build the expected baseline part from the loaded map so a rule that leaks an extra key into Claude entries still fails, and wrap lines over 120 characters.
* fix(bedrock): add GovCloud Kimi K3 runtime prices
* fix(bedrock): omit retired max_tokens from Kimi rows
* fix(bedrock): add GovCloud Haiku and Grok prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): match GovCloud Grok 4.7 limits to the card and date GovCloud Claude 3 Haiku
* fix(bedrock): add priced Grok Mantle base for regional routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* chore(cost-map): set openai gpt-5.6-cyber and Daybreak Red max input to 272K and source the sora-2-pro-high-res deprecation
* Absorb GovCloud Bedrock, Azure GPT-6 Flex and GMI registry rows
Roll the verified registry data and Azure Flex cost coverage into the rolling price sync
Co-authored-by: kerry-berri <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: dguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
* feat: update EmpirioLabs models in model_prices_and_context_window.json
* chore(endpoints): regenerate provider endpoint support for the EmpirioLabs cost map rows
* fix(empiriolabs): drop the retired max_tokens field, route embeddings and remove the unrouted rerank row
The cost map test rejects any row carrying max_tokens, so every EmpirioLabs row keeps max_input_tokens and max_output_tokens only. The provider JSON now lists /v1/embeddings so embedding(model="empiriolabs/text-embedding-v4") reaches the OpenAI-compatible embeddings route instead of failing provider lookup. The qwen3-rerank row goes away because no rerank route exists for the provider. Endpoint support regenerated from the cost map
* fix(empiriolabs): mark audio input and image embedding input on the rows the provider lists as multimodal
The three vision embedding rows get supports_embedding_image_input and the eleven chat rows whose input_modalities on GET https://api.empiriolabs.ai/v1/models include audio get supports_audio_input, so capability checks stop treating them as text only
* fix(empiriolabs): carry the long-context price tiers on the 24 rows the provider bills in bands
EmpirioLabs bills 24 of these models at a higher rate once the prompt passes 32K, 128K, 256K, 272K or 512K tokens, and the rows carried only the first band, so a 300K-token qwen3-7-plus request was priced at a third of the real cost. Each of those rows now carries input_cost_per_token_above_<N>k_tokens, output_cost_per_token_above_<N>k_tokens and, where the row already has cache_read_input_token_cost, cache_read_input_token_cost_above_<N>k_tokens for every band, taken from pricing[].min_context on GET https://api.empiriolabs.ai/v1/models for the base rows and from the Variants rate card on docs.empiriolabs.ai/models/<id> for the six variant rows
* feat(empiriolabs): bill the per-call web search fee reported in usage.tool_usage
EmpirioLabs charges a flat fee per invoked web_search, web_search_image or image_search call and reports the counts in usage.tool_usage on the chat response. The JSON provider config now carries that count as usage.server_tool_use.web_search_requests when providers.json sets special_handling.web_search_requests_from_tool_usage, the per-call web search cost path bills the usage-reported count instead of flooring to one call, and the 39 EmpirioLabs rows the provider lists a per-call search price for carry it as search_context_cost_per_query, from unit_pricing on GET /v1/models for base rows and the Variants rate card on docs.empiriolabs.ai for variant rows
* fix(openai_like): mark the dict signature of transform_parsed_response_dict as matching BaseConfig for LIT001
* fix(openai_like): mark both dict annotations of transform_parsed_response_dict for LIT001
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(openai): bill gov.api.openai.com FedRAMP endpoint with 1.1x uplift
* chore(openai): trim FedRAMP uplift diff
* refactor(openai): bill the FedRAMP endpoint through its own fedramp_uplift_multiplier key
* fix(ui): regenerate fedramp_uplift_multiplier doc comment in schema.d.ts
* style: ruff format utils.py and reorder fedramp_uplift_multiplier in schema.d.ts
* fix(openai): infer data_residency from OPENAI_BASE_URL and litellm.api_base when api_base is unset
* test(cost): cover get_fedramp_uplift_multiplier fallback paths
* fix(cost): apply fedramp_uplift_multiplier in batch_cost_calculator
* fix(openai): derive data_residency from a supplied client's base_url
* test: type the supplied-client data_residency test params
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
