* fix(bedrock): forward the per-turn-control beta to Bedrock Invoke
Claude Code sends `anthropic-beta: per-turn-control-2026-07-01` together
with a mid-conversation `role: system` message carrying a message-level
`output_config` (for example an effort change). The beta header config
mapped the beta to null for `bedrock`, so LiteLLM dropped it while
keeping the message in place on models flagged
`supports_mid_conversation_system`, and Bedrock Invoke rejected the turn
with "messages.N.output_config: Extra inputs are not permitted".
Direct Bedrock InvokeModel probes with a synthetic request show Claude
Opus 5.5 accepts that message shape only with this beta, and that Claude
Haiku 4.5, Sonnet 4.6, Opus 4.8, Opus 5, and Sonnet 5 accept the beta on
a plain request, so map it to itself for `bedrock`. `bedrock_converse`
is unchanged.
Generated with AI
Co-Authored-By: Claude Code
* test(anthropic): expect the per-turn-control beta to reach Bedrock Invoke
Generated with AI
Co-Authored-By: Claude Code
* test(bedrock): drop the duplicate per-turn-control filter test and cite the live check
* fix(rust): allow the per-turn-control beta on bedrock in the Rust beta table
The Rust table must match litellm/anthropic_beta_headers_config.json cell for cell, and the JSON now maps the beta for bedrock. Also type the new Bedrock Invoke test.
* fix(bedrock): forward per-turn-control on Bedrock Invoke only where the cost map flags the model
Bedrock InvokeModel accepts per-turn-control-2026-07-01 on Claude 4.5 and newer and answers 400 invalid beta flag
on Claude 3 Haiku. Released proxies load the beta map from main, so forwarding it in the bedrock row would turn
Claude 3 Haiku requests on every running release from 200 into 400. Keep that row null and forward the client's
beta from the Invoke transformation when a new claude-per-turn-control cost map rule sets supports_per_turn_control.
* fix(bedrock): keep the per-turn-control beta through the messages handler filter on Bedrock Invoke
The /v1/messages handler filtered client betas against the beta map before the
Bedrock Invoke config could add per-turn-control back for models the cost map
flags, so the beta never reached Bedrock. Provider configs now name betas their
model accepts beyond the beta map, and the handler filter keeps those
* fix(bedrock): keep a deployment's per-turn-control flag on that deployment
supports_per_turn_control is now deployment-scoped, so a deployment's opt-in or opt-out no longer changes the beta forward for other deployments of the same Bedrock model. The forward reads the deployment's own model_info first, then the model map.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(bedrock): enable native structured output for Opus 5 models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep JSON-tool streaming coverage on unchanged route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): rolling registry audit, add gpt-6-cyber, Kimi K3.1 Pro rows, Jina rerankers and capability flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): declare gpt-6-cyber PDF input support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add the flex prices over 200k tokens to gemini-3.1-pro-preview and gemini-2.5-pro
* chore(cost-map): absorb GMI catalog rows and EmpirioLabs page prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add grounding prices to the gemini 3.8 live and 3.1 flash live rows
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): stop forcing the response_format json tool when Claude will think
A forced tool call skips Claude's thinking. The json_schema emulation forced
json_tool_call unless thinking.type was enabled, so adaptive thinking and the
models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on
families) lost their thinking on every structured-output request.
AnthropicModelInfo.response_format_tool_can_be_forced is now the one check
the Anthropic, Bedrock Converse (including the native output fallback) and
Databricks paths use. It leaves the tool unforced when thinking is requested
in any form or the model thinks by default, and keeps forcing it when the
request turns thinking off (thinking disabled or reasoning_effort none) on a
model that accepts forced tools.
The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and
Haiku 5.5 on the providers that reach these paths, and a miss-only fallback
rule covers unmapped Claude 5 and later ids.
* fix(bedrock): route native output and thinking token counts by whether Claude will think
Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them.
* fix(anthropic): tell Claude to answer through the unforced response_format tool
Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output
* fix(anthropic): keep open-object schemas on the json tool when Claude will think
Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main
* fix(anthropic): hold streamed prose until the JSON tool decides the answer
In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply.
* test(anthropic): build the stream test events without nested comprehensions
* fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only
* test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model
* fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced
Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse
json_schema stream with the unforced JSON tool never sends Claude's working text as content
* fix(anthropic): drop the thinking False forcing case that completion never reaches
litellm.completion turns thinking False into no thinking param before any route sees it,
so the case was dead on the request path and False keeps meaning the model's default
* test(bedrock): collect the Converse json_schema stream chunks in one assignment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost_map): fill missing Claude capability fields from the family baseline
Mapped Claude entries on Anthropic, Azure Foundry, Bedrock and Vertex now get
supports_parallel_function_calling and supports_system_messages from the
claude-family-baseline rule when the entry leaves them out. Values set on an
entry still win
* test(cost_map): keep the exact fill-missing assertion for Claude thinking rules
Build the expected baseline part from the loaded map so a rule that leaks an extra key into Claude entries still fails, and wrap lines over 120 characters.
* fix(bedrock): add GovCloud Kimi K3 runtime prices
* fix(bedrock): omit retired max_tokens from Kimi rows
* fix(bedrock): add GovCloud Haiku and Grok prices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): match GovCloud Grok 4.7 limits to the card and date GovCloud Claude 3 Haiku
* fix(bedrock): add priced Grok Mantle base for regional routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* chore(cost-map): set openai gpt-5.6-cyber and Daybreak Red max input to 272K and source the sora-2-pro-high-res deprecation
* Absorb GovCloud Bedrock, Azure GPT-6 Flex and GMI registry rows
Roll the verified registry data and Azure Flex cost coverage into the rolling price sync
Co-authored-by: kerry-berri <326648561+kerry-berri@users.noreply.github.com>
Co-authored-by: dguthruf <102602010+danielguthruf@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hisamotodev <hisamotodev@users.noreply.github.com>
* feat: update EmpirioLabs models in model_prices_and_context_window.json
* chore(endpoints): regenerate provider endpoint support for the EmpirioLabs cost map rows
* fix(empiriolabs): drop the retired max_tokens field, route embeddings and remove the unrouted rerank row
The cost map test rejects any row carrying max_tokens, so every EmpirioLabs row keeps max_input_tokens and max_output_tokens only. The provider JSON now lists /v1/embeddings so embedding(model="empiriolabs/text-embedding-v4") reaches the OpenAI-compatible embeddings route instead of failing provider lookup. The qwen3-rerank row goes away because no rerank route exists for the provider. Endpoint support regenerated from the cost map
* fix(empiriolabs): mark audio input and image embedding input on the rows the provider lists as multimodal
The three vision embedding rows get supports_embedding_image_input and the eleven chat rows whose input_modalities on GET https://api.empiriolabs.ai/v1/models include audio get supports_audio_input, so capability checks stop treating them as text only
* fix(empiriolabs): carry the long-context price tiers on the 24 rows the provider bills in bands
EmpirioLabs bills 24 of these models at a higher rate once the prompt passes 32K, 128K, 256K, 272K or 512K tokens, and the rows carried only the first band, so a 300K-token qwen3-7-plus request was priced at a third of the real cost. Each of those rows now carries input_cost_per_token_above_<N>k_tokens, output_cost_per_token_above_<N>k_tokens and, where the row already has cache_read_input_token_cost, cache_read_input_token_cost_above_<N>k_tokens for every band, taken from pricing[].min_context on GET https://api.empiriolabs.ai/v1/models for the base rows and from the Variants rate card on docs.empiriolabs.ai/models/<id> for the six variant rows
* feat(empiriolabs): bill the per-call web search fee reported in usage.tool_usage
EmpirioLabs charges a flat fee per invoked web_search, web_search_image or image_search call and reports the counts in usage.tool_usage on the chat response. The JSON provider config now carries that count as usage.server_tool_use.web_search_requests when providers.json sets special_handling.web_search_requests_from_tool_usage, the per-call web search cost path bills the usage-reported count instead of flooring to one call, and the 39 EmpirioLabs rows the provider lists a per-call search price for carry it as search_context_cost_per_query, from unit_pricing on GET /v1/models for base rows and the Variants rate card on docs.empiriolabs.ai for variant rows
* fix(openai_like): mark the dict signature of transform_parsed_response_dict as matching BaseConfig for LIT001
* fix(openai_like): mark both dict annotations of transform_parsed_response_dict for LIT001
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(openai): bill gov.api.openai.com FedRAMP endpoint with 1.1x uplift
* chore(openai): trim FedRAMP uplift diff
* refactor(openai): bill the FedRAMP endpoint through its own fedramp_uplift_multiplier key
* fix(ui): regenerate fedramp_uplift_multiplier doc comment in schema.d.ts
* style: ruff format utils.py and reorder fedramp_uplift_multiplier in schema.d.ts
* fix(openai): infer data_residency from OPENAI_BASE_URL and litellm.api_base when api_base is unset
* test(cost): cover get_fedramp_uplift_multiplier fallback paths
* fix(cost): apply fedramp_uplift_multiplier in batch_cost_calculator
* fix(openai): derive data_residency from a supplied client's base_url
* test: type the supplied-client data_residency test params
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): remove the 52 rows whose provider retired them before 2026
* test(utils): pin the proxy function-calling parity case to azure/gpt-4.1
* test(batches): price the Azure batch-logging regression case against gpt-4.1
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): record deprecation_source on every dated row that has a known provider notice
* fix(cost-map): keep deprecation_source only where the page lists the row's model and date
* fix(cost-map): drop deprecation_source from 4 rows whose own source page no longer lists the date
* fix(cost-map): give azure_ai/model_router the same deprecation_source as its hyphenated spelling
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks
* fix(rate-limit): count embedding media as media only on models that embed it
A data URI sent to a text-only embedding model is read as text, so it keeps
the text estimate there. On models flagged supports_embedding_image_input,
image and audio use the chat media estimates and PDF and video count at most
the model's max_input_tokens. The same estimate now feeds project and key
limits, the usage-based, latency and cost routers' deployment tpm, and the
deployment itpm check.
* fix(proxy): count embedding media only where the backend reads it as media
Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as
media per item. Cohere counts it as media only when the whole batch is
media, and Voyage, OCI and unknown models keep the text estimate. File
blocks carrying a data URI count like the bare URI.
The router context window pre-call check and the batch limiter now use the
same estimate, resolving each deployment or JSONL row model to its
backends. The cost map gains supports_embedding_image_input on
voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0.
* fix(router): hold each embedding media item to the context window on its own
A PDF or video data URI alone already reserves at most the model's
max_input_tokens, but the context window pre-call check then ran the
blanked input through the Responses transform and added a few framing
tokens, so a PDF that Vertex accepts got a 400. When every input item is
media, the check now returns the largest item's media estimate, since an
embedding request embeds each item separately
* fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope
The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped.
* fix(bedrock): bill Cohere embeddings by Bedrock's input token count header
* fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage
The strict base64 check now scans the payload in place instead of decoding
and re-encoding a full copy. Bedrock's billed token header is used only for
image requests, so text-only Bedrock Cohere usage matches main. The deployment
itpm estimator change is reverted, since the router never passes embedding
input to that check.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cost-map): record the provider's recommended replacement as successor_model on deprecating rows and report it from /model/deprecations
* fix(cost-map): pick the alias row's successor by its own shutdown date when the family has several
* chore(ui): regenerate schema.d.ts for the /model/deprecations successor_model wording
* fix(proxy): prefer a configured successor_model over the cost map's when deployments share a name, and keep the cost map's under a hand-set deprecation_date
* test(cost-map): allow successor_model in the cost map row schema
* feat(cost-map): infer successor_model as the next version in the same family for 136 dated rows with no published replacement
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): mark audio and video input support on the Gemini 3.5 to 3.8 Flash rows
* fix(model_prices): add Vertex AI and Together AI retirement dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct perplexity agent api pricing and add gemini-4-argon, nemotron-3.5-super
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): consolidate verified pricing and Aplomb metadata
Absorb verified pricing from #49547 and #49548 and Aplomb 1 metadata from #49539. Preserve provider tiers and remove an identical duplicate retirement key.
Co-authored-by: Liutasil501 <schmid.martin501@gmail.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(model_prices): sync generated EmpirioLabs endpoint metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): verify Nebius and EmpirioLabs model metadata
Verify canonical EmpirioLabs rows from #49695 against published model pages, preserving pricing tiers and excluding unverified variants and specialty billing. Add Nebius models and cached-input prices from the official catalog and correct the Voyage legacy input limit.
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): absorb Bedrock Opus 5 batch prices, Vertex Omni Flash limits and Gemini 2.5 shutdown dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): drop the EmpirioLabs rows and endpoint flags from this sync
* fix(model_prices): merge main and dedupe the gemini-2.5 deprecation_date keys
* fix(model_prices): set supports_video_input on the two new Nebius video models
* fix(model_prices): add Azure AI Claude Mythos 5.1 and match Foundry max effort support
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Mistral lowered Large 4 to $0.70 in, $0.07 cached in and $2.10 out per million tokens, so both rows in both cost map files move to that price and point their source at the model page that lists it
Breaking change: Large 4 calls are logged at about half the old cost, so budgets last longer and cost-based routing may favor it
* feat(decisions): add Parallel AI web-grounded decisions
* fix(decisions): name the Parallel model decision
* fix(parallel_ai): restrict environment credentials to trusted decision endpoints
* fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence
Only fields a provider names reach the OpenAI response, so existing providers keep
their base shape. A malformed api_base answers 400 on both credential paths. The
complexity router classifier and the Playground handle a provider that omits
confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types
* chore(cost-map): declare the Decisions routes on parallel_ai/decision
The provider endpoint files are generated from the cost map, so the row must
name both routes for the generated System One flag to hold
* fix(decisions): report no classifier confidence when a provider omits it
A choice answer with no confidence is a valid classifier answer, so the
complexity router picks its tier and leaves classifier_confidence and the
confidence signal out instead of reporting a different metric in its place.
Also show the Parallel AI logo in the provider picker and add a Parallel
translation fixture captured from the live API for both Decisions routes.
* fix(decisions): keep a token-priced row on token billing when it also sets a query price
A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR
* fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route
A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions.
* fix(decisions): keep per-query billing and optional confidence on Parallel only
A query price on any other Decisions provider moved that provider off token
billing, and a choice or score answer with no confidence became a 200 with a
null for every provider. Both now apply to Parallel alone, so other providers
bill and fail as they do on main.
A Parallel deployment with explicit zero token rates and a query price read as
free, so every budget check was skipped while it billed per call. It now counts
as priced.
The dashboard key for Parallel is now PARALLEL_AI so a served bundle that
predates the provider still sends parallel_ai as the provider slug, and the
README table lists the provider.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date
* fix(proxy): report a deployment's own provider retirement date before the bare model's
An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as
azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of
the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows
* fix(proxy): read Azure's retirement schedule for azure_ai deployments
An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date.
* fix(proxy): let custom and registered base_model rows keep their deprecation date
Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate
* fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's
* fix(proxy): recognize the router's copied cost-map rows by the loaded catalog
* fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool
Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking
sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model
map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The
GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows
* fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool
Native output without thinking now applies to every Bedrock Claude model that cannot
force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5
needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose.
The invoke chat path keeps the native output_format it built instead of inlining it, and
GovCloud region-path ids read their own false rows.
* fix(bedrock): clean caller output_format schemas before sending them natively on invoke
* fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke
* fix(bedrock): keep the faked Converse stream for native output that replaces the json tool
A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture.
* test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): set the regional DeepSeek V3.2 context to the model card 164K and date GovCloud West Llama 3
* fix(bedrock): reserve DeepSeek V3.2 output capacity and date GovCloud West Llama 3
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The /deepgram/v1/listen WebSocket refuses nova-3-medical with 1008 because the cost map has no
streaming row for it. Deepgram's pricing page does not list Nova-3 Medical, so the new row uses
the $0.0077/min streaming launch price from Deepgram's 2025-03-03 announcement
deepgram/streaming/nova-3 and nova-3-multilingual were $0.0048 and $0.0058 per minute against
$0.0077 and $0.0092 on deepgram.com/pricing as of 2026-11-02, so they are updated too
Resolves LIT-10393
