* feat(model_prices): add NVIDIA Nemotron 3.5 Lightning on OpenRouter and DeepInfra
Nemotron 3.5 Lightning shipped 2026-08-11 with public per-token pricing on
OpenRouter and DeepInfra at $0.05/M in and $0.20/M out. Without cost map
entries both ids raise "This model isn't mapped yet" and log at zero spend.
* fix(model_prices): stop asserting an output cap for Nemotron 3.5 Lightning
262144 is the native context window, not the output budget, and neither
OpenRouter nor DeepInfra publishes an output cap. Keeps max_input_tokens at the
native 256K window: 1M needs VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 on a self-hosted
deployment, so it is not what these hosted endpoints serve.
* chore(tests): drop the Nemotron 3.5 Lightning metadata test
Requested on the review thread: the cost map entries stand on their own.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
`supports_native_structured_output` was set only on the bare `deepseek.v3.2`
and `zai.glm-5` entries, so the cross-region inference profiles and the
region-pinned ids resolved to None. The flag gates the native
`outputConfig.textFormat` branch in BedrockConverseConfig, so callers
addressing the same model as `us.deepseek.v3.2` or
`bedrock/us-west-2/deepseek.v3.2` silently fell back to synthetic tool
injection. `us.` is the form Bedrock steers callers toward, so the most
common way to reach these models was the one missing the capability.
Adds the flag to the 12 affected ids and keeps the packaged backup in sync.
test_get_model_info_bedrock_models already caught the region-pinned ids, but
it filters on `litellm_provider == "bedrock"` and the cross-region profiles
carry `bedrock_converse`, so reverting just `us.deepseek.v3.2` and
`eu.deepseek.v3.2` left it green. The new parity test covers the prefixed
profiles and fails on exactly that mutation.
Record supports_tool_search on the Bedrock Claude entries in both cost
map files and have _supports_tool_search_on_bedrock read it first via
the provider-resolved capability lookup, keeping the name patterns as a
fallback for ARNs and ids the map cannot resolve. Threads the flag
through ModelInfoBase and drops a dated remark from the pattern list
Dated snapshots like o4-mini-2025-04-16 were missing the flex and priority cost keys their base alias carries, so service-tier requests against pinned snapshots were billed at standard rates. Sync the tier keys wherever the snapshot's anchor prices match the base alias, and add a drift regression test.
* fix(bedrock): stop forwarding no-op toolSpec.strict to Converse
`strict: false` is the Chat Completions default, so sending it to Bedrock
Converse communicates nothing the provider does not already assume, while
Bedrock rejects the key by presence rather than by value: any Claude model
routed through its Anthropic-compatible validator 400s with
`tools.0.custom.strict: Extra inputs are not permitted`.
The existing `bedrock_converse_supports_strict_tools` gate only protects
models whose `model_prices_and_context_window.json` entry carries the flag,
which makes every newly released Claude model broken by default until someone
adds it. That is a losing race for a field that carries no information when
false, and it is unrecoverable from the client side on `/v1/responses`, where
the Responses to Chat Completions bridge stamps `strict: false` onto every
function tool even when the caller never sent one. `drop_params` cannot help
there because the caller never supplied the param.
Drop the key when falsy instead. `strict: true` still honors the per-model
gate, so models that accept strict schemas keep the behavior they have today
and the flag keeps doing its job for the values that actually mean something.
* fix(bedrock): flag Claude Sonnet 5 as rejecting toolSpec.strict
Bedrock routes Sonnet 5 through the Anthropic-compatible validator that
rejects `toolSpec.strict`, but its six pricing-map entries never got
`bedrock_converse_supports_strict_tools: false`, so the gate fell back to
forwarding for Anthropic models and every tool call carrying `strict: true`
400'd. Verified live in us-east-1: before this, `strict: true` against
`us.anthropic.claude-sonnet-5` returns
`tools.0.custom.strict: Extra inputs are not permitted`; after, it returns a
real tool call.
Measured the rest of the family the same way rather than trusting the map:
Sonnet 4.5, Sonnet 4.6 and Haiku 4.5 all accept `strict: true`, and Opus 4.8
already carries the flag. Sonnet 5 was the only entry where the map disagreed
with the provider, so it is the only one changed here.
Same shape as the Opus 4.7/4.8 and Sonnet 4 fixes before it.
OpenAI cut Terra 20% and Luna 80% on 2026-07-30; openai and bedrock_mantle
entries already match. Azure global and us/eu data-zone terra/luna rows still
used the pre-cut rates, so spend tracking over-billed those Azure deployments.
Sol is unchanged. Cache-read, priority, and long-context fields scale with the
same multipliers already used for azure gpt-5.6.
OpenAI publishes a long-context column on the Flex tier, at half the standard
long-context rate. We had no field for it, so a >272k flex request fell through
to the standard long-context price and billed 2x: Terra $4/$18 instead of
$2/$9, Luna $0.40/$1.80 instead of $0.20/$0.90, Sol $10/$45 instead of $5/$22.50.
Adding the values to the cost map alone does nothing, because get_model_info
builds ModelInfoBase from an explicit kwargs list and silently drops any key
not named there. Declare the four *_above_272k_tokens_flex fields and wire them
through, then add the values for sol, terra, luna, and the gpt-5.6 alias.
That same gap was already swallowing cache_creation_input_token_cost_flex,
_priority, and _above_272k_tokens, which were present in the cost map but never
reached the calculator; they are wired through here too.
Fast mode (ex-Priority) publishes no long-context column, so nothing is added
there rather than deriving a rate by analogy.
AWS rolled out the 2026-07-30 GPT-5.6 price cut the same day, but the
bedrock_mantle entries still carried values derived from the pre-cut OpenAI
base, so Terra billed 1.25x and Luna 5x over the published rate.
Re-derive both from the AWS Bedrock pricing page, which prices in-region
inference at parity with OpenAI's data residency tier (1.1x base). Sol was
not cut and is unchanged.
Also drop tests/test_litellm/test_gpt_5_6_model_metadata.py; its Azure and
openai pricing assertions are covered by test_llm_cost_calc_utils.py.
gpt-5.4-mini and gpt-5.4-nano are 400K-context models (272K input,
128K output), but their cost map entries carried gpt-5.4's 1.05M window.
The router's pre-call context window check therefore admitted prompts far
past what the models accept, so oversized requests were dispatched to the
provider and failed there instead of being caught locally or routed
through context_window_fallbacks.
The azure_ai entries also inherited gpt-5.4's above-272K tiered pricing.
OpenAI applies that surcharge to the 1.05M-window models only, so those
keys are removed.
Limits per OpenAI's model reference and Azure AI Foundry's model table:
gpt-5.4-mini and gpt-5.4-nano are 400,000 context / 272,000 input /
128,000 output
Fireworks publishes a 262144-token context window for the Kimi K2.5, K2.6
and K2.7 models but caps generation well below that. Every fireworks_ai
Kimi K2.5/K2.6/K2.7 alias had max_output_tokens/max_tokens flattened to
262144 (equal to the context window), so the pre-call context-window check
admitted requests asking for a full 262144-token completion that Fireworks
rejects. Correct max_output_tokens/max_tokens to 32768 while keeping
max_input_tokens at 262144, and add a regression test pinning the limits
for all ten aliases.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dashscope): add qwen3.7-plus and qwen3.7-max to the model cost map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: limit backup cost map diff to the new dashscope entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): adjust tier-only alias assertion for mapped qwen3.7-plus
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(dashscope): drop redundant cost map pinning tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost_calculator): point tier-only alias check at an unmapped model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): point four suites at models the providers still serve
Four llm_translation tests failed against upstream because the model they name
no longer exists. Each replacement was verified against the live stage proxy.
deepseek/deepseek-reasoner is gone; the DeepSeek API now lists only
deepseek-v4-flash and deepseek-v4-pro. Use deepseek/deepseek-v4-pro, which
still returns message.reasoning_content by default and still drops it for both
reasoning_effort="none" and thinking={"type": "disabled"} (litellm maps the
former to the latter, so the provider rejecting a bare "none" does not matter).
amazon.titan-image-generator-v2:0 returns "This model version has reached the
end of its life"; amazon.nova-canvas-v1:0 is the text-to-image model Bedrock
still offers in us-east-1.
Bedrock's Rerank API requires a full model ARN and rejects a bare model id with
"The provided model ARN for reranking is invalid", regardless of model or
region. Pass the ARN for cohere.rerank-v3-5:0, which is available in the
stack's us-east-1.
vertex_ai/gemini-embedding-2 404s as an unknown publisher model on this
project; vertex_ai/text-embedding-005 returns a vector.
* test(e2e): skip the hosted_vllm chat test when its server is unset
test_hosted_vllm_chat_returns_content read os.environ["HOSTED_VLLM_API_BASE"]
directly, so a stack without that env var failed the test with a bare KeyError
instead of reporting an environment gap. The batches suite already skips on the
same variable, and the vertex passthrough tests use pytest.skip for the same
reason, so follow that idiom here.
Drop the HOSTED_VLLM_API_KEY plumbing: the stage vLLM stand-in serves
/v1/chat/completions unauthenticated, and api_key is optional on
LiteLLMParamsBody, so passing it added nothing.
Default the backend to the model that server actually serves,
Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M, rather than a Llama id it never had.
Verified against the live stage proxy: a deployment with just that model and
api_base returns "hello".
* fix(model_map): mark deepseek v4-pro and v4-flash as reasoning-capable
Review on #34567 flagged that deepseek/deepseek-v4-pro is not marked
reasoning-capable while the e2e control case requires reasoning_content back
from it. The behavior premise is inverted, but it surfaced a real data gap: the
model map never gained supports_reasoning for the v4 models when DeepSeek
retired deepseek-reasoner, which did carry the flag.
Both models do reason. Against the live API with no reasoning params, v4-pro
returns 106 chars of reasoning_content and v4-flash returns 54, and both drop
it for thinking={"type":"disabled"}.
The stale flag had a real consequence beyond metadata: DeepSeekChatConfig
._thinking_mode_active() gates on supports_reasoning(), so with the flag unset
it returned False even when a caller passed thinking={"type": "enabled"},
skipping the multi-turn check that reasoning_content be passed back on
assistant messages. Param support itself was never gated, which is why
reasoning_effort="none" still mapped to thinking disabled.
Verified with LITELLM_LOCAL_MODEL_COST_MAP=True: supports_reasoning now
reports True for deepseek/deepseek-v4-pro and deepseek/deepseek-v4-flash.
tencent/deepseek-v4-pro is left alone; that route was not exercised here.
Registers claude-opus-5 across the cost maps and provider lists so the model
prices, reports its real 1M/128K limits, and advertises its capabilities instead
of falling through the generalization patterns at zero cost.
Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp),
Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok
with the usual 1.1x regional premium on the cross-region inference profiles, and
fast mode is priced at 2x through provider_specific_entry on the first-party
entry only.
Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to
512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts
output_config.effort="max" for Opus 5.
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
glm-5p2 (and its fireworks_ai/glm-5p2 alias) carried cache_read_input_token_cost
of 2.6e-07, the GLM 5.1 rate; the entry was seeded from the wrong row. Fireworks'
standard serverless rate for GLM 5.2 is $0.14/1M = 1.4e-07, so every prompt-cache
hit was billed at nearly double the real rate.
Corrects the value in both the canonical map and the bundled backup. The existing
fireworks cost-calculator test now reads the cached rate from the map instead of
hardcoding it, so it tracks the shipped value.
The gpt-realtime family (OpenAI and Azure) only serves /v1/realtime and is rejected by /v1/chat/completions with "This is not a chat model", but the cost map tagged them mode=chat. Retag them mode=realtime (a value already used by gemini-live and handled by the health-check realtime handler) and add realtime to the ModelInfoBase mode literal.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of
tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is
per-model and ranges from 512 to 4096, and it can differ per platform for the same
model, so one constant is wrong in both directions
is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what
optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is
cacheable, async_filter_deployments pins routing to whichever deployment previously
served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5
or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider
never cached it, so the pin cost load balancing for nothing. In the other direction
Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it
had earned
The minimum now resolves from prompt_cache_min_tokens in the model cost map, which
keeps it current with new models and lets the Bedrock override for Fable 5 fall out
of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT
stays as a global escape hatch when explicitly set, and as the fallback for models the
cost map has no entry for
async_filter_deployments only ever receives the model group alias, never a model name,
so it resolves the threshold from healthy_deployments instead. A group may mix models
with different minimums, so it takes the max: a prompt is only treated as cacheable when
it clears every member's minimum, because an unnecessary pin is the defect being fixed
while a missed pin only forfeits an optimization
Gemini context caching shares this gate and has the same defect; its entries are left
unset so they keep today's behavior, tracked separately in LIT-4525
