Restate xAI's usage.cost_in_usd_ticks as usage.cost on chat and responses
replies, streamed ones included, then let the cost calculator own the
figure: a deployment with its own input_cost_per_token and
output_cost_per_token keeps that price, cost margins apply on chat streams
as they already did on non-streamed calls, and only OpenRouter's usage
cost becomes the llm_provider-x-litellm-response-cost header, so xAI
streams no longer skip the calculator through the header or the
stream_chunk_builder hidden response_cost.
Streamed responses through the proxy previously exposed no usable cost:
the x-litellm-response-cost header is unreadable mid-stream and the final
usage chunk carried only tokens, priced against an alias model name the
client cannot resolve. The include_cost_in_streaming_usage flag existed
but was off by default and only fixed the wire, not SDK clients.
Stamp usage.cost into the joined streaming response by default wherever a
final usage object is built: the chat-completions stream_chunk_builder,
the native /v1/responses RESPONSE_COMPLETED event, and synthetic response
events. Provider-reported cost always wins over the computed value, and
only positive computed costs are stamped so unpriceable alias responses
keep deferring to the logging object's own calculation. Per-chunk SSE
cost injection (/v1/messages, generateContent, passthrough) stays behind
the flag.
Also normalize non-litellm usage objects in stream_chunk_builder: openai
CompletionUsage lacks Usage.__contains__, so membership probes silently
returned False and client-side rebuilds dropped the wire cost and
recounted token usage locally. Wire token counts and cost now survive.
Resolves LIT-6427
Router._add_deployment called get_llm_provider without the deployment's api_base, so a config entry with a bare model plus a known OpenAI-compatible endpoint failed startup validation with LLM Provider NOT provided and the proxy returned 400 no healthy deployments for that model group. acompletion had the same gap at request time: it forwarded only base_url into its get_llm_provider call, dropping the api_base kwarg the router passes. Both now forward api_base so endpoint matching resolves the provider the same way sync completion already does
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
* test(cost-calc): stop 182 global writes leaking out of the cost-calc suites
Across test_cost_calculator.py and llm_cost_calc/test_llm_cost_calc_utils.py,
58 tests opened by setting LITELLM_LOCAL_MODEL_COST_MAP in os.environ and
replacing litellm.model_cost, and none of them put the env var back. The
second file already had a _local_model_cost_map fixture doing it by hand with
a try/finally, so both idioms sat in the same file.
Keep that fixture, give it monkeypatch, and have every one of those tests ask
for it. The margin and discount tests drop their hand-rolled
copy-then-restore in favour of monkeypatch.setattr, which also puts the
global back when an assertion fails part way through.
Both files also drop a sys.path.insert whose argument resolves outside the
repo, so it was never what made the imports work.
TQ003 1077 -> 1075, TQ004 768 -> 693, TQ005 2836 -> 2731, and the budget
ceilings come down with them.
* fix(test): make the streamed-cost tests load the map they assert against
The local_cost_map fixture set LITELLM_LOCAL_MODEL_COST_MAP but never reloaded
litellm.model_cost, and reading the variable is not what loads the map. So the
three streaming-cost tests billed against whatever map the process happened to
be holding, and their hardcoded prices only held when something else had
already swapped in the checked-in one. This branch stops the cost-calc tests
leaking that map, which left test_main billing at the ambient prices instead.
The fixture now loads the map it names, so the prices these tests assert hold
on their own.
Rebuilding a streamed response and pricing it is the path a spend row comes
from, and nothing asserted it end to end. Reversing either half of the usage
the provider reported left the file green.
Three cases: the rebuilt response bills the usage the last chunk carried,
streaming and not streaming bill the same usage the same, and a stream that
reported no usage is still billed rather than dropped.
The cost is asserted against the catalog prices the run itself reads, with a
non-zero guard in front of it so an all-zeros lookup cannot satisfy it
vacuously. Pinning the dollar figure as a literal would have made a routine
gpt-4o price update fail a test about usage reconstruction.
completion() aliased the caller's header mapping instead of copying it,
then merged the provider-scoped headers into that same object. The router
shares one header dict across fallback attempts, so the credential written
on an Anthropic attempt was still present when a later Bedrock or Vertex
attempt read the dict, defeating the provider scoping.
Copy the mapping before merging so each attempt sees only its own headers.
A proxy on the default remote cost map never produced a prompt cache
breakpoint: the published map has the gpt-5.6 entries without
supports_prompt_cache_breakpoint, so the model-map gate returned False
for every listed model and only LITELLM_LOCAL_MODEL_COST_MAP=True (the
repo .env, hence the passing unit tests) made the feature work. The hook
now honors the flag when the entry carries one, True or False, and
otherwise applies the GPT-5.6+ version rule to the model name, so a map
that lags the flag still gets the OpenAI dialect. The model-map tests
pin litellm.model_cost to the bundled backup map and a new test drives
the hook against an unflagged gpt-5.6 entry.
completion() and acompletion() take base_url as an alias for api_base
that only lands on api_base after the cache control hook ran, so a
GPT-5.6 call at a non-OpenAI gateway given through base_url still got
the dialect. Both seed calls and the unstamped request-params read now
look at base_url too.
ResponsesAPIRequestUtils.merge_prompt_management_input reshaped hook
output in place, retyping text parts to input_text on the caller's own
message objects. The merge now shapes a copy of each message as it
emits it, so the identity-based merge keeps working on the hook's
objects and nothing the hook or the client owns is mutated.
The cache control hook also runs on litellm.responses() input. On a
GPT-5.6 deployment it wrapped a string-content item into a chat-shaped
{"type": "text"} part, which the Responses API rejects, and it never
marked input_text, input_image or input_file parts, so no breakpoint and
no prompt_cache_options reached the provider. Add the Responses part
types to the eligible block set and translate chat-shaped text parts on
non-assistant items to input_text in
ResponsesAPIRequestUtils.merge_prompt_management_input, which both the
async and the sync prompt management sites go through.
The dialect also fired for any GPT-5.6 name that resolved to provider
openai, including deployments pointed at a custom api_base that does not
understand prompt_cache_breakpoint. Decide it once per request from the
provider, the model map and the resolved api_base (request, then
litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE): only
api.openai.com and *.api.openai.com hosts speak the dialect, a top-level
prompt_cache_options opts a custom target in, and litellm_proxy/ targets
never get it. maybe_seed_default_injection_points takes api_base and
stamps the finished decision on the points as _litellm_openai_dialect so
the sync completion() path, whose hook params do not carry api_base,
honors it; maybe_inject_cache_control takes api_base from the
/v1/messages handler.
Eligibility now comes from a supports_prompt_cache_breakpoint model map
flag on the OpenAI gpt-5.6 entries, exposed through
litellm.utils.supports_prompt_cache_breakpoint, with the GPT version rule
kept only for models the map does not know. The OpenAI dialect no longer
reserves a slot for tool_config points, which OpenAI has no cache block
for, and with_prompt_cache_breakpoint plus the chat bridge helper return
a new block instead of mutating their input.
* fix(main): an explicit provider outranks a known OpenAI model name
completion() picks the OpenAI handler whenever `model in
litellm.open_ai_chat_completion_models`, and that clause is evaluated before the
gemini and vertex_ai branches. get_llm_provider() already resolves those names
to "openai", so the clause only adds anything when the provider is something
else, and then it silently overrides it: the config built for the requested
provider is handed to the OpenAI handler.
For gemini that is fatal. VertexGeminiConfig.transform_request raises
NotImplementedError by design, since Vertex builds its request in its own
handler, so `gemini/gpt-4o` dies in async_transform_request before anything is
sent. register_model() reaches the same state without an odd model id: an entry
claiming litellm_provider "openai" adds its name to
open_ai_chat_completion_models, so one mislabelled pricing entry reroutes every
later call to that model in the process.
The name clause now applies only when no other provider was resolved.
* test(main): move the routing regression into the mapped test file
CLAUDE.md asks bug fixes to extend the mapped test file, so these belong in
tests/test_litellm/test_main.py rather than a module of their own.
They also no longer swap out the provider handler objects. Both Gemini cases
inject an HTTPHandler whose post() answers like generativelanguage does, then
assert the URL the request went to and read the reply back; the OpenAI case
injects an OpenAI client and patches its own raw-response create. That asserts
the endpoint the call reaches instead of which attribute the test replaced, and
matches the neighbouring tests in the file.
The gpt-5.4+ responses-bridge gate classified the endpoint from the call-level
api_base alone, while the OpenAI chat handler resolves arg > global > env >
default. A custom base configured via litellm.api_base or OPENAI_BASE_URL/
OPENAI_API_BASE was therefore invisible to the gate: it read blank as the
default OpenAI endpoint and bridged a request the custom backend has no
/responses route for.
Extract that resolution into one _resolve_openai_api_base() and have both the
gate and _complete_custom_openai() call it, so the gate can never classify an
endpoint the request won't hit. The gate compares the resolved base against the
default (import litellm seeds OPENAI_BASE_URL to the default, so "override is
non-None" is not a safe custom-endpoint signal); whitespace collapses to the
default as before. reasoning_effort="none" remains the escape hatch.
A blank api_base (empty or whitespace) resolves to the default OpenAI
base downstream but is not None, so the constraint-enforcing-endpoint
check misclassified it as a custom backend and skipped the unset-effort
auto-bridge, leaving gpt-5.4+ function-tool requests to 400 at OpenAI.
The check now treats None, empty, and whitespace api_base alike; a real
custom base still opts out. Verified with get_llm_provider, which passes
a blank api_base through while resolving the provider to openai
Chat-only OpenAI-compatible backends registered under the openai
provider with custom api_base and gpt-5.4+ model names served
tools-without-reasoning fine and have no /responses route, so the
unset-effort arm added for real OpenAI would have silently rerouted
previously working deployments. The arm now fires only when api_base is
unset (default OpenAI endpoint) or the provider is azure; an explicit
reasoning_effort keeps its pre-existing bridging behavior on any
api_base. Flagged lines also modernized to PEP 604
The bridge gate compared reasoning_effort against the string "none", so
litellm's dict form ({"effort": "none"}) wrongly bridged; the gate now
reads the effort value from either form and treats a summary inside the
dict as Responses-only regardless of effort. Helicone and lunary
previously skipped custom tool calls entirely; both now serialize them
(helicone as a tool_use block from the custom payload, lunary with the
custom name and input in its function fields, keeping type custom), with
new mapped tests for both integrations
OpenAI's chat completions rejection applies to function tools only;
custom (grammar) tools are served natively with reasoning on, live-proven
by a 200 on a custom-only gpt-5.6 chat request. Gating on any truthy
tools needlessly bridged custom-only requests, and the bridge maps custom
tool calls back function-shaped, so the native chat custom tool_call
surface added earlier in this PR was bypassed exactly where chat serves
it natively. The gate now checks for a function-type tool in either the
nested chat or flat Responses def shape; the same coarseness existed on
the explicit-effort arm before this PR and is fixed by the shared leg
OpenAI enables reasoning by default for gpt-5.4+ (unset reasoning_effort
means medium server-side) and Chat Completions rejects function tools
whenever reasoning is on, so a tools request without an explicit
reasoning_effort 400d instead of auto-bridging to the Responses API; the
bridge heuristic now treats unset effort as reasoning-active and honors
the documented escape hatch by keeping explicit "none" on chat
completions. The cursor input arm also gains the mirror of the
messages-arm normalization: chat-nested tool envelopes, grammar formats,
and object tool_choice flatten to the Responses dialect before dispatch
Chat-completions requests to responses-only Bedrock Mantle models are
bridged to the Responses API, but completion() forwarded only
aws_bedrock_project_id into get_litellm_params, so aws_role_name,
aws_web_identity_token, aws_session_name and the other SigV4 credential
kwargs never reached sign_request and botocore fell back to the default
credential chain ("Bedrock Mantle auth failed: no Bearer token and no
usable AWS credentials"). Forward the whole AWS credential kwarg family,
extracted from the OPTIONAL_KWARGS_KEYS set get_litellm_params already
supports.
* fix: zero out crash-class basedpyright rules across litellm/
* feat(lint): add LIT009 banning inert type: ignore comments
* docs: require bracketed rule and reason on every suppression
* chore(lint): ratchet budgets down and zero crash-class pyright limits
* fix: narrow auto router routelayer through a local before calling
* test: add regression tests for crash-class fixes
* fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
* fix(azure): apply api_version fallback chain to image edit URL
`AzureImageEditConfig.get_complete_url` only read `api_version` from
`litellm_params`. When callers configured it via `litellm.api_version`
or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and
Azure responded `404 Resource not found`.
Apply the same fallback chain the Azure chat path already uses in
`common_utils.py`:
litellm_params > litellm.api_version > AZURE_API_VERSION env >
litellm.AZURE_DEFAULT_API_VERSION
Adds 5 unit tests pinning each layer of the chain plus a regression
guard for `api_base` that already carries `?api-version=`.
* feat(mcp): core sampling and elicitation flow with security hardening
- Add sampling_handler.py: full MCP sampling/createMessage flow with
model selection (hint-based + priority-based), auth enforcement,
budget checks, route restriction gates, and tag policy pre-auth
- Add elicitation_handler.py: MCP elicitation/create relay with
downstream client capability detection
- Wire sampling/elicitation callbacks in mcp_server_manager.py
gated behind allow_sampling/allow_elicitation config flags
- Add allow_sampling/allow_elicitation fields to MCPServer type
- Fix session lock deadlock: skip lock for JSON-RPC response POSTs
(elicitation/sampling replies) with truncated-body heuristic
- Extend client.py with sampling_callback and elicitation_callback
- Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for
spoofing fix, Latin-1 header encoding guard
- Add 4 new test modules (model access, priority selection, request
builder, tool conversion) + update existing MCP tests
* fix(security): run pre-call guardrails before MCP sampling acompletion
Without this, an upstream MCP server with allow_sampling enabled could
send prompts that bypass every guardrail (content filtering, PII
redaction, prompt-injection detection) configured on /chat/completions.
- Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before
llm_router.acompletion so guardrails fire for sampling sub-calls
- Add HTTPException to the re-raise list so guardrail rejections
propagate correctly instead of being swallowed as generic errors
* feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490)
* feat(bedrock_mantle): add Responses API transformation config
* test(bedrock_mantle): cover trailing-slash api_base normalization
* feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig
* feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged)
* feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries
* refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing
Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses;
gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding
gpt-oss (which keeps its chat-completions emulation) and defaulting everything else
to the native Responses config, so future frontier models (gpt-6, etc.) route
correctly without a code change. Verified against the live us-east-2 Mantle endpoint:
gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths.
* test(bedrock_mantle): cover supports_native_websocket opt-out
Closes the one uncovered line flagged by codecov on the Responses config.
The assertion documents that Mantle Responses has no realtime/websocket
transport, so realtime routing must not attempt a socket it cannot serve.
* fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle
BedrockMantleResponsesAPIConfig inherited supports_native_file_search()
-> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no
OpenAI vector stores, so a forwarded file_search tool is rejected with a
400 (verified upstream: Tool type 'file_search' is not supported). Opting
out, like the existing supports_native_websocket override, routes the tool
through LiteLLM's file_search emulation instead.
* fix(bedrock_mantle): only route openai.gpt frontier models to Responses
The previous gate excluded gpt-oss and routed every other model to the
native Responses config. But on Mantle only the OpenAI gpt frontier models
(gpt-5.x) are served on /openai/v1/responses; gpt-oss and the non-OpenAI
families (nvidia, mistral, google, zai, ...) are chat-completions only and
400 on that path. Allow-list the openai.gpt- family (excluding gpt-oss)
instead, so chat-only models fall through to the chat-completions emulation.
Verified against the live us-east-2 endpoint: nvidia.nemotron-nano-9b-v2
returns 400 on /openai/v1/responses and 200 on /v1/chat/completions.
* feat(custom_llm): allow streaming/astreaming to yield ModelResponseStream (#27580)
* fix(custom_llm): allow streaming/astreaming to yield ModelResponseStream directly
* fix(streaming): enhance ModelResponseStream handling for custom LLM providers
* fix(streaming): strip finish_reason from content chunks and ensure tool_calls are preserved
* fix(streaming): add type ignore for finish_reason assignment in CustomStreamWrapper
* fix(proxy): strip stack trace from HTTP 503 responses (CWE-209) (#28330)
* fix(proxy/cwe-209): strip Python traceback from HTTP 503 error responses
The /cache/ping endpoint included a full Python traceback in its 503 error
response body (inside the ProxyException message), leaking internal file
paths, line numbers, and call stacks to any caller. Two MCP route handlers
in proxy_server.py similarly interpolated str(e) into "Internal server
error" detail strings.
Fix: log the traceback server-side via verbose_proxy_logger.exception()
and omit it from the ProxyException payload / HTTPException detail returned
to clients. Tests updated to assert no "traceback" keyword or frame paths
appear in the 503 body, with a new dedicated regression test.
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy/cwe-209): apply Greptile P2 fixes and add MCP exception-path tests
Greptile 4/5 review identified two remaining gaps and Codecov reported
0% coverage on the two MCP handler exception branches:
1. caching_routes.py — str(e) in "Service Unhealthy ({str(e)})" could
still leak Redis hostnames/IPs; replaced with static "Service Unhealthy".
HTTPException is now re-raised before the generic handler so the
"cache not initialized" 503 still reaches callers with its detail.
Removed the redundant str(e) arg from verbose_proxy_logger.exception()
(exception() already appends the traceback automatically).
2. tests — two new unit tests cover the exception paths in
dynamic_mcp_route and toolset_mcp_route that were previously at 0%:
- test_dynamic_mcp_route_unexpected_exception_returns_500_without_traceback
- test_toolset_mcp_route_unexpected_exception_returns_500_without_traceback
All 25 tests pass (9 caching + 16 MCP).
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion in test_cache_ping_no_cache_initialized
The assertion was weakened to `"Cache not initialized" in str(data)`, which
matches the raw string of the entire response dict and would pass even if the
error moved to an unexpected field or changed structure.
Restore a targeted check on the parsed response: assert the exact string in
the correct field `data["detail"]`, matching FastAPI's HTTPException
serialisation format {"detail": "<message>"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion and add CWE-209 no-cache path test
The assertion in test_cache_ping_no_cache_initialized was weakened to
`"Cache not initialized" in str(data)`, which matched against the raw string
representation of the entire response dict. This would pass silently even if
the error message moved to an unexpected field or the structure changed.
Restore a targeted assertion on the parsed field:
assert data["detail"] == "Cache not initialized. litellm.cache is None"
matching FastAPI's HTTPException serialisation format exactly.
Add test_cache_ping_no_cache_does_not_expose_internals to show the code path
is still working correctly after the CWE-209 fix: verifies that the HTTPException
is re-raised as-is (no traceback, no source paths), and asserts the complete
response structure is exactly {"detail": "Cache not initialized. litellm.cache is None"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(caching_routes): restore ProxyException envelope for null-cache 503
The except HTTPException: raise guard (added in the CWE-209 fix) caused
the null-cache HTTPException to escape as FastAPI's {"detail": "..."} shape
instead of the {"error": {...}} ProxyException envelope that callers expect.
Move the null-cache guard before the try block and raise ProxyException
directly so the response structure is consistent with all other /cache/ping
503s, and the except HTTPException: raise guard is only reachable by
unexpected downstream HTTPExceptions.
Update the two no-cache tests to assert the correct ProxyException envelope.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update utils.py (#26609)
* feat(pricing): add Snowflake Cortex REST API model pricing (#26612)
* feat(pricing): add Snowflake Cortex REST API model pricing
## Summary
Adds pricing and context window information for 20+ Snowflake Cortex REST API models to `model_prices_and_context_window.json`.
## What's included
- **7 Claude models** (sonnet-4-5, sonnet-4-6, 4-sonnet, 4-opus, haiku-4-5, 3-7-sonnet, 3-5-sonnet) — with prompt caching rates
- **4 OpenAI models** (gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano) — with prompt caching rates
- **5 Llama models** (3.1-8b, 3.1-70b, 3.1-405b, 3.3-70b, 4-maverick)
- **1 DeepSeek model** (deepseek-r1)
- **1 Mistral model** (mistral-large2)
- **1 Snowflake model** (snowflake-llama-3.3-70b)
- **2 Embedding models** (arctic-embed-l-v2.0, arctic-embed-m-v2.0)
Each entry includes `input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost` (where applicable), `max_input_tokens`, `max_output_tokens`, and capability flags (`supports_function_calling`, `supports_vision`, `supports_prompt_caching`, `supports_reasoning`).
## Pricing source
All prices are in USD per token, sourced from the official [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) — Tables 6(b) (REST API with Prompt Caching) and 6(c) (REST API).
## Context
The existing `snowflake/` provider has zero model entries in the pricing JSON, which means LiteLLM cannot track costs for Snowflake Cortex calls. This PR fills that gap.
## Related
- Existing provider: `litellm/llms/snowflake/`
- Cortex REST API docs: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-rest-api
* Update model_prices_and_context_window.json
Fix the JSON parsing error
* Update model_prices_and_context_window.json
Removed the duplicate entry
* fix(utils): copy extra_body before adding unknown params to prevent model config mutation (#29620)
Fixes#29615. In add_provider_specific_params_to_optional_params, the line:
extra_body = passed_params.pop("extra_body", None) or {}
returns the original dict reference when extra_body is non-empty (truthy).
Subsequent writes like extra_body[k] = passed_params[k] then mutate the
shared model config object held by the router, poisoning /model/info and
all subsequent requests for that deployment.
The or {} short-circuit creates a new dict only when extra_body is falsy
(None or {}), which is why the bug does not reproduce with extra_body: {}.
Fix: wrap in dict() so we always work on a fresh shallow copy.
* fix(vertex_ai): Bake tool_choice into Gemini CachedContent body to prevent silent drop (#29097)
* fix(vertex_ai): bake tool_choice into Gemini CachedContent body to prevent silent drop
* address greptile feedback on tool_choice cache test
* adds test that uses ToolConfig(functionCallingConfig=FunctionCallingConfig(mode=ANY)) instead of a dict literal, mirroring what map_tool_choice_values actually produce
* fix(gemini/veo): move image from parameters into instances[0] (#29501)
* fix(gemini/veo): move image from parameters into instances[0]
Veo's predictLongRunning schema puts image (and prompt) on the
instances element; parameters is for aspectRatio/durationSeconds/etc.
The Gemini path was leaving image in params_copy, so it ended up
nested under parameters and the API silently ignored it.
The Vertex path already builds the instance dict explicitly, so this
just aligns the Gemini path with it.
Fixes#29498
* address greptile: unconditional pop + BytesIO test
- Pop `image` from params_copy unconditionally so it never reaches
GeminiVideoGenerationParameters even when None, removing implicit
reliance on Pydantic's extra-field-ignore.
- Add test_transform_video_create_request_image_filelike_goes_to_instance
covering the BytesIO path (_convert_image_to_gemini_format) — round-trips
the base64 to confirm encoding.
- Add test_transform_video_create_request_image_none_is_dropped covering
the new None branch.
* fix(huggingface): handle special token text in embedding usage (#29660)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params (#29655)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params
ToolPermissionGuardrail builds self.rules and the compiled target/pattern
maps only in __init__. The base update_in_memory_litellm_params re-sets raw
attributes via setattr but never rebuilds those maps, so a guardrail updated
in place (PUT /guardrails, or the immediate in-memory sync) keeps enforcing
the construction-time rules until it is reinitialized (PATCH path, periodic
DB poll, or restart).
Extract the compile step into _load_rules and override
update_in_memory_litellm_params to rebuild from it (dict- and model-safe),
re-normalizing default_action / on_disallowed_action. Mirrors the existing
PresidioGuardrail override of the same method. Adds regression tests.
Fixes#29592.
* fix(guardrails): handle dict params in ToolPermissionGuardrail in-memory update
Delegate to super() only for LitellmParams input (the base setattr loop is
model-only); apply the raw-dict case inline. Fixes the mypy arg-type error
and makes the recompile work when the proxy passes the raw DB dict.
* fix(guardrails): preserve tool-permission rules on a partial in-memory update
A partial update (e.g. a LitellmParams whose rules field is None) ran through
the generic setattr, which set self.rules to None, and the recompile was
skipped, leaving the guardrail with no rules. Snapshot the previous rules and
restore them when the update carries no rules; an explicit empty list still
clears them. Adds a regression test for the rules-absent case.
Addresses the Greptile review note on #29655.
* fix(bedrock): stop base_model label from stripping tools/tool_choice (#29621)
* fix(bedrock): stop base_model label from stripping tools/tool_choice
A Router/proxy Bedrock deployment whose model_info.base_model is a friendly
label (e.g. claude-haiku-4-5) silently lost tools/tool_choice: the outgoing
Converse request was built without toolConfig, so the model behaved as if no
tools were provided. Worked in v1.84.0, regressed in v1.85.0, and with
drop_params=true it failed silently.
Two changes compound into the bug. completion() passed model_info.base_model
as the model argument to get_optional_params, so the real Bedrock model id
never reached supported-param resolution; and get_supported_openai_params
resolved the provider config's params from base_model or model, letting the
label fully replace the real model. For Bedrock the label resolves to no tool
support, so tools/tool_choice were dropped before transformation.
completion() now keeps model as the real deployment model and threads the
resolved base_model (kwarg or model_info) through separately, and
get_supported_openai_params treats base_model as additive: it returns the
union of the params supported by model and by base_model. A hint can only add
capabilities, never strip ones the real model already exposes, which also
preserves the original base_model behavior from #27717 and Azure's base_model
driven model-type detection.
Fixes#29618
* test(main): make base_model param test robust to new parametrize cases
Restore an explicit per-case expected_model_param literal instead of
hardcoding the gemini id, so a future case with a different model can't
produce a misleading assertion failure.
* fix(fireworks_ai): pass response_format json_schema through unchanged (#29606)
FireworksAIConfig.map_openai_params was rewriting the OpenAI strict
`{type: json_schema, json_schema: {name, strict, schema}}` shape into
`{type: json_object, schema: ...}` before sending to Fireworks, dropping
`strict` and `name` and changing the `type`. Per Fireworks' docs json_object
means "force any valid JSON output (no specific schema)", so the schema
constraint was effectively dropped and grammar-guided decoding never ran;
model output silently violated the schema.
The rewrite landed in #7085 (Dec 2024) when Fireworks did not yet accept
native json_schema. Fireworks accepts the OpenAI strict shape natively now,
so the rewrite has become a regression.
Removes the rewrite. Passes response_format through unchanged. Updates the
existing test_map_response_format to assert pass-through. Adds focused
regression tests in tests/test_litellm/ covering preservation of type,
strict, name, and schema body, plus that json_object alone still works.
* fix(types): import Required from typing_extensions in gemini types
* style: reformat sampling_handler.py for py312 black compat
* refactor(mcp-sampling): extract helpers to fix PLR0915 too-many-statements in handle_sampling_create_message
* fix(proxy-server): add explicit ProxyLogging type annotation to proxy_logging_obj to fix mypy inference
* fix(mcp-sampling): suppress mypy assignment error on ImportError fallback for proxy_logging_obj
* fix(test): use .value when comparing LlmProviders enum against string in test_default_api_base
* fix(test): iterate LlmProviders enum in test_default_api_base to avoid str pollution from custom provider registration
litellm.provider_list is a mutable global initialized to list(LlmProviders) but custom_llm_setup() appends plain provider strings to it. When a test_custom_llm.py test runs first in the same xdist worker, provider_list contains a str and calling .value on it raises AttributeError. Iterate the immutable LlmProviders enum instead, which is deterministic and what the check intends.
* fix(mcp): depth-aware JSON-RPC response detection and neutral speed-priority fallback
Replace the flat substring check in the truncated-body routing path with a
top-level-key scan so a JSON-RPC response whose result payload nests a
"method" field is still detected as a response and skips the session lock,
removing a deadlock against the in-flight tool call awaiting it.
Drop the inverse max_output_tokens speed proxy when no model exposes
output_tokens_per_second; context-window size does not track latency, so a
neutral score avoids biasing speedPriority toward the smallest-context model.
* fix(guardrails): make ToolPermission rule reload atomic on invalid regex
_load_rules appended each rule to self.rules before compiling its regex, so an
invalid pattern raised mid-loop after the bad rule was already live but without
a _compiled_rule_targets entry. _matches_regex reads a missing compiled target
as a None pattern and returns True, turning the bad rule into a match-all that
silently applies its decision to every tool. Via update_in_memory_litellm_params
(PUT /guardrails) this corrupted the live guardrail.
Build the parsed rules and compiled maps into locals and swap them in only after
every regex compiles, and restore the previous ruleset if a live update is
rejected, so an invalid regex now fails the update without leaving the guardrail
enforcing a broken policy.
* test(mcp): cover sampling conversion, model resolution, and elicitation relay paths
The MCP sampling and elicitation handlers shipped with partial test
coverage, leaving the response-to-MCP conversion, the model resolution
fallback chain, completion-kwargs assembly, guardrail routing, and the
entire elicitation relay untested. That pulled the PR's diff (patch)
coverage below the codecov threshold even though overall project
coverage rose.
Add focused unit tests for _convert_openai_response_to_mcp_result,
_convert_mcp_tools_to_openai, _convert_mcp_tool_choice_to_openai, image
and audio content conversion, the hint-matching and fallback branches of
_resolve_model_from_preferences, _build_completion_kwargs, the router and
guardrail-rejection paths of _run_guardrails_and_call_llm, the
handle_sampling_create_message success and error-propagation flows, the
marker-hoisting fallback for tool content on unexpected roles, and the
elicitation form/url/generic relay together with its decline paths
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: lengkejun <lengkejun@xd.com>
Co-authored-by: Yug <yugborana000@gmail.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: tanmay958 <53569547+tanmay958@users.noreply.github.com>
Co-authored-by: DrishnaTrivedi <142084770+DrishnaTrivedi@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Navnit Shukla <Navnit.shukla25@gmail.com>
Co-authored-by: PRABHU KIRAN VANDRANKI <72809214+VANDRANKI@users.noreply.github.com>
Co-authored-by: Adrian Lopez <109683617+adriangomez24@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com>
Co-authored-by: Dinesh Girbide <85330597+Dinesh-Girbide@users.noreply.github.com>
Co-authored-by: cloudwiz <22098246+andrey-dubnik@users.noreply.github.com>
Co-authored-by: Ahmad Khan <ahmadkhan2508@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
- Extend responses_api_bridge_check when reasoning_effort + summary aliases
(including nested extra_body) without tools
- Merge summary into reasoning_effort for responses bridge; helpers in utils
- Strip summary aliases in GPT-5 chat mapping when not bridged
- Tests for bridge + merge behavior
Co-authored-by: Cursor <cursoragent@cursor.com>
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds `litellm.route_all_chat_openai_to_responses` (env: `LITELLM_ROUTE_ALL_CHAT_OPENAI_TO_RESPONSES`) to route all OpenAI /chat/completions requests through the Responses API bridge. Also fixes reasoning param dict passthrough in completion transformation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* update bedrock models in tests
* updated more tests and model_prices_and_context_window
* fix model id and pricing
* replace more sonnet models
* update tests
* git push
* update pricing
* flaky total cost
* monkey patch
* relax the cost change
* fix and revert some changes
* revert the pricing
* chore: move cost/pricing changes to bedrock-cost-fixes branch
* chore: split Bedrock file-api beta stripping to separate branch
Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch;
see litellm_bedrock_invoke_strip_file_api_betas for that fix.
Made-with: Cursor
Azure GPT-5.4+ models now get the same auto-routing treatment as OpenAI
when both `reasoning_effort` and `tools` are used in `litellm.completion()`.
Previously, `reasoning_effort` was silently dropped for Azure; now the
request is bridged to the Responses API which supports both parameters.
Fixes#23914
gpt-5.4-pro and gpt-5.4-pro-2026-03-05 do not support the
/v1/chat/completions endpoint — OpenAI returns a 404 with
"This is not a chat model". These models are responses-only,
like o3-pro and o1-pro.
Changes:
- Set mode from "chat" to "responses" for both model entries
- Update supported_endpoints to ["/v1/responses", "/v1/batch"]
- Add regression test for responses API bridge routing
FixesBerriAI/litellm#23014
