* refactor(proxy): inject database repositories through per-repository dependencies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): model Prisma client cleanup
* test(rust): include NativeDatabase in bridge API surface
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): forward and auto-add the compact-2026-09-04 beta on Claude routes
* test(vertex_ai): prove the compact-2026-09-04 beta reaches the wire through both Claude handlers
* fix(vertex_ai): move a client-sent body compact-2026-09-04 beta to the request header
* refactor(vertex_ai): build the Claude beta set without mutation and inject the client in the messages wire test
* fix(vertex_ai): keep client body betas in the body and answer 400 on a malformed anthropic_beta
* fix(rust): allow the compact-2026-09-04 beta on vertex_ai in the Rust beta table
The beta header config now maps compact-2026-09-04 for vertex_ai, so the Rust
table that mirrors it must stop rejecting that beta there, and the Rust Vertex
messages route adds it for a compaction request like the Python route does.
* fix(vertex_ai): keep client body betas out of the Vertex body next to litellm betas, honor drop_params, and send the compaction beta on count_tokens
Litellm's own betas replace a client's body anthropic_beta again, as on main, so a body beta
Vertex rejects no longer turns a 200 into a 400. A malformed beta drops under drop_params and
names its field otherwise. inline-tools-2026-09-15 joins compact-2026-09-04 as header-only, since
Vertex rejects both in the body. Vertex count_tokens sends compact-2026-09-04 when the request
carries compaction.
* fix(logging): log and bill Anthropic messages compaction responses on the Rust route
The Rust route hands the served Anthropic body to the logger, which validates it
against AnthropicResponse. That model had no compaction content block and no
compaction stop reason, so a compaction reply raised and wrote no spend row.
* fix(vertex_ai,azure_ai): forward compact-2026-09-04 to Azure AI Foundry, merge Vertex body betas, and redact compaction summaries
Azure AI Foundry needs the compact-2026-09-04 header for compaction, a replayed compaction block, and count tokens, so the beta map, the native compaction providers, and the Rust beta table now allow it there. Vertex keeps a client's body betas next to the ones litellm adds, as the Anthropic route does, and turn_off_message_logging now redacts compaction summaries too
* fix(vertex_ai,azure_ai): auto-add thinking display and tool change betas, redact streamed compaction summaries
Folds the header changes from #44103 into this PR. Vertex AI and Azure AI Claude
routes now add thinking-display-updates-2026-08-18 when thinking.display is
"updates" and mid-conversation-tool-changes-2026-07-01 when a tool_addition
block is present, matching the Anthropic and Bedrock routes.
With turn_off_message_logging on, compaction summaries from streamed messages
and streamed chat completions are now redacted in spend logs and callback
payloads, like the non-streamed response already was.
* fix(vertex_ai,azure_ai,rust): add tool change betas on count tokens and Rust messages, bill Rust server tool replies
Vertex and Azure count tokens now add mid-conversation-tool-changes-2026-07-01 when the history carries a tool_addition or tool_removal block, through one shared helper with the compaction beta.
The Rust /v1/messages route adds thinking-display-updates-2026-08-18 and mid-conversation-tool-changes-2026-07-01 when a request uses them, on the providers that accept each beta.
Rust-route logging parses the served body with the Anthropic response parser, so web_search, pause_turn, mcp_tool_use and container_upload replies get a spend row.
Legacy enabled thinking keeps display updates on anthropic too, and the Vertex transform reads drop_params through drops_unsupported_params.
* style(rust): rewrap beta map test line with cargo fmt
* test(rust): expect the display beta on Databricks, which main's beta list now accepts
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: openai exception mapper ignored status code for invalid_request_error
_map_openai_exception coerced anything whose body carried type
invalid_request_error into BadRequestError, guarded only by excluding the single
phrase Incorrect API key provided, and this ran before the status-code checks.
OpenAI and openai-compatible providers return 401 and 404 bodies with type
invalid_request_error, so an auth failure like Invalid API key mapped to a 400
instead of AuthenticationError, and a 404 mapped to BadRequestError instead of
NotFoundError. That changes the status returned to proxy clients and router
retry, cooldown, and fallback handling. Mirror the Azure mapper by only coercing
to BadRequestError when status_code is None or 400, so 401 and 404 fall through
to their existing handlers.
* fix(exceptions): map OpenAI-family invalid_request_error 404s to NotFoundError
Move the contributor's test to the mapped unit test file, assert the raised
class, status code and message, and cover the 400 and status-less cases that
must still raise BadRequestError
Add a regression test for the flow the bug was found on. Retrieving a missing
OpenAI vector store, sync or async, now raises NotFoundError with status 404
The router test that pinned an OpenAI 404 typed invalid_request_error as
BadRequestError now expects NotFoundError
* fix(a2a): answer a provider NotFoundError as JSON-RPC invalid params
* fix(exceptions)!: limit the invalid_request_error status guard to 401 and 404
Every other status keeps raising BadRequestError as on main, so 402-415, 429 and 5xx bodies typed invalid_request_error keep their class and router retry behavior. A2A goes back to main's mapping, which answers NotFoundError and AuthenticationError as JSON-RPC internal errors.
* fix(exceptions): keep the provider error fields on mapped 401 and 404 invalid_request_errors
Raise AuthenticationError and NotFoundError straight from the invalid_request_error
branch with the provider body, so code, param and type survive and the proxy's
error.type matches main. The Rust mapper copy skips 401 and 404 the same way.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ocr): support outputContentFormat=markdown for Azure Document Intelligence
* fix(ocr): map only span endpoints when slicing Azure Markdown pages
* fix(ocr): accept any casing of outputContentFormat and report code point offsets
Azure reads outputContentFormat case-insensitively, so Markdown and TEXT
now map to the same values instead of returning 400. When the proxy asks
Azure for code point offsets to split Markdown pages, the response now
carries stringIndexType so callers know the unit of table and key-value
spans. Tests use rstest and the enum derives VariantArray per the Rust
workspace rules.
* fix(ocr): ask Azure for code point offsets only on prebuilt-layout and keep line-joined pages when spans are malformed
* fix(ocr): trim Azure DI outputContentFormat, accept output_content_format, and drop invalid values under drop_params
* fix(ocr): fall back to output_content_format when outputContentFormat is empty
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): omit system prompt from audio transcription
* fix(bedrock): report Converse token usage on audio transcription
Bedrock transcriptions returned usage null and were billed at $0. Map
Converse inputTokens, outputTokens and totalTokens into the transcription
usage so spend uses the model's per-token price. input_token_details is
optional because Converse does not split audio and text tokens.
Also switch the updated request-shape test to rstest.
* fix(bedrock): default transcription temperature to 0 and price ap-southeast-2 Voxtral regionally
Voxtral wrapped some transcripts in quotes at its default sampling
temperature. In direct Converse calls in ap-southeast-2, 0 of 48 calls at
temperature 0 were framed against 7 of 72 at the default, and instruction
wording made no clear difference, so the request now sends temperature 0
when the caller sets none, matching OpenAI's transcription default.
Calls in ap-southeast-2 billed at the us-east-1 price. The cost map gains
bedrock/ap-southeast-2 Voxtral rows with all four service tiers, the Bedrock
transcription response carries the deployment region in its hidden params,
and the transcription token cost uses the region-resolved model key.
The bridge now builds the response with model_copy instead of assigning
usage after construction, and the native wheel test checks the usage the
compiled route returns.
* fix(bedrock): forward transcription language, reject unsupported formats, and bill env and Tokyo/London regions
get_optional_params_transcription had no Bedrock branch, so language,
response_format and timestamp_granularities were dropped before the Rust
transform ever saw them. Bedrock now forwards language, prompt and
temperature, accepts response_format json or text (the only shapes it can
return), and answers 400 for srt, vtt, verbose_json or
timestamp_granularities unless drop_params is set.
The billing region now falls back to AWS_REGION_NAME and then AWS_REGION
when aws_region_name is unset, the same order the Rust resolver uses to
pick the endpoint, so an env-configured region gets its regional price.
Adds ap-northeast-1 and eu-west-2 Voxtral rows (standard, batch, flex and
priority) from the AWS Bedrock offer files, the other two regions AWS
serves Voxtral audio in.
* fix(bedrock): send transcription temperature as a number and name the rejected param
The proxy keeps multipart form values as strings, so -F temperature=0.2
reached Converse as "0.2" and Bedrock answered 400. The Bedrock mapper now
converts temperature to a float and answers 400 (or drops it under
drop_params) when it is not a finite number.
get_optional_params_transcription copied prompt and temperature back from
the raw call after the Bedrock branch ran, which undid that conversion and
any drop. The Bedrock branch now owns every OpenAI transcription param.
An unsupported param such as timestamp_granularities now gets a 400 that
names it instead of the generic "user/encoding format" message.
* refactor(bedrock): return unsupported transcription params as a value and raise at the param step
* fix(bedrock): drop the retired max_tokens field from the regional voxtral price rows
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages
Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes.
* fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body
Vertex reads the API version from the body alone, so the caller's
anthropic-version header now becomes the body's anthropic_version instead
of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party
versions keep vertex-2023-10-16, and the header itself is still never sent.
Same change on the Python proxy and the Rust gateway.
* fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept
Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are
carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no
header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under
drop_params and answers a 400 naming drop_params without it, on the Python proxy
and the Rust gateway.
* refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place
* fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python
The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there
* docs(rust router): name Router::from_config in the crate README
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
* feat(skills): add a litellm-setup agent skill
* fix(skills): run the quickstart for Docker with Postgres in litellm-setup
Use scripts/quickstart.sh for the Docker plus Postgres path instead of a
hand-rolled Postgres container, keep uv with the user's own Postgres, load
.env as data instead of sourcing it, pin the uv Python range, chmod an
existing .env, list bedrock_converse and vertex_ai rows in the cost map
lookup, ask before editing the shell profile, and stop the health wait
when the proxy exits
* fix(skills): guide empty Postgres, docker sudo, and key fixes in litellm-setup
Ask for a new empty database on the uv path and DIRECT_URL behind a pooler,
add the baseline error to troubleshooting, and drop inherited database
settings in the loader line. Check python3 and lsof, explain Docker's
permission denied, give quickstart commands by the folder's path, and fix
provider keys by recreating the container or deleting the quickstart model.
Drop the structure-only test.
* fix(skills): keep shell profiles untouched when litellm-setup installs uv
The uv installer edits shell profiles unless UV_NO_MODIFY_PATH is set, so the skill now sets it and puts ~/.local/bin on PATH for later commands. It also says litellm --setup runs in a separate terminal and how to retire the shared master-key UI login
* fix(skills): offer litellm --setup only on the uv path
The Docker paths install no host litellm CLI, and the old wording also passed the skill's own step number on to the user
* fix(skills): ask before using an env provider key, print model names config.yaml accepts, probe the port with curl
* fix(skills): state the env key default in the question, chmod the wizard config, keep uv's bin dir on PATH
* fix(skills): keep litellm-setup on its own .env and install when the shell has others
* fix(skills): read the setup wizard's config without printing keys and hand off the exact restart command
* fix(skills): hide the database-access dev skill from end-user skill installs
* fix(skills): give the litellm --setup path a restart command that keeps its config
* docs(skills): point litellm --setup users to the master key in litellm_config.yaml
* fix(skills): hide every non-end-user skill, use Azure deployment names, keep the restart block out of the user's shell
* test(skills): find non-end-user skills in-process instead of through git
* docs(skills): price Azure deployments with base_model and put uv on PATH in the start block
* refactor(rust-bridge): require the native extension
_native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader
The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ci): build Rust bridge in Windows smoke tests
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(providers): add ZeroGPU as an OpenAI-compatible provider
Registers zerogpu through the JSON provider registry so zerogpu/<model> routes to
https://api.zerogpu.ai/v1/chat/completions with ZEROGPU_API_KEY and an optional ZEROGPU_API_BASE.
Adds LlmProviders.ZEROGPU, the constants.py provider and endpoint lists, cost map entries for the
10 ZeroGPU text generation models, the endpoint matrix, the Add Model form fields, the README row
and the dashboard provider entry
* feat(zerogpu): add capability flags verified against the live API
Live calls to https://api.zerogpu.ai/v1/chat/completions confirmed tools, tool_choice and json_schema
response_format on 8 of the 10 catalog text models, cached token reporting on deepseek-v4.1-flash,
gpt-4.1-mini, gpt-5.4-nano and gpt-5.6-luna, and reasoning_effort on gpt-5.6-luna. LFM2.5-1.2B-Instruct
answers stream requests with plain JSON instead of SSE and LFM2.5-1.2B-Thinking returns 500 on every
request, so both stay out of the cost map until the API serves them correctly
* feat(zerogpu): price LFM2.5-1.2B-Instruct
Adds zerogpu/LFM2.5-1.2B-Instruct to the cost map with the catalog prices and context window. It
gets no capability flags because the live API ignores tools, response_format and
max_completion_tokens for this model, so LiteLLM drops tool params for it instead of sending them.
LFM2.5-1.2B-Thinking stays out of the cost map
* fix(zerogpu): reject transcription requests before calling the upstream API
ZeroGPU only serves chat completions, so drop it from the OpenAI transcription transport set. A zerogpu transcription request now fails locally instead of hitting an /audio/transcriptions endpoint that does not exist
* test(zerogpu): cover the responses and messages bridges
Both endpoints reach ZeroGPU through chat completions, so assert the request each one sends upstream and the reply it returns
* refactor(zerogpu): move zerogpu out of the provider list tails
New upstream providers append to the end of openai_compatible_endpoints, openai_compatible_providers and providers.json, so an entry at the tail conflicts with every one of them. Placing zerogpu next to long-stable entries keeps those merges clean. No behavior change
* fix(zerogpu): price cached input for qwen3-30b-a3b-fp8
The ZeroGPU model catalog (https://docs.zerogpu.ai/docs/model-catalog, checked 2026-10-09) lists cached input for qwen3-30b-a3b-fp8 at $0.05 per 1M tokens. Without cache_read_input_token_cost, cached tokens were billed at the full $0.10 input rate
* revert(zerogpu): leave the audio transcription provider set unchanged
Per review, keep OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS as upstream defines it. This PR only covers chat completions, and ZeroGPU now serves /audio/transcriptions anyway, so the local refusal and its test are removed
* fix(zerogpu): add max_tokens so trim_messages can size ZeroGPU history
trim_messages reads max_tokens as the fallback for max_input_tokens and evaluates it eagerly, so without it the lookup raised, got swallowed, and the full history came back untrimmed. max_tokens matches each model's maxTokens in ZeroGPU's models API (https://api-dashboard.zerogpu.ai/api/models, checked 2026-10-10)
* test(zerogpu): record mocked requests on a recorder object
The mock handler appended to a list the caller passed in, which AGENTS.md
forbids. A recorder now owns the captured requests as a tuple
* fix(zerogpu): cap max_tokens at each model's output limit
Each cap is the max_output_tokens ZeroGPU stores for the model in its production models table (read 2026-10-20), the same column its /v1/models serves as max_length. That gives 65536 for deepseek-v4.1-flash, 32768 for gpt-4.1-mini and gpt-oss-120b, 128000 for gpt-5.4-nano and gpt-5.6-luna, 8192 for LFM2.5-1.2B-Instruct and llama-3.1-8b-instruct-fast, and 16384 for qwen3-30b-a3b-fp8. glm-5.3-flash is stored as 32786, which reads as a typo for 32768, so that row uses 32768
trim_messages sizes history from max_input_tokens, so lowering max_tokens leaves it unaffected
* fix(zerogpu): flag prompt caching on glm-5.3-flash
Repeating the same ~4.3K-token prompt against ZeroGPU on 2026-10-20 returned cached_tokens of 4320 on some glm-5.3-flash calls. gpt-oss-120b and llama-3.1-8b-instruct-fast returned 0 on all nine calls, and qwen3-30b-a3b-fp8 never returned cached_tokens, so they stay unflagged. Their cached prices stay as ZeroGPU publishes them, so any cache hit is still billed at the right rate
* fix(zerogpu): set gpt-5.4-nano context window to 272000
ZeroGPU accepts a 271001-token gpt-5.4-nano prompt and rejects 273000 tokens as over the context window (checked 2026-10-21), so max_input_tokens is 272000 rather than the 400000 its catalog lists
* fix(zerogpu): list zerogpu models for zerogpu/* wildcards
zerogpu had no models_by_provider entry, so get_provider_models("zerogpu") returned None and a zerogpu/* wildcard listed no models. zerogpu_models is now filled from the cost map rows like openinfer_models and registered in models_by_provider
* fix(zerogpu): flag gpt-5.4-nano as a reasoning model
ZeroGPU honors reasoning_effort on gpt-5.4-nano (checked 2026-10-21). Without it the model spends 0 reasoning tokens, and with low or high it spends 32 or more. LiteLLM only treated it as a reasoning model by falling back to OpenAI's bare gpt-5.4-nano row, so get_model_info and /model/info reported supports_reasoning as null. The row now carries the flag itself
* fix(zerogpu): flag deepseek, glm, gpt-oss and qwen3 as reasoning models
ZeroGPU's /v1/models marks these four as reasoning models and accepts reasoning_effort on all of them (checked 2026-10-21), but LiteLLM rejected the param with UnsupportedParamsError because their rows had no supports_reasoning flag. glm reports reasoning_tokens under reasoning_effort, and gpt-oss and qwen3 spend hidden reasoning tokens on every reply. The gpt-oss test case now expects reasoning_effort to reach ZeroGPU
* fix(zerogpu): set gpt-oss-120b context window to 128000
ZeroGPU routes gpt-oss-120b across several upstream backends, and at least one of them returns 413 past a 128000-token window, so max_input_tokens uses that as the safe floor. The backends reached on 2026-10-21 accepted up to about 131K prompt tokens, so this only gives up a little headroom on the larger ones
* feat(zerogpu): call ZeroGPU's Responses API natively
ZeroGPU serves /v1/responses for all nine catalog models, so LiteLLM no longer bridges Responses requests through chat completions. Native calls return ZeroGPU's reasoning output items for glm, gpt-oss and qwen3, which the bridge dropped. Checked live on 2026-10-22 through litellm.responses for every model, non-streaming and streaming, plus a gpt-oss tool call. LFM2.5-1.2B-Instruct streams correctly now that ZeroGPU's text_generation lane emits Responses SSE
* fix(zerogpu): stop inferring the zerogpu provider from api_base alone
Listing https://api.zerogpu.ai/v1 in openai_compatible_endpoints rerouted bare model names on that api_base from openai to zerogpu. A bare gpt-5.6-luna call with verbosity used to return 200 and started raising UnsupportedParamsError. Drop the entry and its api_base-alone test, so zerogpu is only selected through the zerogpu/ prefix
* fix(zerogpu): price gpt-6-luna, llama-3.2-3b, LFM2.5-1.2B-Thinking and qwen3.6
A zerogpu/* wildcard served these four chat models with no cost-map row, so their calls logged $0 spend. Add rows at ZeroGPU's catalog prices from https://api-dashboard.zerogpu.ai/api/models, with context windows and output caps from ZeroGPU's models table. LFM2.5-1.2B-Thinking has no cached price, like LFM2.5-1.2B-Instruct. gpt-6-luna is flagged supports_reasoning because LiteLLM already routes gpt-6 names as reasoning models. Checked live on 2026-10-28, all four now report nonzero response_cost
* fix(zerogpu): drop the retired max_tokens field from the zerogpu rows
Upstream #48471 retired max_tokens from the cost map in favor of max_input_tokens and max_output_tokens, and test_no_row_carries_the_retired_max_tokens_field failed on all 13 zerogpu rows. Every row already carries the same value in max_output_tokens, so nothing is lost. The zerogpu limits test now checks that each row's output limit fits its context window
* chore(zerogpu): regenerate provider endpoint support for zerogpu
Upstream #48619 generates provider_endpoints_support.json from the cost map and checks it in CI, and the hand-written zerogpu entry no longer matched. Regenerated with scripts/generate_provider_endpoints_support.py. Only the zerogpu entry changed, and it now matches what the generator emits for other JSON-registry providers such as openinfer
* fix(zerogpu): flag tools, json_schema and prompt caching on gpt-6-luna
ZeroGPU serves function calling, tool_choice, json_schema response_format and prompt caching on gpt-6-luna, so set supports_function_calling, supports_tool_choice, supports_response_schema and supports_prompt_caching on its row in both cost maps. Checked live on 2026-10-28, a weather tool call came back with the right arguments, json_schema output validated, and repeated prompts reported 3260 of 3263 tokens cached
* fix(zerogpu): set qwen3.6-35b-a3b context to 262144 and flag its tools and reasoning
ZeroGPU accepts up to 262,144 input tokens on qwen3.6-35b-a3b and serves tools, tool_choice, json_schema response_format and reasoning_effort on it. Raise max_input_tokens from 16384 and set supports_function_calling, supports_tool_choice, supports_response_schema and supports_reasoning in both cost maps. Before this, LiteLLM rejected tools and reasoning_effort for the model. Checked live on 2026-10-28, a 209,905-token prompt succeeded, an oversized one was refused with a 262144-token limit, and tool calls, json_schema output and reasoning_effort all worked
* fix(zerogpu): flag vision on gpt-6-luna and qwen3.6-35b-a3b
ZeroGPU reads image inputs on gpt-6-luna and qwen3.6-35b-a3b, so set supports_vision on both rows in both cost maps. Checked live on 2026-10-28, both models named the animal in an image_url photo, and the image showed up in their prompt token counts
* feat(rust): one typed decline from core up to each host
Rust core says it cannot serve a request with one Decline that travels
unchanged through llms and inference up to the host. The Python bridge
raises RustBridgeDeclined for it in route_failure before the public
exception mapping, so dispatch falls back to legacy Python instead of
failing the call, and the gateway answers 501. Every route collapses
errors after the send with Error::after_send, which keeps a provider
answer and a response fault as they are and nests anything else as the
source of an invalid response, so a billed call can never read as a
decline. Response-side cases that used to decline are now
ErrorDetail::Unhandled response faults.
The rust-error skill documents declined() as the first question both
boundaries ask, and the Python suites pin that a native decline reaches
the dispatcher before any provider IO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): keep post-provider hook failures terminal
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http
The workspace moves to reqwest 0.13, so litellm-http no longer carries a
second reqwest just for rmcp. The mcp feature, its pool and its proxy
conversion go away, and the gateway hands rmcp a pooled NoRedirect client
that already carries the TLS, proxy and timeout settings
The Client test helpers collapse into for_test, a no-redirect client
without TLS or proxy settings, and wrap_for_test for the few tests that
need a hand-built reqwest client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(http): honor transport settings for Rust WebSockets
The Responses WebSocket connected through tokio-tungstenite with its own
native-roots TLS, so it ignored the configured CA bundle, verify flag,
client certificate, proxies and timeouts. The handshake now goes through
a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no
redirects) and upgrades the response, so it shares the settings every
other outbound request uses
The upgrade response is checked for the Sec-WebSocket-Accept key, the
Upgrade and Connection headers and a subprotocol that was offered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(http): build outbound requests as reqwest::Request
OutboundRequest was a hand-rolled copy of reqwest::Request. Routes now build
a reqwest::Request directly and RequestSigner reads the body bytes back out
of it, so the signer still covers exactly what is sent. SigV4 signing moves
onto HeaderMap and takes the method, which drops the BTreeMap plumbing and
the header casing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(http): make litellm_http::Error the one opaque error
transport::Error and media::Error fold into litellm_http::Error, an
Arc<Repr> newtype whose private variants keep the typed reqwest, io,
rustls and tungstenite causes and own every message. Callers can no
longer build an error with a message of their own. They ask through
dispatched, is_request, status, upstream_response and tls_source
One From<reqwest::Error> now classifies connect, network and timeout
and strips the URL. The WebSocket handshake reports each failed check
as its own variant and owns its connect timeout. OCR drops the media
variants it mirrored, and inference::Error maps a header the signer
computes to Unsupported for every route, not only chat
The error rules move out of AGENTS.md into the rust-error skill, which
also records the decisions behind this design
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): split rust-error skill into thin SKILL.md with references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): render causes once and ask predicates at the boundaries
The four WebSocket handshake variants collapse into Handshake(HandshakeFault), and media download failures no longer answer dispatched() since they happen before the provider call
ErrorDetail in llms and auth-types, the inference error and the OCR error now store their cause in a field named cause with no #[source], so to_string() carries it exactly once and source() is None. Tests that walked the chain now destructure the variant and downcast the cause
The gateway asks is_unsupported() and is_missing_credential() on the inference and OCR errors instead of matching through nested enums, and litellm_auth::Error answers is_missing_credential() for both
The rust-error skill gains axum and pyo3 references, boundaries.md describes the real predicate order and the known divergences, the testing example asserts dispatched() correctly, and shape, classification, conversions, messages-and-causes and recipes agree with the code again
* docs(rust): focus the rust-error gateway page on the error flow and add a rust-gateway skill
The former axum page toured gateway-inference: its status match, the envelope
table, an extractor, the unsupported route and the Sse closure. Those details
belong to the gateway crates and go stale with them. The page is now gateway.md
and follows an error from the provider failure through each From into
IntoResponse, where it is classified once and rendered by surface and framing.
The gateway-structure material moves to a new rust-gateway skill: envelopes per
API surface, extractors and path registration, SSE relay, and the router test
harness.
* test(inference): pin provider URL redaction on every route's transport error
Chat, Messages, Responses and transcription each get a route-level test that
fails if a reqwest error reaches the caller with its URL. The Messages stream
decoder also takes body errors through litellm_http::Error like every other
reqwest error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust-error): adopt std source chains and add litellm-error Report
Messages describe their own layer and the cause is the source, boundaries
render the chain once with litellm_error::Report. Corrects skill claims
the audit found stale against the code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): keep causes as sources across http, auth, secrets, cache and utility crates
Messages describe their own layer and every cause is a typed source,
predicates match exhaustively, and boundaries render with
litellm_error::Report. Media downloads no longer count as dispatched,
Error::upstream bounds the provider body itself, and a forwarded
header SigV4 cannot sign is the caller's fault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): keep causes as sources in llms and the gateway crates
ErrorDetail loses its string variant, OCR provider variants become data,
host faults stop counting as the caller's, a tunnelled transport error
classifies the same on every stream path, and each gateway crate has one
Error with one status() and extractors that reject in its envelope.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): nest llms errors in inference and ask predicates in route tests
litellm_inference::Error nests litellm_llms::Error instead of mirroring
its variants, host driver faults get their own variant, every predicate
matches exhaustively, and route tests ask predicates instead of matching
transport variants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): map bridge and gateway-inference errors once and render them with Report
RustUpstreamError has one constructor that always sets headers, argument
checks raise typed errors mapped once, OCR asks upstream_response(), the
Responses stream error event comes from the error type, gateway-inference
loses its string variants, and auth faults classify through one
litellm_auth predicate on every route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-error): carry OCR provider headers as a HeaderMap and keep outages off the caller
The OCR provider error and get_error_class take reqwest's HeaderMap, the
bridge turns it into Python's (name, value) pairs once. Credential
acquisition and provider SDK failures are no longer the caller's fault,
the UI hides auth and backend causes, and chat keeps a provider answer
as is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust-error): correct skill claims against the aligned code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): fix the remaining sweep findings in gateway, llms, auth and cache crates
Auth rejections hide their backend cause on every inference renderer,
path parameters reject in the surface envelope, Textract's unsupported
document and Azure DI's unparseable poll URL get typed variants, the
credential chain exposes its last failure as the source, gateway-mcp
asks predicates over rmcp errors, and the S3 and Azure blob transports
classify through litellm_http::Error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): fix the remaining sweep findings on the Python side
host-python keeps a native error's chain and returns blocking-pool
cancellation instead of panicking, Vault transport failures convert
through litellm_http::Error so no URL reaches Python, the bridge asks
predicates instead of matching three levels into Vault errors, and the
last string-built PyErrs become typed errors mapped once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): refresh compatible dependencies flagged by OSV
* refactor(rust): remove redundant client construction comment
* test(rust): cover configured MCP proxy and CA bundle
* fix(http): preserve native WebSocket trust and optional protocols
* fix(http): allow plain WebSockets without native roots
* fix(http): normalize provider URLs before building outbound requests
* fix(rust): preserve OCR download status and reduce error size
* fix(rust): preserve upstream bodies and bound diagnostics only
* test(rust): preserve bridge serialization boundary in body regression
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(http): build outbound requests as reqwest::Request
OutboundRequest was a hand-rolled copy of reqwest::Request. Routes now build
a reqwest::Request directly and RequestSigner reads the body bytes back out
of it, so the signer still covers exactly what is sent. SigV4 signing moves
onto HeaderMap and takes the method, which drops the BTreeMap plumbing and
the header casing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(http): normalize provider URLs before building outbound requests
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): honor transport settings for Rust WebSockets
The Responses WebSocket connected through tokio-tungstenite with its own
native-roots TLS, so it ignored the configured CA bundle, verify flag,
client certificate, proxies and timeouts. The handshake now goes through
a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no
redirects) and upgrades the response, so it shares the settings every
other outbound request uses
The upgrade response is checked for the Sec-WebSocket-Accept key, the
Upgrade and Connection headers and a subprotocol that was offered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(http): preserve native WebSocket trust and optional protocols
* fix(http): allow plain WebSockets without native roots
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http
The workspace moves to reqwest 0.13, so litellm-http no longer carries a
second reqwest just for rmcp. The mcp feature, its pool and its proxy
conversion go away, and the gateway hands rmcp a pooled NoRedirect client
that already carries the TLS, proxy and timeout settings
The Client test helpers collapse into for_test, a no-redirect client
without TLS or proxy settings, and wrap_for_test for the few tests that
need a hand-built reqwest client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): refresh compatible dependencies flagged by OSV
* refactor(rust): remove redundant client construction comment
* test(rust): cover configured MCP proxy and CA bundle
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages
Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): keep Final variables out of loops in the rename shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): match the warning text in the rename shim test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): mark the alias loader spec restore for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): format the renamed package shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): keep the rebind-ok marker on the spec restore line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): carry the new package search path on alias specs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): declare the shim test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure),
the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py,
anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py,
and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol
names are unchanged.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add circuit-breaker crate and guard redis-client with it
* fix(circuit-breaker): ignore stale probe drops
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): name the timeout phase and the limit that fired instead of 'after None seconds'
Every HTTP verb on both handlers raised litellm.Timeout with the per-call
timeout argument interpolated into the message, which is None whenever the
caller relied on the client default, and always said "Connection" even for a
read timeout. One helper now reads the limit httpx actually enforced from the
failed request and names the phase that fired
Fixes#14635
* fix(rust_bridge): name the timeout phase in the vault timeout message to match the httpx handler
* fix(http_handler): drop the unreachable timeout fallbacks httpx already rules out
* refactor(rust_bridge): drop a doc comment that restated the timeout helper
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Every other crate names its error Error. The format crates already re-exported
it under that name, so only the inference crate and a few importers change
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines
* refactor(rust-bridge): split and document route-specific detectors
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
