Commit graph

528 commits

Author SHA1 Message Date
Mateo Wang
14193435b8
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages

Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): keep Final variables out of loops in the rename shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): match the warning text in the rename shim test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): mark the alias loader spec restore for the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): format the renamed package shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): keep the rebind-ok marker on the spec restore line

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): carry the new package search path on alias specs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): declare the shim test locals Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 10:08:25 -07:00
Mateo Wang
7e14638001
refactor(llms): rename model-version transformation files to family names (#48509)
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure),
the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py,
anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py,
and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol
names are unchanged.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:53:16 -07:00
yujonglee
bdfe7ae195
feat(gateway): connect to Redis from config.yaml (#48284)
* feat(gateway): connect to Redis from config.yaml

* feat(config): own Redis settings resolution

* fix(config): preserve Redis input precedence and presence

* fix(config): box Redis cache settings

* fix(gateway): preserve borrowed redis namespace

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:41:47 -07:00
yujonglee
0c0908c36e
feat(rust): add circuit-breaker crate and guard redis-client with it (#48277)
* feat(rust): add circuit-breaker crate and guard redis-client with it

* fix(circuit-breaker): ignore stale probe drops

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:41:47 -07:00
yujonglee
af0d29030e
feat(redis-client): run Lua scripts and pipelines with per-op errors (#48273)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:41:46 -07:00
yujonglee
a0b5c8b824
feat(redis-client): support Redis Cluster (#48271)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:41:46 -07:00
yujonglee
6edbd27f74
feat(rust): add redis-client crate for standalone Redis (#48269)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:41:46 -07:00
yujonglee
a783f8e78b
feat(db): add typed daily end-user spend upsert (#48237)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:08 -07:00
yujonglee
a2545134f3
feat(db): add typed daily team spend upsert (#48236)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:07 -07:00
yujonglee
1b874ceb85
feat(db): add typed daily user spend upsert (#48235)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:07 -07:00
yujonglee
b446d58909
feat(db): add typed spend-log insert query (#48234)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:06 -07:00
yujonglee
46410c8da9
feat(db): add typed team-member spend upsert (#48233)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:06 -07:00
yujonglee
c531692976
feat(db): add transaction-scoped team lock query (#48232)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:06 -07:00
yujonglee
8f456eb7d5
feat(db): add typed end-user spend query (#48231)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:05 -07:00
yujonglee
3b72c08c4f
feat(db): add typed team spend query (#48230)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:05 -07:00
yujonglee
8e61c22fe0
feat(db): add typed user spend query (#48229)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:04 -07:00
yujonglee
35cd5fc05f
feat(db): add typed key spend query (#48217)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:04 -07:00
yujonglee
554b011787
feat(rust): add typed database query foundation (#48214)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:05:04 -07:00
ryan-crabbe-berri
db9eda19b4
fix(http_handler): name the timeout phase and limit that fired instead of "after None seconds" (#48250)
* fix(http_handler): name the timeout phase and the limit that fired instead of 'after None seconds'

Every HTTP verb on both handlers raised litellm.Timeout with the per-call
timeout argument interpolated into the message, which is None whenever the
caller relied on the client default, and always said "Connection" even for a
read timeout. One helper now reads the limit httpx actually enforced from the
failed request and names the phase that fired

Fixes #14635

* fix(rust_bridge): name the timeout phase in the vault timeout message to match the httpx handler

* fix(http_handler): drop the unreachable timeout fallbacks httpx already rules out

* refactor(rust_bridge): drop a doc comment that restated the timeout helper
2026-10-26 16:04:33 -07:00
yuneng-jiang
e0982d6248
build(deps): bump webpki-roots and webpki-root-certs to 1.0.11 (#48301)
1.0.11 drops emSign Root CA - C1 and emSign ECC Root CA - C3 from the built-in roots
2026-10-26 10:10:57 -07:00
shrey-berri
23ccff69a1
fix(azure_ai): forward the thinking-binding-controls beta header to Foundry (#48033) 2026-10-23 16:34:55 -07:00
ryan-crabbe-berri
f2118fc767
fix(security): stop trusting the compromised eMudhra root CA (#47670)
Some checks failed
unit / enterprise-routing (push) Has been cancelled
unit / guardrails-hooks (push) Has been cancelled
unit / jwt-and-keys (push) Has been cancelled
unit / key-generation (push) Has been cancelled
unit / logging-misc (push) Has been cancelled
unit / mcp-oauth (push) Has been cancelled
unit / proxy-runtime (push) Has been cancelled
unit / proxy-server-core (push) Has been cancelled
unit / root (push) Has been cancelled
unit / proxy-utils (push) Has been cancelled
unit / enterprise-package (push) Has been cancelled
unit / integrations (push) Has been cancelled
unit / llms-anthropic (push) Has been cancelled
unit / llms-bedrock (push) Has been cancelled
unit / llms-openai-meta (push) Has been cancelled
unit / llms-providers (push) Has been cancelled
unit / llms-vertex-ai (push) Has been cancelled
unit / enterprise-managed-files (push) Has been cancelled
unit / redis-py 5.3.1 compatibility (push) Has been cancelled
unit / redis-py 6.4.0 compatibility (push) Has been cancelled
unit / redis-py 7.4.1 compatibility (push) Has been cancelled
unit / redis-py 8.0.1 compatibility (push) Has been cancelled
unit / coverage (push) Has been cancelled
unit / unit passed (push) Has been cancelled
unit / enterprise-repositories-secrets (push) Has been cancelled
unit / proxy-server (push) Has been cancelled
unit / lens-python-310 (push) Has been cancelled
unit / caching-local (push) Has been cancelled
unit / core-utils (push) Has been cancelled
unit / endpoints (push) Has been cancelled
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3

Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility

Co-authored-by: shin-berri <shin-laptop@berri.ai>
2026-10-22 02:09:52 +00:00
yujonglee
b6f91a221b
refactor(inference): rename RouteError to Error (#47771)
Every other crate names its error Error. The format crates already re-exported
it under that name, so only the inference crate and a few importers change
2026-10-21 07:03:18 -07:00
yujonglee
730db4d228
refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines (#46714)
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines

* refactor(rust-bridge): split and document route-specific detectors

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 15:06:25 -07:00
yujonglee
0675c22d56
refactor(rust): inline chat provider resolution (#47218)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 09:03:45 -07:00
yujonglee
97e9300577
refactor(rust): inline Messages provider resolution (#47217)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 09:03:44 -07:00
yujonglee
bfcb9a028f
refactor(rust): update OCR route orchestration (#47216)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 09:03:44 -07:00
yujonglee
ede6d97785
refactor(rust): separate responses provider resolution (#47215)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 09:03:43 -07:00
yujonglee
63b740532b
refactor(rust): inline transcription provider resolution (#47214)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 09:03:42 -07:00
Nate Armstrong
bcd80b6aa1
feat(rust): encode LiteLLM response ids on native responses (#46572)
Native Responses now hands back the same managed ids as Python: the
resp_ id carries the provider and deployment, encrypted-content items get
encitem_ ids and wrapped content under affinity routing, and container ids
become cntr_ ids. Encoded encrypted-content items in the input are restored
before the provider request. The codec lives in inference-responses with
goldens taken from litellm/responses/utils.py
2026-10-18 20:52:34 -07:00
devin-ai-integration[bot]
4bebcdf757
fix(databricks): use the native Anthropic route for Claude /v1/messages (#46091)
* fix(databricks): use the native Anthropic route for Claude /v1/messages

Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge

* fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas

A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again

* fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent

* fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials

Unity Catalog model-service names now reach the AI Gateway's native Anthropic
route instead of a serving-endpoints path that answers 404 for them, and the
native route resolves the key and base through the same order the chat route
uses, so environment-only deployments keep working.

The native route is limited to Databricks foundation-model Claude names, an
unlisted provider forwards safeguards and the beta as sent, and the ten betas
the workspace rejects are marked unsupported for Databricks in the beta map
and its Rust mirror.

* test(rust): expect Databricks to drop the advanced-tool-use beta

* fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback

Only the OpenAI-compatible JSON providers drop safeguards now, so a
deployment opted into /v1/messages passthrough forwards the field and
the beta as sent. When a route drops safeguards and the proxy has a
classifier model, the proxy answers the verdict itself, as the chat
completions bridge did. The Databricks Messages route also reads the
databricks_user_agent param like the chat route.

* fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL

* fix(databricks): drop container on the native Messages route, as the chat completions bridge did

* fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map

Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name

* test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire

* test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas

* fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth

Databricks chat completions, called directly or through the Messages bridge,
no longer receive the client's anthropic-beta and anthropic-version, as at the
merge base. The native Messages route still forwards them, filtered by the
beta map, without counting them as caller credentials, so a deployment with no
api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to
the native route only for Unity Catalog names, matching the chat route.

* test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route

* fix(databricks): keep client anthropic headers off an opted-in passthrough api_base

A databricks Claude deployment whose api_base is no workspace route and that opts
into /v1/messages with supported_endpoints falls back to the OpenAI-like
passthrough. That config does not filter betas through the beta map, yet it
received the native-only Databricks header entry, so Claude Code's
prompt-caching-scope beta reached the upstream raw and got a 400 where the merge
base answered 200.

Each Messages config now says whether it is the provider's native route, the
OpenAI-like passthrough says no, and the handler passes that answer when it
reads provider-specific headers, so native-only entries reach only native configs.

* test(databricks): cover client anthropic betas on an opted-in passthrough api_base

A databricks Claude deployment whose api_base is no workspace route and whose
model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like
passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m
betas through the proxy and asserts the caller gets the upstream's message and
the gateway receives no anthropic-beta, as on the merge base. The scripted
gateway answers 400 on the prompt-caching-scope beta, as the real one does

* fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies

Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do.

* test(databricks): cover a loaded beta map that marks the safeguards beta unsupported

A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict.

* test(anthropic): annotate the MCP beta map recordings as Final

* fix(anthropic): drop safeguards on routes without the classifier only under drop_params

A route whose upstream does not answer safeguards (a JSON provider, or a map
that marks the dangerous-tool-use beta unsupported) now forwards the field and
the beta as the merge base did unless drop_params is set or the proxy runs its
own classifier, matching how litellm treats any unsupported param. Databricks
keeps dropping both, since its merge base bridge never forwarded the field.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-18 02:44:38 +00:00
Aayush Jain
e2295b596f
feat(openinfer): add OpenInfer as an OpenAI-compatible provider (#41034)
* feat(openinfer): add OpenInfer as OpenAI-compatible provider

Register OpenInfer in the JSON provider registry with catalog models,
proxy create fields, and unit coverage so requests route via openinfer/

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(openinfer): assert catalog invariants instead of vendor rates

Pinning per-token prices made CI fail when the vendor list changed. Keep
the backup agreement and positivity checks our code owns.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(openinfer): add openinfer to the Rust OpenAI-compatible provider mirror

* fix(openinfer): align catalog with OpenInfer's live /v1/models

Replace the four catalog rows with the two models OpenInfer serves and
publishes a price for (@oi/GLM-5.3-Flash, @oi/Qwen3.5-27B), using its
listed per-token rates and context windows, and price-check them in tests

* fix(openinfer): use OpenInfer's native Responses and Messages APIs, allow tools on unpriced models, price Qwen3.5-9B

Declare /v1/responses and /v1/messages for openinfer, since OpenInfer serves both natively. Let JSON providers declare provider-wide capabilities in providers.json, read through get_provider_info, so tool calls reach OpenInfer for models without a cost map row. Send max_completion_tokens as is, set output caps to each model's context window, mark the three rows as reasoning models, add Qwen3.5-9B at its listed rate, and wire openinfer into models_by_provider so the openinfer/* wildcard lists its models.

* fix(openinfer): let a model's own capability row win over provider defaults in get_model_info

Provider defaults now only fill capability fields the cost-map row leaves unset, so a row or deployment that turns function calling off stays off. Also marks openinfer_models as Final.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-17 23:22:19 +00:00
devin-ai-integration[bot]
5b042c0db6
fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards (#46102)
* fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards

* test(rust): check the dangerous-tool-use beta reaches the wire with safeguards

* fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column

DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json
keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests.

* fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts

Azure AI and Claude Platform Messages transforms now add the
dangerous-tool-use beta for a non-null safeguards field, through the
Azure AI and Anthropic beta-map columns. The safeguards wire and chaos
tests get unique file names so they do not collide with the Python
fix's tests, the Python-route test asserts the fixed behavior behind a
BUG skip, and the DeepSeek test checks that no beta is added

* fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it

Claude Platform has no beta-map column and no QA account, so it goes back to sending
safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests
before the thread pool waits on them, and mark the no-column test as rstest.

* fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks

Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR.

* fix(rust): drop a null safeguards key instead of sending it without its beta

Anthropic rejects any safeguards key, null included, when the
dangerous-tool-use beta is missing, so an explicit "safeguards": null on the
Rust gateway still answered 400. The shared Messages transform now drops a
null safeguards key on every host, as the proxy already drops None params
before calling Rust, so it answers 200 with no beta.

The scripted peer in the integration wire tests now rejects the key on
presence, matching the real API.

* test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording

The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-17 16:48:05 +00:00
Mateo Wang
ae6f7f6613
ci(cost map): remove the generated model prices JSON schema (#47064)
* ci(cost map): remove the generated model prices JSON schema

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost map): keep schema-independent cost map tests after dropping the schema

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(cost map): keep the schema file until main's guard stops requiring it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(cost map): let bot sync PRs keep touching the schema file until it is deleted

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-17 05:46:31 +00:00
Mateo Wang
d3ed19e369
fix(rust): forward Anthropic headers to DeepSeek (#46959)
Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
2026-10-16 20:07:41 +00:00
yujonglee
d66d8ac127
refactor(rust-bridge): let each route host pick the body params (#46787)
* refactor(rust-bridge): let each route host pick the body params

The native routes projected request body params from a name list, hard-coded in
Rust for Messages and read from `PARAMETERS` for Chat Completions and Responses.
Each route host now answers `body_params(request)` over the keywords the call would
send, so the Python side owns which params reach the provider. No behavior change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust-host): use read-only body parameter types

Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 18:27:06 -07:00
yujonglee
55c8ec4647
refactor(rust-bridge): move the route host modules under host/routes (#46786)
* refactor(rust-bridge): move the route host modules under host/routes

Each native route's Python half lived in its own `<route>/route_host.py`, apart
from the shared host modules Rust also imports by name. They now sit together in
`litellm/rust_bridge/host/routes/<route>.py`, with their unit tests moved to the
mirrored path. No behavior change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust-host): reference messages route alias

Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 18:27:06 -07:00
yujonglee
ebac947907
fix(rust-bridge): honor the configured request_timeout on native Messages (#46749)
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:50:02 -07:00
yujonglee
c0d7264e8b
feat(rust): add native OpenRouter Responses (#46000)
The Responses route now resolves openrouter alongside openai. The
OpenRouter config reuses the OpenAI request and response transforms and
owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and
base URL (OPENROUTER_API_BASE). Wire model ids follow Python's
get_llm_provider, including the openrouter/auto native id with an
explicit provider. The Python route host no longer hands litellm.openai_key
to OpenRouter.

The OpenRouter live test lives in tests/live/openrouter.rs and returns
early when the provider key is not exported, so cargo test stays offline
by default instead of hiding it behind --ignored.

Co-authored-by: nate-berri <nate@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 11:02:03 -07:00
Kolade Fajimi
35970c3eac
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure

Tell a path apart from inline JSON by shape, not os.path.exists().

Fixes #40166

* test(vertex): narrow pytest.raises to the expected message

* fix(vertex): log the credentials path instead of returning it to the caller

* refactor(vertex): log the credential source decision where it is made

* fix(vertex): handle paths open() rejects with a bare ValueError

* fix(vertex): redact the credentials path before it reaches the log

* test(vertex): cover the brace fallback and non-text credentials file

* test(vertex): exclude the unreachable exhaustiveness guard from coverage

* refactor(vertex): read the failure's fields off the matched object

CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.

* fix(vertex): accept a read-only mapping in extract_aws_params

* fix(vertex): keep encoded credentials out of the log and cover the legacy route

* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust

* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file

* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message

* test(vertex): integration cells for credentials file and inline value failures

* test(vertex): drive the Rust messages cell through Vertex Claude

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-15 02:03:56 -07:00
yujonglee
8302f006dd
refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python (#46299)
* refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python

Rust's run_call built Python streams itself and imported
litellm.rust_bridge.streams by name, so sync calls were driven in Rust
while async calls came back as a Rust-made coroutine. Every native
inference entrypoint now returns one unstarted Execution and
public_execution.py drives it with lifecycle.run or lifecycle.drive,
wrapping streams and _hidden_params on the Python side. Rust only
imports the lifecycle step classes through StepTypes, and
python-bridge/src/lifecycle.rs, PythonLifecycle and streams.py are gone.

The adapter trusts the _native.pyi contract with a single cast instead
of a per-request runtime_checkable isinstance. Binding tests assert the
adapter starts the execution and returns its completed value rather
than checking callable identity.

* refactor(rust-bridge): fold lifecycle and public_execution into one execution module

The public stream wrappers only forwarded to the lifecycle streams and added the
header dict, so the streams now carry _hidden_params themselves. lifecycle.py no
longer held any call lifecycle, just the Execution driver, so it is renamed to
execution.py and absorbs the synchronous and asynchronous adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust-bridge): keep execution.py free of third-party imports

The host-python tests load execution.py into a bare interpreter, so the
pydantic header validation broke them. The stream head comes from the
binding's encode_stream_head, which always builds a str-keyed dict

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(host-python): preserve pre-async hook coverage after execution handoff

* style(rust-bridge): sort litellm_host_python imports

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): drive unstarted native calls through execution in the split lifecycle tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* fix(rust-bridge): mark execution TypeVar as Final

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): cover stream close, read failure, and sync suspension in execution

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:50:44 -07:00
yujonglee
f50f14117f
refactor(rust-bridge): generator-style execution protocol with stream step names (#46282)
* refactor(rust-bridge): drive executions with generator-style send, throw and close

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(rust-bridge): name stream steps StreamOpen and StreamChunk and reject out-of-phase steps

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(rust-bridge): drive with ExitStack and require the stream factory

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(host-python): give the hook_chain lifecycle binding the stream factory

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:07 -07:00
yujonglee
7d4be1e217
refactor(host-python): run the Python driver as one async body (#46235)
* refactor(host-python): run the Python driver as one async body

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(host-python): give the driver a PythonCaller with named suspensions

PythonTask becomes PythonCaller with await_python, open_stream and yield_chunk, so the call body no longer builds Suspension values or maps stream replies itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:19:06 -07:00
yujonglee
ab0875b2da
refactor(host-python): collect host failures in a thiserror Error enum (#46361)
* refactor(host-python): collect host failures in a thiserror Error enum

missing_state() and the inline PyRuntimeError/PyTypeError messages become Error variants with one From<Error> for PyErr, and InvokeError derives thiserror so From<PyErr> comes from #[from]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(host-python): note the canonical HostError import alias in AGENTS.md

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:06 -07:00
yujonglee
327bdcaa7f
test(host-python): move driver tests out of the inline module (#46325)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:05 -07:00
Mateo Wang
6b7c77b7d3
fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355)
* fix: lower aiohttp keepalive timeout

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(http): restore the 15s keep-alive default and apply it to the Rust client

15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it.

* test(integration): cover upstream connections dropped after the keep-alive idle window

---------

Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 10:50:11 -07:00
yujonglee
271b4f0306
refactor(rust): one stream codec shape per API format (#45996)
Chat Completions, Messages and Responses each get a streaming module with
the same three names, event_stream, decode and encode, built on a shared
SSE helper in base_llm/sse.rs that owns the [DONE] marker. Callers use the
module path instead of vendor-prefixed names. Encoding matches what the
Python proxy writes to clients: Messages frames carry an event line, the
OpenAI formats are bare data frames.

Responses gets one event type for SSE and WebSocket. ResponsesEventType
names every event on OpenAI's streaming-events reference, with unknown
names riding in Other, and is_terminal and is_failure follow Python's
iterator. The framer adds a bulk decode_all and an SseBlockCodec that
keeps each event's wire bytes. An event the stream ends inside is dropped,
as the SSE spec and the openai SDK's decoder do.

ChatCompletionChunk no longer requires id, created or object, since
compatible hosts omit them.
2026-10-13 15:30:14 -07:00
yujonglee
05cf428761
feat(rust): settle EdenAI Messages reported cost (#45698)
* feat(edenai): settle native Messages reported cost

* test(rust): add live EdenAI Messages contract cases

* refactor(rust): settle call cost once as an ExecutionFacts variant

The route now decides Cost (Reported, CacheHit or Deferred) from the
result source and the provider's reported figure. The legacy callback
adapter only projects a Reported amount onto provider_reported_cost and
no longer writes response_cost, which the Python calculator derives

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(rust): keep EdenAI wire shaping independent of Invoke

* style(rust): format cost settlement imports

* fix(rust): trust provider costs without bypassing configured pricing

* fix(rust): clear untrusted response costs before spend settlement

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 17:39:08 -07:00
yujonglee
8db9037c90
test(rust): add Messages live provider harness (#45823)
* test(rust): add Messages live provider harness

* test(rust): support automatic Vertex tool selection
2026-10-12 17:39:08 -07:00
devin-ai-integration[bot]
a4382657fd
chore: drop Greptile from the PR template, AGENTS.md and CI (#46013)
* chore: drop Greptile from the PR template, AGENTS.md and CI

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: require bot reviews to finish before ticking the review checkbox

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:32:39 +00:00