Commit graph

494 commits

Author SHA1 Message Date
Mateo Wang
d3ed19e369
fix(rust): forward Anthropic headers to DeepSeek (#46959)
Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
2026-10-16 20:07:41 +00:00
yujonglee
d66d8ac127
refactor(rust-bridge): let each route host pick the body params (#46787)
* refactor(rust-bridge): let each route host pick the body params

The native routes projected request body params from a name list, hard-coded in
Rust for Messages and read from `PARAMETERS` for Chat Completions and Responses.
Each route host now answers `body_params(request)` over the keywords the call would
send, so the Python side owns which params reach the provider. No behavior change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust-host): use read-only body parameter types

Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 18:27:06 -07:00
yujonglee
55c8ec4647
refactor(rust-bridge): move the route host modules under host/routes (#46786)
* refactor(rust-bridge): move the route host modules under host/routes

Each native route's Python half lived in its own `<route>/route_host.py`, apart
from the shared host modules Rust also imports by name. They now sit together in
`litellm/rust_bridge/host/routes/<route>.py`, with their unit tests moved to the
mirrored path. No behavior change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust-host): reference messages route alias

Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 18:27:06 -07:00
yujonglee
ebac947907
fix(rust-bridge): honor the configured request_timeout on native Messages (#46749)
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:50:02 -07:00
yujonglee
c0d7264e8b
feat(rust): add native OpenRouter Responses (#46000)
The Responses route now resolves openrouter alongside openai. The
OpenRouter config reuses the OpenAI request and response transforms and
owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and
base URL (OPENROUTER_API_BASE). Wire model ids follow Python's
get_llm_provider, including the openrouter/auto native id with an
explicit provider. The Python route host no longer hands litellm.openai_key
to OpenRouter.

The OpenRouter live test lives in tests/live/openrouter.rs and returns
early when the provider key is not exported, so cargo test stays offline
by default instead of hiding it behind --ignored.

Co-authored-by: nate-berri <nate@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 11:02:03 -07:00
Kolade Fajimi
35970c3eac
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure

Tell a path apart from inline JSON by shape, not os.path.exists().

Fixes #40166

* test(vertex): narrow pytest.raises to the expected message

* fix(vertex): log the credentials path instead of returning it to the caller

* refactor(vertex): log the credential source decision where it is made

* fix(vertex): handle paths open() rejects with a bare ValueError

* fix(vertex): redact the credentials path before it reaches the log

* test(vertex): cover the brace fallback and non-text credentials file

* test(vertex): exclude the unreachable exhaustiveness guard from coverage

* refactor(vertex): read the failure's fields off the matched object

CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.

* fix(vertex): accept a read-only mapping in extract_aws_params

* fix(vertex): keep encoded credentials out of the log and cover the legacy route

* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust

* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file

* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message

* test(vertex): integration cells for credentials file and inline value failures

* test(vertex): drive the Rust messages cell through Vertex Claude

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-15 02:03:56 -07:00
yujonglee
8302f006dd
refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python (#46299)
* refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python

Rust's run_call built Python streams itself and imported
litellm.rust_bridge.streams by name, so sync calls were driven in Rust
while async calls came back as a Rust-made coroutine. Every native
inference entrypoint now returns one unstarted Execution and
public_execution.py drives it with lifecycle.run or lifecycle.drive,
wrapping streams and _hidden_params on the Python side. Rust only
imports the lifecycle step classes through StepTypes, and
python-bridge/src/lifecycle.rs, PythonLifecycle and streams.py are gone.

The adapter trusts the _native.pyi contract with a single cast instead
of a per-request runtime_checkable isinstance. Binding tests assert the
adapter starts the execution and returns its completed value rather
than checking callable identity.

* refactor(rust-bridge): fold lifecycle and public_execution into one execution module

The public stream wrappers only forwarded to the lifecycle streams and added the
header dict, so the streams now carry _hidden_params themselves. lifecycle.py no
longer held any call lifecycle, just the Execution driver, so it is renamed to
execution.py and absorbs the synchronous and asynchronous adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust-bridge): keep execution.py free of third-party imports

The host-python tests load execution.py into a bare interpreter, so the
pydantic header validation broke them. The stream head comes from the
binding's encode_stream_head, which always builds a str-keyed dict

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(host-python): preserve pre-async hook coverage after execution handoff

* style(rust-bridge): sort litellm_host_python imports

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): drive unstarted native calls through execution in the split lifecycle tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* fix(rust-bridge): mark execution TypeVar as Final

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): cover stream close, read failure, and sync suspension in execution

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:50:44 -07:00
yujonglee
f50f14117f
refactor(rust-bridge): generator-style execution protocol with stream step names (#46282)
* refactor(rust-bridge): drive executions with generator-style send, throw and close

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(rust-bridge): name stream steps StreamOpen and StreamChunk and reject out-of-phase steps

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(rust-bridge): drive with ExitStack and require the stream factory

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(host-python): give the hook_chain lifecycle binding the stream factory

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:07 -07:00
yujonglee
7d4be1e217
refactor(host-python): run the Python driver as one async body (#46235)
* refactor(host-python): run the Python driver as one async body

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(host-python): give the driver a PythonCaller with named suspensions

PythonTask becomes PythonCaller with await_python, open_stream and yield_chunk, so the call body no longer builds Suspension values or maps stream replies itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:19:06 -07:00
yujonglee
ab0875b2da
refactor(host-python): collect host failures in a thiserror Error enum (#46361)
* refactor(host-python): collect host failures in a thiserror Error enum

missing_state() and the inline PyRuntimeError/PyTypeError messages become Error variants with one From<Error> for PyErr, and InvokeError derives thiserror so From<PyErr> comes from #[from]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(host-python): note the canonical HostError import alias in AGENTS.md

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:06 -07:00
yujonglee
327bdcaa7f
test(host-python): move driver tests out of the inline module (#46325)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:19:05 -07:00
Mateo Wang
6b7c77b7d3
fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355)
* fix: lower aiohttp keepalive timeout

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(http): restore the 15s keep-alive default and apply it to the Rust client

15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it.

* test(integration): cover upstream connections dropped after the keep-alive idle window

---------

Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 10:50:11 -07:00
yujonglee
271b4f0306
refactor(rust): one stream codec shape per API format (#45996)
Chat Completions, Messages and Responses each get a streaming module with
the same three names, event_stream, decode and encode, built on a shared
SSE helper in base_llm/sse.rs that owns the [DONE] marker. Callers use the
module path instead of vendor-prefixed names. Encoding matches what the
Python proxy writes to clients: Messages frames carry an event line, the
OpenAI formats are bare data frames.

Responses gets one event type for SSE and WebSocket. ResponsesEventType
names every event on OpenAI's streaming-events reference, with unknown
names riding in Other, and is_terminal and is_failure follow Python's
iterator. The framer adds a bulk decode_all and an SseBlockCodec that
keeps each event's wire bytes. An event the stream ends inside is dropped,
as the SSE spec and the openai SDK's decoder do.

ChatCompletionChunk no longer requires id, created or object, since
compatible hosts omit them.
2026-10-13 15:30:14 -07:00
yujonglee
05cf428761
feat(rust): settle EdenAI Messages reported cost (#45698)
* feat(edenai): settle native Messages reported cost

* test(rust): add live EdenAI Messages contract cases

* refactor(rust): settle call cost once as an ExecutionFacts variant

The route now decides Cost (Reported, CacheHit or Deferred) from the
result source and the provider's reported figure. The legacy callback
adapter only projects a Reported amount onto provider_reported_cost and
no longer writes response_cost, which the Python calculator derives

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(rust): keep EdenAI wire shaping independent of Invoke

* style(rust): format cost settlement imports

* fix(rust): trust provider costs without bypassing configured pricing

* fix(rust): clear untrusted response costs before spend settlement

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 17:39:08 -07:00
yujonglee
8db9037c90
test(rust): add Messages live provider harness (#45823)
* test(rust): add Messages live provider harness

* test(rust): support automatic Vertex tool selection
2026-10-12 17:39:08 -07:00
devin-ai-integration[bot]
a4382657fd
chore: drop Greptile from the PR template, AGENTS.md and CI (#46013)
* chore: drop Greptile from the PR template, AGENTS.md and CI

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: require bot reviews to finish before ticking the review checkbox

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:32:39 +00:00
devin-ai-integration[bot]
102619d47a
test: migrate proxy security, schema migration and MCP legacy tests (#45937)
Some checks are pending
unit / key-generation (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / assert-shard-coverage (push) Waiting to run
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / coverage (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* test: migrate proxy security, schema migration and MCP legacy tests

* test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp

* test: warm the baseline store before timing the locked projection

* test(e2e): hold the stack exclusively while the semantic filter flag is flipped

* test(mcp): scope the in-process proxy e2e fixtures to their module

The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite

* test(e2e): assert the semantic filter keeps listed tools instead of a ranking

Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k

* test(mcp): keep the in-process proxy e2e tests on one worker

loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out

* test(mcp): name the upstream requests that carried the wrong token

The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable

---------

Co-authored-by: yuneng <yuneng@berri.ai>
2026-10-11 16:17:24 -07:00
yujonglee
51164d9c32
feat(rust): add native Bedrock Claude Platform and Mantle Messages (#45908)
* feat(rust): support Claude Platform and Mantle Messages

* fix(ci): regenerate .greptile/files.json

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): preserve Mantle body betas

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): resolve Mantle default region

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): prefer regional endpoint host regions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(rust): format Bedrock regression tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 15:55:47 -07:00
yujonglee
a85db3c68b
feat(rust): add native OpenRouter Messages (#45910)
* feat(rust): add native OpenRouter Messages

* chore(greptile): regenerate files.json for OpenRouter Messages AGENTS.md

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 15:48:50 -07:00
yujonglee
3d86a51ab7
feat(rust): support Tencent Messages API (#45899)
* feat(rust): support Tencent Messages API

* fix(rust): ignore blank Tencent Messages credentials
2026-10-11 22:40:20 +00:00
yujonglee
462962c094
feat(rust): support MiniMax Messages API (#45896)
* feat(rust): support MiniMax Messages API on main

* fix(rust): ignore blank MiniMax Messages credentials

* fix(rust): enable MiniMax Messages SDK opt-in

* revert: remove MiniMax SDK rollout changes
2026-10-11 15:12:48 -07:00
yujonglee
dff8131f14
refactor(rust_bridge): route Python state reads through host modules (#45689)
* refactor(rust_bridge): group the Python modules Rust imports by name under rust_bridge/host

Rust reads Python-owned state (legacy callback registries, model capabilities) through
modules it imports by path. Put them in one folder with an AGENTS.md that states the
contract, and update the module path constants on the Rust side.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(rust_bridge): route Python state reads through host accessors

* docs(rust_bridge): point test_support at the moved host contract test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(rust_bridge): move secret manager, transport and timeout accessors under host

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(greptile): register rust_bridge/host AGENTS.md in files.json

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 14:10:16 -07:00
devin-ai-integration[bot]
4c532a5713
feat(decisions): accept decisions as a model mode alongside evaluation (#45474)
* refactor(decisions): rename the evaluation model mode to decisions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): canonicalize the health check mode without rebinding

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): replace canonical_model_mode with is_decisions_model_mode

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): preserve evaluation mode compatibility

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: drop the rebuilt dashboard bundle from this PR

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(dashboard): format decision mode helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 19:18:53 -07:00
yujonglee
705bd84523
refactor(rust): share provider constants via llms-types and restructure AGENTS.md (#45850)
* wip

* refactor(rust): share Bedrock paths and Converse/Invoke wire types via llms-types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): share Vertex AI locations, base URL and rawPredict paths via llms-types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): give every llms-types format and provider its own folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): keep single-adapter projections in llms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): restructure llms and llms-types AGENTS.md into rules and references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): keep provider mod.rs files as re-export entrypoints

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): split llms and llms-types AGENTS.md rules into topic sections

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust): move llms-types tests inline and default the workspace to inline tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust): move llms tests inline beside the code they cover

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): cover every provider format with AGENTS.md and keep each upstream URL in one file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: drop the check_rust_agents_md step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: regenerate greptile files.json for the new AGENTS.md files

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): make every llms-types mod.rs a re-export entrypoint and drop check_rust_agents_md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(rust): register Messages providers in one match and drop the nested AGENTS.md list

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:53:35 -07:00
yujonglee
6830a5c338
fix(python-bridge): preserve callback keyword deletions in native requests (#45692)
* fix(messages): preserve callback keyword deletions in native requests

* fix(python-bridge): share hook-aware request view across native route hosts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 02:36:49 +00:00
Mateo Wang
3069a467d0
docs(agents): ban semicolons and colons in human-facing text (#45684)
* docs(agents): ban semicolons and colons in human-facing text

Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."

* docs(agents): rewrite semicolons in nested AGENTS.md files

* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines

* docs(agents): point at the PR template path that exists

* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
2026-10-09 16:36:32 -07:00
yujonglee
4cb47edd2a
feat(rust): add the DeepSeek Anthropic Messages config (#45614)
* feat(rust): add the DeepSeek Anthropic Messages config

* refactor(rust): drop the tool discriminator without mutation and use rstest in DeepSeek tests
2026-10-09 16:29:46 -07:00
yujonglee
a0b3c46cf0
feat(rust): add the Vertex AI Anthropic Messages config (#45626)
* feat(rust): add the Vertex AI Anthropic Messages config

* fix(rust): normalize Vertex system turns the way Python does

Vertex uses the shared normalize_system_role_messages: a leading run is
hoisted, a later turn stays or is converted in place by the model's
supports_mid_conversation_system, and billing blocks are filtered after the
hoist so one inside a system-role message no longer reaches Vertex.

* fix(rust_bridge): keep non-Claude Vertex Messages calls on Python

The catalog admits every vertex_ai Messages call, but Rust serves only Claude
there and reports anything else as a terminal InvalidProvider instead of a
decline. The dispatcher now bypasses Rust for those models, as
get_provider_anthropic_messages_config does.

* refactor(rust): template the Vertex location and project errors

The location failure is the shared InvalidType detail and the missing project
is MissingSetting with the provider, setting, param and variable name, so no
sentence is spelled at the return site.

* test(rust): use rstest in the Vertex AI Messages tests

* refactor(rust): derive the missing Vertex project error from its spec
2026-10-09 16:29:45 -07:00
yujonglee
ee295aea68
refactor(rust): give Messages configs typed litellm params (#45611)
* refactor(rust): give Messages configs typed litellm params

* feat(rust): port _normalize_system_role_messages for Messages hosts

mid_conversation_system mirrors Python's module function for function: a
leading run of system turns is hoisted into the top-level system, a later
turn stays in place when the model map flags supports_mid_conversation_system
and otherwise becomes a user turn where it was, never between a tool_use and
its tool_result, and billing blocks are stripped from the hoisted system. The
capability joins MessagesModelCapabilities and the bridge projection. Azure AI
uses it in place of fold_system_role_messages, which hoisted every turn.

* fix(rust): keep the Bedrock runtime endpoint behind a blank api_base

* refactor(rust): import LitellmParams from its crate instead of a re-export

* refactor(rust): template the Messages request decode errors

Both decode failures go through ErrorDetail::invalid with the subject and the
serde error as source instead of a preformatted sentence.

* refactor(rust_bridge): read the module globals the param specs name

The Messages host no longer keeps its own list of litellm globals; a spec
whose spellings the call leaves out names the global to read, so the next
family that has one needs no host change.

* refactor(rust): give each route lookup Python's implicit None

ProviderConfigManager's per-route methods list only the providers the route
serves and fall through to None; the Rust lookups now do the same, so a new
LlmProviders variant touches only the route that serves it.
2026-10-09 16:29:45 -07:00
yujonglee
3d2abf9a44
feat(rust): one LitellmParams type for a config and a caller (#45642)
litellm-router-types mirrors litellm/types/router.py: LitellmParams holds the
model, the credentials, one flattened group per credential family from
auth-types (aws, vertex), the deployment settings a config spells, and every
other key in extra, as Python's extra="allow" keeps it. Config parses it
directly, so config::LiteLlmParams and its untyped additional_fields are gone
and a mistyped provider param is a parse error. fields() and specs() derive
from the families' param specs for hosts that project kwargs and fold module
globals. Spelled<T> is the one untagged shape for a typed value or the text a
config spells, organization takes the list Python's router expands, and the
callback shorthand keeps a value-level one-or-many. No credentials wrapper
group: serde's flatten only consumes keys for struct-shaped children.
2026-10-09 16:29:45 -07:00
yujonglee
04a95f60c1
feat(rust): type the Vertex AI connection params and where Python reads them (#45641)
VertexParams in auth-types holds the vertex_* fields of CredentialLiteLLMParams
in both spellings, with one ParamSpec per setting: the wire names, the litellm
module global VertexBase.get_vertex_ai_* consults, and the environment names.
A credential deserializes from text or the JSON object a config spells, an
empty object counts as absent, and both credential fields are redacted in
Debug. auth-gcp is split so lib.rs is the entrypoint (config, auth,
constants), its lookups resolve through the specs, secret_names() derives
from them, and get_vertex_ai_project_from_credentials reads a service
account's project_id for hosts that need it before a token exchange. The OCR
host folds the module globals in by spec at projection, so OcrSettings no
longer carries vertex_project and vertex_location.
2026-10-09 16:29:44 -07:00
yujonglee
0410abea8b
feat(rust): type the AWS connection params and where Python reads them (#45610)
AwsParams in auth-types holds the aws_* fields of CredentialLiteLLMParams
with one ParamSpec per field: its wire name and the environment names it
falls back to. fields() and secret_names() derive from the specs, the AWS
helpers take the struct instead of a map, aws_auth_config and the region
resolution read through the specs, and a missing region is
AwsParams::REGION.missing("AWS"), whose message is rendered from the spec.
Bedrock Converse, Bedrock transcription, Textract OCR and the Bedrock
Messages config build the struct at their one untyped boundary.
2026-10-09 16:29:44 -07:00
moe-berri
22ad3c5fff
refactor(lens): connect LiteLLM to the independent Lens service (#45529)
* feat(lens): consume the independent Lens service and shared UI

* fix(lens): keep embedded setup stable and vendor UI before builds

* chore(lens): pin embedded UI provenance to published Lens source

* fix(lens): complete embedded UI extraction across CI builds

* docs(lens): record latest main and removal-boundary validation

* fix(lens): wire external chart credentials and ingestion modes

* test(lens): cover adapter failures and fix extraction CI

* docs(lens): refresh bundled chart setup instructions

* test(lens): restore gateway adapter test package marker

* build(lens): pin clean-cut UI and supported storage chart

* test(ui): await guardrail scope menu before checking options

* perf(lens): omit unused trace-team lookup from product requests

* docs(lens): record final-image adapter permission checks

* test(lens): assert delegated investigation authorization

* refactor(lens): remove copied trace runtime and preserve spend logging

* test(lens): verify independent gateway image upgrades and outages

* test(lens): pin the chart qualification database image

* fix(lens): update embedded UI metadata filters

* test: wait for Lens chart pod identity convergence

* fix(e2e): select Lens chart pods by deployment ownership

* fix(e2e): select Helm migration ownership for Lens upgrades

* test(lens): retain migration Jobs through rollout assertions

* docs(helm): require existing database for migration hooks

* test(lens): qualify release boundaries with embedded navigation update

* docs(lens): record browser qualification for embedded tabs

* fix(lens): remove unrelated CI and guardrail test changes

* test(lens): give qualification tenants unique key aliases

* feat(lens): embed the latest canonical Lens interface

* fix(lens): qualify split inference through the gateway

* fix(lens): preserve scoped auth and isolate delegated credentials

* chore(lens): remove unrelated documentation and lint changes

* fix(lens): bound concurrent forwarding buffers through response delivery

* fix(lens): preserve OAuth2 dispatch without inference policies

* chore(lens): adopt current shared onboarding UI

* fix(lens): refresh shared setup UI and review context

* fix(lens): bound uploads before gateway authentication

* docs(lens): remove extraction evidence from the gateway repo

* docs(lens): refresh shared setup prompt for paired releases
2026-10-09 16:17:05 -07:00
moe-berri
3ebc71be2a
fix(rust): update serde_with for serialization advisory (#45498) 2026-10-09 13:31:41 -07:00
yuneng-jiang
3250ccc802
test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554)
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files
2026-10-09 09:48:10 -07:00
yujonglee
1bc85fad40
refactor(python-bridge): ship a signature base and read the resolved call (#45450)
* refactor(python-bridge): ship a signature base and read the resolved call

NativeCall carries base (positionals by name plus signature defaults) instead
of the fully bound dict. resolved lays kwargs over base, which is what bound
held, so every pre-hook read and every route host keeps seeing the same values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(bedrock): build the transcription NativeCall with an empty base

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dispatch): read the resolved call instead of bound

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(host-python): rename effective to effective_py_args and note the shallow copy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 17:20:29 -07:00
yujonglee
abee1c14e7
refactor(python-bridge): derive NativeCall extraction (#45449)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:20:28 -07:00
devin-ai-integration[bot]
3822947b0d
fix(rust): add the inline-tools-2026-09-15 beta to AnthropicBeta (#45439)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 15:25:17 -07:00
yujonglee
0a98c7be10
refactor(rust): derive strum VariantArray and string conversions (#45434)
* refactor(rust): derive strum VariantArray and string conversions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(rust): use strum conversions directly with explicit spellings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust): use rstest values for key management systems

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 21:24:49 +00:00
yujonglee
0721cffab2
refactor(python-bridge): take NativeCall directly and fold routes into per-route folders (#45413)
* refactor(python-bridge): take NativeCall directly and fold routes into per-route folders

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(python-bridge): use rstest for updated tests and keep embedding's stub parameter name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:48:28 -07:00
yujonglee
9505af0639
feat(rust): add the Anthropic beta header policy (#45419)
* feat(rust): add the Anthropic beta header policy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rust): resolve known betas held as Other in AnthropicBeta::on

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust): use named rstest cases for Other beta resolution

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(rust): make the rstest named-case rule explicit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust): move BetaPolicy tests to the public API test crate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 20:22:58 +00:00
nate-berri
e843cb7aca
fix(rust): path-qualify attribute aliases so rust-analyzer resolves them (#45416)
rust-analyzer cannot resolve the bare alias names emitted by
macro_rules_attribute::apply, so every aliased type was invisible to it
(no go-to-definition or find-references). Referencing the aliases through
crate:: resolves them via the re-export that attribute_alias! generates.

Co-authored-by: Nate Armstrong <narmstrong@Nates-MBP.localdomain>
2026-10-08 11:51:58 -07:00
nate-berri
a81f507bca
docs(rust): add ADRs for the Rust core (#45205)
* docs(rust): add ADR folder with ADR 000 and ADR 001 scaffold

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): add ADR 000 and ADR 001 on typing requests inside Rust

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): list four ADR sections in ADR 000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): move ADR process doc from adr_000 to AGENTS.md

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 10:21:56 -07:00
yujonglee
35b0992533
feat(rust): add standalone typed LLM wire contracts (#45188)
* refactor(rust): add typed LLM wire contracts to llms-types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(rust): drop blank lines left in new wire types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(rust): complete additive wire type contracts

* feat(rust): complete additive nested wire contracts

* feat(rust): give Messages content blocks concrete per-block contracts

Replace the shared all-optional ContentBlockPayload with one struct per
documented block and nested tagged unions for server tool results, so
required fields from the official Messages reference are enforced.

Make CustomTool a plain struct with required name and input_schema, add
MessagesToolParam plus the toolset and undated tool-search tags, model
MiniMax media sources as a tagged union, require documented fields on
chat audio, image URLs, and logprobs, and rename the new block enum to
MessagesContentPart so it no longer shadows the streaming struct.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust): tighten new wire contracts against the official references

Give each Messages builtin tool a concrete contract with its documented
required fields, fixed names, and typed allowed callers in place of the
shared all-optional ToolDefinition bag. Make MCP tool result content and
fallback triggers optional as the request params document, and narrow
tool search references, web fetch content, and bash results to their
documented block types.

Keep unmodeled Responses output items and new usage iteration and stop
detail types as Recognized values instead of rejecting the whole payload,
accept find_in_page and optional open_page URLs, preserve JSON Schema
property order, and move prompt_cache_breakpoint to Chat Completions
content parts where OpenAI documents it.

Drop OCR table and key/value types that only matched Azure Document
Intelligence and would reject other providers' normalized passthrough,
along with unsourced bounding box and applied edit fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(rust): type every documented Responses output item and close wire gaps

Add the 18 remaining documented Responses output item variants with
concrete nested actions, outputs, and safety checks, and add the
documented namespace, async, caller, and image generation fields to the
existing items.

Restrict MCP tool result content to a string or text blocks, model the
Messages diagnostics cache-miss union and its request param with null
distinct from absent, and accept draft-07 tuple items and prefixItems in
JSON Schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust): keep Messages contract assertions in test bodies

Compare built-in tool decodes against fully built expected values and
split the required-field rejections per type, so no case carries its
assertions in a callback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rust): cite the Mistral OCR reference beside the OCR wire types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:27:12 +00:00
tin-berri
057034d0d3
fix(lens): refresh runs until gateway costs are complete (#45105) 2026-10-08 00:24:41 -07:00
ishaan-berri
03ba79d261
feat(lens): show end-user feedback on traces, stored in ClickHouse (#45171)
* feat(lens): add lens_feedback ClickHouse table for human trace scores

ReplacingMergeTree(UpdatedAt, IsDeleted) keyed like agent_traces_by_key so
feedback joins traces in sort order, with a bloom filter on TraceId, a score
CHECK, and the same retention as traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add scoped ClickHouse reads for trace feedback

feedback_target resolves a visible trace's team, key and trace_ref before a
write; feedback lists the latest live entry per author; feedback_summary
returns count, average and lowest score for a batch of traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lens): pin the Claude session to trace id hash shared with feedback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): expose feedback queries to the Python trace bridge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add typed feedback models and ClickHouse feedback store

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add /lens/feedback API for 0-10 human scores on traces

PUT saves or replaces the caller's score and comment, GET lists every
entry, DELETE removes the caller's, POST /summary feeds the trace list.
Callers can target a trace_id or a Claude Code session_id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): add typed feedback calls to the traces API

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): flag rated traces and filter the list by feedback

A Feedback column shows each run's average score and rating count, marked
red when any score is 4 or below, and a filter narrows to rated or
low-score runs through the URL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show and edit human feedback inside a trace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): let app keys post end-user feedback on their own traces

Any key can now PUT/DELETE feedback on traces its team or key sent, naming
the end user in an optional user field. Reading feedback stays with Lens
admins

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show end-user feedback first in a run and flag low-scored rows

Replace the edit popover and feedback filter with a read-only view: the
list shows each run's score in a fixed-width badge and tints the row red
when a user scored it 4 or below, and opening a run shows what users said
with their score right under the header

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:14:30 -07:00
ishaan-berri
b85756104f
feat(lens): show agent, user and slack thread first in the run header (#45261)
* feat(lens): add who started the run to RunSource

* feat(lens): carry agent.source.user on trace span rows

* feat(lens): resolve the run source user from the span row

* feat(lens): read agent.source.user in trace_spans query

* feat(lens): read agent.source.user in trace_page_spans query

* feat(lens): read agent.source.user in trace_span_batch query

* feat(lens): read agent.source.user in trace_list_span_batch query

* feat(lens): decode the source user from clickhouse span rows

* test(lens): add source user to trace cache test rows

* test(lens): add source user to trace cache read fixtures

* test(lens): add source user to trace cache snapshot fixtures

* test(lens): add source user to capture fixture rows

* test(lens): round-trip source user on span row contract

* test(lens): cover run source user resolution

* chore(lens): regenerate python trace types with source user

* chore(lens): regenerate trace json schema with source user

* chore(lens): regenerate trace page json schema with source user

* chore(ui): add source user to api types

* feat(lens): add run user chip and slack thread chip

* feat(lens): put agent, user and thread on the first row of the run header

* test(lens): cover the run header identity row and compact totals

* feat(lens): start demo runs from a slack thread
2026-10-07 20:01:55 -07:00
ishaan-berri
b970e412d9
perf(lens): faster trace opens and list pages at scale (#45228)
* perf(lens): read trace spans in larger batches

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* perf(lens): evaluate the trace list page once per identity lookup

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(lens): queue trace reads instead of rejecting past 8 in flight

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 01:44:19 +00:00
moe-berri
fdb1ce8d78
fix(lens): preserve numeric tags and report OTLP error codes (#45206) 2026-10-08 00:31:48 +00:00
yujonglee
e638540de9
refactor(rust-bridge): share field and response marshaling (#45187)
* refactor(rust-bridge): share field and response marshaling

* test(rust-bridge): isolate response factory test modules per case

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs(rust-bridge): document shared marshaling helpers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 17:22:23 -07:00