Adds a spend writer observer that records every gateway request in the
proxy's own Postgres tables with the same statements the Python proxy
uses: spend logs, key/user/team/team member/end user spend, and the
daily user, team and end-user activity tables. Writes are batched on the
proxy's timers and are additive, so replicas and Python proxies can
write side by side. The database pool is now shared between virtual
keys and the writer.
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(gateway): hooks and observers around inference requests, with usage and cost
Adds the litellm-gateway-hooks crate: request facts built only when something
reads them, on-path hooks that declare what they read and what they may
change and run in config order, off-path observers on bounded queues, a
usage tap for SSE and JSON responses, and per-deployment prices from the
model cost map overlaid by config prices. The messages and chat handlers run
through it when anything is installed; with nothing installed requests take
the old path.
The key store now also returns the key alias, user, team alias and key and
team metadata, and router attempts carry the deployment id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* fix(model-catalog): read token counts the cost map writes as whole floats
The published cost map now has prompt_cache_min_tokens: 4096.0, which the
typed u64 fields rejected, so the gateway dropped the whole map at startup
and priced every request at zero. Counts accept any whole non-negative
number, as Python reads any JSON number, and a test keeps the checked-in
map parsing.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gateway replicas now share state through the coordination Redis the
Python proxy uses (general_settings.coordination_redis, a redis
response cache, or REDIS_*):
- A deployment one replica cools down is written under Python's
deployment:<id>:cooldown key and applied by every other replica's
background poll (one MGET a second), so the fleet stops sending to a
failing deployment together. allowed_fails counts failures across
replicas, as Python's v1 rule does; the v2 failure-rate rule stays
per replica, as in Python. Deployments are named by model_info.id, or
an id derived from the config that is the same on every replica.
- The virtual key cache follows the proxy's
litellm_proxy.auth_cache_invalidation channel, so a key blocked,
deleted or edited through the admin API (or its team or user) is
re-read on every replica at once instead of after up to 60 s.
general_settings.user_api_key_cache_ttl sets the key TTL.
Nothing is added to the path of a successful request. Without Redis,
or while it is down, each replica keeps its own state as before.
Redis Cluster and Sentinel are refused at startup for now.
The gateway now authenticates LiteLLM virtual keys as well as the master key. Keys are looked up by
sha256 hash in the proxy's LiteLLM_VerificationToken table (general_settings.database_url or
DATABASE_URL), honoring expires, blocked, models and allowed_routes, and are read from
Authorization: Bearer, x-api-key or api-key
The same query LEFT JOINs the key's team. A blocked or missing team refuses the key and the team's
models narrow the key's, as in the proxy's can_key_call_model and can_team_access_model. The key's
team also stays in its grant: a deployment whose model_info.team_id names a team serves only that
team's keys, so routing and retries skip it for any other virtual key, as the proxy's
filter_team_based_models does. The master key routes as before
Keys with budgets pass without spend tracking: the gateway does not record spend for its own
requests, so a key or team gets the proxy's budget-exceeded error only once the spend the Python
proxy has stored reaches its max_budget
Every other restriction the proxy enforces and this gateway does not yet fails closed. The query also
joins the key's team membership, user and budget tables and reduces each restriction column to
whether it is set, and a key carrying one (rate limits and max_parallel_requests, including those in
a budget table or per model in model_max_budget, a membership's allowed_models, projects,
organizations, agents, guardrails, tags, tool allowlists, enforced params and the like, on the key,
its team, its membership or its user) gets a 401 saying that kind of key is not supported by this
gateway yet
Lookups sit behind a moka cache (60 s for found keys, 5 s for unknown ones) with single-flight
misses. When the database fails, recently seen keys keep working from a stale copy and store reads
pause briefly; the master key never needs the database. Auth failures return the proxy's
ProxyException JSON, including 403 key_model_access_denied for a model outside the key's list
The load-test README documents running with virtual keys and x-api-key, and key-churn asks for the
4 model groups genconfig creates by default
Deployments sharing a model name now form a model group instead of
overwriting each other. router_settings drive selection (simple-shuffle
weighted by weight, rpm or tpm, or least-busy), num_retries on a
different deployment for connect errors, 408, 429 and 5xx received
before any byte reaches the client, and allowed_fails / cooldown_time
cooldown with fallback to cooling deployments when no other is left.
Per-deployment health is lock-free atomics. /v1/messages (Anthropic and
Bedrock deployments) and /v1/chat/completions (native OpenAI passthrough
and the core chat route) route through it, and the metrics label records
the deployment each attempt used. Other endpoints pick one deployment
per request without retries.
Implements the Bedrock InvokeModel request shaping that returned 501: anthropic_version, the top-level allowlist, model/stream moved to the URL, header and feature betas in the body's anthropic_beta, clear_thinking, cache_control, structured output and tool fixes in Python's order. Streams decode the AWS event stream into Anthropic SSE. genconfig and Terraform now default to half Bedrock deployments signed with SigV4.
Add the openai provider to /v1/chat/completions. The caller's body is passed
through with only the model renamed and LiteLLM-owned keys removed, and the
provider's JSON or SSE bytes are relayed as received. Streams have no total
timeout, only a per-read idle limit, so 30+ minute generations are not cut off.
openai_like and the translated providers keep their existing path.
Adds /health/liveliness, /health/liveness and /health/readiness with the Python proxy's bodies,
a /metrics endpoint (request count, latency and time to first byte histograms by route, model
group and status, in-flight gauge, upstream failures by deployment) and a drain on SIGTERM or
SIGINT: readiness turns 503, the listener closes after LITELLM_UNREADY_DELAY_SECONDS and open
requests and streams get up to LITELLM_DRAIN_SECONDS (default 900) to finish
Adds litellm-rust/loadtest: a mock LLM (OpenAI chat, Anthropic messages, Bedrock invoke with
event-stream framing) that takes its timing from the prompt, an open-loop load generator with
workloads covering 10k to 1M token contexts, 6 minute non-streaming calls and 30 minute streams,
a config and virtual key generator, a summarizer and a /proc sampler, plus Terraform for an AWS
rig with load generators, gateway and mocks on separate machines
The gateway sets TCP_NODELAY on accepted sockets, removing a ~40 ms stall per streamed response,
and its Docker build takes LTO and CODEGEN_UNITS build args for faster iteration builds
* refactor(rust-bridge): let each route host pick the body params
The native routes projected request body params from a name list, hard-coded in
Rust for Messages and read from `PARAMETERS` for Chat Completions and Responses.
Each route host now answers `body_params(request)` over the keywords the call would
send, so the Python side owns which params reach the provider. No behavior change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-host): use read-only body parameter types
Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): move the route host modules under host/routes
Each native route's Python half lived in its own `<route>/route_host.py`, apart
from the shared host modules Rust also imports by name. They now sit together in
`litellm/rust_bridge/host/routes/<route>.py`, with their unit tests moved to the
mirrored path. No behavior change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-host): reference messages route alias
Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Responses route now resolves openrouter alongside openai. The
OpenRouter config reuses the OpenAI request and response transforms and
owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and
base URL (OPENROUTER_API_BASE). Wire model ids follow Python's
get_llm_provider, including the openrouter/auto native id with an
explicit provider. The Python route host no longer hands litellm.openai_key
to OpenRouter.
The OpenRouter live test lives in tests/live/openrouter.rs and returns
early when the provider key is not exported, so cargo test stays offline
by default instead of hiding it behind --ignored.
Co-authored-by: nate-berri <nate@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(vertex): report an unreadable credentials file as a read failure
Tell a path apart from inline JSON by shape, not os.path.exists().
Fixes#40166
* test(vertex): narrow pytest.raises to the expected message
* fix(vertex): log the credentials path instead of returning it to the caller
* refactor(vertex): log the credential source decision where it is made
* fix(vertex): handle paths open() rejects with a bare ValueError
* fix(vertex): redact the credentials path before it reaches the log
* test(vertex): cover the brace fallback and non-text credentials file
* test(vertex): exclude the unreachable exhaustiveness guard from coverage
* refactor(vertex): read the failure's fields off the matched object
CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.
* fix(vertex): accept a read-only mapping in extract_aws_params
* fix(vertex): keep encoded credentials out of the log and cover the legacy route
* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust
* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file
* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message
* test(vertex): integration cells for credentials file and inline value failures
* test(vertex): drive the Rust messages cell through Vertex Claude
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust-bridge): return an unstarted Execution from native entrypoints and drive it in Python
Rust's run_call built Python streams itself and imported
litellm.rust_bridge.streams by name, so sync calls were driven in Rust
while async calls came back as a Rust-made coroutine. Every native
inference entrypoint now returns one unstarted Execution and
public_execution.py drives it with lifecycle.run or lifecycle.drive,
wrapping streams and _hidden_params on the Python side. Rust only
imports the lifecycle step classes through StepTypes, and
python-bridge/src/lifecycle.rs, PythonLifecycle and streams.py are gone.
The adapter trusts the _native.pyi contract with a single cast instead
of a per-request runtime_checkable isinstance. Binding tests assert the
adapter starts the execution and returns its completed value rather
than checking callable identity.
* refactor(rust-bridge): fold lifecycle and public_execution into one execution module
The public stream wrappers only forwarded to the lifecycle streams and added the
header dict, so the streams now carry _hidden_params themselves. lifecycle.py no
longer held any call lifecycle, just the Execution driver, so it is renamed to
execution.py and absorbs the synchronous and asynchronous adapters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-bridge): keep execution.py free of third-party imports
The host-python tests load execution.py into a bare interpreter, so the
pydantic header validation broke them. The stream head comes from the
binding's encode_stream_head, which always builds a str-keyed dict
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(host-python): preserve pre-async hook coverage after execution handoff
* style(rust-bridge): sort litellm_host_python imports
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): drive unstarted native calls through execution in the split lifecycle tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* fix(rust-bridge): mark execution TypeVar as Final
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): cover stream close, read failure, and sync suspension in execution
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): drive executions with generator-style send, throw and close
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): name stream steps StreamOpen and StreamChunk and reject out-of-phase steps
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): drive with ExitStack and require the stream factory
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(host-python): give the hook_chain lifecycle binding the stream factory
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(host-python): run the Python driver as one async body
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(host-python): give the driver a PythonCaller with named suspensions
PythonTask becomes PythonCaller with await_python, open_stream and yield_chunk, so the call body no longer builds Suspension values or maps stream replies itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(host-python): collect host failures in a thiserror Error enum
missing_state() and the inline PyRuntimeError/PyTypeError messages become Error variants with one From<Error> for PyErr, and InvokeError derives thiserror so From<PyErr> comes from #[from]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(host-python): note the canonical HostError import alias in AGENTS.md
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: lower aiohttp keepalive timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): restore the 15s keep-alive default and apply it to the Rust client
15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it.
* test(integration): cover upstream connections dropped after the keep-alive idle window
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Chat Completions, Messages and Responses each get a streaming module with
the same three names, event_stream, decode and encode, built on a shared
SSE helper in base_llm/sse.rs that owns the [DONE] marker. Callers use the
module path instead of vendor-prefixed names. Encoding matches what the
Python proxy writes to clients: Messages frames carry an event line, the
OpenAI formats are bare data frames.
Responses gets one event type for SSE and WebSocket. ResponsesEventType
names every event on OpenAI's streaming-events reference, with unknown
names riding in Other, and is_terminal and is_failure follow Python's
iterator. The framer adds a bulk decode_all and an SseBlockCodec that
keeps each event's wire bytes. An event the stream ends inside is dropped,
as the SSE spec and the openai SDK's decoder do.
ChatCompletionChunk no longer requires id, created or object, since
compatible hosts omit them.
* feat(edenai): settle native Messages reported cost
* test(rust): add live EdenAI Messages contract cases
* refactor(rust): settle call cost once as an ExecutionFacts variant
The route now decides Cost (Reported, CacheHit or Deferred) from the
result source and the provider's reported figure. The legacy callback
adapter only projects a Reported amount onto provider_reported_cost and
no longer writes response_cost, which the Python calculator derives
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust): keep EdenAI wire shaping independent of Invoke
* style(rust): format cost settlement imports
* fix(rust): trust provider costs without bypassing configured pricing
* fix(rust): clear untrusted response costs before spend settlement
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* chore: drop Greptile from the PR template, AGENTS.md and CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: require bot reviews to finish before ticking the review checkbox
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: migrate proxy security, schema migration and MCP legacy tests
* test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp
* test: warm the baseline store before timing the locked projection
* test(e2e): hold the stack exclusively while the semantic filter flag is flipped
* test(mcp): scope the in-process proxy e2e fixtures to their module
The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite
* test(e2e): assert the semantic filter keeps listed tools instead of a ranking
Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k
* test(mcp): keep the in-process proxy e2e tests on one worker
loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out
* test(mcp): name the upstream requests that carried the wrong token
The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(rust): support Claude Platform and Mantle Messages
* fix(ci): regenerate .greptile/files.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): preserve Mantle body betas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): resolve Mantle default region
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): prefer regional endpoint host regions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(rust): format Bedrock regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust_bridge): group the Python modules Rust imports by name under rust_bridge/host
Rust reads Python-owned state (legacy callback registries, model capabilities) through
modules it imports by path. Put them in one folder with an AGENTS.md that states the
contract, and update the module path constants on the Rust side.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust_bridge): route Python state reads through host accessors
* docs(rust_bridge): point test_support at the moved host contract test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust_bridge): move secret manager, transport and timeout accessors under host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(greptile): register rust_bridge/host AGENTS.md in files.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the evaluation model mode to decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): canonicalize the health check mode without rebinding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): replace canonical_model_mode with is_decisions_model_mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): preserve evaluation mode compatibility
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop the rebuilt dashboard bundle from this PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(dashboard): format decision mode helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wip
* refactor(rust): share Bedrock paths and Converse/Invoke wire types via llms-types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): share Vertex AI locations, base URL and rawPredict paths via llms-types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): give every llms-types format and provider its own folder
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): keep single-adapter projections in llms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): restructure llms and llms-types AGENTS.md into rules and references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): keep provider mod.rs files as re-export entrypoints
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): split llms and llms-types AGENTS.md rules into topic sections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust): move llms-types tests inline and default the workspace to inline tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust): move llms tests inline beside the code they cover
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): cover every provider format with AGENTS.md and keep each upstream URL in one file
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: drop the check_rust_agents_md step
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: regenerate greptile files.json for the new AGENTS.md files
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): make every llms-types mod.rs a re-export entrypoint and drop check_rust_agents_md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust): register Messages providers in one match and drop the nested AGENTS.md list
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(agents): ban semicolons and colons in human-facing text
Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."
* docs(agents): rewrite semicolons in nested AGENTS.md files
* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines
* docs(agents): point at the PR template path that exists
* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
* feat(rust): add the DeepSeek Anthropic Messages config
* refactor(rust): drop the tool discriminator without mutation and use rstest in DeepSeek tests
* feat(rust): add the Vertex AI Anthropic Messages config
* fix(rust): normalize Vertex system turns the way Python does
Vertex uses the shared normalize_system_role_messages: a leading run is
hoisted, a later turn stays or is converted in place by the model's
supports_mid_conversation_system, and billing blocks are filtered after the
hoist so one inside a system-role message no longer reaches Vertex.
* fix(rust_bridge): keep non-Claude Vertex Messages calls on Python
The catalog admits every vertex_ai Messages call, but Rust serves only Claude
there and reports anything else as a terminal InvalidProvider instead of a
decline. The dispatcher now bypasses Rust for those models, as
get_provider_anthropic_messages_config does.
* refactor(rust): template the Vertex location and project errors
The location failure is the shared InvalidType detail and the missing project
is MissingSetting with the provider, setting, param and variable name, so no
sentence is spelled at the return site.
* test(rust): use rstest in the Vertex AI Messages tests
* refactor(rust): derive the missing Vertex project error from its spec
* refactor(rust): give Messages configs typed litellm params
* feat(rust): port _normalize_system_role_messages for Messages hosts
mid_conversation_system mirrors Python's module function for function: a
leading run of system turns is hoisted into the top-level system, a later
turn stays in place when the model map flags supports_mid_conversation_system
and otherwise becomes a user turn where it was, never between a tool_use and
its tool_result, and billing blocks are stripped from the hoisted system. The
capability joins MessagesModelCapabilities and the bridge projection. Azure AI
uses it in place of fold_system_role_messages, which hoisted every turn.
* fix(rust): keep the Bedrock runtime endpoint behind a blank api_base
* refactor(rust): import LitellmParams from its crate instead of a re-export
* refactor(rust): template the Messages request decode errors
Both decode failures go through ErrorDetail::invalid with the subject and the
serde error as source instead of a preformatted sentence.
* refactor(rust_bridge): read the module globals the param specs name
The Messages host no longer keeps its own list of litellm globals; a spec
whose spellings the call leaves out names the global to read, so the next
family that has one needs no host change.
* refactor(rust): give each route lookup Python's implicit None
ProviderConfigManager's per-route methods list only the providers the route
serves and fall through to None; the Rust lookups now do the same, so a new
LlmProviders variant touches only the route that serves it.
litellm-router-types mirrors litellm/types/router.py: LitellmParams holds the
model, the credentials, one flattened group per credential family from
auth-types (aws, vertex), the deployment settings a config spells, and every
other key in extra, as Python's extra="allow" keeps it. Config parses it
directly, so config::LiteLlmParams and its untyped additional_fields are gone
and a mistyped provider param is a parse error. fields() and specs() derive
from the families' param specs for hosts that project kwargs and fold module
globals. Spelled<T> is the one untagged shape for a typed value or the text a
config spells, organization takes the list Python's router expands, and the
callback shorthand keeps a value-level one-or-many. No credentials wrapper
group: serde's flatten only consumes keys for struct-shaped children.
VertexParams in auth-types holds the vertex_* fields of CredentialLiteLLMParams
in both spellings, with one ParamSpec per setting: the wire names, the litellm
module global VertexBase.get_vertex_ai_* consults, and the environment names.
A credential deserializes from text or the JSON object a config spells, an
empty object counts as absent, and both credential fields are redacted in
Debug. auth-gcp is split so lib.rs is the entrypoint (config, auth,
constants), its lookups resolve through the specs, secret_names() derives
from them, and get_vertex_ai_project_from_credentials reads a service
account's project_id for hosts that need it before a token exchange. The OCR
host folds the module globals in by spec at projection, so OcrSettings no
longer carries vertex_project and vertex_location.
AwsParams in auth-types holds the aws_* fields of CredentialLiteLLMParams
with one ParamSpec per field: its wire name and the environment names it
falls back to. fields() and secret_names() derive from the specs, the AWS
helpers take the struct instead of a map, aws_auth_config and the region
resolution read through the specs, and a missing region is
AwsParams::REGION.missing("AWS"), whose message is rendered from the spec.
Bedrock Converse, Bedrock transcription, Textract OCR and the Bedrock
Messages config build the struct at their one untyped boundary.
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files
* refactor(python-bridge): ship a signature base and read the resolved call
NativeCall carries base (positionals by name plus signature defaults) instead
of the fully bound dict. resolved lays kwargs over base, which is what bound
held, so every pre-hook read and every route host keeps seeing the same values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): build the transcription NativeCall with an empty base
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(dispatch): read the resolved call instead of bound
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(host-python): rename effective to effective_py_args and note the shallow copy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): derive strum VariantArray and string conversions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): use strum conversions directly with explicit spellings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): use rstest values for key management systems
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(python-bridge): take NativeCall directly and fold routes into per-route folders
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(python-bridge): use rstest for updated tests and keep embedding's stub parameter name
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
