This commit is contained in:
yujonglee 2026-11-06 15:51:12 -05:00 • committed by GitHub
commit 73ec83b75a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 428 additions and 1 deletions

View file

@ -0,0 +1,98 @@
---
name: rust-hosts
description: Decide where code goes between the shared Rust crates, python-bridge and Python for the python-sdk, python-proxy and rust-proxy hosts, design a public API Rust can implement, move a module into a shared crate, or prove parity between old and new code. Use when adding a pyclass or host call in python-bridge, porting a module, refactoring a module behind an entry point, or touching cache, Redis, database, secret manager, router, model catalog, HTTP client or callback code on the Rust path
metadata:
internal: true
---
# Rust hosts
One implementation lives in shared Rust crates, and each host runs it. python-sdk on the native extension is a permanent host, so design for it as an end state, not a step on the way to rust-proxy
These are migration rules and target boundaries, not a claim that every subsystem is already ported. `crates/` paths are relative to `litellm-rust/`. A `TBD` is an unresolved decision, not permission to invent a new seam
## The three hosts
Read the page for each host the change runs on
- [python-sdk](references/python-sdk.md)
- Permanent. Runs the core through `python-bridge`, and may ask Python through host calls
- [python-proxy](references/python-proxy.md)
- Runs behind rust-proxy for a long time and serves every request Rust can't serve yet. Retired once rust-proxy covers everything
- [rust-proxy](references/rust-proxy.md)
- No interpreter. Sits in front, serves what it supports end to end in Rust, forwards the rest to python-proxy
## Subsystems
Read the page for each subsystem the change touches
- [cache](references/cache.md)
- [redis](references/redis.md)
- [database](references/database.md)
- [secret manager](references/secret-manager.md)
- [router](references/router.md)
- [model catalog](references/model-catalog.md)
- [http](references/http.md)
- [callbacks](references/callbacks.md)
## Classes and values
What Python exposes decides how the core reaches it
| Python exposes | python-sdk and python-proxy | rust-proxy |
|---|---|---|
| A class users configure or swap (`litellm.cache`, the secret manager, `credential_provider`) | The core makes a host call, and the Python object answers | The shared crate, built from `config.yaml` |
| Values (kwargs, `litellm.*` globals, config fields) | Read through a host call or passed in at construction | Read from `config.yaml` |
- The built-in implementations of those Python classes become thin shims over the shared crates. A Python object answering a host call may call back into Rust, and that is fine while the logic exists once
- Resolving values (Redis settings, cloud tokens) is always a shared crate, never Python
- Data Rust can load itself is loaded by Rust on every host, never asked of Python. The model catalog is read and refetched natively on the extension too
- A user-supplied Python object with no Rust equivalent stays SDK-only, with a clear error where it is unsupported
## Host calls and injected traits
- A host call is for anything the Python hosts may answer with Python, such as the cache, the secret manager and `litellm.*` settings. The core yields it through `crates/host` and the host's driver answers (`host-python` or `host-native`)
- An injected trait is for things that are Rust on every host, such as the Redis client, token providers, clocks and sinks. Built once at construction, never yielded per call
- The core never names Python in a shared type. Whether Python answers is the host's business
- Speed of the Python answer doesn't matter. Requests that need speed run end to end on rust-proxy
## Where code goes
| Code | Where it lives |
|---|---|
| Used by rust-proxy | A shared crate |
| Used by python-sdk and rust-proxy | A shared crate, plus a pyclass in `python-bridge` |
| Used only by python-proxy | Stays in Python. It serves the long tail until rust-proxy supports it |
| Python-owned state the core asks about | A `litellm/rust_bridge/host/` module, called only by the extension |
- Product behavior lives in shared crates and never needs Python to be correct
- Hosts differ in how input arrives, never in how it is resolved
- Follow the subsystem's migration gates. In particular, [database](references/database.md) moves operations behind repositories and ports them incrementally even while python-proxy remains
## Moving a module into a shared crate
1. Define a public API. Map what the module imports and what imports it, bucket each outside use by what the caller wants, then make DB and cache access, mutability and ownership explicit
2. Route every caller through it. The new API implements the old one, old names forward with a `DeprecationWarning` that names the replacement, and a shrink-only allowlist in CI stops new coupling (`tests/code_coverage_tests/router_surface_check.py` is the model)
3. Build the Rust crate behind that API
4. Make the Python module a thin shim over Rust
5. Prove exact parity between old and new
6. Delete the Python implementation. No fallback copy stays behind
Rebase often while a refactor is in flight, and check what landed on main for changes to behavior already moved
## Surfaces Rust can implement
| Avoid | Prefer |
|---|---|
| Shared mutable dicts | Typed values in, values out |
| Live references handed out | One owner, snapshot reads, writes through one `apply` |
| Hidden cache and DB access | Named reads and writes |
| `litellm.*` globals read at call time | Config passed in at construction |
| Callbacks that can touch the whole request | Hooks that declare what they read and write |
| Callers parsing error text | Typed errors |
## Proving parity
- Fuzz old against new with [BerriAI/litefuzz](https://github.com/BerriAI/litefuzz). Its README covers suites, levels, oracles and coverage
- Check public values, errors and side effects, including the subsystem's shared-state contracts
- For Redis and database ports, also run the compatibility checks against real Redis or PostgreSQL described in their reference pages

View file

@ -0,0 +1,35 @@
# Cache
## Python surface today
- `litellm.Cache` (`litellm/caching/caching.py`) is the response cache. It owns the cache key (`get_cache_key`), which call types are cached and TTL
- It wraps a backend in `self.cache`, a `BaseCache` such as `RedisCache`, `RedisClusterCache`, `S3Cache`, `GCSCache`, `AzureBlobCache`, `InMemoryCache` or a semantic cache
- `Cache` and `ResponseCache` are not the same thing. `BaseCache` and `DualCache` are also used as plain key-value state (cooldowns, spend queues, auth cache), which is not response caching
- Users can configure the cache or pass their own object
## Shared crates
- `crates/cache` mirrors Python's `BaseCache`, including its counter, claim and TTL capabilities. It keeps Python's shape on purpose
- `crates/cache-response` holds the one response-cache key builder (`cache_key`). Python's `Cache.get_cache_key` becomes a shim over it, never a third builder
- Backend crates (`cache-redis`, `cache-s3`, `cache-gcs`, `cache-azure-blob`, `cache-disk`, `cache-memory`, the semantic caches) implement the backends
- Redis-backed storage follows [redis](redis.md)
## Per host
- python-sdk and python-proxy
- The core asks for the cache through a host call, and the configured Python object answers (`litellm/rust_bridge/host/cache.py`)
- The built-in Python cache classes keep their API and become shims over the shared crates, so the answer may land in Rust again. That is fine because the logic exists once
- A user subclass or custom cache is just another Python object answering the call
- No shortcut that unwraps the native store inside the host driver. Speed on this path doesn't matter
- rust-proxy
- Always the Rust cache, built from `config.yaml`. No other way
## Wire contracts
- Response-cache entries in Redis follow the Redis view rule in [redis](redis.md)
- Entries in other backends (S3, GCS, Azure Blob, disk): TBD
## Order and gates
- Python's `get_cache_key` moves onto `cache_response::cache_key` before Python and Rust pods share a response cache
- Order of backend moves: TBD

View file

@ -0,0 +1,29 @@
# Callbacks
Callbacks will be redesigned later. Until then, don't extend this page or design new callback seams from it
## Python surface today
- Legacy Python callbacks and `CustomLogger` hooks, which can read and change the whole request
- TBD
## Shared crates
- `crates/callbacks-legacy-python` runs the legacy callbacks from Rust
- TBD
## Per host
- python-sdk and python-proxy
- Legacy callbacks run through `litellm/rust_bridge/host/callbacks_legacy_python.py`
- rust-proxy
- A config with callbacks Rust doesn't have makes the request forward to python-proxy
- TBD
## Wire contracts
- TBD
## Order and gates
- TBD

View file

@ -0,0 +1,42 @@
# Database
Use [database-access](../../database-access/SKILL.md) to keep repository extraction, query-only ports and caller integration scoped to the requested work
Background in the [Database design](https://app.notion.com/p/Database-40543b8acdab81bd9738d924c7617302)
## Python surface today
- python-proxy only. python-sdk doesn't use the database
- Callers (routes, auth, spend, jobs) reach PostgreSQL partly through `litellm/repositories` and partly through direct Prisma calls and raw SQL
- `PrismaClient` holds the writer and an optional reader
- Everything moves behind repositories
## Shared crates
- `crates/db` exposes typed operations and owns SQL in `queries/`. Rust callers use it directly
- Python passes typed arguments through PyO3, never SQL strings
## Per host
- python-proxy
- Routes receive `Repositories` and call domain APIs such as `repositories.banners.get_banner()`
- Each operation has one implementation, Prisma or native. Operations in the same repository may use different backends
- No duplicate query, backend switch or Prisma fallback
- A whole transaction stays on one backend
- `Database` and FastAPI dependency injection are optional conveniences, not prerequisites. Port an operation without changing its callers
- Existing startup and shutdown code manages Prisma, and the SQLx pool joins it when introduced. Close opened connections if startup fails, finish active work before disconnecting
- rust-proxy
- Calls `crates/db` through its own SQLx pool
- Prisma and SQLx keep separate pools, even with PgBouncer. Size both layers together and keep reader and writer routing
## Wire contracts
- Same schema for both. Every row both sides write is a wire contract
## Order and gates
1. Extract Prisma calls into typed repositories, alone or in the same PR as a port
2. Port one operation or a whole transaction. Validate against real PostgreSQL and refresh SQLx metadata
3. Switch all callers and delete the Prisma query in the same change
4. Ship native support for ported operations. Roll back releases if needed
5. Retire Prisma queries first, then migrations through a separate SQLx cutover

View file

@ -0,0 +1,26 @@
# HTTP
## Python surface today
- TBD: the Python HTTP clients and transports, and which of them users configure
## Shared crates
- `crates/http` owns outbound clients, TLS, proxies and timeouts, with `HttpClientPool` in `pool.rs`
- Rust routes take clients from `HttpClientPool` and never build their own per route
## Per host
- python-sdk and python-proxy
- `python-bridge` injects the pool
- Which async transport is in use and timeout conversion are host calls today (`http_transport`, `timeouts` in `litellm/rust_bridge/host/AGENTS.md`). TBD whether they stay host calls
- rust-proxy
- Builds the pool from `config.yaml`
## Wire contracts
- TBD
## Order and gates
- TBD

View file

@ -0,0 +1,33 @@
# Model catalog
## Python surface today
- `model_prices_and_context_window.json` at the repo root, loaded into `litellm.model_cost` and refetched from a remote URL
- Capability and price questions (`supports_*`, effort tiers, token prices) answered by Python helpers such as `AnthropicModelInfo` and `litellm.get_model_info`
- `litellm.register_model` and direct writes to `litellm.model_cost` change entries at runtime
- `litellm/rust_bridge/host/model_capabilities.py` answers capability questions for the extension by running those Python helpers
## Shared crates
- `crates/model-catalog` reads the document, indexes models and aliases, lowers entries into typed prices and capabilities, and holds the in-memory store. Its rules are in its `AGENTS.md`
- `crates/model-catalog-refresh` owns refetching (source, policy, steps)
- Reading, refetching, lookups and capability rules all live in Rust
## Per host
- python-sdk and python-proxy
- The extension loads and refetches the catalog natively, the same way rust-proxy does
- Rust exposes the catalog to Python through `python-bridge`. Python's model info and capability helpers become shims over it
- No host call. `host/model_capabilities.py` is deleted once the native catalog answers capability questions
- `litellm.model_cost` stays the same writable dict and global for now. How it and `litellm.register_model` stay in sync with the Rust store: TBD
- rust-proxy
- Same crates, loaded and refetched natively
## Wire contracts
- The JSON document is the shared input. `Catalog::problems` defines a mergeable cost map and `check-registry` guards it on pull requests
- What Python serves today for a model (`Catalog::served`) is what the Rust API returns
## Order and gates
- TBD

View file

@ -0,0 +1,15 @@
# python-proxy on the Rust core
The FastAPI proxy. It reaches Rust through the same extension as python-sdk, so everything in [python-sdk](python-sdk.md) applies. For a long time it runs behind rust-proxy and serves every request rust-proxy forwards to it. It is retired once rust-proxy supports everything
## What gets ported
- Code only python-proxy uses stays in Python. It serves the long tail until rust-proxy supports that request
- When rust-proxy needs the same behavior, write it in a shared crate for rust-proxy and leave python-proxy's code in place, with wire parity tests between the two
- Code python-sdk also uses (response cache, router cooldowns, affinity, router usage) goes through the shared crate from both Python hosts
## Shared state
- python-proxy and rust-proxy run against one Redis and one database, both in the same pod (rust-proxy forwarding to local Python workers) and across pods
- Every Redis key and value and every DB row both sides write is a wire contract. See [redis](redis.md) and [database](database.md)
- A behavior change to a shared key or row lands on both sides in the same change

View file

@ -0,0 +1,22 @@
# python-sdk on the Rust core
`import litellm` in a user's process (`litellm.completion`, `Router`, `litellm.Cache`). It is a permanent host. It runs the shared crates through the PyO3 extension, `crates/python-bridge`
## What reaches Rust
- Values arrive as Python already resolved them, such as the kwargs `RedisCache` was built with. The bridge converts them and hands them to the same shared resolver rust-proxy feeds from `config.yaml` (`RedisSettings::resolve` in `crates/config`)
- `litellm.*` globals such as `litellm.drop_params` are read through a host call, because users set them at runtime and Python owns them
- Rust exposes a shared-crate operation as a pyclass. Python never sees the underlying protocol, for example one pyclass per Redis keyspace and no Redis commands
- Results become Python values and errors map to the existing public exceptions. Signatures of public Python entry points stay the same
## What only this host can do
- Rust may call Python through `litellm/rust_bridge/host/`, one module per question, under the contract in that directory's `AGENTS.md`
- That covers classes users configure or swap (`litellm.cache`, the secret manager) and Python-owned state such as `litellm.*` settings
- The built-in classes behind those calls are shims over shared crates, so a host call that lands in Rust again is expected
- A user-supplied Python object with no Rust equivalent (`credential_provider`) stays an SDK-only feature, with a clear error where it is unsupported
## Where the rules live
- `litellm/rust_bridge/AGENTS.md` covers choosing Rust or Python per call and falling back to Python
- No product rule lives in `python-bridge`. A rule the extension needs that rust-proxy also needs moves into a shared crate

View file

@ -0,0 +1,44 @@
# Redis
Background in the [Redis design](https://app.notion.com/p/Redis-40543b8acdab810b8ca6dbc39c3cf521). The native-parity gates and Rust token generation below supersede that page's earlier redis-py fallback and Python-token proposal
## Python surface today
- Cache callers (routing, auth cache, spend queues, cooldowns) go through `RedisCache` methods and `RedisBatch`, which apply the namespace and breaker
- Raw-access callers (rate limit, budget, lock, affinity and login Lua, config and auth-cache pub/sub, MCP OAuth) take clients and scripts from `RedisCache`. Raw clients skip the namespace and breaker, so callers namespace keys by hand
- Scripts from `async_register_script` keep the namespace and breaker
- Vector search (`RedisSemanticCache`, `ValkeySemanticCache`, the Valkey vector store) builds its own clients
- `_redis.py` merges kwargs and `REDIS_*` env, picks auth and topology (standalone, cluster, Sentinel) and builds redis-py clients
- Keys, encoding, TTLs, Lua and reply decoding live in Python, spread across modules
## Shared crates
- `crates/config` resolves Redis settings for every host (`RedisSettings::resolve`)
- `crates/redis-client` owns connections, commands, pipelines, pub/sub, the namespace, the circuit breaker, TLS, Sentinel, cluster and cloud token providers. It knows nothing about domains
- `crates/redis-store` (planned) owns each keyspace's key layout, codec, expiry, Lua and reply decoding, like a `crates/db` repository
- Backend-independent cache identity stays out of stores, see [cache](cache.md)
## Per host
- python-sdk and python-proxy
- Python passes the resolved kwargs, and the bridge feeds them to `RedisSettings::resolve`
- Each migrated keyspace is a facade in `litellm/redis/stores/<keyspace>.py` over a `#[pyclass]` in `crates/python-bridge/src/redis/` that wraps the store. Transport and store pyclasses share one native client per resolved target
- Cloud auth tokens are generated in Rust. A user-supplied `credential_provider` is the only Python token source and stays SDK-only
- rust-proxy
- Builds the same settings from `config.yaml` and env, and calls `redis-store` directly
## Wire contracts
- What Redis sees stays exactly what Python writes today, during the migration and after it. This is permanent, not only until python-proxy is retired
- That covers key names (namespace and hash tags included), value bytes, TTLs, atomic units, and pub/sub channel names and payloads. The exact command sequence may differ when nothing above can tell
- So each keyspace keeps the shape Python uses today. A Lua keyspace stays one `EVAL`, a `DualCache` keyspace keeps its in-memory layer. Record the shape, atomic unit and who applies the namespace per keyspace
- Each move ships compatibility tests against one Redis. Old and new paths read each other's writes, and keys, values, expiry and atomicity hold through a rolling deploy
## Order and gates
1. `crates/redis-client` reaches parity with every config Python supports (standalone, cluster, Sentinel, TLS options, cloud auth, breaker)
2. Swap the transport. No redis-py fallback
3. Move keyspaces into `redis-store` one at a time. No Python fallback for a moved keyspace
4. Admin commands, pub/sub and vector search: TBD
5. Drop `redis-py` and `redisvl`, or keep them as optional extras: TBD

View file

@ -0,0 +1,34 @@
# Router
## Python surface today
- `litellm.Router` in `litellm/router.py`. Before the refactor, anything reachable was public, with 165 members used by 120 files outside the router
- The new API keeps 37 endpoint methods (`acompletion`, `aresponses`, ...) and adds eight entry points with 36 methods behind them
- `registry()`, `settings()`, `health()`, `select()`, `apply()` and `aapply()`, `observe()`, `register_plugin()`, `retire()`
- Reads go through `registry()` snapshots and every write goes through `apply(ChangeSet(...))`
- Old members forward through `DeprecatedRouterMembers` with a `DeprecationWarning` that names the replacement
- `tests/code_coverage_tests/router_surface_check.py` is a shrink-only allowlist of outside uses of router internals
- The router stops importing the proxy. Proxy-only questions go through a host the proxy installs
## Shared crates
- `crates/router`, `crates/router-core` and `crates/router-types`
- TBD: which entry point maps to which crate
## Per host
- python-sdk and python-proxy
- `Router` keeps its public API and becomes a shim over the Rust router core
- Routing plugins and strategies that exist only in Python: TBD
- rust-proxy
- Builds the router from `config.yaml` and the database
- A routing plugin Rust doesn't have makes the request forward to python-proxy
## Wire contracts
- Cooldowns, affinity and router usage in Redis follow [redis](redis.md)
- Others: TBD
## Order and gates
- TBD

View file

@ -0,0 +1,19 @@
# rust-proxy on the Rust core
The `gateway` crate tree. It links the shared crates directly and has no interpreter. Routes, extractors and envelopes follow the [`rust-gateway`](../../../../litellm-rust/.agents/skills/rust-gateway/SKILL.md) skill
## In front of python-proxy
- rust-proxy takes every request. It serves what it supports end to end in Rust and forwards the rest to python-proxy workers
- The decision is made once per request, before any limit or budget is charged, so nothing is counted twice. A forwarded request runs entirely in Python
- The forward list (routes, providers, config callbacks, unknown headers and body fields) shrinks as Rust supports more. The end state is rust-proxy alone
## Everything exists in Rust
- rust-proxy can't reach `litellm/rust_bridge/host/` or any Python state, unlike the extension embedded in Python. It also doesn't want to, so the end-to-end path stays fast
- It reads `config.yaml` and the database itself and resolves secrets with the `secrets-*` crates, then feeds the same shared resolvers the extension feeds with Python values
- It answers every host call natively through `host-native`, and supplies its own implementation of every injected trait
## Matching python-proxy
- Anything python-proxy writes that rust-proxy also writes (Redis keys and values, DB rows) is a wire contract, see [python-proxy](python-proxy.md)

View file

@ -0,0 +1,28 @@
# Secret manager
## Python surface today
- A secret manager class users configure. It resolves secret references in config, such as `os.environ/...` and manager refs
- TBD: the exact classes, how users swap them, and which caching rules apply
## Shared crates
- `crates/secrets` and the `secrets-*` crates (AWS, Azure, Google, HashiCorp, CyberArk) resolve secrets
- `crates/secrets-types` holds the shared types
## Per host
- python-sdk and python-proxy
- Same pattern as [cache](cache.md). The core asks through a host call (`secret_manager` in `litellm/rust_bridge/host/`), and the configured Python object answers
- The built-in Python secret managers keep their API and become shims over the `secrets-*` crates
- A user-supplied Python secret manager with no Rust equivalent is answered by Python on the SDK. TBD whether that is supported
- rust-proxy
- Resolves secrets with the `secrets-*` crates from `config.yaml`
## Wire contracts
- TBD
## Order and gates
- TBD

View file

@ -4,7 +4,9 @@ Python state Rust needs but does not own, behind one module per question. Rust i
- One module answers one question: `model_capabilities` (what a model can do), `callbacks_legacy_python` (the legacy callback registries), `settings` (configured snapshots), `cache` (the live configured cache), `diagnostics` (legacy failure logging), `secret_manager` (the configured secret manager and native config capture), `http_transport` (which async transport is in use), `timeouts` (timeout to seconds), `redis_events` (reporting native Redis calls and breaker transitions to `ServiceLogging`), `redis_credentials` (what a caller's redis-py `CredentialProvider` authenticates with)
- A module is the only place Rust reads that Python state. Rust never does `py.import("litellm").getattr(...)` on a global; it asks here
- Each module reuses the Python implementation the Python route runs (`AnthropicModelInfo`, `Logging`). Do not re-derive those rules here or in Rust; one resolver, two callers
- Product-state questions are limited to a class users configure or swap (`cache`, `secret_manager`) or Python-owned values (`settings`). The test is the "Classes and values" rule in the [`rust-hosts`](../../../.agents/skills/rust-hosts/SKILL.md) skill
- Python-only compatibility glue (`diagnostics`, route response construction and exception mapping) also stays here. It preserves Python's public API, never shared product rules
- Rules and data rust-proxy also needs live in a shared crate, never here. One resolver, two callers, and that resolver is the shared crate. `model_capabilities` goes away once the native model catalog answers it
- The module path and function signature are a contract. The Rust side pins the path in a `const MODULE` in the crate that asks (`python-bridge/src/routes/messages/host.rs`, `python-bridge/src/secrets/`, `callbacks-legacy-python/src/python.rs`); change both sides together and keep `tests/unit/rust_bridge/host/` green
- Inputs and outputs use values and frozen snapshots. Retain caller-owned clients and caches as opaque references when identity is required; Rust never discovers them through module globals
- Moving a question into Rust means replacing the function body with a native call, not adding a second resolver. The Python signature stays