From 3035422db8f502c40e3805a12f12900aad29d46e Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Fri, 30 Oct 2026 04:25:40 +0000 Subject: [PATCH 1/2] docs: add rust-hosts architecture and migration skill --- .agents/skills/rust-hosts/SKILL.md | 98 +++++++++++++++++++ .agents/skills/rust-hosts/references/cache.md | 35 +++++++ .../skills/rust-hosts/references/callbacks.md | 29 ++++++ .../skills/rust-hosts/references/database.md | 42 ++++++++ .agents/skills/rust-hosts/references/http.md | 26 +++++ .../rust-hosts/references/model-catalog.md | 33 +++++++ .../rust-hosts/references/python-proxy.md | 15 +++ .../rust-hosts/references/python-sdk.md | 22 +++++ .agents/skills/rust-hosts/references/redis.md | 44 +++++++++ .../skills/rust-hosts/references/router.md | 34 +++++++ .../rust-hosts/references/rust-proxy.md | 19 ++++ .../rust-hosts/references/secret-manager.md | 28 ++++++ litellm/rust_bridge/host/AGENTS.md | 3 +- 13 files changed, 427 insertions(+), 1 deletion(-) create mode 100644 .agents/skills/rust-hosts/SKILL.md create mode 100644 .agents/skills/rust-hosts/references/cache.md create mode 100644 .agents/skills/rust-hosts/references/callbacks.md create mode 100644 .agents/skills/rust-hosts/references/database.md create mode 100644 .agents/skills/rust-hosts/references/http.md create mode 100644 .agents/skills/rust-hosts/references/model-catalog.md create mode 100644 .agents/skills/rust-hosts/references/python-proxy.md create mode 100644 .agents/skills/rust-hosts/references/python-sdk.md create mode 100644 .agents/skills/rust-hosts/references/redis.md create mode 100644 .agents/skills/rust-hosts/references/router.md create mode 100644 .agents/skills/rust-hosts/references/rust-proxy.md create mode 100644 .agents/skills/rust-hosts/references/secret-manager.md diff --git a/.agents/skills/rust-hosts/SKILL.md b/.agents/skills/rust-hosts/SKILL.md new file mode 100644 index 00000000000..633e6983e04 --- /dev/null +++ b/.agents/skills/rust-hosts/SKILL.md @@ -0,0 +1,98 @@ +--- +name: rust-hosts +description: Decide where code goes between the shared Rust crates, python-bridge and Python for the python-sdk, python-proxy and rust-proxy hosts, design a public API Rust can implement, move a module into a shared crate, or prove parity between old and new code. Use when adding a pyclass or host call in python-bridge, porting a module, refactoring a module behind an entry point, or touching cache, Redis, database, secret manager, router, model catalog, HTTP client or callback code on the Rust path +metadata: + internal: true +--- + +# Rust hosts + +One implementation lives in shared Rust crates, and each host runs it. python-sdk on the native extension is a permanent host, so design for it as an end state, not a step on the way to rust-proxy + +These are migration rules and target boundaries, not a claim that every subsystem is already ported. `crates/` paths are relative to `litellm-rust/`. A `TBD` is an unresolved decision, not permission to invent a new seam + +## The three hosts + +Read the page for each host the change runs on + +- [python-sdk](references/python-sdk.md) + - Permanent. Runs the core through `python-bridge`, and may ask Python through host calls +- [python-proxy](references/python-proxy.md) + - Runs behind rust-proxy for a long time and serves every request Rust can't serve yet. Retired once rust-proxy covers everything +- [rust-proxy](references/rust-proxy.md) + - No interpreter. Sits in front, serves what it supports end to end in Rust, forwards the rest to python-proxy + +## Subsystems + +Read the page for each subsystem the change touches + +- [cache](references/cache.md) +- [redis](references/redis.md) +- [database](references/database.md) +- [secret manager](references/secret-manager.md) +- [router](references/router.md) +- [model catalog](references/model-catalog.md) +- [http](references/http.md) +- [callbacks](references/callbacks.md) + +## Classes and values + +What Python exposes decides how the core reaches it + +| Python exposes | python-sdk and python-proxy | rust-proxy | +|---|---|---| +| A class users configure or swap (`litellm.cache`, the secret manager, `credential_provider`) | The core makes a host call, and the Python object answers | The shared crate, built from `config.yaml` | +| Values (kwargs, `litellm.*` globals, config fields) | Read through a host call or passed in at construction | Read from `config.yaml` | + +- The built-in implementations of those Python classes become thin shims over the shared crates. A Python object answering a host call may call back into Rust, and that is fine while the logic exists once +- Resolving values (Redis settings, cloud tokens) is always a shared crate, never Python +- Data Rust can load itself is loaded by Rust on every host, never asked of Python. The model catalog is read and refetched natively on the extension too +- A user-supplied Python object with no Rust equivalent stays SDK-only, with a clear error where it is unsupported + +## Host calls and injected traits + +- A host call is for anything the Python hosts may answer with Python, such as the cache, the secret manager and `litellm.*` settings. The core yields it through `crates/host` and the host's driver answers (`host-python` or `host-native`) +- An injected trait is for things that are Rust on every host, such as the Redis client, token providers, clocks and sinks. Built once at construction, never yielded per call +- The core never names Python in a shared type. Whether Python answers is the host's business +- Speed of the Python answer doesn't matter. Requests that need speed run end to end on rust-proxy + +## Where code goes + +| Code | Where it lives | +|---|---| +| Used by rust-proxy | A shared crate | +| Used by python-sdk and rust-proxy | A shared crate, plus a pyclass in `python-bridge` | +| Used only by python-proxy | Stays in Python. It serves the long tail until rust-proxy supports it | +| Python-owned state the core asks about | A `litellm/rust_bridge/host/` module, called only by the extension | + +- Product behavior lives in shared crates and never needs Python to be correct +- Hosts differ in how input arrives, never in how it is resolved +- Follow the subsystem's migration gates. In particular, [database](references/database.md) moves operations behind repositories and ports them incrementally even while python-proxy remains + +## Moving a module into a shared crate + +1. Define a public API. Map what the module imports and what imports it, bucket each outside use by what the caller wants, then make DB and cache access, mutability and ownership explicit +2. Route every caller through it. The new API implements the old one, old names forward with a `DeprecationWarning` that names the replacement, and a shrink-only allowlist in CI stops new coupling (`tests/code_coverage_tests/router_surface_check.py` is the model) +3. Build the Rust crate behind that API +4. Make the Python module a thin shim over Rust +5. Prove exact parity between old and new +6. Delete the Python implementation. No fallback copy stays behind + +Rebase often while a refactor is in flight, and check what landed on main for changes to behavior already moved + +## Surfaces Rust can implement + +| Avoid | Prefer | +|---|---| +| Shared mutable dicts | Typed values in, values out | +| Live references handed out | One owner, snapshot reads, writes through one `apply` | +| Hidden cache and DB access | Named reads and writes | +| `litellm.*` globals read at call time | Config passed in at construction | +| Callbacks that can touch the whole request | Hooks that declare what they read and write | +| Callers parsing error text | Typed errors | + +## Proving parity + +- Fuzz old against new with [BerriAI/litefuzz](https://github.com/BerriAI/litefuzz). Its README covers suites, levels, oracles and coverage +- Check public values, errors and side effects, including the subsystem's shared-state contracts +- For Redis and database ports, also run the compatibility checks against real Redis or PostgreSQL described in their reference pages diff --git a/.agents/skills/rust-hosts/references/cache.md b/.agents/skills/rust-hosts/references/cache.md new file mode 100644 index 00000000000..caf8df2c36b --- /dev/null +++ b/.agents/skills/rust-hosts/references/cache.md @@ -0,0 +1,35 @@ +# Cache + +## Python surface today + +- `litellm.Cache` (`litellm/caching/caching.py`) is the response cache. It owns the cache key (`get_cache_key`), which call types are cached and TTL +- It wraps a backend in `self.cache`, a `BaseCache` such as `RedisCache`, `RedisClusterCache`, `S3Cache`, `GCSCache`, `AzureBlobCache`, `InMemoryCache` or a semantic cache +- `Cache` and `ResponseCache` are not the same thing. `BaseCache` and `DualCache` are also used as plain key-value state (cooldowns, spend queues, auth cache), which is not response caching +- Users can configure the cache or pass their own object + +## Shared crates + +- `crates/cache` mirrors Python's `BaseCache`, including its counter, claim and TTL capabilities. It keeps Python's shape on purpose +- `crates/cache-response` holds the one response-cache key builder (`cache_key`). Python's `Cache.get_cache_key` becomes a shim over it, never a third builder +- Backend crates (`cache-redis`, `cache-s3`, `cache-gcs`, `cache-azure-blob`, `cache-disk`, `cache-memory`, the semantic caches) implement the backends +- Redis-backed storage follows [redis](redis.md) + +## Per host + +- python-sdk and python-proxy + - The core asks for the cache through a host call, and the configured Python object answers (`litellm/rust_bridge/host/cache.py`) + - The built-in Python cache classes keep their API and become shims over the shared crates, so the answer may land in Rust again. That is fine because the logic exists once + - A user subclass or custom cache is just another Python object answering the call + - No shortcut that unwraps the native store inside the host driver. Speed on this path doesn't matter +- rust-proxy + - Always the Rust cache, built from `config.yaml`. No other way + +## Wire contracts + +- Response-cache entries in Redis follow the Redis view rule in [redis](redis.md) +- Entries in other backends (S3, GCS, Azure Blob, disk): TBD + +## Order and gates + +- Python's `get_cache_key` moves onto `cache_response::cache_key` before Python and Rust pods share a response cache +- Order of backend moves: TBD diff --git a/.agents/skills/rust-hosts/references/callbacks.md b/.agents/skills/rust-hosts/references/callbacks.md new file mode 100644 index 00000000000..c763662bae8 --- /dev/null +++ b/.agents/skills/rust-hosts/references/callbacks.md @@ -0,0 +1,29 @@ +# Callbacks + +Callbacks will be redesigned later. Until then, don't extend this page or design new callback seams from it + +## Python surface today + +- Legacy Python callbacks and `CustomLogger` hooks, which can read and change the whole request +- TBD + +## Shared crates + +- `crates/callbacks-legacy-python` runs the legacy callbacks from Rust +- TBD + +## Per host + +- python-sdk and python-proxy + - Legacy callbacks run through `litellm/rust_bridge/host/callbacks_legacy_python.py` +- rust-proxy + - A config with callbacks Rust doesn't have makes the request forward to python-proxy +- TBD + +## Wire contracts + +- TBD + +## Order and gates + +- TBD diff --git a/.agents/skills/rust-hosts/references/database.md b/.agents/skills/rust-hosts/references/database.md new file mode 100644 index 00000000000..7b60e7c00c3 --- /dev/null +++ b/.agents/skills/rust-hosts/references/database.md @@ -0,0 +1,42 @@ +# Database + +Use [database-access](../../database-access/SKILL.md) to keep repository extraction, query-only ports and caller integration scoped to the requested work + +Background in the [Database design](https://app.notion.com/p/Database-40543b8acdab81bd9738d924c7617302) + +## Python surface today + +- python-proxy only. python-sdk doesn't use the database +- Callers (routes, auth, spend, jobs) reach PostgreSQL partly through `litellm/repositories` and partly through direct Prisma calls and raw SQL +- `PrismaClient` holds the writer and an optional reader +- Everything moves behind repositories + +## Shared crates + +- `crates/db` exposes typed operations and owns SQL in `queries/`. Rust callers use it directly +- Python passes typed arguments through PyO3, never SQL strings + +## Per host + +- python-proxy + - Routes receive `Repositories` and call domain APIs such as `repositories.banners.get_banner()` + - Each operation has one implementation, Prisma or native. Operations in the same repository may use different backends + - No duplicate query, backend switch or Prisma fallback + - A whole transaction stays on one backend + - `Database` and FastAPI dependency injection are optional conveniences, not prerequisites. Port an operation without changing its callers + - Existing startup and shutdown code manages Prisma, and the SQLx pool joins it when introduced. Close opened connections if startup fails, finish active work before disconnecting +- rust-proxy + - Calls `crates/db` through its own SQLx pool +- Prisma and SQLx keep separate pools, even with PgBouncer. Size both layers together and keep reader and writer routing + +## Wire contracts + +- Same schema for both. Every row both sides write is a wire contract + +## Order and gates + +1. Extract Prisma calls into typed repositories, alone or in the same PR as a port +2. Port one operation or a whole transaction. Validate against real PostgreSQL and refresh SQLx metadata +3. Switch all callers and delete the Prisma query in the same change +4. Ship native support for ported operations. Roll back releases if needed +5. Retire Prisma queries first, then migrations through a separate SQLx cutover diff --git a/.agents/skills/rust-hosts/references/http.md b/.agents/skills/rust-hosts/references/http.md new file mode 100644 index 00000000000..6d8427eac77 --- /dev/null +++ b/.agents/skills/rust-hosts/references/http.md @@ -0,0 +1,26 @@ +# HTTP + +## Python surface today + +- TBD: the Python HTTP clients and transports, and which of them users configure + +## Shared crates + +- `crates/http` owns outbound clients, TLS, proxies and timeouts, with `HttpClientPool` in `pool.rs` +- Rust routes take clients from `HttpClientPool` and never build their own per route + +## Per host + +- python-sdk and python-proxy + - `python-bridge` injects the pool + - Which async transport is in use and timeout conversion are host calls today (`http_transport`, `timeouts` in `litellm/rust_bridge/host/AGENTS.md`). TBD whether they stay host calls +- rust-proxy + - Builds the pool from `config.yaml` + +## Wire contracts + +- TBD + +## Order and gates + +- TBD diff --git a/.agents/skills/rust-hosts/references/model-catalog.md b/.agents/skills/rust-hosts/references/model-catalog.md new file mode 100644 index 00000000000..de9cd381ae3 --- /dev/null +++ b/.agents/skills/rust-hosts/references/model-catalog.md @@ -0,0 +1,33 @@ +# Model catalog + +## Python surface today + +- `model_prices_and_context_window.json` at the repo root, loaded into `litellm.model_cost` and refetched from a remote URL +- Capability and price questions (`supports_*`, effort tiers, token prices) answered by Python helpers such as `AnthropicModelInfo` and `litellm.get_model_info` +- `litellm.register_model` and direct writes to `litellm.model_cost` change entries at runtime +- `litellm/rust_bridge/host/model_capabilities.py` answers capability questions for the extension by running those Python helpers + +## Shared crates + +- `crates/model-catalog` reads the document, indexes models and aliases, lowers entries into typed prices and capabilities, and holds the in-memory store. Its rules are in its `AGENTS.md` +- `crates/model-catalog-refresh` owns refetching (source, policy, steps) +- Reading, refetching, lookups and capability rules all live in Rust + +## Per host + +- python-sdk and python-proxy + - The extension loads and refetches the catalog natively, the same way rust-proxy does + - Rust exposes the catalog to Python through `python-bridge`. Python's model info and capability helpers become shims over it + - No host call. `host/model_capabilities.py` is deleted once the native catalog answers capability questions + - `litellm.model_cost` stays the same writable dict and global for now. How it and `litellm.register_model` stay in sync with the Rust store: TBD +- rust-proxy + - Same crates, loaded and refetched natively + +## Wire contracts + +- The JSON document is the shared input. `Catalog::problems` defines a mergeable cost map and `check-registry` guards it on pull requests +- What Python serves today for a model (`Catalog::served`) is what the Rust API returns + +## Order and gates + +- TBD diff --git a/.agents/skills/rust-hosts/references/python-proxy.md b/.agents/skills/rust-hosts/references/python-proxy.md new file mode 100644 index 00000000000..9909e6a11c6 --- /dev/null +++ b/.agents/skills/rust-hosts/references/python-proxy.md @@ -0,0 +1,15 @@ +# python-proxy on the Rust core + +The FastAPI proxy. It reaches Rust through the same extension as python-sdk, so everything in [python-sdk](python-sdk.md) applies. For a long time it runs behind rust-proxy and serves every request rust-proxy forwards to it. It is retired once rust-proxy supports everything + +## What gets ported + +- Code only python-proxy uses stays in Python. It serves the long tail until rust-proxy supports that request +- When rust-proxy needs the same behavior, write it in a shared crate for rust-proxy and leave python-proxy's code in place, with wire parity tests between the two +- Code python-sdk also uses (response cache, router cooldowns, affinity, router usage) goes through the shared crate from both Python hosts + +## Shared state + +- python-proxy and rust-proxy run against one Redis and one database, both in the same pod (rust-proxy forwarding to local Python workers) and across pods +- Every Redis key and value and every DB row both sides write is a wire contract. See [redis](redis.md) and [database](database.md) +- A behavior change to a shared key or row lands on both sides in the same change diff --git a/.agents/skills/rust-hosts/references/python-sdk.md b/.agents/skills/rust-hosts/references/python-sdk.md new file mode 100644 index 00000000000..2a52427dd77 --- /dev/null +++ b/.agents/skills/rust-hosts/references/python-sdk.md @@ -0,0 +1,22 @@ +# python-sdk on the Rust core + +`import litellm` in a user's process (`litellm.completion`, `Router`, `litellm.Cache`). It is a permanent host. It runs the shared crates through the PyO3 extension, `crates/python-bridge` + +## What reaches Rust + +- Values arrive as Python already resolved them, such as the kwargs `RedisCache` was built with. The bridge converts them and hands them to the same shared resolver rust-proxy feeds from `config.yaml` (`RedisSettings::resolve` in `crates/config`) +- `litellm.*` globals such as `litellm.drop_params` are read through a host call, because users set them at runtime and Python owns them +- Rust exposes a shared-crate operation as a pyclass. Python never sees the underlying protocol, for example one pyclass per Redis keyspace and no Redis commands +- Results become Python values and errors map to the existing public exceptions. Signatures of public Python entry points stay the same + +## What only this host can do + +- Rust may call Python through `litellm/rust_bridge/host/`, one module per question, under the contract in that directory's `AGENTS.md` +- That covers classes users configure or swap (`litellm.cache`, the secret manager) and Python-owned state such as `litellm.*` settings +- The built-in classes behind those calls are shims over shared crates, so a host call that lands in Rust again is expected +- A user-supplied Python object with no Rust equivalent (`credential_provider`) stays an SDK-only feature, with a clear error where it is unsupported + +## Where the rules live + +- `litellm/rust_bridge/AGENTS.md` covers choosing Rust or Python per call and falling back to Python +- No product rule lives in `python-bridge`. A rule the extension needs that rust-proxy also needs moves into a shared crate diff --git a/.agents/skills/rust-hosts/references/redis.md b/.agents/skills/rust-hosts/references/redis.md new file mode 100644 index 00000000000..70d3a14abd7 --- /dev/null +++ b/.agents/skills/rust-hosts/references/redis.md @@ -0,0 +1,44 @@ +# Redis + +Background in the [Redis design](https://app.notion.com/p/Redis-40543b8acdab810b8ca6dbc39c3cf521). The native-parity gates and Rust token generation below supersede that page's earlier redis-py fallback and Python-token proposal + +## Python surface today + +- Cache callers (routing, auth cache, spend queues, cooldowns) go through `RedisCache` methods and `RedisBatch`, which apply the namespace and breaker +- Raw-access callers (rate limit, budget, lock, affinity and login Lua, config and auth-cache pub/sub, MCP OAuth) take clients and scripts from `RedisCache`. Raw clients skip the namespace and breaker, so callers namespace keys by hand + - Scripts from `async_register_script` keep the namespace and breaker +- Vector search (`RedisSemanticCache`, `ValkeySemanticCache`, the Valkey vector store) builds its own clients +- `_redis.py` merges kwargs and `REDIS_*` env, picks auth and topology (standalone, cluster, Sentinel) and builds redis-py clients +- Keys, encoding, TTLs, Lua and reply decoding live in Python, spread across modules + +## Shared crates + +- `crates/config` resolves Redis settings for every host (`RedisSettings::resolve`) +- `crates/redis-client` owns connections, commands, pipelines, pub/sub, the namespace, the circuit breaker, TLS, Sentinel, cluster and cloud token providers. It knows nothing about domains +- `crates/redis-store` (planned) owns each keyspace's key layout, codec, expiry, Lua and reply decoding, like a `crates/db` repository +- Backend-independent cache identity stays out of stores, see [cache](cache.md) + +## Per host + +- python-sdk and python-proxy + - Python passes the resolved kwargs, and the bridge feeds them to `RedisSettings::resolve` + - Each migrated keyspace is a facade in `litellm/redis/stores/.py` over a `#[pyclass]` in `crates/python-bridge/src/redis/` that wraps the store. Transport and store pyclasses share one native client per resolved target + - Cloud auth tokens are generated in Rust. A user-supplied `credential_provider` is the only Python token source and stays SDK-only +- rust-proxy + - Builds the same settings from `config.yaml` and env, and calls `redis-store` directly + +## Wire contracts + +- What Redis sees stays exactly what Python writes today, during the migration and after it. This is permanent, not only until python-proxy is retired +- That covers key names (namespace and hash tags included), value bytes, TTLs, atomic units, and pub/sub channel names and payloads. The exact command sequence may differ when nothing above can tell +- So each keyspace keeps the shape Python uses today. A Lua keyspace stays one `EVAL`, a `DualCache` keyspace keeps its in-memory layer. Record the shape, atomic unit and who applies the namespace per keyspace + +- Each move ships compatibility tests against one Redis. Old and new paths read each other's writes, and keys, values, expiry and atomicity hold through a rolling deploy + +## Order and gates + +1. `crates/redis-client` reaches parity with every config Python supports (standalone, cluster, Sentinel, TLS options, cloud auth, breaker) +2. Swap the transport. No redis-py fallback +3. Move keyspaces into `redis-store` one at a time. No Python fallback for a moved keyspace +4. Admin commands, pub/sub and vector search: TBD +5. Drop `redis-py` and `redisvl`, or keep them as optional extras: TBD diff --git a/.agents/skills/rust-hosts/references/router.md b/.agents/skills/rust-hosts/references/router.md new file mode 100644 index 00000000000..16fffff86fb --- /dev/null +++ b/.agents/skills/rust-hosts/references/router.md @@ -0,0 +1,34 @@ +# Router + +## Python surface today + +- `litellm.Router` in `litellm/router.py`. Before the refactor, anything reachable was public, with 165 members used by 120 files outside the router +- The new API keeps 37 endpoint methods (`acompletion`, `aresponses`, ...) and adds eight entry points with 36 methods behind them + - `registry()`, `settings()`, `health()`, `select()`, `apply()` and `aapply()`, `observe()`, `register_plugin()`, `retire()` +- Reads go through `registry()` snapshots and every write goes through `apply(ChangeSet(...))` +- Old members forward through `DeprecatedRouterMembers` with a `DeprecationWarning` that names the replacement +- `tests/code_coverage_tests/router_surface_check.py` is a shrink-only allowlist of outside uses of router internals +- The router stops importing the proxy. Proxy-only questions go through a host the proxy installs + +## Shared crates + +- `crates/router`, `crates/router-core` and `crates/router-types` +- TBD: which entry point maps to which crate + +## Per host + +- python-sdk and python-proxy + - `Router` keeps its public API and becomes a shim over the Rust router core + - Routing plugins and strategies that exist only in Python: TBD +- rust-proxy + - Builds the router from `config.yaml` and the database + - A routing plugin Rust doesn't have makes the request forward to python-proxy + +## Wire contracts + +- Cooldowns, affinity and router usage in Redis follow [redis](redis.md) +- Others: TBD + +## Order and gates + +- TBD diff --git a/.agents/skills/rust-hosts/references/rust-proxy.md b/.agents/skills/rust-hosts/references/rust-proxy.md new file mode 100644 index 00000000000..cdc695e23d7 --- /dev/null +++ b/.agents/skills/rust-hosts/references/rust-proxy.md @@ -0,0 +1,19 @@ +# rust-proxy on the Rust core + +The `gateway` crate tree. It links the shared crates directly and has no interpreter. Routes, extractors and envelopes follow the [`rust-gateway`](../../../../litellm-rust/.agents/skills/rust-gateway/SKILL.md) skill + +## In front of python-proxy + +- rust-proxy takes every request. It serves what it supports end to end in Rust and forwards the rest to python-proxy workers +- The decision is made once per request, before any limit or budget is charged, so nothing is counted twice. A forwarded request runs entirely in Python +- The forward list (routes, providers, config callbacks, unknown headers and body fields) shrinks as Rust supports more. The end state is rust-proxy alone + +## Everything exists in Rust + +- rust-proxy can't reach `litellm/rust_bridge/host/` or any Python state, unlike the extension embedded in Python. It also doesn't want to, so the end-to-end path stays fast +- It reads `config.yaml` and the database itself and resolves secrets with the `secrets-*` crates, then feeds the same shared resolvers the extension feeds with Python values +- It answers every host call natively through `host-native`, and supplies its own implementation of every injected trait + +## Matching python-proxy + +- Anything python-proxy writes that rust-proxy also writes (Redis keys and values, DB rows) is a wire contract, see [python-proxy](python-proxy.md) diff --git a/.agents/skills/rust-hosts/references/secret-manager.md b/.agents/skills/rust-hosts/references/secret-manager.md new file mode 100644 index 00000000000..b2136243556 --- /dev/null +++ b/.agents/skills/rust-hosts/references/secret-manager.md @@ -0,0 +1,28 @@ +# Secret manager + +## Python surface today + +- A secret manager class users configure. It resolves secret references in config, such as `os.environ/...` and manager refs +- TBD: the exact classes, how users swap them, and which caching rules apply + +## Shared crates + +- `crates/secrets` and the `secrets-*` crates (AWS, Azure, Google, HashiCorp, CyberArk) resolve secrets +- `crates/secrets-types` holds the shared types + +## Per host + +- python-sdk and python-proxy + - Same pattern as [cache](cache.md). The core asks through a host call (`secret_manager` in `litellm/rust_bridge/host/`), and the configured Python object answers + - The built-in Python secret managers keep their API and become shims over the `secrets-*` crates + - A user-supplied Python secret manager with no Rust equivalent is answered by Python on the SDK. TBD whether that is supported +- rust-proxy + - Resolves secrets with the `secrets-*` crates from `config.yaml` + +## Wire contracts + +- TBD + +## Order and gates + +- TBD diff --git a/litellm/rust_bridge/host/AGENTS.md b/litellm/rust_bridge/host/AGENTS.md index 9b751a1ada2..4d421734aea 100644 --- a/litellm/rust_bridge/host/AGENTS.md +++ b/litellm/rust_bridge/host/AGENTS.md @@ -4,7 +4,8 @@ Python state Rust needs but does not own, behind one module per question. Rust i - One module answers one question: `model_capabilities` (what a model can do), `callbacks_legacy_python` (the legacy callback registries), `settings` (configured snapshots), `cache` (the live configured cache), `diagnostics` (legacy failure logging), `secret_manager` (the configured secret manager and native config capture), `http_transport` (which async transport is in use), `timeouts` (timeout to seconds) - A module is the only place Rust reads that Python state. Rust never does `py.import("litellm").getattr(...)` on a global; it asks here -- Each module reuses the Python implementation the Python route runs (`AnthropicModelInfo`, `Logging`). Do not re-derive those rules here or in Rust; one resolver, two callers +- A module exists only for a class users configure or swap (`cache`, `secret_manager`) or for Python-owned values (`settings`). The test is the "Classes and values" rule in the [`rust-hosts`](../../../.agents/skills/rust-hosts/SKILL.md) skill +- Rules and data rust-proxy also needs live in a shared crate, never here. One resolver, two callers, and that resolver is the shared crate. `model_capabilities` goes away once the native model catalog answers it - The module path and function signature are a contract. The Rust side pins the path in a `const MODULE` in the crate that asks (`python-bridge/src/routes/messages/host.rs`, `python-bridge/src/secrets/`, `callbacks-legacy-python/src/python.rs`); change both sides together and keep `tests/unit/rust_bridge/host/` green - Inputs and outputs use values and frozen snapshots. Retain caller-owned clients and caches as opaque references when identity is required; Rust never discovers them through module globals - Moving a question into Rust means replacing the function body with a native call, not adding a second resolver. The Python signature stays From 044ec43891d08a5f26e566aa1f9238dad760bb10 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Fri, 30 Oct 2026 04:32:24 +0000 Subject: [PATCH 2/2] docs: clarify Python-only host compatibility adapters --- litellm/rust_bridge/host/AGENTS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/litellm/rust_bridge/host/AGENTS.md b/litellm/rust_bridge/host/AGENTS.md index 4d421734aea..3645df0959f 100644 --- a/litellm/rust_bridge/host/AGENTS.md +++ b/litellm/rust_bridge/host/AGENTS.md @@ -4,7 +4,8 @@ Python state Rust needs but does not own, behind one module per question. Rust i - One module answers one question: `model_capabilities` (what a model can do), `callbacks_legacy_python` (the legacy callback registries), `settings` (configured snapshots), `cache` (the live configured cache), `diagnostics` (legacy failure logging), `secret_manager` (the configured secret manager and native config capture), `http_transport` (which async transport is in use), `timeouts` (timeout to seconds) - A module is the only place Rust reads that Python state. Rust never does `py.import("litellm").getattr(...)` on a global; it asks here -- A module exists only for a class users configure or swap (`cache`, `secret_manager`) or for Python-owned values (`settings`). The test is the "Classes and values" rule in the [`rust-hosts`](../../../.agents/skills/rust-hosts/SKILL.md) skill +- Product-state questions are limited to a class users configure or swap (`cache`, `secret_manager`) or Python-owned values (`settings`). The test is the "Classes and values" rule in the [`rust-hosts`](../../../.agents/skills/rust-hosts/SKILL.md) skill +- Python-only compatibility glue (`diagnostics`, route response construction and exception mapping) also stays here. It preserves Python's public API, never shared product rules - Rules and data rust-proxy also needs live in a shared crate, never here. One resolver, two callers, and that resolver is the shared crate. `model_capabilities` goes away once the native model catalog answers it - The module path and function signature are a contract. The Rust side pins the path in a `const MODULE` in the crate that asks (`python-bridge/src/routes/messages/host.rs`, `python-bridge/src/secrets/`, `callbacks-legacy-python/src/python.rs`); change both sides together and keep `tests/unit/rust_bridge/host/` green - Inputs and outputs use values and frozen snapshots. Retain caller-owned clients and caches as opaque references when identity is required; Rust never discovers them through module globals