mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
Merge 044ec43891 into aeede53112
This commit is contained in:
commit
73ec83b75a
13 changed files with 428 additions and 1 deletions
98
.agents/skills/rust-hosts/SKILL.md
Normal file
98
.agents/skills/rust-hosts/SKILL.md
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
---
|
||||
name: rust-hosts
|
||||
description: Decide where code goes between the shared Rust crates, python-bridge and Python for the python-sdk, python-proxy and rust-proxy hosts, design a public API Rust can implement, move a module into a shared crate, or prove parity between old and new code. Use when adding a pyclass or host call in python-bridge, porting a module, refactoring a module behind an entry point, or touching cache, Redis, database, secret manager, router, model catalog, HTTP client or callback code on the Rust path
|
||||
metadata:
|
||||
internal: true
|
||||
---
|
||||
|
||||
# Rust hosts
|
||||
|
||||
One implementation lives in shared Rust crates, and each host runs it. python-sdk on the native extension is a permanent host, so design for it as an end state, not a step on the way to rust-proxy
|
||||
|
||||
These are migration rules and target boundaries, not a claim that every subsystem is already ported. `crates/` paths are relative to `litellm-rust/`. A `TBD` is an unresolved decision, not permission to invent a new seam
|
||||
|
||||
## The three hosts
|
||||
|
||||
Read the page for each host the change runs on
|
||||
|
||||
- [python-sdk](references/python-sdk.md)
|
||||
- Permanent. Runs the core through `python-bridge`, and may ask Python through host calls
|
||||
- [python-proxy](references/python-proxy.md)
|
||||
- Runs behind rust-proxy for a long time and serves every request Rust can't serve yet. Retired once rust-proxy covers everything
|
||||
- [rust-proxy](references/rust-proxy.md)
|
||||
- No interpreter. Sits in front, serves what it supports end to end in Rust, forwards the rest to python-proxy
|
||||
|
||||
## Subsystems
|
||||
|
||||
Read the page for each subsystem the change touches
|
||||
|
||||
- [cache](references/cache.md)
|
||||
- [redis](references/redis.md)
|
||||
- [database](references/database.md)
|
||||
- [secret manager](references/secret-manager.md)
|
||||
- [router](references/router.md)
|
||||
- [model catalog](references/model-catalog.md)
|
||||
- [http](references/http.md)
|
||||
- [callbacks](references/callbacks.md)
|
||||
|
||||
## Classes and values
|
||||
|
||||
What Python exposes decides how the core reaches it
|
||||
|
||||
| Python exposes | python-sdk and python-proxy | rust-proxy |
|
||||
|---|---|---|
|
||||
| A class users configure or swap (`litellm.cache`, the secret manager, `credential_provider`) | The core makes a host call, and the Python object answers | The shared crate, built from `config.yaml` |
|
||||
| Values (kwargs, `litellm.*` globals, config fields) | Read through a host call or passed in at construction | Read from `config.yaml` |
|
||||
|
||||
- The built-in implementations of those Python classes become thin shims over the shared crates. A Python object answering a host call may call back into Rust, and that is fine while the logic exists once
|
||||
- Resolving values (Redis settings, cloud tokens) is always a shared crate, never Python
|
||||
- Data Rust can load itself is loaded by Rust on every host, never asked of Python. The model catalog is read and refetched natively on the extension too
|
||||
- A user-supplied Python object with no Rust equivalent stays SDK-only, with a clear error where it is unsupported
|
||||
|
||||
## Host calls and injected traits
|
||||
|
||||
- A host call is for anything the Python hosts may answer with Python, such as the cache, the secret manager and `litellm.*` settings. The core yields it through `crates/host` and the host's driver answers (`host-python` or `host-native`)
|
||||
- An injected trait is for things that are Rust on every host, such as the Redis client, token providers, clocks and sinks. Built once at construction, never yielded per call
|
||||
- The core never names Python in a shared type. Whether Python answers is the host's business
|
||||
- Speed of the Python answer doesn't matter. Requests that need speed run end to end on rust-proxy
|
||||
|
||||
## Where code goes
|
||||
|
||||
| Code | Where it lives |
|
||||
|---|---|
|
||||
| Used by rust-proxy | A shared crate |
|
||||
| Used by python-sdk and rust-proxy | A shared crate, plus a pyclass in `python-bridge` |
|
||||
| Used only by python-proxy | Stays in Python. It serves the long tail until rust-proxy supports it |
|
||||
| Python-owned state the core asks about | A `litellm/rust_bridge/host/` module, called only by the extension |
|
||||
|
||||
- Product behavior lives in shared crates and never needs Python to be correct
|
||||
- Hosts differ in how input arrives, never in how it is resolved
|
||||
- Follow the subsystem's migration gates. In particular, [database](references/database.md) moves operations behind repositories and ports them incrementally even while python-proxy remains
|
||||
|
||||
## Moving a module into a shared crate
|
||||
|
||||
1. Define a public API. Map what the module imports and what imports it, bucket each outside use by what the caller wants, then make DB and cache access, mutability and ownership explicit
|
||||
2. Route every caller through it. The new API implements the old one, old names forward with a `DeprecationWarning` that names the replacement, and a shrink-only allowlist in CI stops new coupling (`tests/code_coverage_tests/router_surface_check.py` is the model)
|
||||
3. Build the Rust crate behind that API
|
||||
4. Make the Python module a thin shim over Rust
|
||||
5. Prove exact parity between old and new
|
||||
6. Delete the Python implementation. No fallback copy stays behind
|
||||
|
||||
Rebase often while a refactor is in flight, and check what landed on main for changes to behavior already moved
|
||||
|
||||
## Surfaces Rust can implement
|
||||
|
||||
| Avoid | Prefer |
|
||||
|---|---|
|
||||
| Shared mutable dicts | Typed values in, values out |
|
||||
| Live references handed out | One owner, snapshot reads, writes through one `apply` |
|
||||
| Hidden cache and DB access | Named reads and writes |
|
||||
| `litellm.*` globals read at call time | Config passed in at construction |
|
||||
| Callbacks that can touch the whole request | Hooks that declare what they read and write |
|
||||
| Callers parsing error text | Typed errors |
|
||||
|
||||
## Proving parity
|
||||
|
||||
- Fuzz old against new with [BerriAI/litefuzz](https://github.com/BerriAI/litefuzz). Its README covers suites, levels, oracles and coverage
|
||||
- Check public values, errors and side effects, including the subsystem's shared-state contracts
|
||||
- For Redis and database ports, also run the compatibility checks against real Redis or PostgreSQL described in their reference pages
|
||||
35
.agents/skills/rust-hosts/references/cache.md
Normal file
35
.agents/skills/rust-hosts/references/cache.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Cache
|
||||
|
||||
## Python surface today
|
||||
|
||||
- `litellm.Cache` (`litellm/caching/caching.py`) is the response cache. It owns the cache key (`get_cache_key`), which call types are cached and TTL
|
||||
- It wraps a backend in `self.cache`, a `BaseCache` such as `RedisCache`, `RedisClusterCache`, `S3Cache`, `GCSCache`, `AzureBlobCache`, `InMemoryCache` or a semantic cache
|
||||
- `Cache` and `ResponseCache` are not the same thing. `BaseCache` and `DualCache` are also used as plain key-value state (cooldowns, spend queues, auth cache), which is not response caching
|
||||
- Users can configure the cache or pass their own object
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/cache` mirrors Python's `BaseCache`, including its counter, claim and TTL capabilities. It keeps Python's shape on purpose
|
||||
- `crates/cache-response` holds the one response-cache key builder (`cache_key`). Python's `Cache.get_cache_key` becomes a shim over it, never a third builder
|
||||
- Backend crates (`cache-redis`, `cache-s3`, `cache-gcs`, `cache-azure-blob`, `cache-disk`, `cache-memory`, the semantic caches) implement the backends
|
||||
- Redis-backed storage follows [redis](redis.md)
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- The core asks for the cache through a host call, and the configured Python object answers (`litellm/rust_bridge/host/cache.py`)
|
||||
- The built-in Python cache classes keep their API and become shims over the shared crates, so the answer may land in Rust again. That is fine because the logic exists once
|
||||
- A user subclass or custom cache is just another Python object answering the call
|
||||
- No shortcut that unwraps the native store inside the host driver. Speed on this path doesn't matter
|
||||
- rust-proxy
|
||||
- Always the Rust cache, built from `config.yaml`. No other way
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- Response-cache entries in Redis follow the Redis view rule in [redis](redis.md)
|
||||
- Entries in other backends (S3, GCS, Azure Blob, disk): TBD
|
||||
|
||||
## Order and gates
|
||||
|
||||
- Python's `get_cache_key` moves onto `cache_response::cache_key` before Python and Rust pods share a response cache
|
||||
- Order of backend moves: TBD
|
||||
29
.agents/skills/rust-hosts/references/callbacks.md
Normal file
29
.agents/skills/rust-hosts/references/callbacks.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# Callbacks
|
||||
|
||||
Callbacks will be redesigned later. Until then, don't extend this page or design new callback seams from it
|
||||
|
||||
## Python surface today
|
||||
|
||||
- Legacy Python callbacks and `CustomLogger` hooks, which can read and change the whole request
|
||||
- TBD
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/callbacks-legacy-python` runs the legacy callbacks from Rust
|
||||
- TBD
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- Legacy callbacks run through `litellm/rust_bridge/host/callbacks_legacy_python.py`
|
||||
- rust-proxy
|
||||
- A config with callbacks Rust doesn't have makes the request forward to python-proxy
|
||||
- TBD
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- TBD
|
||||
|
||||
## Order and gates
|
||||
|
||||
- TBD
|
||||
42
.agents/skills/rust-hosts/references/database.md
Normal file
42
.agents/skills/rust-hosts/references/database.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Database
|
||||
|
||||
Use [database-access](../../database-access/SKILL.md) to keep repository extraction, query-only ports and caller integration scoped to the requested work
|
||||
|
||||
Background in the [Database design](https://app.notion.com/p/Database-40543b8acdab81bd9738d924c7617302)
|
||||
|
||||
## Python surface today
|
||||
|
||||
- python-proxy only. python-sdk doesn't use the database
|
||||
- Callers (routes, auth, spend, jobs) reach PostgreSQL partly through `litellm/repositories` and partly through direct Prisma calls and raw SQL
|
||||
- `PrismaClient` holds the writer and an optional reader
|
||||
- Everything moves behind repositories
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/db` exposes typed operations and owns SQL in `queries/`. Rust callers use it directly
|
||||
- Python passes typed arguments through PyO3, never SQL strings
|
||||
|
||||
## Per host
|
||||
|
||||
- python-proxy
|
||||
- Routes receive `Repositories` and call domain APIs such as `repositories.banners.get_banner()`
|
||||
- Each operation has one implementation, Prisma or native. Operations in the same repository may use different backends
|
||||
- No duplicate query, backend switch or Prisma fallback
|
||||
- A whole transaction stays on one backend
|
||||
- `Database` and FastAPI dependency injection are optional conveniences, not prerequisites. Port an operation without changing its callers
|
||||
- Existing startup and shutdown code manages Prisma, and the SQLx pool joins it when introduced. Close opened connections if startup fails, finish active work before disconnecting
|
||||
- rust-proxy
|
||||
- Calls `crates/db` through its own SQLx pool
|
||||
- Prisma and SQLx keep separate pools, even with PgBouncer. Size both layers together and keep reader and writer routing
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- Same schema for both. Every row both sides write is a wire contract
|
||||
|
||||
## Order and gates
|
||||
|
||||
1. Extract Prisma calls into typed repositories, alone or in the same PR as a port
|
||||
2. Port one operation or a whole transaction. Validate against real PostgreSQL and refresh SQLx metadata
|
||||
3. Switch all callers and delete the Prisma query in the same change
|
||||
4. Ship native support for ported operations. Roll back releases if needed
|
||||
5. Retire Prisma queries first, then migrations through a separate SQLx cutover
|
||||
26
.agents/skills/rust-hosts/references/http.md
Normal file
26
.agents/skills/rust-hosts/references/http.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# HTTP
|
||||
|
||||
## Python surface today
|
||||
|
||||
- TBD: the Python HTTP clients and transports, and which of them users configure
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/http` owns outbound clients, TLS, proxies and timeouts, with `HttpClientPool` in `pool.rs`
|
||||
- Rust routes take clients from `HttpClientPool` and never build their own per route
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- `python-bridge` injects the pool
|
||||
- Which async transport is in use and timeout conversion are host calls today (`http_transport`, `timeouts` in `litellm/rust_bridge/host/AGENTS.md`). TBD whether they stay host calls
|
||||
- rust-proxy
|
||||
- Builds the pool from `config.yaml`
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- TBD
|
||||
|
||||
## Order and gates
|
||||
|
||||
- TBD
|
||||
33
.agents/skills/rust-hosts/references/model-catalog.md
Normal file
33
.agents/skills/rust-hosts/references/model-catalog.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Model catalog
|
||||
|
||||
## Python surface today
|
||||
|
||||
- `model_prices_and_context_window.json` at the repo root, loaded into `litellm.model_cost` and refetched from a remote URL
|
||||
- Capability and price questions (`supports_*`, effort tiers, token prices) answered by Python helpers such as `AnthropicModelInfo` and `litellm.get_model_info`
|
||||
- `litellm.register_model` and direct writes to `litellm.model_cost` change entries at runtime
|
||||
- `litellm/rust_bridge/host/model_capabilities.py` answers capability questions for the extension by running those Python helpers
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/model-catalog` reads the document, indexes models and aliases, lowers entries into typed prices and capabilities, and holds the in-memory store. Its rules are in its `AGENTS.md`
|
||||
- `crates/model-catalog-refresh` owns refetching (source, policy, steps)
|
||||
- Reading, refetching, lookups and capability rules all live in Rust
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- The extension loads and refetches the catalog natively, the same way rust-proxy does
|
||||
- Rust exposes the catalog to Python through `python-bridge`. Python's model info and capability helpers become shims over it
|
||||
- No host call. `host/model_capabilities.py` is deleted once the native catalog answers capability questions
|
||||
- `litellm.model_cost` stays the same writable dict and global for now. How it and `litellm.register_model` stay in sync with the Rust store: TBD
|
||||
- rust-proxy
|
||||
- Same crates, loaded and refetched natively
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- The JSON document is the shared input. `Catalog::problems` defines a mergeable cost map and `check-registry` guards it on pull requests
|
||||
- What Python serves today for a model (`Catalog::served`) is what the Rust API returns
|
||||
|
||||
## Order and gates
|
||||
|
||||
- TBD
|
||||
15
.agents/skills/rust-hosts/references/python-proxy.md
Normal file
15
.agents/skills/rust-hosts/references/python-proxy.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# python-proxy on the Rust core
|
||||
|
||||
The FastAPI proxy. It reaches Rust through the same extension as python-sdk, so everything in [python-sdk](python-sdk.md) applies. For a long time it runs behind rust-proxy and serves every request rust-proxy forwards to it. It is retired once rust-proxy supports everything
|
||||
|
||||
## What gets ported
|
||||
|
||||
- Code only python-proxy uses stays in Python. It serves the long tail until rust-proxy supports that request
|
||||
- When rust-proxy needs the same behavior, write it in a shared crate for rust-proxy and leave python-proxy's code in place, with wire parity tests between the two
|
||||
- Code python-sdk also uses (response cache, router cooldowns, affinity, router usage) goes through the shared crate from both Python hosts
|
||||
|
||||
## Shared state
|
||||
|
||||
- python-proxy and rust-proxy run against one Redis and one database, both in the same pod (rust-proxy forwarding to local Python workers) and across pods
|
||||
- Every Redis key and value and every DB row both sides write is a wire contract. See [redis](redis.md) and [database](database.md)
|
||||
- A behavior change to a shared key or row lands on both sides in the same change
|
||||
22
.agents/skills/rust-hosts/references/python-sdk.md
Normal file
22
.agents/skills/rust-hosts/references/python-sdk.md
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# python-sdk on the Rust core
|
||||
|
||||
`import litellm` in a user's process (`litellm.completion`, `Router`, `litellm.Cache`). It is a permanent host. It runs the shared crates through the PyO3 extension, `crates/python-bridge`
|
||||
|
||||
## What reaches Rust
|
||||
|
||||
- Values arrive as Python already resolved them, such as the kwargs `RedisCache` was built with. The bridge converts them and hands them to the same shared resolver rust-proxy feeds from `config.yaml` (`RedisSettings::resolve` in `crates/config`)
|
||||
- `litellm.*` globals such as `litellm.drop_params` are read through a host call, because users set them at runtime and Python owns them
|
||||
- Rust exposes a shared-crate operation as a pyclass. Python never sees the underlying protocol, for example one pyclass per Redis keyspace and no Redis commands
|
||||
- Results become Python values and errors map to the existing public exceptions. Signatures of public Python entry points stay the same
|
||||
|
||||
## What only this host can do
|
||||
|
||||
- Rust may call Python through `litellm/rust_bridge/host/`, one module per question, under the contract in that directory's `AGENTS.md`
|
||||
- That covers classes users configure or swap (`litellm.cache`, the secret manager) and Python-owned state such as `litellm.*` settings
|
||||
- The built-in classes behind those calls are shims over shared crates, so a host call that lands in Rust again is expected
|
||||
- A user-supplied Python object with no Rust equivalent (`credential_provider`) stays an SDK-only feature, with a clear error where it is unsupported
|
||||
|
||||
## Where the rules live
|
||||
|
||||
- `litellm/rust_bridge/AGENTS.md` covers choosing Rust or Python per call and falling back to Python
|
||||
- No product rule lives in `python-bridge`. A rule the extension needs that rust-proxy also needs moves into a shared crate
|
||||
44
.agents/skills/rust-hosts/references/redis.md
Normal file
44
.agents/skills/rust-hosts/references/redis.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# Redis
|
||||
|
||||
Background in the [Redis design](https://app.notion.com/p/Redis-40543b8acdab810b8ca6dbc39c3cf521). The native-parity gates and Rust token generation below supersede that page's earlier redis-py fallback and Python-token proposal
|
||||
|
||||
## Python surface today
|
||||
|
||||
- Cache callers (routing, auth cache, spend queues, cooldowns) go through `RedisCache` methods and `RedisBatch`, which apply the namespace and breaker
|
||||
- Raw-access callers (rate limit, budget, lock, affinity and login Lua, config and auth-cache pub/sub, MCP OAuth) take clients and scripts from `RedisCache`. Raw clients skip the namespace and breaker, so callers namespace keys by hand
|
||||
- Scripts from `async_register_script` keep the namespace and breaker
|
||||
- Vector search (`RedisSemanticCache`, `ValkeySemanticCache`, the Valkey vector store) builds its own clients
|
||||
- `_redis.py` merges kwargs and `REDIS_*` env, picks auth and topology (standalone, cluster, Sentinel) and builds redis-py clients
|
||||
- Keys, encoding, TTLs, Lua and reply decoding live in Python, spread across modules
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/config` resolves Redis settings for every host (`RedisSettings::resolve`)
|
||||
- `crates/redis-client` owns connections, commands, pipelines, pub/sub, the namespace, the circuit breaker, TLS, Sentinel, cluster and cloud token providers. It knows nothing about domains
|
||||
- `crates/redis-store` (planned) owns each keyspace's key layout, codec, expiry, Lua and reply decoding, like a `crates/db` repository
|
||||
- Backend-independent cache identity stays out of stores, see [cache](cache.md)
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- Python passes the resolved kwargs, and the bridge feeds them to `RedisSettings::resolve`
|
||||
- Each migrated keyspace is a facade in `litellm/redis/stores/<keyspace>.py` over a `#[pyclass]` in `crates/python-bridge/src/redis/` that wraps the store. Transport and store pyclasses share one native client per resolved target
|
||||
- Cloud auth tokens are generated in Rust. A user-supplied `credential_provider` is the only Python token source and stays SDK-only
|
||||
- rust-proxy
|
||||
- Builds the same settings from `config.yaml` and env, and calls `redis-store` directly
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- What Redis sees stays exactly what Python writes today, during the migration and after it. This is permanent, not only until python-proxy is retired
|
||||
- That covers key names (namespace and hash tags included), value bytes, TTLs, atomic units, and pub/sub channel names and payloads. The exact command sequence may differ when nothing above can tell
|
||||
- So each keyspace keeps the shape Python uses today. A Lua keyspace stays one `EVAL`, a `DualCache` keyspace keeps its in-memory layer. Record the shape, atomic unit and who applies the namespace per keyspace
|
||||
|
||||
- Each move ships compatibility tests against one Redis. Old and new paths read each other's writes, and keys, values, expiry and atomicity hold through a rolling deploy
|
||||
|
||||
## Order and gates
|
||||
|
||||
1. `crates/redis-client` reaches parity with every config Python supports (standalone, cluster, Sentinel, TLS options, cloud auth, breaker)
|
||||
2. Swap the transport. No redis-py fallback
|
||||
3. Move keyspaces into `redis-store` one at a time. No Python fallback for a moved keyspace
|
||||
4. Admin commands, pub/sub and vector search: TBD
|
||||
5. Drop `redis-py` and `redisvl`, or keep them as optional extras: TBD
|
||||
34
.agents/skills/rust-hosts/references/router.md
Normal file
34
.agents/skills/rust-hosts/references/router.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Router
|
||||
|
||||
## Python surface today
|
||||
|
||||
- `litellm.Router` in `litellm/router.py`. Before the refactor, anything reachable was public, with 165 members used by 120 files outside the router
|
||||
- The new API keeps 37 endpoint methods (`acompletion`, `aresponses`, ...) and adds eight entry points with 36 methods behind them
|
||||
- `registry()`, `settings()`, `health()`, `select()`, `apply()` and `aapply()`, `observe()`, `register_plugin()`, `retire()`
|
||||
- Reads go through `registry()` snapshots and every write goes through `apply(ChangeSet(...))`
|
||||
- Old members forward through `DeprecatedRouterMembers` with a `DeprecationWarning` that names the replacement
|
||||
- `tests/code_coverage_tests/router_surface_check.py` is a shrink-only allowlist of outside uses of router internals
|
||||
- The router stops importing the proxy. Proxy-only questions go through a host the proxy installs
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/router`, `crates/router-core` and `crates/router-types`
|
||||
- TBD: which entry point maps to which crate
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- `Router` keeps its public API and becomes a shim over the Rust router core
|
||||
- Routing plugins and strategies that exist only in Python: TBD
|
||||
- rust-proxy
|
||||
- Builds the router from `config.yaml` and the database
|
||||
- A routing plugin Rust doesn't have makes the request forward to python-proxy
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- Cooldowns, affinity and router usage in Redis follow [redis](redis.md)
|
||||
- Others: TBD
|
||||
|
||||
## Order and gates
|
||||
|
||||
- TBD
|
||||
19
.agents/skills/rust-hosts/references/rust-proxy.md
Normal file
19
.agents/skills/rust-hosts/references/rust-proxy.md
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# rust-proxy on the Rust core
|
||||
|
||||
The `gateway` crate tree. It links the shared crates directly and has no interpreter. Routes, extractors and envelopes follow the [`rust-gateway`](../../../../litellm-rust/.agents/skills/rust-gateway/SKILL.md) skill
|
||||
|
||||
## In front of python-proxy
|
||||
|
||||
- rust-proxy takes every request. It serves what it supports end to end in Rust and forwards the rest to python-proxy workers
|
||||
- The decision is made once per request, before any limit or budget is charged, so nothing is counted twice. A forwarded request runs entirely in Python
|
||||
- The forward list (routes, providers, config callbacks, unknown headers and body fields) shrinks as Rust supports more. The end state is rust-proxy alone
|
||||
|
||||
## Everything exists in Rust
|
||||
|
||||
- rust-proxy can't reach `litellm/rust_bridge/host/` or any Python state, unlike the extension embedded in Python. It also doesn't want to, so the end-to-end path stays fast
|
||||
- It reads `config.yaml` and the database itself and resolves secrets with the `secrets-*` crates, then feeds the same shared resolvers the extension feeds with Python values
|
||||
- It answers every host call natively through `host-native`, and supplies its own implementation of every injected trait
|
||||
|
||||
## Matching python-proxy
|
||||
|
||||
- Anything python-proxy writes that rust-proxy also writes (Redis keys and values, DB rows) is a wire contract, see [python-proxy](python-proxy.md)
|
||||
28
.agents/skills/rust-hosts/references/secret-manager.md
Normal file
28
.agents/skills/rust-hosts/references/secret-manager.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Secret manager
|
||||
|
||||
## Python surface today
|
||||
|
||||
- A secret manager class users configure. It resolves secret references in config, such as `os.environ/...` and manager refs
|
||||
- TBD: the exact classes, how users swap them, and which caching rules apply
|
||||
|
||||
## Shared crates
|
||||
|
||||
- `crates/secrets` and the `secrets-*` crates (AWS, Azure, Google, HashiCorp, CyberArk) resolve secrets
|
||||
- `crates/secrets-types` holds the shared types
|
||||
|
||||
## Per host
|
||||
|
||||
- python-sdk and python-proxy
|
||||
- Same pattern as [cache](cache.md). The core asks through a host call (`secret_manager` in `litellm/rust_bridge/host/`), and the configured Python object answers
|
||||
- The built-in Python secret managers keep their API and become shims over the `secrets-*` crates
|
||||
- A user-supplied Python secret manager with no Rust equivalent is answered by Python on the SDK. TBD whether that is supported
|
||||
- rust-proxy
|
||||
- Resolves secrets with the `secrets-*` crates from `config.yaml`
|
||||
|
||||
## Wire contracts
|
||||
|
||||
- TBD
|
||||
|
||||
## Order and gates
|
||||
|
||||
- TBD
|
||||
|
|
@ -4,7 +4,9 @@ Python state Rust needs but does not own, behind one module per question. Rust i
|
|||
|
||||
- One module answers one question: `model_capabilities` (what a model can do), `callbacks_legacy_python` (the legacy callback registries), `settings` (configured snapshots), `cache` (the live configured cache), `diagnostics` (legacy failure logging), `secret_manager` (the configured secret manager and native config capture), `http_transport` (which async transport is in use), `timeouts` (timeout to seconds), `redis_events` (reporting native Redis calls and breaker transitions to `ServiceLogging`), `redis_credentials` (what a caller's redis-py `CredentialProvider` authenticates with)
|
||||
- A module is the only place Rust reads that Python state. Rust never does `py.import("litellm").getattr(...)` on a global; it asks here
|
||||
- Each module reuses the Python implementation the Python route runs (`AnthropicModelInfo`, `Logging`). Do not re-derive those rules here or in Rust; one resolver, two callers
|
||||
- Product-state questions are limited to a class users configure or swap (`cache`, `secret_manager`) or Python-owned values (`settings`). The test is the "Classes and values" rule in the [`rust-hosts`](../../../.agents/skills/rust-hosts/SKILL.md) skill
|
||||
- Python-only compatibility glue (`diagnostics`, route response construction and exception mapping) also stays here. It preserves Python's public API, never shared product rules
|
||||
- Rules and data rust-proxy also needs live in a shared crate, never here. One resolver, two callers, and that resolver is the shared crate. `model_capabilities` goes away once the native model catalog answers it
|
||||
- The module path and function signature are a contract. The Rust side pins the path in a `const MODULE` in the crate that asks (`python-bridge/src/routes/messages/host.rs`, `python-bridge/src/secrets/`, `callbacks-legacy-python/src/python.rs`); change both sides together and keep `tests/unit/rust_bridge/host/` green
|
||||
- Inputs and outputs use values and frozen snapshots. Retain caller-owned clients and caches as opaque references when identity is required; Rust never discovers them through module globals
|
||||
- Moving a question into Rust means replacing the function body with a native call, not adding a second resolver. The Python signature stays
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue