diff --git a/.agents/skills/database-access/SKILL.md b/.agents/skills/database-access/SKILL.md index 796dfafe4f6..c439d0fe352 100644 --- a/.agents/skills/database-access/SKILL.md +++ b/.agents/skills/database-access/SKILL.md @@ -1,52 +1,73 @@ --- name: database-access -description: Add database queries, extract existing Prisma calls into repositories, or explicitly port queries to Rust and SQLx without expanding migration scope +description: Organize database repositories and proxy dependency wiring, manage connection ownership, or port Prisma operations to Rust and SQLx without expanding migration scope +metadata: + internal: true --- # Database access +## Situations + +- Adding a query or an operation to a repository, read [adding a query](references/python-repository.md#adding-a-query) +- Moving existing Prisma calls behind a repository, read [moving existing access](references/python-repository.md#moving-existing-access-into-a-repository) +- Porting a repository from Prisma to Rust and SQLx, read [porting to SQLx](references/python-repository.md#porting-a-repository-to-sqlx) +- Injecting a repository into a route, read [Python and FastAPI](references/python-fastapi.md) or [Rust and Axum](references/rust-axum.md) + ## Adding new database access - Put Python database access behind an operation in `litellm/repositories` -- Give callers typed inputs and outputs, not Prisma clients, models, or query dictionaries -- Keep query execution and backend selection inside the repository +- Give each repository its own FastAPI provider, so a route builds only the repositories it uses +- A provider borrows existing handles and performs no database I/O during construction +- Define the operations a route calls as a Protocol in `litellm/models/.py`, next to its domain types + - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads + - Parsing and serialization of stored values live in the repository + - Add a Protocol when a repository gets a provider, not up front for every repository +- Keep query execution inside the repository, with one implementation per repository and no backend switch or fallback +- A repository is either all Prisma or all native, never a mix of both - Prisma is fine for the implementation, adding a query does not require a Rust port - Native Rust callers use `litellm-rust/crates/db` directly -## Asked to move existing access behind a repository +## Framework wiring -- Identify the calls being moved and their transaction boundary -- Move the existing Prisma implementation into the relevant repository -- Replace those direct database calls with repository operations -- Preserve return values, errors, and transaction behavior, with focused regression tests -- Keep this an extraction, not a backend migration or unrelated cleanup +- Python routes receive a per-repository dependency such as `UserBannerRepositoryDep` and call its operations, such as `banners.get_banner()` +- `DatabaseDep` lives in `litellm/proxy/deps/database.py`, repository providers in `litellm/proxy/deps/repositories.py` + - A provider returns the feature's Protocol, so routes never see which backend implements it + - Dependencies used by one feature only, like the banner audit binding, stay next to that feature's routes +- Repositories never import FastAPI, so jobs and helpers can use them without a request +- Use named FastAPI providers with `Annotated[..., Depends(...)]`, ordinary helpers and jobs receive explicit arguments +- Preserve each route's no-database behavior, authorization precedence and background-task scheduling +- Read [Python and FastAPI](references/python-fastapi.md) for provider examples, lifespan state and route verification +- Read [Rust and Axum](references/rust-axum.md) when adding database access to Rust handlers -## An equivalent Rust or SQLx query already exists +## Connection ownership -- Leave it and the current backend selection unchanged unless integration is explicitly requested -- Its existence alone does not justify switching a Prisma caller to Rust -- Repository extraction can keep using Prisma even when the Rust query is available +- Keep connection setup and shutdown separate from repositories, which own no pools +- Reuse existing handles, never connect per request or disconnect from a repository operation +- Give each pool one owner for setup and shutdown, borrowed handles never close it +- Finish requests, jobs and queued database work before closing connections +- Clean up resources opened before a startup failure, without a second disconnect fallback +- Add a backend to the existing lifecycle owner and pass its handle only to repositories that need it +- Preserve cancellation, timeout, error and transaction semantics across the native bridge -## Asked to port a Prisma query to Rust or SQLx +## Changing ownership or reloading resources -- Identify the existing inputs, outputs, errors, and transaction boundary -- Check `litellm-rust/crates/db` for an equivalent query and reuse it if its behavior matches -- Otherwise follow `litellm-rust/crates/db/AGENTS.md` - - Put static SQL in `queries/` and compile-check it with SQLx - - Keep typed inputs and reusable row projections in `src/` - - Refresh SQLx query metadata after SQL changes -- Keep Python conversion in the Python bridge, not in `litellm-rust/crates/db` -- When caller integration is requested, expose the Rust operation through the bridge and call it from the repository - - A query-only port can land without switching callers -- Verify behavior against real PostgreSQL before switching callers - - Check relevant null handling, conflicts, ordering, errors, and transaction semantics -- Remove the old implementation only when no callers need it +- Inventory initialization, reload, tests, background jobs and shutdown together +- Move every writer when changing authoritative storage, copying a replaceable global into state is insufficient +- A frozen repository does not make its clients immutable +- Decide whether each request captures one resource generation or reads it again during processing +- Build and validate replacements before publishing them, preserving the old generation on failure +- Serialize reloads and coordinate publication with shutdown +- Drain streams and background users before retiring old resources, retaining every generation grows resource use +- Define multi-worker coordination separately from an in-process swap +- Choose mechanisms such as `ArcSwap` only after defining publication, capture and retirement ## Migration context and boundaries -- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations incrementally with Rust APIs -- Mixed Prisma and Rust implementations are expected during the transition, unrelated existing calls can stay as they are -- Both backends may have separate pools through PgBouncer, size them together and verify pooling-mode compatibility +- Long-term direction is to centralize Python database access in repositories, then replace their Prisma implementations with Rust APIs one repository at a time +- Prisma and native repositories coexist during the transition, unrelated existing calls can stay as they are +- Prisma and SQLx keep separate client pools to PgBouncer, which pools connections to PostgreSQL + - Size both layers together and preserve reader/writer routing and pooling-mode compatibility - PgBouncer does not share transactions between Prisma and SQLx, keep each atomic unit of work on one backend and transaction - Never retry a failed Rust write through Prisma when it may already have committed - Prisma owns schema migrations today, eventual SQLx migration ownership and Prisma removal are separate planned steps, not part of a query port diff --git a/.agents/skills/database-access/references/python-fastapi.md b/.agents/skills/database-access/references/python-fastapi.md new file mode 100644 index 00000000000..2f6d1b001fc --- /dev/null +++ b/.agents/skills/database-access/references/python-fastapi.md @@ -0,0 +1,101 @@ +# Python and FastAPI + +## Existing pattern + +- `UserBannerStore` in `litellm/models/user_banner.py` is the Protocol the banner routes call +- `get_database` in `litellm/proxy/deps/database.py` reads the lifespan-owned handle from request state +- `get_user_banner_repository` in `litellm/proxy/deps/repositories.py` builds the banner repository from that handle +- Choose each repository's replica routing once in its provider, never at call sites +- A handler receives `UserBannerStore | None`, never a Prisma client or a concrete repository +- `UserBannerRepository` implements `UserBannerStore` structurally, keeping the existing Prisma table, serialization and configuration-sync behavior +- The banner audit provider binds the legacy audit callable at request time, without performing audit I/O +- A native repository can later implement the same Protocol without changing handler signatures, see [porting a repository to SQLx](python-repository.md#porting-a-repository-to-sqlx) + +## Dependency injection + +- Add one provider and one `Annotated` alias per repository in `litellm/proxy/deps/repositories.py` +- Annotate the provider and alias with the feature's Protocol, not the concrete repository +- A route depends only on the repositories it calls, so a request never builds repositories it does not use +- Do not build repositories once at lifespan, a reconnect can replace the Prisma client they wrap + +```python +class UserBannerStore(Protocol): + async def get_banner(self) -> UserBanner: ... + + async def save_banner(self, banner: UserBanner) -> None: ... +``` + +```python +async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None: + return None if database is None else UserBannerRepository(database.prisma) + + +UserBannerRepositoryDep: TypeAlias = Annotated[UserBannerStore | None, Depends(get_user_banner_repository)] +``` + +- FastAPI caches each provider within a request, so routes and sub-dependencies share one instance +- Ordinary helpers and jobs receive explicit arguments, never unresolved `Depends` defaults +- A route still reading the `prisma_client` global or building a repository inline switches to the repository's `...RepositoryDep`, keeping its no-database behavior and error precedence + +```python +async def get_user_banner(banners: UserBannerRepositoryDep) -> UserBanner: + if banners is None: + return UserBanner() + return await banners.get_banner() +``` + +## Lifespan state + +- Prisma startup owns connection setup, authentication refresh, health checks and migrations +- `Database` holds the initialized Prisma client and the `NativeDatabase` handle, without holding fields per feature + - The native handle opens no pool yet, its `connect` and `close` are noops +- The proxy lifespan builds the handle with `Database.open`, registers `Database.aclose` on its exit stack and publishes it in request state + - Callers only open and close `Database`, which closes Prisma and the native handle together, and `open` closes Prisma itself when the native connect fails +- Starlette shallow-copies lifespan state into request state, which is distinct from `app.state` +- The database module never imports a repository, legacy globals borrow its handle without owning shutdown +- Drain requests, jobs, logging, spend and audit work before database teardown +- The exit stack closes the handle on exit, including startup failures after the client is created +- When adding native resources, initialize them in the serving process, respecting runtime fork restrictions +- Await native shutdown instead of depending on Python garbage collection +- Apply the [connection ownership rules](../SKILL.md#connection-ownership) when changing the owner + +## Timing and errors + +- FastAPI resolves dependencies before entering the handler +- Handler exception handling cannot catch failures raised during dependency resolution +- A universal required-database dependency can override authorization and validation behavior +- Banner GET without a database returns the default disabled banner +- Banner PATCH checks the admin role before returning the existing missing-database error +- Invalid stored banner data falls back to the default, but storage failures still propagate +- Banner writes finish before the success audit is queued as a FastAPI background task +- Queue post-response work with `BackgroundTasks`, not bare `asyncio.create_task`, so the in-flight drain waits for it before the database closes +- Do not move background work into an awaited provider merely to remove an import + +## Verification + +- Exercise real FastAPI handlers using `dependency_overrides` for external boundaries +- Keep production parsing and persistence code real +- Restore overrides after each test +- Verify complete response bodies, status codes and OpenAPI against the base +- Include combined failure conditions to pin error precedence +- Check saved values, fresh revisions, audit contents and failed-write behavior +- Check lifespan state with a running lifespan, not only a TestClient that skips startup +- Verify providers do not connect, query, disconnect or audit eagerly +- Use real PostgreSQL and live proxy requests when changing persistence behavior + +## Scope + +- Preserve direct Python call sites or migrate them explicitly +- Inventory imports, state writers, side effects and error ordering before migrating a feature +- Preserve configuration sync and serialization, and state which legacy reads remain +- Do not migrate every route in a large module just to establish the pattern +- Do not invent providers for resources without consumers + +## Framework references + +- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/) and [Starlette lifespan state](https://starlette.dev/lifespan/) +- [`SessionDep`-style `Annotated` aliases](https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-session-dependency), which `DatabaseDep` and `UserBannerRepositoryDep` follow +- [Dependency composition and caching](https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/) +- [Background tasks and dependencies with `yield`](https://fastapi.tiangolo.com/advanced/advanced-dependencies/#background-tasks-and-dependencies-with-yield-technical-details) +- [Dependency overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/) +- [Testing lifespan](https://fastapi.tiangolo.com/advanced/testing-events/) diff --git a/.agents/skills/database-access/references/python-repository.md b/.agents/skills/database-access/references/python-repository.md new file mode 100644 index 00000000000..3b16ef24de6 --- /dev/null +++ b/.agents/skills/database-access/references/python-repository.md @@ -0,0 +1,119 @@ +# Python repositories + +## Adding a query + +Use this when a feature needs a read or write that no repository offers yet, including one more operation on an existing repository + +### Before starting + +- Search `litellm/repositories` for a repository that already owns the table, and extend it instead of adding a second one +- Check `litellm/models/.py` for an existing Protocol and domain types +- Decide the route's behavior when no database is configured, and where that check sits relative to authorization and validation + +### Steps + +1. Domain types and Protocol in `litellm/models/.py` + - The Protocol takes and returns domain types, never Prisma models, raw rows or serialized payloads + - Add the Protocol together with the provider, not for repositories nothing injects +2. Repository in `litellm/repositories/_repository.py` + - Implement the operation with Prisma, a Rust port is not required + - Compose the Prisma client and type the tables it touches with `TableActions` from `prisma_protocols.py`, never inherit `PrismaTableRepository` + - Parse and serialize stored values here, so callers only see domain types + - Never import FastAPI, connect, disconnect or choose replicas per call +3. Provider and `Annotated` alias in `litellm/proxy/deps/repositories.py`, see [dependency injection](python-fastapi.md#dependency-injection) +4. The route takes `RepositoryDep`, handles `None` with the behavior decided above, and calls the operation +5. Jobs and helpers that need the operation receive the repository as an explicit argument, built by their caller from the handle it already holds + +### Done when + +- Handler tests drive the real route with `dependency_overrides` and the real repository code, see [verification](python-fastapi.md#verification) +- Tests cover the no-database path, error precedence, and what was actually stored +- Persistence changes are checked against real PostgreSQL through a live proxy + +### Out of scope + +- Porting the query to Rust, see [porting a repository to SQLx](#porting-a-repository-to-sqlx) +- Moving unrelated direct Prisma calls in the same module + +## Moving existing access into a repository + +Use this when code calls Prisma directly, through `prisma_client.db.` or a helper, and the calls should live behind a repository. The backend stays Prisma + +### Before starting + +- List every call being moved, its callers, and the transaction or batch it runs in +- List the module's imports, global state writers, side effects such as audit and config sync, and error ordering +- Record current return values, raised errors and response bodies, these are what the change must preserve + +### Steps + +1. Move the existing Prisma code into the relevant repository in `litellm/repositories` without rewriting the queries +2. Add domain types and a Protocol in `litellm/models/.py` when a route will inject the repository +3. Replace each direct call with the repository operation, through a provider for routes and explicit arguments for jobs and helpers, see [Python and FastAPI](python-fastapi.md) +4. Calls sharing a transaction move into the same repository or unit of work in one change +5. State which legacy reads remain and why, when the change does not move all of them + +### Done when + +- Focused regression tests pin the preserved return values, errors and transaction behavior +- Response bodies, status codes and OpenAPI match the base +- Configuration sync and serialization behave the same, including invalid stored data + +### Out of scope + +- Switching the backend, even when an equivalent Rust query exists +- Converting other routes in the module just to establish the pattern +- Cleanups unrelated to the moved calls + +## Porting a repository to SQLx + +Status: TBD. No repository has been ported yet. `database.native` is a `NativeDatabase` from `crates/python-bridge/src/db.rs` whose `connect` and `close` are noops that open no pool, and the PyO3 repository surface does not exist. A real pool takes typed writer/reader settings and a token provider, never the Prisma `DATABASE_URL`, which carries Prisma-only params and rotating IAM tokens. The constraints below are settled, the steps are a sketch to revise after the first port lands + +Use this only when a port is explicitly requested. Repository extraction does not require one + +### When not to port + +- An equivalent Rust or SQLx query existing is not a reason to switch a Prisma caller +- Leave existing backend selection unchanged unless integration is explicitly requested +- Never port part of a repository, it is either all Prisma or all native + +### Before starting + +- Identify the repository's inputs, outputs, errors and transaction boundary +- List repositories that share a transaction with it, they move together +- Check `litellm-rust/crates/db` for an equivalent query and reuse it when its behavior matches + +### Steps + +1. Rust queries, following `litellm-rust/crates/db/AGENTS.md` + - Static SQL in `queries/`, compile-checked with SQLx, typed inputs and row projections in `src/` + - Dynamic filters and sort keys become closed Rust enums that select static SQL, every value is bound + - Refresh SQLx query metadata after SQL changes +2. PyO3 surface exposing typed operations, never SQL text, table names or column names from Python. TBD: where it lives and how the handle reaches it +3. Native repository implementing the same Protocol in `litellm/repositories` + - Converts typed PyO3 results to domain types, preserving defaults, invalid-data handling and storage errors + - Python conversion stays in the bridge, not in `crates/db` +4. Switch the provider in `litellm/proxy/deps/repositories.py`, the Protocol and routes stay unchanged + +```diff + async def get_user_banner_repository(database: DatabaseDep) -> UserBannerStore | None: +- return None if database is None else UserBannerRepository(database.prisma) ++ return None if database is None else NativeUserBannerRepository(database.native) +``` + +5. Delete the Prisma repository in the same change and switch every caller, including jobs and helpers + +### Done when + +- Behavior is verified against real PostgreSQL, covering null handling, conflicts, ordering, errors and transaction semantics +- `DATABASE_URL=... cargo test -p litellm-db --locked -- --include-ignored` passes against a Prisma-migrated database +- Live proxy requests show unchanged responses. TBD: a Python-side real-PostgreSQL test harness + +### Rollback + +- Roll back the release, never retry an uncertain native write through Prisma + +### Out of scope + +- SQLx owning schema migrations, and removing Prisma +- Converting unrelated routes to dependency injection diff --git a/.agents/skills/database-access/references/rust-axum.md b/.agents/skills/database-access/references/rust-axum.md new file mode 100644 index 00000000000..83368d27233 --- /dev/null +++ b/.agents/skills/database-access/references/rust-axum.md @@ -0,0 +1,4 @@ +# Rust and Axum + +- This should use same `crates/db` what is exposed to Python world. +- Details are TBD. diff --git a/.circleci/AGENTS.md b/.circleci/AGENTS.md new file mode 100644 index 00000000000..b3591145a3e --- /dev/null +++ b/.circleci/AGENTS.md @@ -0,0 +1,7 @@ +# .circleci + +The `integration` workflow on `main` runs from a scheduled pipeline in the CircleCI project settings, cron `9 0-8,14-23 * * *` (UTC) on `main` with no pipeline parameters. It has no `triggers:` block in this file because a workflow with one runs only on its schedule, so the API pipelines that set `run_migration_tests` or `routing_parity_base` would stop running it. If main stops getting integration runs, check that the schedule still exists and recreate it with these values + +Jobs clone with the `checkout_over_https` command, never the built-in `checkout` step, because CircleCI falls back to the project's SSH checkout key without its GitHub App access, GitHub rejects that key, and the public repo needs no key. It fetches the job's commit with full history and no file contents (`--filter=blob:none`), like the built-in step did, so `scripts/path_filter.sh` can find a merge base + +A step that checks out files from another commit fetches that commit with `--no-filter`, as `routing_parity` does, because the clone has no file contents and git would otherwise download each missing file in its own request, which took 210 to 896 seconds for the routing parity base diff --git a/.circleci/config.yml b/.circleci/config.yml index 4ce5ea8b6fe..95ee0a6412c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -21,19 +21,30 @@ orbs: win: circleci/windows@5.0 # Add Windows orb commands: - checkout_migration_source: + checkout_over_https: + parameters: + revision: + type: string + default: "" + shell: + type: string + default: /bin/bash -eo pipefail steps: - run: - name: Select the requested migration test revision + name: Check out the public repository over HTTPS + shell: << parameters.shell >> environment: - MIGRATION_SOURCE_SHA: << pipeline.parameters.migration_source_sha >> + REQUESTED_REVISION: << parameters.revision >> command: | - revision="${MIGRATION_SOURCE_SHA:-$CIRCLE_SHA1}" + revision="${REQUESTED_REVISION:-$CIRCLE_SHA1}" [[ "$revision" =~ ^[0-9a-f]{40}$ ]] || exit 1 git init git remote add origin https://github.com/BerriAI/litellm.git - git fetch --depth 1 origin "$revision" - git checkout --detach FETCH_HEAD + for attempt in 1 2 3; do + git fetch --filter=blob:none origin "$revision" && git checkout --detach FETCH_HEAD && break + [[ $attempt -lt 3 ]] || exit 1 + sleep 5 + done skip_if_unrelated_changes: parameters: category: @@ -347,7 +358,7 @@ commands: - when: condition: << parameters.checkout >> steps: - - checkout + - checkout_over_https - setup_google_dns - install_uv - install_rust @@ -377,7 +388,8 @@ jobs: CARGO_HTTP_MULTIPLEXING: "false" CARGO_NET_RETRY: "5" steps: - - checkout + - checkout_over_https: + shell: bash.exe - run: name: Skip job when no windows-release-relevant files changed shell: bash.exe @@ -451,7 +463,7 @@ jobs: password: ${DOCKERHUB_PASSWORD} working_directory: ~/project steps: - - checkout + - checkout_over_https - setup_google_dns - install_uv - install_rust @@ -482,7 +494,7 @@ jobs: working_directory: ~/project steps: - - checkout + - checkout_over_https - skip_if_unrelated_changes - attach_workspace: at: ~/project @@ -559,7 +571,7 @@ jobs: resource_class: medium working_directory: ~/project steps: - - checkout + - checkout_over_https - run: name: Generate LiteLLM master key command: | @@ -659,7 +671,7 @@ jobs: resource_class: large working_directory: ~/project steps: - - checkout + - checkout_over_https - run: name: Generate LiteLLM master key command: | @@ -740,7 +752,7 @@ jobs: resource_class: large working_directory: ~/project steps: - - checkout + - checkout_over_https - skip_if_unrelated_changes: category: client - setup_google_dns @@ -794,7 +806,7 @@ jobs: # the rest of the suite: nothing else performs a logout. PROXY_LOGOUT_URL: "https://www.example.com" steps: - - checkout + - checkout_over_https - run: name: Generate LiteLLM master key command: | @@ -940,7 +952,7 @@ jobs: # and by migration.serverRootPath.config.ts, which refuses to run without it. SERVER_ROOT_PATH: "/litellm" steps: - - checkout + - checkout_over_https - run: name: Generate LiteLLM master key command: | @@ -1066,11 +1078,12 @@ jobs: - when: condition: << parameters.migration_qualification >> steps: - - checkout_migration_source + - checkout_over_https: + revision: << pipeline.parameters.migration_source_sha >> - unless: condition: << parameters.migration_qualification >> steps: - - checkout + - checkout_over_https - skip_if_unrelated_changes - run: @@ -1122,7 +1135,8 @@ jobs: MIGRATION_TEST_OUTPUT: /tmp/migration-results PYTHONPATH: tests/e2e steps: - - checkout_migration_source + - checkout_over_https: + revision: << pipeline.parameters.migration_source_sha >> - run: name: Generate LiteLLM master key command: | @@ -1197,7 +1211,7 @@ jobs: resource_class: medium working_directory: ~/project steps: - - checkout + - checkout_over_https - skip_if_unrelated_changes - attach_workspace: at: ~/project @@ -1371,7 +1385,7 @@ jobs: environment: UV_PYTHON_DOWNLOADS: never steps: - - checkout + - checkout_over_https - install_uv - run: name: Install the locked shard verifier and coverage dependencies @@ -1419,7 +1433,7 @@ jobs: ROUTING_PARITY_BASE: << pipeline.parameters.routing_parity_base >> command: | [[ "$ROUTING_PARITY_BASE" =~ ^[0-9a-f]{40}$ ]] || exit 1 - git fetch --depth 1 origin "$ROUTING_PARITY_BASE" + git fetch --depth 1 --no-filter origin "$ROUTING_PARITY_BASE" git rm -r -f --quiet litellm enterprise litellm-proxy-extras git checkout "$ROUTING_PARITY_BASE" -- litellm enterprise litellm-proxy-extras git reset --quiet @@ -1494,7 +1508,7 @@ jobs: environment: DATABASE_URL: postgresql://postgres:postgres@localhost:5432/litellm_test steps: - - checkout + - checkout_over_https - skip_if_unrelated_changes - setup_litellm_test_deps: checkout: false diff --git a/.circleci/scripts/run_integration.sh b/.circleci/scripts/run_integration.sh index 00b3c5f8fdd..fb91bcce77f 100644 --- a/.circleci/scripts/run_integration.sh +++ b/.circleci/scripts/run_integration.sh @@ -1,14 +1,14 @@ #!/usr/bin/env bash set -euo pipefail -if [ "${GITHUB_ACTIONS:-}" = true ]; then - echo "Integration contracts are owned by CircleCI" >&2 - exit 1 -fi - suite="${1:?integration suite required}" mode="${2:-standard}" side="${3:-}" +requested_workers="${INTEGRATION_WORKERS:-}" +license_env=() +if [ -n "${LITELLM_LICENSE:-}" ]; then + license_env=("LITELLM_LICENSE=$LITELLM_LICENSE") +fi if [ "$mode" = replica ]; then results="test-results/integration-${suite}-replica" elif [ "$mode" = parity ]; then @@ -84,7 +84,7 @@ if [ "$suite" = browser ]; then export LITELLM_UI_PATH="$PWD/ui/litellm-dashboard/out" test -f "$LITELLM_UI_PATH/index.html" fi -export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256((os.environ.get("CIRCLE_SHA1", "local") + os.environ.get("CIRCLE_WORKFLOW_ID", "local")).encode()).hexdigest()[:8],16))')" +export INTEGRATION_SEED="$(.venv/bin/python -c 'import hashlib,os; print(int(hashlib.sha256(((os.environ.get("CIRCLE_SHA1") or os.environ.get("GITHUB_SHA") or "local") + (os.environ.get("CIRCLE_WORKFLOW_ID") or os.environ.get("GITHUB_RUN_ID") or "local")).encode()).hexdigest()[:8],16))')" export INTEGRATION_ORDER_SEED="$INTEGRATION_SEED" uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma > "$results/prisma-generate.log" 2>&1 @@ -107,7 +107,7 @@ sudo iptables -N integration_only guard_created=true sudo iptables -A integration_only -o lo -j ACCEPT sudo iptables -A integration_only -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT -for service in postgres-db redis-cache; do +for service in "${INTEGRATION_POSTGRES_CONTAINER:-postgres-db}" "${INTEGRATION_REDIS_CONTAINER:-redis-cache}"; do address="$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$service")" sudo iptables -A integration_only -d "$address" -j ACCEPT done @@ -128,6 +128,8 @@ fi sudo iptables -L integration_only -n -v -x > "$results/egress-guard.txt" awk '$3 == "REJECT" && $1 > 0 { rejected=1 } END { exit !rejected }' "$results/egress-guard.txt" +export UV_OFFLINE=true + if [ "$mode" = parity ]; then export INTEGRATION_WORKERS=1 else @@ -145,6 +147,9 @@ fi if [ "$suite" = mcp ]; then export INTEGRATION_WORKERS=2 INTEGRATION_COVERAGE=1 fi +if [ -n "$requested_workers" ]; then + export INTEGRATION_WORKERS="$requested_workers" +fi isolated_workers=false if [ "${INTEGRATION_WORKERS:-1}" -gt 1 ]; then isolated_workers=true @@ -185,7 +190,7 @@ start_proxy() { "${database_env[@]}" REDIS_HOST="$REDIS_HOST" REDIS_PORT="$REDIS_PORT" \ INTEGRATION_UPSTREAM_URL="$INTEGRATION_UPSTREAM_URL" \ LITELLM_MASTER_KEY="$LITELLM_MASTER_KEY" LITELLM_SALT_KEY="$LITELLM_SALT_KEY" LITELLM_UI_PATH="$LITELLM_UI_PATH" PROXY_BASE_URL="http://127.0.0.1:$port" \ - LITELLM_LICENSE="${LITELLM_LICENSE:-}" \ + "${license_env[@]}" \ LITELLM_MODE=PRODUCTION STORE_MODEL_IN_DB=True LITELLM_ENABLE_MCP_STDIO=true "${cost_map_env[@]}" \ AWS_EC2_METADATA_DISABLED=true DO_NOT_TRACK=1 COVERAGE_FILE="$coverage_data" \ "${proxy_command[@]}" --config tests/integration/proxy_config.yaml \ @@ -257,7 +262,7 @@ env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \ "${shared_services[@]}" "${worker_coverage[@]}" \ INTEGRATION_WORKERS="${INTEGRATION_WORKERS:-1}" \ LITELLM_MODE=PRODUCTION \ - LITELLM_LICENSE="${LITELLM_LICENSE:-}" \ + "${license_env[@]}" \ INTEGRATION_SEED="$INTEGRATION_SEED" \ INTEGRATION_ORDER_SEED="$INTEGRATION_ORDER_SEED" \ INTEGRATION_CASE_SHARDS="${CIRCLE_NODE_TOTAL:-1}" INTEGRATION_CASE_SHARD_INDEX="${CIRCLE_NODE_INDEX:-0}" \ diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f6c06f7d83f..2826098fd1f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,4 +1,6 @@ /model_prices_and_context_window.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri /litellm/model_prices_and_context_window_backup.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri +/ci_cd/cost_map_pins.json @mateo-berri @ryan-crabbe-berri @kerry-berri @joshua-berri /helm/ @yassin-berriai /terraform/ @yassin-berriai +/.github/workflows/zizmor.yml @yuneng-berri diff --git a/.github/PULL_REQUEST_TEMPLATE/general.md b/.github/PULL_REQUEST_TEMPLATE/general.md index 243d121d626..f815601a29c 100644 --- a/.github/PULL_REQUEST_TEMPLATE/general.md +++ b/.github/PULL_REQUEST_TEMPLATE/general.md @@ -4,9 +4,14 @@ Drop every section you have nothing to put in, heading included: a bare "## Relevant issues" or "## Affected release" with nothing under it must not appear in the final description --> - + No bullets, tables, or code names here. Everything else goes inside the Details block below --> + +[insert handwritten description here]
Details @@ -84,9 +89,11 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac ## Screenshots / Proof of Fix \n\n| a |", + "contentFormat": "markdown", + "stringIndexType": "unicodeCodePoint", + "pages": [ + {"pageNumber": 1, "spans": [{"offset": 0, "length": 29}], "lines": [{"content": "Tïtle"}]}, + {"pageNumber": 2, "spans": [{"offset": 29, "length": 5}], "lines": [{"content": "a"}]} + ] + } + }))]) + .await; + + let response = perform(ocr_request( + LAYOUT_MODEL, + &upstream.uri(), + json!({"outputContentFormat": output_content_format, "pages": [0, 1]}), + )) + .await + .unwrap(); + + let sent = only_request(&upstream).await; + assert_eq!( + sent.query("outputContentFormat").as_deref(), + Some("markdown") + ); + assert_eq!( + sent.query("stringIndexType").as_deref(), + Some("unicodeCodePoint") + ); + assert_eq!(sent.query("pages").as_deref(), Some("1,2")); + assert_eq!(sent.json(), json!({"base64Source": "YWJj"})); + assert_eq!( + response.content.as_deref(), + Some("# Tïtle\n\n\n\n| a |") + ); + let pages = response + .pages + .iter() + .map(|page| page.markdown.as_str()) + .collect::>(); + assert_eq!(pages, ["# Tïtle\n\n\n\n", "| a |"]); + assert_eq!( + response.extra_fields.get("stringIndexType"), + Some(&json!("unicodeCodePoint")) + ); + assert_eq!( + response.usage_info.as_ref().unwrap().pages_processed, + Some(2) + ); +} + +#[rstest] +#[case::native_markdown(LAYOUT_MODEL, json!({"outputContentFormat": "markdown", "req_format": "native"}), Some("markdown"))] +#[case::markdown_on_read(MODEL, json!({"outputContentFormat": "markdown"}), Some("markdown"))] +#[case::text(LAYOUT_MODEL, json!({"outputContentFormat": "text"}), Some("text"))] +#[case::omitted(LAYOUT_MODEL, json!({}), None)] +#[tokio::test] +async fn only_layout_markdown_in_litellm_format_asks_for_code_point_offsets( + #[case] model: &str, + #[case] options: Value, + #[case] output_content_format: Option<&str>, +) { + let upstream = upstream([json_response(json!({ + "status": "succeeded", + "analyzeResult": { + "contentFormat": "text", + "stringIndexType": "textElements", + "pages": [{"pageNumber": 1, "lines": [{"content": "hello"}]}] + } + }))]) + .await; + + let response = perform(ocr_request(model, &upstream.uri(), options)) + .await + .unwrap(); + + let sent = only_request(&upstream).await; + assert_eq!( + sent.query("outputContentFormat").as_deref(), + output_content_format + ); + assert_eq!(sent.query("stringIndexType"), None); + assert_eq!(sent.json(), json!({"base64Source": "YWJj"})); + assert_eq!(response.pages[0].markdown, "hello"); + assert_eq!(response.extra_fields.get("stringIndexType"), None); +} + +#[rstest] +#[case::snake_case(json!({"output_content_format": "markdown"}), Some("markdown"))] +#[case::padded(json!({"outputContentFormat": " markdown "}), Some("markdown"))] +#[case::empty(json!({"outputContentFormat": ""}), None)] +#[case::empty_camel_case_falls_back_to_snake_case( + json!({"outputContentFormat": "", "output_content_format": "markdown"}), + Some("markdown") +)] +#[case::dropped(json!({"outputContentFormat": "html", "drop_params": true}), None)] +#[tokio::test] +async fn the_requested_output_content_format_reaches_azure_in_its_query_spelling( + #[case] options: Value, + #[case] output_content_format: Option<&str>, +) { + let upstream = upstream([json_response(json!({ + "status": "succeeded", + "analyzeResult": {"pages": [{"pageNumber": 1, "lines": [{"content": "hello"}]}]} + }))]) + .await; + + let response = perform(ocr_request(MODEL, &upstream.uri(), options)) + .await + .unwrap(); + + let sent = only_request(&upstream).await; + assert_eq!( + sent.query("outputContentFormat").as_deref(), + output_content_format + ); + assert_eq!(sent.query("output_content_format"), None); + assert_eq!(sent.query("drop_params"), None); + assert_eq!(sent.json(), json!({"base64Source": "YWJj"})); + assert_eq!(response.pages[0].markdown, "hello"); +} + #[tokio::test] async fn native_format_normalizes_pages_and_keeps_the_provider_response() { let operation = json!({ @@ -315,7 +448,11 @@ async fn polling_does_not_follow_redirects() { .await .unwrap_err(); - assert!(error.to_string().contains("status 302"), "{error}"); + assert_eq!( + error.upstream_response().map(|(status, ..)| status), + Some(302), + "{error}" + ); assert_eq!(received(&upstream).await.len(), 2); } @@ -389,12 +526,15 @@ async fn malformed_provider_pages_report_the_response_path( } #[rstest] -#[case::missing(None)] -#[case::relative(Some("/relative"))] -#[case::cross_origin(Some("http://example.com/operation"))] -#[case::with_userinfo(Some("http://user:password@127.0.0.1/operation"))] +#[case::missing(None, false)] +#[case::relative(Some("/relative"), true)] +#[case::cross_origin(Some("http://example.com/operation"), false)] +#[case::with_userinfo(Some("http://user:password@127.0.0.1/operation"), false)] #[tokio::test] -async fn an_unusable_operation_location_is_rejected(#[case] location: Option<&str>) { +async fn an_unusable_operation_location_is_rejected( + #[case] location: Option<&str>, + #[case] unparseable: bool, +) { let response = location .into_iter() .fold(ResponseTemplate::new(202), |response, location| { @@ -407,6 +547,11 @@ async fn an_unusable_operation_location_is_rejected(#[case] location: Option<&st .unwrap_err(); assert!(error.to_string().contains("operation-location"), "{error}"); + assert!(error.is_response()); + assert_eq!( + litellm_error::find::(&error).is_some(), + unparseable + ); assert_eq!(received(&upstream).await.len(), 1); } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs b/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs index 007aa49a2fd..a5076cf0112 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/cohere.rs @@ -37,6 +37,15 @@ async fn a_non_image_document_is_rejected_before_sending( .await .unwrap_err(); - assert!(matches!(error, Error::CohereImageOnly), "{error:?}"); + assert!( + matches!( + error, + Error::UnsupportedDocument { + provider: "Cohere Parse", + .. + } + ), + "{error:?}" + ); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs b/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs index 338a47c5f57..35de1326386 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/documents.rs @@ -176,8 +176,8 @@ async fn a_missing_path_document_fails_before_sending() { assert!( matches!( &error, - Error::FileRead { path: failed, source } - if *failed == path && source.kind() == std::io::ErrorKind::NotFound + Error::FileRead { path: failed, cause } + if *failed == path && cause.kind() == std::io::ErrorKind::NotFound ), "{error:?}" ); @@ -216,7 +216,8 @@ async fn configured_client_preserves_document_url_policy(#[case] allowed: bool) .await; if !allowed { - assert!(matches!(result, Err(Error::BlockedDocumentUrl))); + let error = result.unwrap_err(); + assert!(error.is_request(), "{error:?}"); assert!(received(&documents).await.is_empty()); assert!(received(&upstream).await.is_empty()); return; diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs index e413781f1a3..9f59bb9a2a2 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/lifecycle.rs @@ -38,7 +38,7 @@ fn recording_host( .with_before_send(move |wire, _| { interceptions.fetch_add(1, Ordering::SeqCst); match block { - true => Err(Error::InvalidRequest("blocked".into())), + true => Err(Error::MissingField("blocked")), false => Ok(wire), } }) @@ -95,10 +95,7 @@ async fn a_blocking_before_send_prevents_the_call_and_emits_one_failure() { .await .unwrap_err(); - assert!( - matches!(&error, Error::InvalidRequest(message) if message == "blocked"), - "{error:?}" - ); + assert!(matches!(error, Error::MissingField("blocked")), "{error:?}"); assert_eq!(interceptions.load(Ordering::SeqCst), 1); assert_eq!(*events.lock().unwrap(), ["started", "failure"]); assert!(received(&upstream).await.is_empty()); @@ -242,7 +239,7 @@ impl litellm_host_native::services::HostCallHandler for CallerTokenHost { OcrOp::AcquireAzureAdToken(reply) => { self.trace.lock().unwrap().push("token".into()); reply.send(litellm_auth::ResolvedCredential::Static( - litellm_auth::SecretValue::new("caller-token"), + litellm_secret_value::SecretValue::new("caller-token"), )); Ok(()) } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs b/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs index 4f9266c4a8d..c7a5d58ad3d 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/machine.rs @@ -41,11 +41,10 @@ async fn drive_until( Err(error) => break Err(error), }; let answer = match op { - HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => host - .result_ready(facts) - .await - .map(|()| reply.send(())) - .map_err(HostFailure::Error), + HostRequest::Intercept(InterceptRequest::ResultReady { facts, reply }) => { + reply.send(host.result_ready(facts).await); + Ok(()) + } HostRequest::Stream(stream) => match stream { litellm_host::protocol::StreamDelivery::Open(head, _) => match head {}, litellm_host::protocol::StreamDelivery::Chunk(chunk, _) => match chunk {}, @@ -64,10 +63,8 @@ async fn drive_until( } HostRequest::Intercept(InterceptRequest::AfterProviderResponse { raw, reply }) => { ops.push("response"); - host.after_provider_response(raw) - .await - .map(|()| reply.send(())) - .map_err(HostFailure::Error) + reply.send(host.after_provider_response(raw).await); + Ok(()) } }; if let Err(failure) = answer { @@ -85,10 +82,10 @@ async fn drive_until_notified(machine: &mut OcrMachine, host: &LocalOcrHost, sto _ = stop.notified() => break, step = machine.resume() => { match step.unwrap() { - MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::ResultReady { reply, .. })) => reply.send(()), + MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::ResultReady { reply, .. })) => reply.send(Ok(())), MachineStep::Suspended(HostRequest::HostCall(op)) => host.handle_host_call(op).await.unwrap(), MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::BeforeProviderRequest { wire, reply, .. })) => reply.send(*wire), - MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::AfterProviderResponse { reply, .. })) => reply.send(()), + MachineStep::Suspended(HostRequest::Intercept(InterceptRequest::AfterProviderResponse { reply, .. })) => reply.send(Ok(())), MachineStep::Suspended(HostRequest::Stream(stream)) => match stream { litellm_host::protocol::StreamDelivery::Open(head, _) => match head {}, litellm_host::protocol::StreamDelivery::Chunk(chunk, _) => match chunk {}, @@ -117,10 +114,14 @@ async fn a_hand_driven_machine_performs_the_same_call() { assert_eq!(outcome.unwrap().pages[0].markdown, "native"); assert_eq!(received(&upstream).await.len(), 1); assert_eq!(ops, ["BeforeSend", "response"]); - assert!(matches!( - machine.resume().await, - Err(Error::InvalidRequest(_)) - )); + let Err(error) = machine.resume().await else { + panic!("resuming a finished call is a host protocol fault"); + }; + assert!(!error.is_request(), "{error:?}"); + assert!( + litellm_error::find::(&error).is_some(), + "{error:?}" + ); } #[rstest::rstest] @@ -153,8 +154,11 @@ async fn a_path_document_is_read_by_core_without_a_host_operation() { } #[rstest] -#[case::failed(HostFailure::Error(Error::InvalidRequest("before_provider_request failed".into())), "before_provider_request failed")] -#[case::cancelled(HostFailure::Cancelled(Error::InvalidRequest("cancelled".into())), "cancelled")] +#[case::failed( + HostFailure::Error(Error::MissingField("before_provider_request failed")), + "before_provider_request failed" +)] +#[case::cancelled(HostFailure::Cancelled(Error::MissingField("cancelled")), "cancelled")] #[tokio::test] async fn a_before_send_failure_ends_the_call_without_reaching_transport( #[case] failure: HostFailure, @@ -174,7 +178,7 @@ async fn a_before_send_failure_ends_the_call_without_reaching_transport( .await; assert!( - matches!(&outcome, Err(Error::InvalidRequest(actual)) if actual == message), + matches!(&outcome, Err(Error::MissingField(actual)) if *actual == message), "{outcome:?}" ); assert_eq!(ops, ["BeforeSend"]); @@ -255,17 +259,17 @@ async fn interrupt_drops_provider_captures_before_returning() { drive_until_notified(&mut machine, &host, &entered).await; assert!(!dropped.load(Ordering::SeqCst)); - let acknowledgement = machine.interrupt(HostFailure::Cancelled(Error::InvalidRequest( - "cancelled".into(), - ))); + let acknowledgement = + machine.interrupt(HostFailure::Cancelled(Error::MissingField("cancelled"))); assert!( dropped.load(Ordering::SeqCst), "interrupt returned while provider captures were still alive" ); - assert!( - matches!(acknowledgement.await, Err(Error::InvalidRequest(message)) if message == "cancelled") - ); + assert!(matches!( + acknowledgement.await, + Err(Error::MissingField("cancelled")) + )); } #[rstest::rstest] @@ -290,7 +294,7 @@ async fn interrupting_an_in_flight_provider_request_closes_its_connection() { let mut machine = ocr_route().machine(host.request().unwrap(), None); drive_until_notified(&mut machine, &host, &received).await; - let cancelled = Error::InvalidRequest("cancelled".into()); + let cancelled = Error::MissingField("cancelled"); assert!( machine diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs index 5a905f907d2..6871d1ca1f6 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/main.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/main.rs @@ -80,7 +80,7 @@ fn wire(model: &str, base: &str, document: Value, options: Value) -> OcrWireRequ OcrWireRequest { model: model.into(), document, - api_key: Some(litellm_auth::SecretValue::new("test-key")), + api_key: Some(litellm_secret_value::SecretValue::new("test-key")), api_base: Some(base.into()), custom_llm_provider: None, extra_headers: None, @@ -209,7 +209,7 @@ impl LocalOcrHost { request, caller_token: false, }) - .ok_or_else(|| Error::InvalidRequest("OCR request was already projected".into())) + .ok_or(Error::MissingField("OCR request")) } pub fn runtime(&self) -> litellm_host_native::in_process::Host<'_, Self, Self, ()> { litellm_host_native::in_process::Host { @@ -223,11 +223,11 @@ impl LocalOcrHost { impl litellm_host_native::services::HostCallHandler for LocalOcrHost { async fn handle_host_call(&self, op: OcrOp) -> Result<(), Error> { match op { - OcrOp::AcquireAzureAdToken(_) => { - Err(Error::Auth(litellm_auth::Error::CredentialAcquisition( - "OCR host has no Azure AD token provider".into(), - ))) - } + OcrOp::AcquireAzureAdToken(_) => Err(Error::Auth( + litellm_auth::Error::CredentialAcquisition(litellm_auth::ErrorDetail::Unresolved { + subject: "OCR host Azure AD token provider", + }), + )), } } } diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs b/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs index 528adc7feaf..21cf4ce455d 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/mistral.rs @@ -92,28 +92,13 @@ async fn an_upstream_error_keeps_its_status_whole_body_and_headers( .unwrap_err(); assert_eq!(received(&upstream).await.len(), 1); - let Error::Provider { - status, - body, - headers, - } = error - else { - panic!("expected provider error, got {error:?}"); - }; + let (status, body, headers) = error.upstream_response().expect("the provider answered"); assert_eq!(status, 422); - for (name, value) in [ - ("retry-after", "17"), - ("x-request-id", "request-123"), - ("x-future-header", "retained"), - ] { - assert!( - headers - .iter() - .any(|(key, actual)| key.eq_ignore_ascii_case(name) && actual == value), - "{name} missing from {headers:?}" - ); - } assert_eq!(body, expected_body); + let header = |name: &str| headers.get(name).and_then(|value| value.to_str().ok()); + assert_eq!(header("retry-after"), Some("17")); + assert_eq!(header("x-request-id"), Some("request-123")); + assert_eq!(header("x-future-header"), Some("retained")); } #[rstest] diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs b/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs index 649dca3d690..f6f458a4916 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/reducto.rs @@ -175,8 +175,11 @@ async fn a_failed_upload_stops_before_parse(#[case] upload: ResponseTemplate) { } #[rstest] -#[case::remote_url("https://example.com/a.pdf", Error::ReductoSource)] -#[case::empty_file_id("reducto://", Error::RequestField { path: "document file id".into() })] +#[case::remote_url( + "https://example.com/a.pdf", + Error::UnsupportedDocument { provider: "Reducto", guidance: "" } +)] +#[case::empty_file_id("reducto://", Error::RequestField { path: "document file id".into(), cause: None })] #[case::data_uri_without_payload("data:application/pdf;base64", Error::InvalidDataUri)] #[case::invalid_base64("data:application/pdf;base64,INVALID!", Error::InvalidDataUri)] #[tokio::test] @@ -202,7 +205,6 @@ async fn invalid_document_sources_are_rejected_before_sending( std::mem::discriminant(&expected) ); assert_eq!(error.http_status_code(), Some(400)); - assert_eq!(error.to_string(), expected.to_string()); } #[tokio::test] diff --git a/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs b/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs index 81b52d076a5..dba12de60f6 100644 --- a/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs +++ b/litellm-rust/crates/inference-ocr/tests/ocr/vertex_ai.rs @@ -71,7 +71,8 @@ async fn invalid_credentials_fail_before_sending() { .await .unwrap_err(); - assert!(error.to_string().contains("vertex_credentials"), "{error}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("vertex_credentials"), "{report}"); } #[rstest] @@ -91,11 +92,10 @@ async fn a_request_controlled_api_base_is_rejected_before_vertex_auth( let error = perform(request).await.unwrap_err(); + let report = litellm_error::Report::new(&error).to_string(); assert!( - error - .to_string() - .contains("request-controlled Vertex AI endpoint"), - "{error}" + report.contains("request-controlled Vertex AI endpoint"), + "{report}" ); } diff --git a/litellm-rust/crates/inference-ocr/tests/resources.rs b/litellm-rust/crates/inference-ocr/tests/resources.rs index d618647e89b..3132a837eaf 100644 --- a/litellm-rust/crates/inference-ocr/tests/resources.rs +++ b/litellm-rust/crates/inference-ocr/tests/resources.rs @@ -7,7 +7,7 @@ use std::sync::{ use litellm_auth::AuthServices; use litellm_auth_gcp::{ - CredentialSource, VertexAuth, VertexAuthFuture, VertexProviderLoader, VertexTokenSource, + CredentialSource, GcpAuthFuture, GcpProviderLoader, GcpTokenSource, VertexAuth, }; use litellm_http::{HttpSettings, Resolution}; use litellm_inference::resources::CoreResources; @@ -20,28 +20,28 @@ use support::{ReceivedRequest, json_response, upstream}; struct TokenSource(String); -impl VertexTokenSource for TokenSource { - fn project_id(&self) -> VertexAuthFuture<'_, String> { +impl GcpTokenSource for TokenSource { + fn project_id(&self) -> GcpAuthFuture<'_, String> { Box::pin(async { Ok(self.0.clone()) }) } - fn token(&self) -> VertexAuthFuture<'_, String> { - Box::pin(async { Ok(self.0.clone()) }) + fn token(&self) -> GcpAuthFuture<'_, litellm_secret_value::SecretValue> { + Box::pin(async { Ok(litellm_secret_value::SecretValue::new(self.0.clone())) }) } } #[derive(Default)] struct Loader(AtomicUsize); -impl VertexProviderLoader for Loader { - fn load(&self, source: CredentialSource) -> VertexAuthFuture<'_, Arc> { +impl GcpProviderLoader for Loader { + fn load(&self, source: CredentialSource) -> GcpAuthFuture<'_, Arc> { Box::pin(async move { self.0.fetch_add(1, Ordering::SeqCst); let identity = match source { CredentialSource::Trusted(secret) => secret.expose().to_string(), other => panic!("unexpected credential source: {other:?}"), }; - Ok(Arc::new(TokenSource(identity)) as Arc) + Ok(Arc::new(TokenSource(identity)) as Arc) }) } } diff --git a/litellm-rust/crates/inference-responses/Cargo.toml b/litellm-rust/crates/inference-responses/Cargo.toml index 793483ee76d..f90651634c5 100644 --- a/litellm-rust/crates/inference-responses/Cargo.toml +++ b/litellm-rust/crates/inference-responses/Cargo.toml @@ -25,8 +25,11 @@ tokio-tungstenite.workspace = true tracing.workspace = true [dev-dependencies] +litellm-secret-value.workspace = true +litellm-http = { workspace = true, features = ["test-support"] } litellm-cache.workspace = true litellm-cache-memory.workspace = true +litellm-error.workspace = true litellm-host-native.workspace = true litellm-inference-testing.workspace = true litellm-tracing.workspace = true diff --git a/litellm-rust/crates/inference-responses/src/handler.rs b/litellm-rust/crates/inference-responses/src/handler.rs index 02182d23ae3..f81a51ad192 100644 --- a/litellm-rust/crates/inference-responses/src/handler.rs +++ b/litellm-rust/crates/inference-responses/src/handler.rs @@ -72,28 +72,29 @@ pub(super) async fn execute( let stream = match wire.body.get("stream") { None => false, Some(serde_json::Value::Bool(value)) => *value, - Some(_) => return Err(Error::InvalidRequest("stream must be a boolean".into())), + Some(_) => { + return Err(Error::invalid_request( + litellm_llms::ErrorDetail::Expected { + subject: "stream", + expected: "a boolean", + }, + )); + } }; let outbound = litellm_inference::outbound::outbound_request( Authenticated { headers: wire.headers, signer: authenticated.signer, }, - wire.url, + &wire.url, &wire.body, Some(call.timeout.unwrap_or(Duration::from_secs(600))), )?; - let response = litellm_inference::outbound::send(outbound, &route.http) - .await - .map_err(network)?; + let response = litellm_inference::outbound::send(outbound, &route.http).await?; let status = response.status().as_u16(); if !response.status().is_success() { - let body = response.text().await.map_err(network)?; - return Err(litellm_http::transport::Error::Http { - status, - body: litellm_http::request::truncate_error_body(&body), - } - .into()); + let body = response.text().await.map_err(litellm_http::Error::from)?; + return Err(litellm_http::Error::upstream(status, body).into()); } if stream { let headers = response @@ -105,15 +106,15 @@ pub(super) async fn execute( .collect(); let chunks = response .bytes_stream() - .map(|chunk| chunk.map_err(network)) + .map(|chunk| chunk.map_err(|error| litellm_http::Error::from(error).into())) .boxed(); return Ok(ResponsesOutput::Stream { head: ResponsesStreamHead { headers }, chunks, }); } - let body = response.text().await.map_err(network)?; - let raw = RawResponse { body: body.clone() }; + let body = response.text().await.map_err(litellm_http::Error::from)?; + let raw = RawResponse::new(body.clone()); if let Some(observers) = observers { observers.emit(litellm_host::lifecycle::CallEvent::Execution( ExecutionEvent::ProviderResponseReceived { raw: raw.clone() }, @@ -123,17 +124,18 @@ pub(super) async fn execute( .after_provider_response(raw) .await .map_err(Error::post_call)?; - let value = serde_json::from_str(&body) - .map_err(|error| Error::InvalidResponse(error.to_string().into()))?; + let value = serde_json::from_str(&body).map_err(|error| { + Error::invalid_response(litellm_llms::ErrorDetail::invalid( + "Responses API response JSON", + error, + )) + })?; config .transform_response_api_response(value) .map(ResponsesOutput::Complete) .map_err(Error::from) + .map_err(Error::after_send) }, ) .await } - -fn network(error: reqwest::Error) -> Error { - litellm_http::transport::Error::Network(error.to_string()).into() -} diff --git a/litellm-rust/crates/inference-responses/src/lib.rs b/litellm-rust/crates/inference-responses/src/lib.rs index b6e151ecec3..9af5dea1fe8 100644 --- a/litellm-rust/crates/inference-responses/src/lib.rs +++ b/litellm-rust/crates/inference-responses/src/lib.rs @@ -11,6 +11,7 @@ mod handler; use std::sync::Arc; use litellm_auth::AuthServices; +use litellm_core_utils::decline::Decline; use litellm_core_utils::get_llm_provider_logic::{ CustomLlmProvider, LlmProviders, get_custom_llm_provider, }; @@ -24,6 +25,8 @@ use litellm_llms::{ use litellm_secrets::source::SecretSource; use types::{ResponsesCall, ResponsesOutput, ResponsesProvider}; +const NOT_NATIVE: Decline = Decline::new("native HTTP responses provider"); + fn responses_provider(provider: LlmProviders) -> Option<&'static dyn BaseResponsesApiConfig> { match provider { LlmProviders::Openai => Some(&OpenAiResponsesApiConfig), @@ -111,9 +114,8 @@ impl ResponsesRoute { } let provider = provider_name .parse::() - .map_err(|_| Error::Unsupported("native HTTP responses provider"))?; - let config = responses_provider(provider) - .ok_or(Error::Unsupported("native HTTP responses provider"))?; + .map_err(|_| Error::Declined(NOT_NATIVE))?; + let config = responses_provider(provider).ok_or(Error::Declined(NOT_NATIVE))?; let wire_model = match provider { LlmProviders::Openai if resolved_model.contains('/') => { return Err(Error::InvalidProvider(call.model)); @@ -123,7 +125,7 @@ impl ResponsesRoute { format!("openrouter/{resolved_model}") } LlmProviders::Openrouter => resolved_model.into(), - _ => return Err(Error::Unsupported("native HTTP responses provider")), + _ => return Err(Error::Declined(NOT_NATIVE)), }; let provider = ResponsesProvider { identity: ProviderIdentity { diff --git a/litellm-rust/crates/inference-responses/src/websocket.rs b/litellm-rust/crates/inference-responses/src/websocket.rs index 213a9a95831..2afcde0a7a4 100644 --- a/litellm-rust/crates/inference-responses/src/websocket.rs +++ b/litellm-rust/crates/inference-responses/src/websocket.rs @@ -1,14 +1,18 @@ use std::{collections::HashMap, sync::Arc, time::Duration}; use futures_util::{SinkExt, StreamExt}; -use litellm_http::websocket::{UpstreamWebSocket, connect_upstream}; +use litellm_http::{ + Client, + websocket::{self, UpstreamWebSocket}, +}; use tokio::sync::Mutex; use tokio_tungstenite::tungstenite::{ Message, - client::IntoClientRequest, - http::{HeaderName, HeaderValue}, + http::{HeaderMap, HeaderName, HeaderValue}, }; +use litellm_llms::ErrorDetail; + use super::Error; #[derive(Clone)] @@ -24,42 +28,32 @@ impl ResponsesWebSocketConnection { fields(outcome) )] pub async fn connect_url( + client: &Client, url: &str, headers: &HashMap, timeout: Option, ) -> Result { litellm_inference::diagnostic::operation("litellm.websocket.connect_url", async { - let mut request = url.into_client_request().map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; - for (name, value) in headers { - let header_name = name - .parse::() - .map_err(|error| Error::InvalidRequest(error.to_string().into()))?; - let header_value = HeaderValue::from_str(value) - .map_err(|error| Error::InvalidRequest(error.to_string().into()))?; - request.headers_mut().insert(header_name, header_value); - } - let connect = connect_upstream(request); - let result = match timeout { - Some(timeout) => tokio::time::timeout(timeout, connect).await.map_err(|_| { - Error::Transport(litellm_http::transport::Error::Network( - "Responses WebSocket connection timed out".into(), + let headers = headers + .iter() + .map(|(name, value)| { + Ok(( + name.parse::().map_err(|error| { + Error::invalid_request(ErrorDetail::invalid( + "WebSocket header name", + error, + )) + })?, + HeaderValue::from_str(value).map_err(|error| { + Error::invalid_request(ErrorDetail::invalid( + "WebSocket header value", + error, + )) + })?, )) - })?, - None => connect.await, - }; - let (socket, _) = result.map_err(|error| match *error { - tokio_tungstenite::tungstenite::Error::Http(response) => { - Error::Transport(litellm_http::transport::Error::Http { - status: response.status().as_u16(), - body: String::new(), - }) - } - other => { - Error::Transport(litellm_http::transport::Error::Network(other.to_string())) - } - })?; + }) + .collect::>()?; + let socket = websocket::connect(client, url, headers, timeout).await?; Ok(Self { socket: Arc::new(Mutex::new(Some(socket))), }) @@ -77,13 +71,12 @@ impl ResponsesWebSocketConnection { litellm_inference::diagnostic::operation("litellm.websocket.send_text", async { let mut socket = self.socket.lock().await; let Some(socket) = socket.as_mut() else { - return Err(Error::Transport(litellm_http::transport::Error::Network( - "Responses WebSocket is closed".into(), - ))); + return Err(litellm_http::Error::websocket_closed().into()); }; - socket.send(Message::Text(text)).await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - }) + socket + .send(Message::Text(text)) + .await + .map_err(|error| litellm_http::Error::from(error).into()) }) .await } @@ -104,12 +97,12 @@ impl ResponsesWebSocketConnection { Some(Ok(Message::Text(text))) => Ok(Some(text)), Some(Ok(Message::Binary(bytes))) => String::from_utf8(bytes.to_vec()) .map(Some) - .map_err(|error| Error::InvalidResponse(error.to_string().into())), + .map_err(|error| { + Error::invalid_response(ErrorDetail::invalid("WebSocket text frame", error)) + }), Some(Ok(Message::Close(_))) | None => Ok(None), Some(Ok(_)) => Ok(None), - Some(Err(error)) => Err(Error::Transport(litellm_http::transport::Error::Network( - error.to_string(), - ))), + Some(Err(error)) => Err(litellm_http::Error::from(error).into()), } }) .await @@ -125,9 +118,10 @@ impl ResponsesWebSocketConnection { litellm_inference::diagnostic::operation("litellm.websocket.close", async { let mut socket = self.socket.lock().await; if let Some(socket) = socket.as_mut() { - socket.close(None).await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + socket + .close(None) + .await + .map_err(litellm_http::Error::from)?; } *socket = None; Ok(()) diff --git a/litellm-rust/crates/inference-responses/tests/caching.rs b/litellm-rust/crates/inference-responses/tests/caching.rs index 462461695c3..56e25fbcb98 100644 --- a/litellm-rust/crates/inference-responses/tests/caching.rs +++ b/litellm-rust/crates/inference-responses/tests/caching.rs @@ -98,7 +98,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets { name: &'a str, ) -> futures_util::future::BoxFuture< 'a, - Result, litellm_secrets::Error>, + Result, litellm_secrets::Error>, > { Box::pin(async move { let revision = self.revision.load(Ordering::SeqCst); @@ -112,7 +112,7 @@ impl litellm_secrets::source::SecretSource for ChangingSecrets { } else { None }; - Ok(value.map(litellm_secrets::SecretValue::new)) + Ok(value.map(litellm_secret_value::SecretValue::new)) }) } } diff --git a/litellm-rust/crates/inference-responses/tests/openrouter.rs b/litellm-rust/crates/inference-responses/tests/openrouter.rs index e52cdba8ce8..8cdb68c376f 100644 --- a/litellm-rust/crates/inference-responses/tests/openrouter.rs +++ b/litellm-rust/crates/inference-responses/tests/openrouter.rs @@ -1,10 +1,7 @@ use std::sync::Arc; use futures_util::TryStreamExt; -use litellm_inference_responses::{ - Error, - types::{ResponsesCall, ResponsesOutput}, -}; +use litellm_inference_responses::types::{ResponsesCall, ResponsesOutput}; use litellm_inference_testing::{RecordingSecrets, no_secrets}; use rstest::{fixture, rstest}; use serde_json::json; @@ -108,13 +105,12 @@ async fn a_missing_key_fails_before_transport(call: ResponsesCall) { None, ) .await; - assert!(matches!( - result, - Err(Error::Auth(litellm_auth::Error::MissingApiKey { - provider: "OpenRouter", - .. - })) - )); + let Err(error) = result else { + panic!("the call fails"); + }; + assert!(error.is_missing_credential(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("OpenRouter"), "{report}"); assert!(received(&upstream).await.is_empty()); } diff --git a/litellm-rust/crates/inference-responses/tests/responses.rs b/litellm-rust/crates/inference-responses/tests/responses.rs index ae07a71dabd..89368a941ef 100644 --- a/litellm-rust/crates/inference-responses/tests/responses.rs +++ b/litellm-rust/crates/inference-responses/tests/responses.rs @@ -103,10 +103,7 @@ async fn unsupported_providers_fail_before_transport( ) .await; - assert!(matches!( - result, - Err(Error::Unsupported("native HTTP responses provider")) - )); + assert!(matches!(result, Err(ref error) if error.declined().is_some())); assert!(received(&upstream).await.is_empty()); } @@ -304,6 +301,25 @@ async fn provider_failures_emit_failure_once( ); } +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url(call: ResponsesCall) { + let host = RecordingCall::::new(ResponsesCall { + api_base: Some(format!("{UNREACHABLE_BASE}/tenant-token-1234")), + ..call + }); + let call = host.request.lock().unwrap().take().unwrap(); + + let Err(error) = responses_route(no_secrets()) + .execute(call, &host, None) + .await + else { + panic!("nothing is listening"); + }; + + assert!(!error.to_string().contains("tenant-token-1234"), "{error}"); +} + #[rstest] #[case::explicit(true)] #[case::from_secrets(false)] @@ -538,6 +554,7 @@ async fn websocket_operations_trace_outcomes_without_capturing_frames_or_credent .logger() .instrument(async { let connection = ResponsesWebSocketConnection::connect_url( + &litellm_http::Client::for_test(), &format!("ws://{address}/responses"), &std::collections::HashMap::from([( "authorization".into(), @@ -556,12 +573,11 @@ async fn websocket_operations_trace_outcomes_without_capturing_frames_or_credent Some("private-frame-sentinel") ); connection.close().await.unwrap(); - assert!( - connection - .send_text("private-frame-sentinel".into()) - .await - .is_err() - ); + let error = connection + .send_text("private-frame-sentinel".into()) + .await + .expect_err("a closed connection sends nothing"); + assert!(!error.dispatched(), "{error:?}"); }) .await; server.await.unwrap(); diff --git a/litellm-rust/crates/inference-responses/tests/support/mod.rs b/litellm-rust/crates/inference-responses/tests/support/mod.rs index 81f133a0471..984b878eb21 100644 --- a/litellm-rust/crates/inference-responses/tests/support/mod.rs +++ b/litellm-rust/crates/inference-responses/tests/support/mod.rs @@ -13,6 +13,9 @@ use litellm_secrets::source::SecretSource; use serde_json::Value; use wiremock::{Mock, MockServer, Request, ResponseTemplate, matchers::any}; +/// A port nothing listens on, for calls that must fail before any request is sent. +pub const UNREACHABLE_BASE: &str = "http://127.0.0.1:1"; + pub fn responses_route( secrets: Arc, ) -> litellm_inference_responses::ResponsesRoute { diff --git a/litellm-rust/crates/inference-testing/Cargo.toml b/litellm-rust/crates/inference-testing/Cargo.toml index 748b1df0d3b..5f690532d9c 100644 --- a/litellm-rust/crates/inference-testing/Cargo.toml +++ b/litellm-rust/crates/inference-testing/Cargo.toml @@ -7,6 +7,7 @@ repository.workspace = true publish = false [dependencies] +litellm-secret-value.workspace = true litellm-auth.workspace = true futures-util.workspace = true litellm-http.workspace = true diff --git a/litellm-rust/crates/inference-testing/src/lib.rs b/litellm-rust/crates/inference-testing/src/lib.rs index 2e20d5a9e91..f330e02e99a 100644 --- a/litellm-rust/crates/inference-testing/src/lib.rs +++ b/litellm-rust/crates/inference-testing/src/lib.rs @@ -5,7 +5,8 @@ use litellm_http::{ ClientVariant, HttpClientConfig, HttpClientPool, HttpSettings, Resolution, media::PublicDnsResolver, }; -use litellm_secrets::{SecretValue, source::SecretSource}; +use litellm_secret_value::SecretValue; +use litellm_secrets::source::SecretSource; use litellm_inference::resources::CoreResources; diff --git a/litellm-rust/crates/inference-transcription/Cargo.toml b/litellm-rust/crates/inference-transcription/Cargo.toml index c501d329852..cf50b3f84fd 100644 --- a/litellm-rust/crates/inference-transcription/Cargo.toml +++ b/litellm-rust/crates/inference-transcription/Cargo.toml @@ -17,6 +17,7 @@ serde_json.workspace = true tracing.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-inference-testing.workspace = true litellm-tracing.workspace = true rstest.workspace = true diff --git a/litellm-rust/crates/inference-transcription/src/handler.rs b/litellm-rust/crates/inference-transcription/src/handler.rs index 5a7238c5e36..2ed3b1eba7b 100644 --- a/litellm-rust/crates/inference-transcription/src/handler.rs +++ b/litellm-rust/crates/inference-transcription/src/handler.rs @@ -2,7 +2,7 @@ use std::time::Duration; use litellm_http::{ Client, - request::{string_headers, truncate_error_body, with_default_headers}, + request::{string_headers, with_default_headers}, }; use litellm_llms::base_llm::{ audio_transcription::transformation::BaseAudioTranscriptionConfig, @@ -48,35 +48,27 @@ pub(super) async fn execute( let authenticated = resolve_auth(auth, environment, &env_lookup).await?; let outbound = litellm_inference::outbound::outbound_request( authenticated, - url, + &url, &body, Some( call.timeout .unwrap_or(Duration::from_secs(AUDIO_TRANSCRIPTION_TIMEOUT_SECS)), ), )?; - let response = litellm_inference::outbound::send(outbound, http) - .await - .map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + let response = litellm_inference::outbound::send(outbound, http).await?; let status = response.status(); - let text = response.text().await.map_err(|error| { - Error::Transport(litellm_http::transport::Error::Network(error.to_string())) - })?; + let text = response.text().await.map_err(litellm_http::Error::from)?; if !status.is_success() { - return Err(Error::Transport(litellm_http::transport::Error::Http { - status: status.as_u16(), - body: truncate_error_body(&text), - })); + return Err(litellm_http::Error::upstream(status.as_u16(), text).into()); } let response_json = serde_json::from_str(&text).map_err(|error| { - Error::InvalidResponse(litellm_llms::ErrorDetail::invalid( + Error::invalid_response(litellm_llms::ErrorDetail::invalid( "audio response JSON", error, )) })?; - Ok(config - .transform_audio_transcription_response(provider.model, response_json)? - .into_json()) + config + .transform_audio_transcription_response(provider.model, response_json) + .map(|response| response.into_json()) + .map_err(|error| Error::from(error).after_send()) } diff --git a/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs b/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs index eb252bdff7e..9e48edce0c0 100644 --- a/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs +++ b/litellm-rust/crates/inference-transcription/tests/audio_transcription.rs @@ -152,13 +152,7 @@ async fn invalid_audio_is_rejected_before_sending( .await .expect_err("invalid audio is rejected"); - assert!( - matches!( - error, - Error::InvalidRequest(_) | Error::MissingField(_) | Error::InvalidType { .. } - ), - "{error:?}" - ); + assert!(error.is_request(), "{error:?}"); assert!(received(&upstream).await.is_empty()); } @@ -185,7 +179,9 @@ async fn unsupported_providers_are_rejected_before_sending( .await .expect_err("unsupported provider errors"); - assert_eq!(error, Error::InvalidProvider(reported.into())); + assert!(error.is_request(), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains(reported), "{report}"); } #[rstest] @@ -222,14 +218,28 @@ async fn an_upstream_error_keeps_its_status_and_body( .expect_err("upstream error propagates"); assert_eq!( - error, - Error::Transport(litellm_http::transport::Error::Http { - status, - body: "upstream said no".into() - }) + error.upstream_response(), + Some((status, "upstream said no")) ); } +#[rstest] +#[tokio::test] +async fn a_transport_error_does_not_echo_the_provider_url( + request: AudioTranscriptionRequest<'static>, +) { + let base = format!("{UNREACHABLE_BASE}/tenant-token-1234"); + + let error = transcribe(AudioTranscriptionRequest { + api_base: Some(&base), + ..request + }) + .await + .expect_err("nothing is listening"); + + assert!(!error.to_string().contains("tenant-token-1234"), "{error}"); +} + #[rstest] #[case::not_json(ResponseTemplate::new(200).set_body_string("not json"))] #[case::no_output(json_response(json!({"unexpected": true})))] @@ -248,7 +258,7 @@ async fn an_unreadable_success_body_is_an_invalid_response( .await .expect_err("an unreadable body fails"); - assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!(error.is_response(), "{error:?}"); } #[rstest] diff --git a/litellm-rust/crates/inference/AGENTS.md b/litellm-rust/crates/inference/AGENTS.md index 7f2bd0387be..bcbc98a209e 100644 --- a/litellm-rust/crates/inference/AGENTS.md +++ b/litellm-rust/crates/inference/AGENTS.md @@ -53,10 +53,10 @@ ## Error placement -- The workspace `Error definitions` rules shape each crate's error. This section decides which crate and module a failure belongs to +- The [rust-error skill](../../.agents/skills/rust-error/SKILL.md) shapes each crate's error. This section decides which crate and module a failure belongs to - A failure is declared once, by the lowest crate that raises it - - every crate above nests it unchanged (`#[error(transparent)] Auth(#[from] litellm_auth::Error)`) or maps it once at its boundary, as `src/error.rs` does for `litellm_llms::Error` - - `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises + - every crate above nests it in one variant and delegates its predicates, as `Error::Llm`, `Error::Auth` and `Error::Http` do, or translates it once in the shared `From` when its meaning changes, as `From` does for `Unsupported` and `From` does for a header the signer computes + - `Error` collects route failures for every format crate and never re-declares a variant a lower crate raises. Routes build transformation failures with `Error::invalid_request` and `Error::invalid_response` and collapse anything raised after the send with `Error::after_send`, and callers ask `declined()`, `is_request()`, `is_response()`, `is_missing_credential()`, `dispatched()`, `upstream_response()` and `rejected_request()` instead of matching nested variants - Scope follows the concept, not the first caller - an error type under `litellm-llms`'s `/` is private to that provider: no other provider and nothing in `base_llm` may import it - a failure two providers or two routes can hit (wire framing, stream event decoding, a malformed provider response) belongs to the crate that owns the concept: `litellm-framer` for framing, `litellm_llms::Error` for the transformation layer diff --git a/litellm-rust/crates/inference/Cargo.toml b/litellm-rust/crates/inference/Cargo.toml index acc211f214b..807a3ea85d3 100644 --- a/litellm-rust/crates/inference/Cargo.toml +++ b/litellm-rust/crates/inference/Cargo.toml @@ -29,6 +29,7 @@ time.workspace = true url.workspace = true [dev-dependencies] +litellm-error.workspace = true litellm-cache-memory.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-llms = { workspace = true, features = ["test-support"] } diff --git a/litellm-rust/crates/inference/src/caching.rs b/litellm-rust/crates/inference/src/caching.rs index 09d03c06215..d9b5b891015 100644 --- a/litellm-rust/crates/inference/src/caching.rs +++ b/litellm-rust/crates/inference/src/caching.rs @@ -377,5 +377,66 @@ async fn publish( facts: facts.clone(), })); } - interceptors.result_ready(facts).await + let from_provider = facts.source == ResultSource::Provider; + interceptors.result_ready(facts).await.map_err(|error| { + if from_provider { + Error::post_call(error) + } else { + error + } + }) +} + +#[cfg(test)] +mod tests { + use litellm_core_utils::decline::Decline; + use litellm_host::interceptors::{RawResponse, RequestContext, WireRequest}; + + use super::*; + + struct RejectResult; + + impl Interceptors for RejectResult { + async fn before_provider_request( + &self, + wire: WireRequest, + _: RequestContext, + ) -> Result { + Ok(wire) + } + + async fn after_provider_response(&self, _: RawResponse) -> Result<(), Error> { + Ok(()) + } + + async fn result_ready(&self, _: ExecutionFacts) -> Result<(), Error> { + Err(Error::Declined(Decline::new("result rejected"))) + } + } + + #[rstest::rstest] + #[case::provider(ResultSource::Provider, false)] + #[case::cache(ResultSource::Cache { key: "cached".into() }, true)] + #[tokio::test] + async fn result_rejection_only_declines_without_a_provider_call( + #[case] source: ResultSource, + #[case] may_decline: bool, + ) { + let facts = ExecutionFacts { + provider: ProviderIdentity { + model: "model".into(), + provider: "provider".into(), + }, + cost: Cost::settle(&source, None), + source, + }; + let error = publish(facts, &RejectResult, None).await.unwrap_err(); + assert_eq!(error.declined().is_some(), may_decline); + assert_eq!(error.dispatched(), !may_decline); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("result rejected") + ); + } } diff --git a/litellm-rust/crates/inference/src/context.rs b/litellm-rust/crates/inference/src/context.rs index 655b768dde1..ef2635d24a1 100644 --- a/litellm-rust/crates/inference/src/context.rs +++ b/litellm-rust/crates/inference/src/context.rs @@ -1,6 +1,6 @@ use litellm_cache_response::CachePolicy; use litellm_host::{ - interceptors::{ExecutionFacts, Interceptors, RawResponse}, + interceptors::{ExecutionFacts, Interceptors, RawResponse, ResultSource}, lifecycle::{CallEvent, ExecutionEvent}, observation::ObservationSender, }; @@ -28,13 +28,25 @@ impl<'a, I: Interceptors> CallContext<'a, I> { facts: facts.clone(), })); } - self.interceptors.result_ready(facts).await + let from_provider = facts.source == ResultSource::Provider; + self.interceptors + .result_ready(facts) + .await + .map_err(|error| { + if from_provider { + Error::post_call(error) + } else { + error + } + }) } pub async fn response_received(&self, body: &str) -> Result<(), Error> { - let raw = RawResponse { - body: body.to_owned(), - }; + self.raw_response_received(RawResponse::new(body.to_owned())) + .await + } + + pub async fn raw_response_received(&self, raw: RawResponse) -> Result<(), Error> { if let Some(observers) = &self.observers { observers.emit(CallEvent::Execution( ExecutionEvent::ProviderResponseReceived { raw: raw.clone() }, diff --git a/litellm-rust/crates/inference/src/error.rs b/litellm-rust/crates/inference/src/error.rs index 895c8c55e01..841e0c53799 100644 --- a/litellm-rust/crates/inference/src/error.rs +++ b/litellm-rust/crates/inference/src/error.rs @@ -1,57 +1,48 @@ -//! One error for every route in this crate. OCR still carries its own, richer enum. -//! -//! A variant is declared by the layer that produces it and nested here as is: -//! credentials by `litellm_auth` (AWS folds into it at that crate's boundary), the wire by -//! `litellm_http`, secrets by `litellm_secrets`. The transformation layer's [`LlmError`] -//! maps onto the same-named variants once, here, so no route re-declares them. - use std::sync::Arc; -use litellm_http::transport::Error as TransportError; -use litellm_llms::Error as LlmError; +use litellm_core_utils::decline::Decline; +use litellm_host::machine::MachineFault; +use litellm_llms::{Error as LlmError, ErrorDetail}; #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum Error { - #[error("expected {expected}, got {actual}")] - InvalidType { - expected: &'static str, - actual: &'static str, - }, - #[error("missing required field: {0}")] - MissingField(&'static str), #[error("invalid provider: {0}")] InvalidProvider(String), - #[error("invalid request: {0}")] - InvalidRequest(#[source] litellm_llms::ErrorDetail), - #[error("invalid response: {0}")] - InvalidResponse(#[source] litellm_llms::ErrorDetail), - #[error("unsupported by the rust path: {0}")] - Unsupported(&'static str), + #[error("invalid provider: unable to resolve custom_llm_provider for {route} request")] + UnresolvedProvider { route: &'static str }, + #[error(transparent)] + Declined(Decline), + #[error(transparent)] + Llm(LlmError), #[error(transparent)] Auth(#[from] litellm_auth::Error), #[error(transparent)] - Transport(#[from] TransportError), - #[error(transparent)] Headers(#[from] litellm_http::request::HeaderError), #[error(transparent)] - Http(#[from] litellm_http::Error), + Http(litellm_http::Error), #[error(transparent)] Secret(#[from] SecretError), - #[error("post-call hook failed: {0}")] + #[error("the host driver failed")] + Host(#[source] MachineFault), + #[error("post-call hook failed")] PostCallHook(#[source] Arc), } -impl From for Error { - fn from(fault: litellm_host::machine::MachineFault) -> Self { - use litellm_host::machine::MachineFault; - Self::InvalidRequest(match fault { - MachineFault::Abandoned => "host driver was abandoned".into(), - MachineFault::Protocol(message) => format!("host {message}").into(), - }) +impl From for Error { + fn from(fault: MachineFault) -> Self { + Self::Host(fault) } } impl Error { + pub fn invalid_request(detail: ErrorDetail) -> Self { + Self::Llm(LlmError::InvalidRequest(detail)) + } + + pub fn invalid_response(detail: ErrorDetail) -> Self { + Self::Llm(LlmError::InvalidResponse(detail)) + } + pub fn post_call(error: Self) -> Self { Self::PostCallHook(Arc::new(error)) } @@ -60,31 +51,166 @@ impl Error { /// the provider's answer. pub fn is_request(&self) -> bool { match self { - Self::InvalidType { .. } - | Self::MissingField(_) - | Self::InvalidProvider(_) - | Self::InvalidRequest(_) - | Self::Unsupported(_) - | Self::Headers(_) => true, - Self::Auth(error) => !matches!(error, litellm_auth::Error::MissingApiKey { .. }), - Self::InvalidResponse(_) - | Self::Transport(_) - | Self::Http(_) + Self::InvalidProvider(_) | Self::UnresolvedProvider { .. } | Self::Headers(_) => true, + Self::Llm(error) => error.is_request(), + Self::Auth(error) => error.is_request(), + Self::Http(error) => error.is_request(), + Self::Declined(_) | Self::Secret(_) | Self::Host(_) | Self::PostCallHook(_) => false, + } + } + + pub fn is_response(&self) -> bool { + match self { + Self::Llm(error) => error.is_response(), + Self::Http(error) => error.is_response(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Declined(_) + | Self::Auth(_) + | Self::Headers(_) | Self::Secret(_) + | Self::Host(_) | Self::PostCallHook(_) => false, } } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::Llm(error) => error.is_missing_credential(), + Self::Auth(error) => error.is_missing_credential(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Declined(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => false, + } + } + + pub fn declined(&self) -> Option<&Decline> { + match self { + Self::Declined(decline) => Some(decline), + Self::Llm(error) => error.declined(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Auth(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => None, + } + } + + /// Re-tag an error raised while handling an answer the provider already returned. + /// + /// A config reports the same variants on either side of the call, so a missing field or + /// a decline can mean "this request cannot be translated" before the send and "this + /// response cannot be normalized" after it. Only the second kind was billed, and a host + /// that keeps a reference implementation must not retry it. + pub fn after_send(self) -> Self { + if self.is_response() || self.upstream_response().is_some() { + return self; + } + Self::invalid_response(ErrorDetail::invalid("provider response", self)) + } + + pub fn dispatched(&self) -> bool { + match self { + Self::Llm(error) => error.dispatched(), + Self::Http(error) => error.dispatched(), + Self::Host(_) | Self::PostCallHook(_) => true, + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Declined(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Secret(_) => false, + } + } + + pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Llm(error) => error.rejected_request(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Declined(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Http(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => None, + } + } + + /// The provider's response headers on an upstream failure; empty for every other error. + pub fn upstream_headers(&self) -> &[(String, String)] { + match self { + Self::Llm(error) => error.upstream_headers(), + Self::Http(error) => error.upstream_headers(), + _ => &[], + } + } + + /// A read of a streamed body timed out after the response head arrived. + pub fn body_read_timeout(&self) -> bool { + match self { + Self::Llm(error) => error.is_body_read_timeout(), + Self::Http(error) => error.is_body_read_timeout(), + _ => false, + } + } + + /// `Some(connect)` when the call timed out, where `connect` says the connection was never + /// established; `None` for every other error. + pub fn timeout(&self) -> Option { + match self { + Self::Llm(error) => error.is_timeout(), + Self::Http(error) if error.is_timeout() => Some(error.is_connect_timeout()), + _ => None, + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str)> { + match self { + Self::Llm(error) => error.upstream_response(), + Self::Http(error) => error.upstream_response(), + Self::InvalidProvider(_) + | Self::UnresolvedProvider { .. } + | Self::Declined(_) + | Self::Auth(_) + | Self::Headers(_) + | Self::Secret(_) + | Self::Host(_) + | Self::PostCallHook(_) => None, + } + } +} + +impl From for Error { + fn from(error: litellm_http::Error) -> Self { + // Python drops the caller's copy and prefers a forwarded Authorization + // over the signature, so leave the request to it. + if error.is_computed_header() { + return Self::Declined(Decline::new("request forwards a header AWS SigV4 computes")); + } + Self::Http(error) + } } impl From for Error { fn from(error: LlmError) -> Self { match error { - LlmError::InvalidType { expected, actual } => Self::InvalidType { expected, actual }, - LlmError::MissingField(field) => Self::MissingField(field), - LlmError::InvalidRequest(message) => Self::InvalidRequest(message), - LlmError::InvalidResponse(message) => Self::InvalidResponse(message), - LlmError::Unsupported(reason) => Self::Unsupported(reason), + LlmError::Declined(decline) => Self::Declined(decline), + LlmError::Http(error) => Self::from(error), LlmError::Auth(error) => Self::Auth(error), + error @ (LlmError::InvalidType { .. } + | LlmError::MissingField(_) + | LlmError::InvalidRequest(_) + | LlmError::InvalidResponse(_)) => Self::Llm(error), } } } @@ -116,26 +242,205 @@ impl Eq for SecretError {} #[cfg(test)] mod tests { use super::Error; + use litellm_core_utils::decline::Decline; + use litellm_host::machine::MachineFault; use litellm_llms::{Error as LlmError, ErrorDetail}; use rstest::rstest; - #[test] - fn a_missing_api_key_is_the_environment_not_the_request() { - assert!( - !Error::Auth(litellm_auth::Error::MissingApiKey { - provider: "Anthropic", - environment_variable: "ANTHROPIC_API_KEY", - }) - .is_request() - ); - assert!(Error::Auth(litellm_auth::Error::InvalidHeader).is_request()); - assert!(Error::InvalidRequest("top_k".into()).is_request()); - assert!(!Error::InvalidResponse("bad json".into()).is_request()); + #[rstest] + #[case::missing_api_key( + litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + } + .into(), + false + )] + #[case::invalid_auth_header(litellm_auth::Error::InvalidHeader.into(), true)] + #[case::transformation_missing_api_key( + LlmError::Auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + }) + .into(), + false + )] + #[case::invalid_request( + LlmError::InvalidRequest(ErrorDetail::Expected { + subject: "top_k", + expected: "an integer", + }) + .into(), + true + )] + #[case::invalid_response( + LlmError::InvalidResponse(ErrorDetail::MissingField { + subject: "response", + field: "usage", + }) + .into(), + false + )] + #[case::host_driver(MachineFault::Abandoned.into(), false)] + #[case::post_call_hook_over_a_request_fault( + Error::post_call(LlmError::MissingField("usage").into()), + false + )] + fn only_the_callers_input_is_a_request_fault(#[case] error: Error, #[case] is_request: bool) { + assert_eq!(error.is_request(), is_request); } + + #[rstest] + #[case::own_auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + } + .into())] + #[case::transformation_auth( + LlmError::Auth(litellm_auth::Error::MissingApiKey { + provider: "Anthropic", + environment_variable: "ANTHROPIC_API_KEY", + }) + .into() + )] + fn a_missing_api_key_is_a_missing_credential_wherever_it_is_raised(#[case] error: Error) { + assert!(error.is_missing_credential()); + } + + #[rstest] + #[case::direct(litellm_http::Error::computed_header("authorization").into())] + #[case::through_a_transformation( + LlmError::Http(litellm_http::Error::computed_header("authorization")).into() + )] + fn a_header_the_signer_computes_is_left_to_the_host_on_every_route(#[case] error: Error) { + assert!(error.declined().is_some(), "{error:?}"); + assert!(!error.dispatched()); + } + + #[test] + fn a_transformation_decline_survives_into_the_route_error() { + let decline = Decline::new("unrecognized request parameter"); + let error = Error::from(LlmError::Declined(decline)); + assert_eq!(error.declined(), Some(&decline)); + assert!(!error.is_request()); + assert!(!error.dispatched()); + } + + #[test] + fn a_post_call_hook_failure_is_never_a_decline() { + let hook = Error::post_call(Error::Declined(Decline::new("hook"))); + assert_eq!(hook.declined(), None); + } + + #[rstest] + #[case::declined(Error::Declined(Decline::new("non-text response content block")))] + #[case::transformation_decline(LlmError::Declined(Decline::new("streaming")).into())] + #[case::missing_field(LlmError::MissingField("usage").into())] + #[case::invalid_request(Error::invalid_request(ErrorDetail::Expected { + subject: "whatever", + expected: "anything else", + }))] + #[case::auth(Error::Auth(litellm_auth::Error::InvalidHeader))] + #[case::host_driver(MachineFault::Abandoned.into())] + fn errors_after_the_send_are_response_faults_and_never_a_decline(#[case] error: Error) { + let report = litellm_error::Report::new(&error).to_string(); + let collapsed = error.after_send(); + assert!(collapsed.is_response(), "{collapsed:?}"); + assert!(!collapsed.is_request(), "{collapsed:?}"); + assert!(collapsed.dispatched(), "{collapsed:?}"); + assert_eq!(collapsed.declined(), None); + let collapsed_report = litellm_error::Report::new(&collapsed).to_string(); + assert!(collapsed_report.ends_with(&report), "{collapsed_report}"); + } + + #[test] + fn a_response_fault_is_not_wrapped_twice_after_the_send() { + let original = Error::invalid_response(ErrorDetail::MissingField { + subject: "response", + field: "usage", + }); + let report = litellm_error::Report::new(&original).to_string(); + assert_eq!( + litellm_error::Report::new(&original.after_send()).to_string(), + report + ); + } + + #[test] + fn a_provider_answer_stays_as_is_after_the_send() { + let upstream = Error::from(litellm_http::Error::upstream(429, "slow down".into())); + assert_eq!( + upstream.after_send().upstream_response(), + Some((429, "slow down")) + ); + } + + #[test] + fn an_upstream_failure_through_a_transformation_keeps_the_provider_answer() { + let error = Error::from(LlmError::Http(litellm_http::Error::upstream( + 429, + "slow down".into(), + ))); + assert_eq!(error.upstream_response(), Some((429, "slow down"))); + assert!(error.dispatched()); + } + + #[rstest] + #[case::invalid_request( + Error::invalid_request(ErrorDetail::Expected { + subject: "messages", + expected: "non-empty", + }), + Some("messages must be non-empty") + )] + #[case::missing_field( + LlmError::MissingField("max_tokens").into(), + Some("missing required field: max_tokens") + )] + #[case::invalid_response( + Error::invalid_response(ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + None + )] + #[case::provider_rejection(litellm_http::Error::upstream(400, "bad".into()).into(), None)] + #[case::unresolved_provider(Error::InvalidProvider("unknown".into()), None)] + fn a_rejected_request_names_only_what_the_caller_got_wrong( + #[case] error: Error, + #[case] expected: Option<&str>, + ) { + let rejection = error + .rejected_request() + .map(|rejection| litellm_error::Report::new(rejection).to_string()); + assert_eq!(rejection.as_deref(), expected); + } + + #[test] + fn a_post_call_hook_failure_is_not_the_providers_answer() { + let error = + Error::post_call(litellm_http::Error::upstream(500, "hook backend down".into()).into()); + assert_eq!(error.upstream_response(), None); + assert!(!error.is_request()); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.starts_with("post-call hook failed: "), "{report}"); + assert!(report.contains("hook backend down"), "{report}"); + } + + #[test] + fn a_host_driver_fault_keeps_the_fault_as_its_source() { + let error = Error::from(MachineFault::Abandoned); + assert_eq!( + litellm_error::find::(&error), + Some(&MachineFault::Abandoned) + ); + assert!(error.dispatched()); + } + #[rstest] #[case::request(true)] #[case::response(false)] - fn contextual_errors_preserve_sources_and_route_classification(#[case] request: bool) { + fn contextual_errors_keep_their_cause_and_route_classification(#[case] request: bool) { let source = serde_json::from_str::("{").unwrap_err(); let source_message = source.to_string(); let detail = ErrorDetail::invalid("test payload", source); @@ -145,16 +450,10 @@ mod tests { LlmError::InvalidResponse(detail) }); assert_eq!(error.is_request(), request); - let category = if request { "request" } else { "response" }; - assert_eq!( - error.to_string(), - format!("invalid {category}: invalid test payload: {source_message}") - ); - let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| { - error.source() - }) - .find_map(|error| error.downcast_ref::()) - .expect("the original JSON error remains available"); - assert_eq!(source.to_string(), source_message); + assert_eq!(error.is_response(), !request); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("test payload"), "{report}"); + assert_eq!(report.matches(&source_message).count(), 1, "{report}"); + assert!(litellm_error::find::(&error).is_some()); } } diff --git a/litellm-rust/crates/inference/src/outbound.rs b/litellm-rust/crates/inference/src/outbound.rs index 41fe3aa62b8..b0cc71d425d 100644 --- a/litellm-rust/crates/inference/src/outbound.rs +++ b/litellm-rust/crates/inference/src/outbound.rs @@ -1,6 +1,6 @@ use std::time::Duration; -use litellm_http::outbound::OutboundRequest; +use litellm_http::outbound::{json, signed}; use litellm_llms::base_llm::auth::Authenticated; use serde_json::Value; @@ -11,25 +11,26 @@ use serde_json::Value; fields(status) )] pub async fn send( - request: OutboundRequest, + request: reqwest::Request, client: &litellm_http::Client, -) -> Result { - request.send(client).await.inspect(|response| { - tracing::Span::current().record("status", response.status().as_u16()); - }) +) -> Result { + let response = client.execute(request).await?; + tracing::Span::current().record("status", response.status().as_u16()); + Ok(response) } /// Header credentials are already in `headers`; SigV4 is applied here, over the /// bytes that are sent. pub fn outbound_request( authenticated: Authenticated, - url: String, + url: &str, body: &Value, timeout: Option, -) -> Result { +) -> Result { let Authenticated { headers, signer } = authenticated; + let request = json(url, &headers, body, timeout)?; match signer { - None => OutboundRequest::json(url, headers, body, timeout), - Some(signer) => OutboundRequest::signed_json(url, headers, body, timeout, &signer), + None => Ok(request), + Some(signer) => signed(request, &signer), } } diff --git a/litellm-rust/crates/inference/src/provider.rs b/litellm-rust/crates/inference/src/provider.rs index 305aca85f97..fa226a4ef8d 100644 --- a/litellm-rust/crates/inference/src/provider.rs +++ b/litellm-rust/crates/inference/src/provider.rs @@ -25,11 +25,7 @@ pub fn resolve_llm_provider<'a>( custom_llm_provider: provider, }) }) - .ok_or_else(|| { - Error::InvalidProvider(format!( - "unable to resolve custom_llm_provider for {route} request" - )) - })?; + .ok_or(Error::UnresolvedProvider { route })?; let provider = custom_llm_provider .parse() .map_err(|_| Error::InvalidProvider(custom_llm_provider.to_string()))?; diff --git a/litellm-rust/crates/inference/tests/caching.rs b/litellm-rust/crates/inference/tests/caching.rs index a0a6a757565..4e8b2349ac8 100644 --- a/litellm-rust/crates/inference/tests/caching.rs +++ b/litellm-rust/crates/inference/tests/caching.rs @@ -14,6 +14,7 @@ use litellm_cache_memory::InMemoryCache; use litellm_cache_response::{ CacheOptions, CachePolicy, CacheScope, ResponseCache, ResponseCacheConfig, ResponseCacheService, }; +use litellm_core_utils::decline::Decline; use litellm_host::{ call::{CallOutput, OutputOf}, interceptors::{ @@ -204,7 +205,8 @@ async fn streamed( .chunks(3) .map(|bytes| Ok(Bytes::copy_from_slice(bytes))) .collect::>(); - let ending = fail.then_some(Err(Error::Unsupported("test transport failure"))); + let ending = + fail.then_some(Err(Error::Declined(Decline::new("test transport failure")))); Ok(CallOutput::Stream { head: (), chunks: stream::iter(chunks.into_iter().chain(ending)).boxed(), @@ -315,7 +317,7 @@ async fn a_provider_failure_never_populates_the_cache(cache: Arc Result, litellm_cache::Error> { - Err(litellm_cache::Error::Unavailable) + Err(litellm_cache::Error::Unavailable(None)) } fn set_cache( @@ -426,7 +428,7 @@ impl litellm_cache::BaseCache for UnavailableCache { _: Self::Value, _: &Self::Context, ) -> Result<(), litellm_cache::Error> { - Err(litellm_cache::Error::Unavailable) + Err(litellm_cache::Error::Unavailable(None)) } } @@ -482,7 +484,7 @@ impl Interceptors for CacheHitAccounting { ); self.key.set(key).unwrap(); if self.reject { - return Err(Error::Unsupported("cache accounting rejected")); + return Err(Error::Declined(Decline::new("cache accounting rejected"))); } Ok(()) } @@ -569,10 +571,10 @@ async fn cache_hits_notify_accounting_once_and_propagate_its_failure( .await }; if reject { - assert!(matches!( - result, - Err(Error::Unsupported("cache accounting rejected")) - )); + assert_eq!( + result.unwrap_err().declined(), + Some(&Decline::new("cache accounting rejected")) + ); } else { assert_eq!(result.unwrap(), expected); } diff --git a/litellm-rust/crates/llms-types/src/formats/audio_transcription/response.rs b/litellm-rust/crates/llms-types/src/formats/audio_transcription/response.rs index 624eb7be4e4..211d4d0f648 100644 --- a/litellm-rust/crates/llms-types/src/formats/audio_transcription/response.rs +++ b/litellm-rust/crates/llms-types/src/formats/audio_transcription/response.rs @@ -1,14 +1,31 @@ use serde_json::Value; +#[macro_rules_attribute::apply(crate::wire_type)] +pub struct AudioTranscriptionUsage { + pub input_tokens: u64, + pub output_tokens: u64, + pub total_tokens: u64, +} + #[macro_rules_attribute::apply(crate::wire_type)] pub struct AudioTranscriptionResponseData { pub text: String, + pub usage: Option, } impl AudioTranscriptionResponseData { pub fn into_json(self) -> Value { - serde_json::json!({ - "text": self.text, - }) + match self.usage { + None => serde_json::json!({"text": self.text}), + Some(usage) => serde_json::json!({ + "text": self.text, + "usage": { + "type": "tokens", + "input_tokens": usage.input_tokens, + "output_tokens": usage.output_tokens, + "total_tokens": usage.total_tokens, + }, + }), + } } } diff --git a/litellm-rust/crates/llms-types/src/formats/messages/request.rs b/litellm-rust/crates/llms-types/src/formats/messages/request.rs index 9dbfa068025..d30d21e74e7 100644 --- a/litellm-rust/crates/llms-types/src/formats/messages/request.rs +++ b/litellm-rust/crates/llms-types/src/formats/messages/request.rs @@ -264,7 +264,12 @@ pub enum ThinkingDisplay { #[macro_rules_attribute::apply(crate::wire_type)] #[derive(Default)] pub struct EnabledThinking { - #[serde(default, skip_serializing_if = "Option::is_none")] + /// An explicit `null` stays `Unrecognized(Null)`: Python rejects it, but defaults a missing key to 0. + #[serde( + default, + skip_serializing_if = "Option::is_none", + deserialize_with = "crate::recognized::present" + )] pub budget_tokens: Option>, #[serde(default, skip_serializing_if = "Option::is_none")] pub display: Option>, diff --git a/litellm-rust/crates/llms-types/src/formats/messages/response.rs b/litellm-rust/crates/llms-types/src/formats/messages/response.rs index 9b0c5d31b3a..276a30a6cb4 100644 --- a/litellm-rust/crates/llms-types/src/formats/messages/response.rs +++ b/litellm-rust/crates/llms-types/src/formats/messages/response.rs @@ -10,14 +10,29 @@ pub struct MessagesResponse { pub content: Vec, pub stop_reason: Option, pub stop_sequence: Option, - #[serde(skip_serializing_if = "Option::is_none")] + /// `Some(Value::Null)` when the provider sent an explicit `null`, which Python keeps. + #[serde( + default, + deserialize_with = "present_value", + skip_serializing_if = "Option::is_none" + )] pub usage: Option, - #[serde(skip_serializing_if = "Option::is_none")] + #[serde( + default, + deserialize_with = "present_value", + skip_serializing_if = "Option::is_none" + )] pub container: Option, #[serde(flatten)] pub extra: Map, } +fn present_value<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + >::deserialize(deserializer).map(Some) +} + #[cfg(test)] mod tests { use super::*; @@ -73,4 +88,21 @@ mod tests { assert_eq!(body.get("usage").cloned(), usage); assert_eq!(body.get("container").cloned(), container); } + + #[test] + fn explicit_null_usage_and_container_survive_a_round_trip() { + let body = json!({"id": "msg_1", "type": "message", "role": "assistant", "model": "m", + "content": [], "stop_reason": null, "stop_sequence": null, + "usage": null, "container": null}); + let parsed: MessagesResponse = serde_json::from_value(body.clone()).expect("parses"); + assert_eq!(serde_json::to_value(parsed).expect("serializable"), body); + } + + #[test] + fn absent_usage_and_container_stay_absent() { + let body = json!({"id": "msg_1", "type": "message", "role": "assistant", "model": "m", + "content": [], "stop_reason": null, "stop_sequence": null}); + let parsed: MessagesResponse = serde_json::from_value(body.clone()).expect("parses"); + assert_eq!(serde_json::to_value(parsed).expect("serializable"), body); + } } diff --git a/litellm-rust/crates/llms-types/src/providers/anthropic/beta.rs b/litellm-rust/crates/llms-types/src/providers/anthropic/beta.rs index 8429c98a88a..46c14a21c59 100644 --- a/litellm-rust/crates/llms-types/src/providers/anthropic/beta.rs +++ b/litellm-rust/crates/llms-types/src/providers/anthropic/beta.rs @@ -124,12 +124,86 @@ pub enum AnthropicBeta { ToolSearchTool20251019, #[strum(serialize = "inline-tools-2026-09-15")] InlineTools20260915, + #[strum(serialize = "afk-mode-2026-01-31")] + AfkMode20260131, + #[strum(serialize = "agent-memory-2026-07-22")] + AgentMemory20260722, + #[strum(serialize = "analytics-usage-cost-2026-08-24")] + AnalyticsUsageCost20260824, + #[strum(serialize = "cache-diagnosis-2026-04-07")] + CacheDiagnosis20260407, + #[strum(serialize = "ce-plugins-2026-09-01")] + CePlugins20260901, + #[strum(serialize = "ce-user-management-2026-07-13")] + CeUserManagement20260713, + #[strum(serialize = "code-execution-2025-05-22")] + CodeExecution20250522, + #[strum(serialize = "computer-use-2024-10-22")] + ComputerUse20241022, + #[strum(serialize = "dev-full-thinking-2025-05-14")] + DevFullThinking20250514, + #[strum(serialize = "dreaming-2026-04-21")] + Dreaming20260421, + #[strum(serialize = "experimental-claude-tag-2026-10-21")] + ExperimentalClaudeTag20261021, + #[strum(serialize = "extended-cache-ttl-2025-04-11")] + ExtendedCacheTtl20250411, + #[strum(serialize = "fallback-credit-2026-06-01")] + FallbackCredit20260601, + #[strum(serialize = "fallback-credit-2026-07-01")] + FallbackCredit20260701, + #[strum(serialize = "managed-agents-2026-04-01")] + ManagedAgents20260401, + #[strum(serialize = "mcp-client-2026-09-15")] + McpClient20260915, + #[strum(serialize = "mcp-tunnels-2026-06-22")] + McpTunnels20260622, + #[strum(serialize = "message-batches-2024-09-24")] + MessageBatches20240924, + #[strum(serialize = "message-threads-2026-08-12")] + MessageThreads20260812, + #[strum(serialize = "mid-conversation-system-clear-at-2026-08-21")] + MidConversationSystemClearAt20260821, + #[strum(serialize = "model-context-window-exceeded-2025-08-26")] + ModelContextWindowExceeded20250826, + #[strum(serialize = "output-300k-2026-03-24")] + Output300k20260324, + #[strum(serialize = "pdfs-2024-09-25")] + Pdfs20240925, + #[strum(serialize = "prompt-caching-2024-07-31")] + PromptCaching20240731, + #[strum(serialize = "redact-thinking-2026-02-12")] + RedactThinking20260212, + #[strum(serialize = "server-side-fallback-2026-06-01")] + ServerSideFallback20260601, + #[strum(serialize = "server-side-fallback-2026-07-01")] + ServerSideFallback20260701, + #[strum(serialize = "spend-limit-reads-2026-09-26")] + SpendLimitReads20260926, + #[strum(serialize = "structured-outputs-2025-12-15")] + StructuredOutputs20251215, + #[strum(serialize = "task-budgets-2026-03-13")] + TaskBudgets20260313, + #[strum(serialize = "telemetry-destinations-2026-08-11")] + TelemetryDestinations20260811, + #[strum(serialize = "thinking-token-count-2026-05-13")] + ThinkingTokenCount20260513, + #[strum(serialize = "token-counting-2024-11-01")] + TokenCounting20241101, + #[strum(serialize = "user-profiles-2026-03-24")] + UserProfiles20260324, + #[strum(serialize = "user-profiles-2026-08-18")] + UserProfiles20260818, + #[strum(serialize = "user-profiles-2026-09-04")] + UserProfiles20260904, + #[strum(serialize = "webhooks-2026-10-07")] + Webhooks20261007, #[strum(default, transparent)] Other(String), } impl AnthropicBeta { - pub const KNOWN: [Self; 40] = [ + pub const KNOWN: [Self; 77] = [ Self::Oauth20250420, Self::WebFetch20250910, Self::WebSearch20250305, @@ -170,6 +244,43 @@ impl AnthropicBeta { Self::ToolExamples20251029, Self::ToolSearchTool20251019, Self::InlineTools20260915, + Self::AfkMode20260131, + Self::AgentMemory20260722, + Self::AnalyticsUsageCost20260824, + Self::CacheDiagnosis20260407, + Self::CePlugins20260901, + Self::CeUserManagement20260713, + Self::CodeExecution20250522, + Self::ComputerUse20241022, + Self::DevFullThinking20250514, + Self::Dreaming20260421, + Self::ExperimentalClaudeTag20261021, + Self::ExtendedCacheTtl20250411, + Self::FallbackCredit20260601, + Self::FallbackCredit20260701, + Self::ManagedAgents20260401, + Self::McpClient20260915, + Self::McpTunnels20260622, + Self::MessageBatches20240924, + Self::MessageThreads20260812, + Self::MidConversationSystemClearAt20260821, + Self::ModelContextWindowExceeded20250826, + Self::Output300k20260324, + Self::Pdfs20240925, + Self::PromptCaching20240731, + Self::RedactThinking20260212, + Self::ServerSideFallback20260601, + Self::ServerSideFallback20260701, + Self::SpendLimitReads20260926, + Self::StructuredOutputs20251215, + Self::TaskBudgets20260313, + Self::TelemetryDestinations20260811, + Self::ThinkingTokenCount20260513, + Self::TokenCounting20241101, + Self::UserProfiles20260324, + Self::UserProfiles20260818, + Self::UserProfiles20260904, + Self::Webhooks20261007, ]; pub fn as_str(&self) -> &str { @@ -195,11 +306,10 @@ impl AnthropicBeta { fn rejected_by(&self) -> &'static [BetaProvider] { match self { - Self::AdvancedToolUse20251120 => { + Self::AdvancedToolUse20251120 | Self::ToolSearchTool20251019 => { &[BetaProvider::BedrockConverse, BetaProvider::Databricks] } - Self::ContextManagement20250627 => &[BetaProvider::BedrockConverse], - Self::AdvisorTool20260301 | Self::Compact20260904 => &[ + Self::AdvisorTool20260301 => &[ BetaProvider::AzureAi, BetaProvider::BedrockConverse, BetaProvider::Bedrock, @@ -207,88 +317,99 @@ impl AnthropicBeta { BetaProvider::VertexAi, BetaProvider::Databricks, ], - Self::Bash20241022 + Self::AfkMode20260131 + | Self::CePlugins20260901 + | Self::CeUserManagement20260713 + | Self::Compact20260904 + | Self::Dreaming20260421 + | Self::ExperimentalClaudeTag20261021 + | Self::ExtendedCacheTtl20250411 + | Self::FallbackCredit20260601 + | Self::FallbackCredit20260701 + | Self::InlineTools20260915 + | Self::ManagedAgents20260401 + | Self::McpTunnels20260622 + | Self::MidConversationSystemClearAt20260821 + | Self::ModelContextWindowExceeded20250826 + | Self::Oauth20250420 + | Self::Pdfs20240925 + | Self::PromptCaching20240731 + | Self::PromptCachingScope20260105 + | Self::RedactThinking20260212 + | Self::ServerSideFallback20260601 + | Self::SpendLimitReads20260926 + | Self::StructuredOutputs20251215 + | Self::TaskBudgets20260313 + | Self::TelemetryDestinations20260811 + | Self::ThinkingTokenCount20260513 + | Self::TokenCounting20241101 + | Self::UserProfiles20260324 + | Self::UserProfiles20260818 + | Self::UserProfiles20260904 => &[ + BetaProvider::BedrockConverse, + BetaProvider::Bedrock, + BetaProvider::BedrockMantle, + BetaProvider::Databricks, + ], + Self::AgentMemory20260722 + | Self::Bash20241022 | Self::Bash20250124 + | Self::DevFullThinking20250514 | Self::McpServers20251204 | Self::StructuredOutput20240301 | Self::TextEditor20241022 | Self::TextEditor20250124 => BetaProvider::VARIANTS, - Self::ClaudeCode20250219 | Self::ToolExamples20251029 => &[ - BetaProvider::Anthropic, - BetaProvider::AzureAi, - BetaProvider::BedrockConverse, - BetaProvider::Bedrock, - BetaProvider::VertexAi, - BetaProvider::Databricks, - ], - Self::CodeExecution20250825 + Self::AnalyticsUsageCost20260824 + | Self::CacheDiagnosis20260407 + | Self::CodeExecution20250522 + | Self::CodeExecution20250825 + | Self::ComputerUse20241022 + | Self::FastMode20260201 | Self::FilesApi20250414 | Self::McpClient20250404 | Self::McpClient20251120 - | Self::PromptCachingScope20260105 + | Self::McpClient20260915 + | Self::MessageBatches20240924 + | Self::MessageThreads20260812 + | Self::Output300k20260324 + | Self::ServerSideFallback20260701 | Self::Skills20251002 - | Self::WebFetch20250910 => &[ + | Self::WebFetch20250910 + | Self::Webhooks20261007 => &[ BetaProvider::BedrockConverse, BetaProvider::Bedrock, BetaProvider::BedrockMantle, BetaProvider::VertexAi, BetaProvider::Databricks, ], - Self::Compact20260112 => &[BetaProvider::AzureAi, BetaProvider::BedrockConverse], + Self::ClaudeCode20250219 + | Self::PerTurnControl20260701 + | Self::ToolExamples20251029 => &[ + BetaProvider::BedrockConverse, + BetaProvider::Bedrock, + BetaProvider::Databricks, + ], + Self::Compact20260112 + | Self::ContextManagement20250627 + | Self::DangerousToolUse20260903 => &[BetaProvider::BedrockConverse], Self::ComputerUse20250124 | Self::ComputerUse20251124 | Self::Context1m20250807 - | Self::ThinkingBindingControls20260801 => &[], - Self::DangerousToolUse20260903 => &[BetaProvider::BedrockConverse], - Self::Effort20251124 => &[BetaProvider::VertexAi], - Self::FastMode20260201 | Self::Oauth20250420 => &[ - BetaProvider::AzureAi, - BetaProvider::BedrockConverse, - BetaProvider::Bedrock, - BetaProvider::BedrockMantle, - BetaProvider::VertexAi, - BetaProvider::Databricks, - ], - Self::FineGrainedToolStreaming20250514 => { - &[BetaProvider::AzureAi, BetaProvider::VertexAi] - } - Self::InterleavedThinking20250514 | Self::WebSearch20250305 => { - &[BetaProvider::BedrockConverse, BetaProvider::Bedrock] - } + | Self::Effort20251124 + | Self::FineGrainedToolStreaming20250514 + | Self::ThinkingBindingControls20260801 + | Self::ThinkingDisplayUpdates20260818 => &[], + Self::InterleavedThinking20250514 + | Self::Output128k20250219 + | Self::TokenEfficientTools20250219 + | Self::WebSearch20250305 => &[BetaProvider::BedrockConverse, BetaProvider::Bedrock], Self::MidConversationOutputConfig20260701 | Self::MidConversationToolChanges20260701 => &[ - BetaProvider::AzureAi, BetaProvider::BedrockConverse, BetaProvider::BedrockMantle, - BetaProvider::VertexAi, - BetaProvider::Databricks, - ], - Self::Output128k20250219 | Self::TokenEfficientTools20250219 => &[ - BetaProvider::AzureAi, - BetaProvider::BedrockConverse, - BetaProvider::Bedrock, - BetaProvider::VertexAi, - ], - Self::PerTurnControl20260701 => &[ - BetaProvider::BedrockConverse, - BetaProvider::Bedrock, - BetaProvider::Databricks, - ], - Self::StructuredOutputs20251113 => &[BetaProvider::Bedrock, BetaProvider::VertexAi], - Self::ThinkingDisplayUpdates20260818 => &[BetaProvider::Databricks], - Self::ToolSearchTool20251019 => &[ - BetaProvider::Anthropic, - BetaProvider::AzureAi, - BetaProvider::BedrockConverse, - BetaProvider::Databricks, - ], - Self::InlineTools20260915 => &[ - BetaProvider::AzureAi, - BetaProvider::BedrockConverse, - BetaProvider::Bedrock, - BetaProvider::BedrockMantle, BetaProvider::Databricks, ], + Self::StructuredOutputs20251113 => &[BetaProvider::Bedrock], Self::Other(_) => BetaProvider::VARIANTS, } } @@ -441,7 +562,44 @@ mod tests { AnthropicBeta::TokenEfficientTools20250219, AnthropicBeta::ToolExamples20251029, AnthropicBeta::ToolSearchTool20251019, - AnthropicBeta::InlineTools20260915 + AnthropicBeta::InlineTools20260915, + AnthropicBeta::AfkMode20260131, + AnthropicBeta::AgentMemory20260722, + AnthropicBeta::AnalyticsUsageCost20260824, + AnthropicBeta::CacheDiagnosis20260407, + AnthropicBeta::CePlugins20260901, + AnthropicBeta::CeUserManagement20260713, + AnthropicBeta::CodeExecution20250522, + AnthropicBeta::ComputerUse20241022, + AnthropicBeta::DevFullThinking20250514, + AnthropicBeta::Dreaming20260421, + AnthropicBeta::ExperimentalClaudeTag20261021, + AnthropicBeta::ExtendedCacheTtl20250411, + AnthropicBeta::FallbackCredit20260601, + AnthropicBeta::FallbackCredit20260701, + AnthropicBeta::ManagedAgents20260401, + AnthropicBeta::McpClient20260915, + AnthropicBeta::McpTunnels20260622, + AnthropicBeta::MessageBatches20240924, + AnthropicBeta::MessageThreads20260812, + AnthropicBeta::MidConversationSystemClearAt20260821, + AnthropicBeta::ModelContextWindowExceeded20250826, + AnthropicBeta::Output300k20260324, + AnthropicBeta::Pdfs20240925, + AnthropicBeta::PromptCaching20240731, + AnthropicBeta::RedactThinking20260212, + AnthropicBeta::ServerSideFallback20260601, + AnthropicBeta::ServerSideFallback20260701, + AnthropicBeta::SpendLimitReads20260926, + AnthropicBeta::StructuredOutputs20251215, + AnthropicBeta::TaskBudgets20260313, + AnthropicBeta::TelemetryDestinations20260811, + AnthropicBeta::ThinkingTokenCount20260513, + AnthropicBeta::TokenCounting20241101, + AnthropicBeta::UserProfiles20260324, + AnthropicBeta::UserProfiles20260818, + AnthropicBeta::UserProfiles20260904, + AnthropicBeta::Webhooks20261007 )] beta: AnthropicBeta, ) { @@ -483,7 +641,7 @@ mod tests { let config = beta_headers_config(); let config_columns: Vec = config .keys() - .filter(|column| column.as_str() != "description") + .filter(|column| column.as_str() != "description" && !column.ends_with("_header_only")) .cloned() .collect(); let beta_provider_columns: Vec = BetaProvider::VARIANTS @@ -502,12 +660,16 @@ mod tests { let config = beta_headers_config(); for provider in BetaProvider::VARIANTS { for beta in &AnthropicBeta::KNOWN { - let expected = config - .get(provider.as_ref()) - .and_then(serde_json::Value::as_object) - .and_then(|column| column.get(beta.as_str())) - .and_then(serde_json::Value::as_str) - .map(str::to_string); + let cell = |column: &str| { + config + .get(column) + .and_then(serde_json::Value::as_object) + .and_then(|values| values.get(beta.as_str())) + .and_then(serde_json::Value::as_str) + .map(str::to_string) + }; + let expected = + cell(&format!("{provider}_header_only")).or_else(|| cell(provider.as_ref())); let actual = beta.on(*provider).map(|name| name.to_string()); assert_eq!(actual, expected, "provider {provider}, beta {beta}"); if matches!(beta, AnthropicBeta::AdvancedToolUse20251120) @@ -537,7 +699,7 @@ mod tests { BetaProvider::Anthropic, Some(AnthropicBeta::Oauth20250420) )] - #[case::rejected("effort-2025-11-24", BetaProvider::VertexAi, None)] + #[case::rejected("skills-2025-10-02", BetaProvider::VertexAi, None)] #[case::unknown("example-beta-2099-01-01", BetaProvider::Anthropic, None)] fn on_resolves_known_spellings_held_as_other( #[case] raw: &str, @@ -566,12 +728,11 @@ mod tests { #[test] fn unknown_values_are_kept_verbatim() { - let parsed: AnthropicBeta = "claude-code-20250219".parse().unwrap(); - assert_eq!( - parsed, - AnthropicBeta::Other("claude-code-20250219".to_string()) + let parsed: AnthropicBeta = "example-beta-2099-01-01".parse().unwrap(); + assert!( + matches!(&parsed, AnthropicBeta::Other(value) if value == "example-beta-2099-01-01") ); - assert_eq!(parsed.to_string(), "claude-code-20250219"); + assert_eq!(parsed.to_string(), "example-beta-2099-01-01"); } #[test] diff --git a/litellm-rust/crates/llms-types/src/recognized.rs b/litellm-rust/crates/llms-types/src/recognized.rs index 58e8dd7a2e7..0545389afbc 100644 --- a/litellm-rust/crates/llms-types/src/recognized.rs +++ b/litellm-rust/crates/llms-types/src/recognized.rs @@ -7,6 +7,16 @@ pub enum Recognized { Unrecognized(Value), } +/// `deserialize_with` for an optional `Recognized` field whose explicit `null` must stay +/// distinguishable from an absent key: `null` becomes `Some(Unrecognized(Null))`. +pub fn present<'de, D, T>(deserializer: D) -> Result>, D::Error> +where + D: serde::Deserializer<'de>, + Recognized: serde::Deserialize<'de>, +{ + as serde::Deserialize<'de>>::deserialize(deserializer).map(Some) +} + impl Recognized { pub fn known(&self) -> Option<&T> { match self { diff --git a/litellm-rust/crates/llms/Cargo.toml b/litellm-rust/crates/llms/Cargo.toml index 2d3ea85d1fc..79ee17e8b97 100644 --- a/litellm-rust/crates/llms/Cargo.toml +++ b/litellm-rust/crates/llms/Cargo.toml @@ -9,8 +9,10 @@ repository.workspace = true test-support = ["litellm-http/test-support"] [dependencies] +litellm-secret-value.workspace = true litellm-llms-types.workspace = true litellm-core-utils.workspace = true +litellm-error.workspace = true litellm-auth = { workspace = true, features = ["aws", "azure", "gcp"] } litellm-auth-aws.workspace = true litellm-auth-azure.workspace = true diff --git a/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs b/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs index b8ea701af99..0513f88038d 100644 --- a/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/batches/transformation.rs @@ -1,3 +1,4 @@ +use litellm_core_utils::decline::Decline; use litellm_llms_types::formats::batches::{BatchRequestCounts, BatchResponse, BatchStatus}; use litellm_llms_types::formats::messages::MessagesResponse; use litellm_llms_types::providers::anthropic::{BATCHES_PATH, MESSAGES_PATH}; @@ -127,7 +128,9 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { } fn transform_create_batch_request(&self) -> Result { - Err(Error::Unsupported("Anthropic message batch creation")) + Err(Error::Declined(Decline { + reason: "Anthropic message batch creation", + })) } fn transform_create_batch_response( @@ -135,7 +138,9 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { _response: AnthropicMessageBatch, _now: i64, ) -> Result { - Err(Error::Unsupported("Anthropic message batch creation")) + Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "Anthropic message batch creation", + ))) } fn retrieve_batch_url( @@ -149,7 +154,12 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { } let mut url = batches_base_url(api_base, env_lookup)?; url.path_segments_mut() - .map_err(|_| Error::InvalidRequest("Anthropic API base cannot be a base URL".into()))? + .map_err(|()| { + Error::InvalidRequest(crate::ErrorDetail::Expected { + subject: "Anthropic API base", + expected: "a base URL", + }) + })? .push(batch_id); Ok(url.into()) } @@ -217,11 +227,11 @@ impl AnthropicBatchesConfig for AnthropicBatchesTransformation { .map(|(index, line)| { let record: BatchResultRecord = serde_json::from_str(line.trim()).map_err(|error| { - Error::InvalidResponse(crate::ErrorDetail::InvalidLine { - subject: "Anthropic batch result", - line: index + 1, - source: crate::ErrorSource::new(error), - }) + Error::InvalidResponse(crate::ErrorDetail::invalid_line( + "Anthropic batch result", + index + 1, + error, + )) })?; match record.result { BatchResult::Succeeded { message } => Ok(*message), @@ -314,30 +324,40 @@ mod tests { assert_eq!(messages[0].id, "msg_1"); } - #[test] - fn reports_malformed_and_unsuccessful_batch_results() { - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_batch_results("not-json"), - Err(Error::InvalidResponse(message)) if message.to_string().contains("line 1") - )); - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_batch_results( - r#"{"result":{"type":"errored","error":{"type":"invalid_request_error"}}}"# - ), - Err(Error::InvalidResponse(message)) if message.to_string().contains("request failed") - )); + #[rstest::rstest] + #[case::malformed("not-json", "line 1")] + #[case::errored( + r#"{"result":{"type":"errored","error":{"type":"invalid_request_error"}}}"#, + "request failed" + )] + fn reports_malformed_and_unsuccessful_batch_results(#[case] body: &str, #[case] fact: &str) { + let error = ANTHROPIC_BATCHES_TRANSFORMATION + .transform_batch_results(body) + .unwrap_err(); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains(fact) + ); } - #[test] - fn preserves_python_placeholder_for_batch_creation() { - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_create_batch_request(), - Err(Error::Unsupported("Anthropic message batch creation")) - )); + #[rstest::rstest] + fn batch_creation_only_declines_before_a_provider_response() { + let error = ANTHROPIC_BATCHES_TRANSFORMATION + .transform_create_batch_request() + .unwrap_err(); + assert!(!error.dispatched()); + assert_eq!( + error.declined().map(|decline| decline.reason), + Some("Anthropic message batch creation") + ); let response: AnthropicMessageBatch = serde_json::from_value(json!({})).unwrap(); - assert!(matches!( - ANTHROPIC_BATCHES_TRANSFORMATION.transform_create_batch_response(response, 0), - Err(Error::Unsupported("Anthropic message batch creation")) - )); + let error = ANTHROPIC_BATCHES_TRANSFORMATION + .transform_create_batch_response(response, 0) + .unwrap_err(); + assert!(error.dispatched()); + assert!(error.is_response()); + assert!(error.declined().is_none()); } } diff --git a/litellm-rust/crates/llms/src/anthropic/beta_headers.rs b/litellm-rust/crates/llms/src/anthropic/beta_headers.rs index a0843621043..7842b478dce 100644 --- a/litellm-rust/crates/llms/src/anthropic/beta_headers.rs +++ b/litellm-rust/crates/llms/src/anthropic/beta_headers.rs @@ -96,7 +96,10 @@ mod tests { } #[rstest] - #[case::azure_ai(BetaProvider::AzureAi, "fast-mode-2026-02-01,example-beta-2099-01-01")] + #[case::azure_ai( + BetaProvider::AzureAi, + "advisor-tool-2026-03-01,example-beta-2099-01-01" + )] #[case::anthropic(BetaProvider::Anthropic, "bash_20241022")] #[case::vertex_ai(BetaProvider::VertexAi, " , ")] fn filter_removes_header_when_nothing_survives( diff --git a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs b/litellm-rust/crates/llms/src/anthropic/chat/handler.rs deleted file mode 100644 index eaee006f228..00000000000 --- a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs +++ /dev/null @@ -1,163 +0,0 @@ -use std::collections::HashMap; - -use litellm_llms_types::formats::{ - chat_completions::{ - ChatCompletionChunk, ChatCompletionThinkingBlock, ChatCompletionToolCallChunk, - ChatCompletionsUsage, - }, - messages::streaming::{ - MessagesContentBlock, MessagesContentBlockDelta, MessagesStreamEvent, MessagesStreamUsage, - }, -}; -use serde_json::Value; - -use crate::{ - Error, - base_llm::{base_model_iterator::StreamTransformer, chat::streaming::StreamShape}, -}; - -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum AnthropicJsonChunkType { - ValidJson, - AccumulatedJson, -} - -#[derive(Clone, Debug, Eq, PartialEq)] -pub enum AnthropicContentBlockType { - Text, - ToolUse, - ServerToolUse, - Thinking, - RedactedThinking, - Compaction, - ToolResult(String), - Other(String), -} - -#[derive(Clone, Debug, PartialEq)] -pub struct AnthropicContentBlockDeltaEvent { - pub index: u64, - pub delta: MessagesContentBlockDelta, -} - -pub struct ModelResponseIterator { - pub content_blocks: Vec, - pub tool_index: i64, - pub json_mode: bool, - pub speed: Option, - pub tool_name_reverse_map: HashMap, - pub response_id: String, - pub served_model: Option, - pub is_response_format_tool: bool, - pub converted_response_format_tool: bool, - pub accumulated_json: String, - pub chunk_type: AnthropicJsonChunkType, - pub current_content_block_type: Option, - pub web_search_results: Vec, - pub web_search_calls: HashMap, - pub compaction_blocks: Vec, - pub reasoning_content_chunks: Vec, - pub server_tool_inputs: HashMap, - pub tool_results: Vec, - pub current_server_tool_id: Option, - pub container_id: Option, -} - -impl ModelResponseIterator { - pub fn new(_shape: StreamShape) -> Self { - todo!() - } - - pub fn check_empty_tool_call_args(&self) -> bool { - todo!() - } - - pub fn handle_usage(&mut self, _usage: MessagesStreamUsage) -> ChatCompletionsUsage { - todo!() - } - - pub fn handle_content_block_delta( - &mut self, - _index: u64, - _delta: MessagesContentBlockDelta, - ) -> ( - String, - Option, - Vec, - Option, - Option, - ) { - todo!() - } - - pub fn handle_content_block_start( - &mut self, - _index: u64, - _content_block: MessagesContentBlock, - ) -> Result { - todo!() - } - - pub fn handle_json_mode_chunk( - &mut self, - _text: String, - _tool_use: Option, - ) -> (String, Option) { - todo!() - } - - pub fn handle_accumulated_json_chunk( - &mut self, - _data: &str, - _is_final: bool, - ) -> Result, Error> { - todo!() - } - - pub fn handle_redacted_thinking_content( - &mut self, - _content_block: &MessagesContentBlock, - ) -> Vec { - todo!() - } - - pub fn web_search_call_snapshot(&self) -> HashMap { - todo!() - } - - pub fn complete_web_search_call(&mut self, _result: Value) { - todo!() - } - - pub fn build_code_interpreter_results(&self) -> Vec { - todo!() - } - - pub fn handle_message_delta( - &mut self, - _event: MessagesStreamEvent, - ) -> (Option, Option, Option) { - todo!() - } - - pub fn chunk_parser( - &mut self, - _event: MessagesStreamEvent, - ) -> Result { - todo!() - } -} - -impl StreamTransformer for ModelResponseIterator { - type Input = MessagesStreamEvent; - type Output = ChatCompletionChunk; - type Error = Error; - - fn transform(&mut self, _input: Self::Input) -> Result, Self::Error> { - todo!() - } - - fn finish(&mut self) -> Result, Self::Error> { - todo!() - } -} diff --git a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs index f0050b7dc71..f239b6921fa 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs @@ -1,2 +1 @@ -pub mod handler; pub mod transformation; diff --git a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs index 91baa247ac8..164dbce1264 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs @@ -1,4 +1,4 @@ -use litellm_auth::SecretValue; +use litellm_core_utils::decline::Decline; use litellm_core_utils::{ core_helpers::{finish_reason_for, unix_now, usage_from_parts}, prompt_templates::factory::{Conversation, build_conversation}, @@ -9,28 +9,25 @@ use litellm_llms_types::{ }, providers::anthropic::DEFAULT_HEADERS, }; +use litellm_secret_value::SecretValue; use serde::Deserialize; use serde_json::{Map, Value, json}; use crate::{ Error, - anthropic::{ - chat::handler::ModelResponseIterator, - common_utils::{ - API_KEY_PLACEMENT, complete_anthropic_url, forwarded_oauth_bearer, - resolve_anthropic_api_key, - }, + anthropic::common_utils::{ + API_KEY_PLACEMENT, complete_anthropic_url, forwarded_oauth_bearer, + resolve_anthropic_api_key, }, base_llm::{ auth::AuthScheme, chat::{ - streaming::{ChatStream, StreamShape}, + streaming::StreamShape, transformation::{ BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, - Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param, + ValidatedEnvironment, unsupported_message, unsupported_param, }, }, - messages::streaming::event_stream as messages_event_stream, }, }; @@ -130,15 +127,14 @@ impl BaseConfig for AnthropicConfig { serde_json::from_value(response.body).map_err(|error| { Error::InvalidResponse(crate::ErrorDetail::invalid("messages response", error)) })?; - // The route declines tool and thinking requests, so a non-text block - // means the response carries something this path never asked for. - // Decline rather than silently dropping it; the host falls back. if body .content .iter() .any(|block| matches!(block, TextResponseBlock::Other)) { - return Err(Error::Unsupported("non-text response content block")); + return Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "non-text response content block", + ))); } let text: String = body .content @@ -194,13 +190,6 @@ impl BaseConfig for AnthropicConfig { Ok(ValidatedEnvironment { headers, auth }) } - fn model_response_iterator(&self, shape: StreamShape) -> Option { - Some(ChatStream::new( - messages_event_stream, - ModelResponseIterator::new(shape), - )) - } - fn default_headers(&self) -> &'static [(&'static str, &'static str)] { DEFAULT_HEADERS } @@ -214,7 +203,7 @@ impl BaseConfig for AnthropicConfig { &self, messages: &[ChatMessage], optional_params: &Map, - ) -> Option { + ) -> Option { unsupported_param(self.supported_openai_param_mappings(), &[], optional_params) .or_else(|| messages.iter().find_map(unsupported_message)) // Anthropic rejects a request whose first turn is not a user turn. @@ -222,7 +211,7 @@ impl BaseConfig for AnthropicConfig { // core cannot observe, so decline instead of guessing. .or_else(|| { (!build_conversation(messages).opens_on_user_turn()) - .then_some(Unsupported("conversation does not open on a user turn")) + .then_some(Decline::new("conversation does not open on a user turn")) }) } } @@ -271,9 +260,10 @@ mod tests { anthropic::chat::transformation::ANTHROPIC_CHAT_COMPLETIONS_CONFIG, base_llm::{ auth::AuthScheme, - chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported}, + chat::transformation::{BaseConfig, ProviderChatResponseData}, }, }; + use litellm_core_utils::decline::Decline; use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage}; use rstest::rstest; use serde_json::{Map, Value, json}; @@ -301,7 +291,7 @@ mod tests { .transform_response("claude-sonnet-4-5", ProviderChatResponseData { body }) } - fn reason(msgs: Value, opts: Value) -> Option { + fn reason(msgs: Value, opts: Value) -> Option { ANTHROPIC_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts)) } @@ -409,7 +399,7 @@ mod tests { json!([{"role": "user", "content": "hi"}]), json!({"top_k": 40}) ), - Some(Unsupported("unrecognized request parameter")) + Some(Decline::new("unrecognized request parameter")) ); } @@ -420,7 +410,7 @@ mod tests { json!([{"role": "user", "content": "hi"}]), json!({"stream": true, "max_tokens": 16}) ), - Some(Unsupported("streaming")) + Some(Decline::new("streaming")) ); } @@ -445,7 +435,7 @@ mod tests { fn declines_any_param_outside_the_allowlist(#[case] param: Value) { assert_eq!( reason(json!([{"role": "user", "content": "hi"}]), param.clone()), - Some(Unsupported("unrecognized request parameter")), + Some(Decline::new("unrecognized request parameter")), "expected {param} to decline" ); } @@ -463,7 +453,7 @@ mod tests { ]), json!({}) ), - Some(Unsupported("unrecognized message field")) + Some(Decline::new("unrecognized message field")) ); assert_eq!( reason( @@ -473,7 +463,7 @@ mod tests { ]), json!({}) ), - Some(Unsupported("unrecognized message field")) + Some(Decline::new("unrecognized message field")) ); assert_eq!( reason( @@ -483,7 +473,7 @@ mod tests { ]}]), json!({}) ), - Some(Unsupported("non-text message content")) + Some(Decline::new("non-text message content")) ); assert_eq!( reason( @@ -493,7 +483,7 @@ mod tests { ]}]), json!({}) ), - Some(Unsupported("non-text message content")) + Some(Decline::new("non-text message content")) ); } @@ -504,7 +494,7 @@ mod tests { // than declining to Python before it. assert_eq!( reason(json!([{"role": "user", "content": []}]), json!({})), - Some(Unsupported("message without content")) + Some(Decline::new("message without content")) ); assert_eq!( reason( @@ -525,7 +515,7 @@ mod tests { ]), json!({}) ), - Some(Unsupported("conversation does not open on a user turn")) + Some(Decline::new("conversation does not open on a user turn")) ); } @@ -658,7 +648,7 @@ mod tests { } #[test] - fn declines_a_response_carrying_a_non_text_block() { + fn a_response_carrying_a_non_text_block_is_invalid() { let err = transform_response(json!({ "model": "claude-sonnet-4-5", "content": [{"type": "tool_use", "id": "t1", "name": "f", "input": {}}], @@ -666,7 +656,12 @@ mod tests { "usage": {"input_tokens": 1, "output_tokens": 1} })) .expect_err("non-text block"); - assert_eq!(err, Error::Unsupported("non-text response content block")); + assert_eq!( + err, + Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "non-text response content block" + )) + ); } #[rstest::rstest] @@ -676,16 +671,11 @@ mod tests { #[case::missing_model(json!({"content": [], "usage": {"input_tokens": 1, "output_tokens": 1}}))] fn errors_on_a_response_missing_required_fields(#[case] body: Value) { let error = transform_response(body).expect_err("invalid response"); - assert!(matches!(error, Error::InvalidResponse(_))); - let source = std::iter::successors(Some(&error as &dyn std::error::Error), |error| { - error.source() - }) - .find_map(|error| error.downcast_ref::()) - .expect("the JSON decoding source is preserved"); - assert_eq!( - error.to_string(), - format!("invalid response: invalid messages response: {source}") - ); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + let cause = litellm_error::find::(&error).expect("JSON cause"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("invalid messages response"), "{report}"); + assert_eq!(report.matches(&cause.to_string()).count(), 1, "{report}"); } #[test] diff --git a/litellm-rust/crates/llms/src/anthropic/common_utils.rs b/litellm-rust/crates/llms/src/anthropic/common_utils.rs index 7991a57353e..13653f438f7 100644 --- a/litellm-rust/crates/llms/src/anthropic/common_utils.rs +++ b/litellm-rust/crates/llms/src/anthropic/common_utils.rs @@ -1,5 +1,7 @@ use crate::base_llm::messages::context::MessagesModelCapabilities; -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_http::request::{ has_header, header_value, header_values, with_header, without_headers, @@ -152,11 +154,42 @@ pub fn complete_anthropic_url( ) -> String { let api_base = resolve_anthropic_api_base(api_base, env_lookup); - let api_base = api_base.trim_end_matches('/'); - if api_base.ends_with(MESSAGES_PATH) { + ensure_api_route(&api_base, MESSAGES_PATH) +} + +/// Python's `urlsplit(url).path`, for the URLs an `api_base` holds. +fn url_path(url: &str) -> &str { + let before_fragment = url.split('#').next().unwrap_or_default(); + let before_query = before_fragment.split('?').next().unwrap_or_default(); + match before_query.split_once("://") { + Some((_, rest)) => rest.find('/').map_or("", |slash| &rest[slash..]), + None => before_query, + } +} + +/// Python's `_strip_trailing_slashes_keeping_the_scheme`. +fn strip_trailing_slashes(api_base: &str) -> String { + let stripped = api_base.trim_end_matches('/'); + if stripped.ends_with(':') && !stripped.contains('/') && !stripped.contains('?') { return api_base.to_string(); } - format!("{api_base}{MESSAGES_PATH}") + match stripped.split_once('?') { + Some((before, query)) if url_path(stripped).ends_with('/') => { + format!("{}?{query}", before.trim_end_matches('/')) + } + _ => stripped.to_string(), + } +} + +/// Python's `ensure_api_route`: an `api_base` whose path already ends in the route is kept as +/// it is, query string included; anything else gets the route appended to the whole string. +fn ensure_api_route(api_base: &str, route: &str) -> String { + let base = strip_trailing_slashes(api_base); + let route = route.trim_start_matches('/'); + if url_path(&base).ends_with(&format!("/{route}")) { + return base; + } + format!("{}/{route}", strip_trailing_slashes(&base)) } pub fn existing_betas(headers: &[(String, String)]) -> BetaSet { @@ -1920,4 +1953,24 @@ mod tests { expected ); } + + #[rstest] + #[case::bare_host("https://api.example.com", "https://api.example.com/v1/messages")] + #[case::trailing_slash("https://api.example.com/", "https://api.example.com/v1/messages")] + #[case::route_present( + "https://api.example.com/v1/messages/", + "https://api.example.com/v1/messages" + )] + #[case::route_and_query( + "https://api.example.com/v1/messages?beta=true", + "https://api.example.com/v1/messages?beta=true" + )] + #[case::query_without_route( + "https://api.example.com/?x=1", + "https://api.example.com?x=1/v1/messages" + )] + #[case::scheme_only("https://", "https:///v1/messages")] + fn the_messages_url_follows_python_ensure_api_route(#[case] api_base: &str, #[case] url: &str) { + assert_eq!(complete_anthropic_url(Some(api_base), &|_| None), url); + } } diff --git a/litellm-rust/crates/llms/src/anthropic/messages/handler.rs b/litellm-rust/crates/llms/src/anthropic/messages/handler.rs index 6e8de65d6fa..54f924ddec6 100644 --- a/litellm-rust/crates/llms/src/anthropic/messages/handler.rs +++ b/litellm-rust/crates/llms/src/anthropic/messages/handler.rs @@ -208,24 +208,29 @@ mod tests { #[case::empty(json!({}), Ok(json!({})))] #[case::numeric_user_id( json!({"user_id": 123}), - Err(Error::InvalidRequest("metadata.user_id must be a string, got 123".to_string().into())), + Err(invalid_value("metadata.user_id", "a string", json!(123))), )] #[case::boolean_user_id( json!({"user_id": true}), - Err(Error::InvalidRequest("metadata.user_id must be a string, got true".to_string().into())), + Err(invalid_value("metadata.user_id", "a string", json!(true))), )] #[case::not_an_object( json!(["u-1"]), - Err(Error::InvalidRequest(r#"metadata must be an object, got ["u-1"]"#.to_string().into())), + Err(invalid_value("metadata", "an object", json!(["u-1"]))), )] fn validate_anthropic_api_metadata_passes_only_a_string_user_id( #[case] metadata: Value, #[case] expected: Result, ) { - assert_eq!( - validate_anthropic_api_metadata(&metadata).map_err(|error| error.to_string()), - expected.map_err(|error| error.to_string()), - ); + assert_eq!(validate_anthropic_api_metadata(&metadata), expected); + } + + fn invalid_value(field: &'static str, expected: &'static str, actual: Value) -> Error { + Error::InvalidRequest(crate::ErrorDetail::InvalidValue { + field, + expected, + actual, + }) } #[rstest] diff --git a/litellm-rust/crates/llms/src/anthropic/messages/thinking.rs b/litellm-rust/crates/llms/src/anthropic/messages/thinking.rs index df1ae13fd28..ae3488a21f9 100644 --- a/litellm-rust/crates/llms/src/anthropic/messages/thinking.rs +++ b/litellm-rust/crates/llms/src/anthropic/messages/thinking.rs @@ -2,8 +2,8 @@ use litellm_llms_types::{ formats::{ chat_completions::ReasoningEffort, messages::{ - EffortLevel, MessagesOptionalParams, MessagesRequest, OutputConfig, ThinkingConfig, - ThinkingDisplay, + AdaptiveThinking, EffortLevel, MessagesOptionalParams, MessagesRequest, OutputConfig, + ThinkingConfig, ThinkingDisplay, }, }, recognized::Recognized, @@ -36,16 +36,17 @@ fn budget_for_effort(budgets: &ThinkingBudgets, effort: ReasoningEffort) -> Opti fn effort_for_budget( budgets: &ThinkingBudgets, - budget_tokens: u64, + budget_tokens: i64, capabilities: &MessagesModelCapabilities, ) -> EffortLevel { - if budget_tokens >= budgets.xhigh && capabilities.effort_tiers.xhigh { + let reaches = |tier: u64| i128::from(budget_tokens) >= i128::from(tier); + if reaches(budgets.xhigh) && capabilities.effort_tiers.xhigh { return EffortLevel::Xhigh; } - if budget_tokens >= budgets.high { + if reaches(budgets.high) { return EffortLevel::High; } - if budget_tokens >= budgets.medium { + if reaches(budgets.medium) { return EffortLevel::Medium; } EffortLevel::Low @@ -231,32 +232,129 @@ fn drop_disabled_thinking(request: MessagesRequest, context: &ThinkingContext) - } } +/// `_numeric_budget_tokens`: Python reads any int, float (truncated) or int-parseable string +/// as a budget, and nothing else. +fn numeric_budget_tokens(budget: &Recognized) -> Option { + let value = match budget { + Recognized::Known(budget) => return Some(i64::try_from(*budget).unwrap_or(i64::MAX)), + Recognized::Unrecognized(value) => value, + }; + match value { + Value::Number(number) => number + .as_i64() + .or_else(|| number.as_f64().map(|float| float.trunc() as i64)), + Value::String(text) => python_int(text), + Value::Null | Value::Bool(_) | Value::Array(_) | Value::Object(_) => None, + } +} + +/// Python's `int(str)`: surrounding whitespace, a sign, and single underscores between digits. +fn python_int(text: &str) -> Option { + let trimmed = text.trim(); + let digits = trimmed.trim_start_matches(['+', '-']); + let well_formed = trimmed.len() - digits.len() <= 1 + && !digits.is_empty() + && !digits.starts_with('_') + && !digits.ends_with('_') + && !digits.contains("__") + && digits.chars().all(|ch| ch.is_ascii_digit() || ch == '_'); + if !well_formed { + return None; + } + let cleaned: String = trimmed.chars().filter(|ch| *ch != '_').collect(); + cleaned.parse::().ok().or_else(|| { + Some(if cleaned.starts_with('-') { + i64::MIN + } else { + i64::MAX + }) + }) +} + +fn python_type_name(value: &Value) -> &'static str { + match value { + Value::Null => "NoneType", + Value::Bool(_) => "bool", + Value::Number(number) if number.is_f64() => "float", + Value::Number(_) => "int", + Value::String(_) => "str", + Value::Array(_) => "list", + Value::Object(_) => "dict", + } +} + +fn kept_display( + display: Option<&Recognized>, + context: &ThinkingContext, +) -> Option> { + match display? { + Recognized::Known(ThinkingDisplay::Summarized | ThinkingDisplay::Omitted) => { + display.cloned() + } + Recognized::Known(ThinkingDisplay::Updates) if context.keeps_display_updates => { + display.cloned() + } + Recognized::Known(ThinkingDisplay::Updates) | Recognized::Unrecognized(_) => None, + } +} + +/// Keys Python's `carried_thinking_keys` keeps on the rebuilt adaptive config: the adaptive +/// keys always, and any other key unless `drop_params` is set. +fn carried_thinking_keys( + extra: &serde_json::Map, + context: &ThinkingContext, +) -> serde_json::Map { + extra + .iter() + .filter(|(key, _)| key.as_str() == "block_binding" || !context.drop_params) + .map(|(key, value)| (key.clone(), value.clone())) + .collect() +} + +/// `AnthropicModelInfo.translate_legacy_thinking_for_adaptive_model_silently`. fn translate_legacy_thinking_for_adaptive_model( request: MessagesRequest, context: &ThinkingContext, -) -> MessagesRequest { +) -> Result { let capabilities = &context.capabilities; if !capabilities.supports_adaptive_thinking || capabilities.supports_legacy_thinking { - return request; + return Ok(request); } let Some(ThinkingConfig::Enabled(enabled)) = known_thinking(&request) else { - return request; + return Ok(request); }; - let budget = enabled - .budget_tokens - .as_ref() - .and_then(Recognized::known) - .copied() - .unwrap_or(0); - let level = effort_for_budget(&context.budgets, budget, capabilities); - MessagesRequest { + let budget = match &enabled.budget_tokens { + None => Some(0), + Some(budget) => numeric_budget_tokens(budget), + }; + if budget.is_none() && !context.drop_params { + let actual = match &enabled.budget_tokens { + Some(Recognized::Unrecognized(value)) => python_type_name(value), + Some(Recognized::Known(_)) | None => "int", + }; + return Err(Error::InvalidRequest( + crate::ErrorDetail::NonNumericThinkingBudget { actual }, + )); + } + let thinking = ThinkingConfig::Adaptive(AdaptiveThinking { + display: kept_display(enabled.display.as_ref(), context), + extra: carried_thinking_keys(&enabled.extra, context), + }); + let output_config = match budget { + Some(budget) => with_default_effort( + request.params.output_config, + effort_for_budget(&context.budgets, budget, capabilities), + ), + None => request.params.output_config, + }; + Ok(MessagesRequest { params: MessagesOptionalParams { - thinking: Some(Recognized::Known(ThinkingConfig::adaptive(None))), - output_config: with_default_effort(request.params.output_config, level), + thinking: Some(Recognized::Known(thinking)), + output_config, ..request.params }, ..request - } + }) } fn translate_adaptive_effort_for_non_adaptive_model( @@ -337,7 +435,7 @@ pub fn translate_thinking( ) -> Result { let request = translate_reasoning_effort(request, context)?; let request = drop_disabled_thinking(request, context); - let request = translate_legacy_thinking_for_adaptive_model(request, context); + let request = translate_legacy_thinking_for_adaptive_model(request, context)?; let request = translate_adaptive_effort_for_non_adaptive_model(request, context)?; Ok(drop_incompatible_temperature_for_thinking(request, context)) } @@ -363,6 +461,7 @@ mod tests { ThinkingContext { capabilities, budgets: ThinkingBudgets::default(), + ..ThinkingContext::default() } } @@ -1239,10 +1338,102 @@ mod tests { let context = ThinkingContext { capabilities, budgets: overridden_budgets(overrides), + ..ThinkingContext::default() }; assert_eq!( translate_thinking(request(input), &context), Ok(request(expected)) ); } + + use serde_json::json; + + fn legacy_context(drop_params: bool, keeps_display_updates: bool) -> ThinkingContext { + ThinkingContext { + drop_params, + keeps_display_updates, + ..context(opus_4_7()) + } + } + + #[rstest] + #[case::float_budget(json!(8192.0), "xhigh")] + #[case::float_below_a_tier(json!(1023.9), "low")] + #[case::string_budget(json!(" 16_000 "), "xhigh")] + #[case::negative_budget(json!(-5), "low")] + fn legacy_budgets_read_like_python_numbers(#[case] budget: Value, #[case] effort: &str) { + let translated = translate_thinking( + request( + json!({"max_tokens": 64000, "thinking": {"type": "enabled", "budget_tokens": budget}}), + ), + &legacy_context(false, false), + ); + assert_eq!( + translated, + Ok(request(json!({ + "max_tokens": 64000, + "thinking": {"type": "adaptive"}, + "output_config": {"effort": effort}, + }))) + ); + } + + #[rstest] + #[case::text(json!("lots"), "str")] + #[case::list(json!([1]), "list")] + #[case::null(Value::Null, "NoneType")] + #[case::boolean(json!(true), "bool")] + fn a_non_numeric_legacy_budget_is_rejected_unless_dropped( + #[case] budget: Value, + #[case] actual: &'static str, + ) { + let fields = + json!({"max_tokens": 8192, "thinking": {"type": "enabled", "budget_tokens": budget}}); + assert_eq!( + translate_thinking(request(fields.clone()), &legacy_context(false, false)), + Err(Error::InvalidRequest( + crate::ErrorDetail::NonNumericThinkingBudget { actual } + )) + ); + assert_eq!( + translate_thinking(request(fields), &legacy_context(true, false)), + Ok(request( + json!({"max_tokens": 8192, "thinking": {"type": "adaptive"}}) + )) + ); + } + + #[rstest] + #[case::summarized("summarized", false, Some("summarized"))] + #[case::updates_kept("updates", true, Some("updates"))] + #[case::updates_dropped("updates", false, None)] + #[case::unknown("verbose", true, None)] + fn legacy_display_survives_only_where_python_keeps_it( + #[case] display: &str, + #[case] keeps_display_updates: bool, + #[case] kept: Option<&str>, + ) { + let translated = translate_thinking( + request(json!({"max_tokens": 4096, "thinking": {"type": "enabled", "budget_tokens": 2048, "display": display}})), + &legacy_context(false, keeps_display_updates), + ) + .unwrap(); + let thinking = serde_json::to_value(translated.params.thinking).unwrap(); + assert_eq!(thinking.get("display").and_then(Value::as_str), kept); + } + + #[rstest] + #[case::kept(false, json!({"type": "adaptive", "block_binding": "strict", "note": "x"}))] + #[case::dropped(true, json!({"type": "adaptive", "block_binding": "strict"}))] + fn legacy_extra_keys_follow_drop_params(#[case] drop_params: bool, #[case] expected: Value) { + let translated = translate_thinking( + request(json!({"max_tokens": 4096, "thinking": {"type": "enabled", "budget_tokens": 2048, "block_binding": "strict", "note": "x"}})), + &legacy_context(drop_params, false), + ) + .unwrap(); + assert_eq!( + serde_json::to_value(translated.params.thinking).unwrap(), + expected + ); + } } diff --git a/litellm-rust/crates/llms/src/anthropic/messages/transformation.rs b/litellm-rust/crates/llms/src/anthropic/messages/transformation.rs index 6657cbf262f..18789e5b2be 100644 --- a/litellm-rust/crates/llms/src/anthropic/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/messages/transformation.rs @@ -1,7 +1,8 @@ use litellm_auth::CredentialPlacement; use litellm_llms_types::{ formats::messages::{ - ContextEdit, ContextManagement, Message, MessagesOptionalParams, MessagesRequest, Speed, + ContentBlock, ContentBlockType, ContextEdit, ContextManagement, Message, + MessagesOptionalParams, MessagesRequest, Speed, ThinkingConfig, ThinkingDisplay, }, providers::anthropic::{AnthropicBeta, BetaProvider, BetaSet, DEFAULT_HEADERS}, recognized::Recognized, @@ -55,7 +56,7 @@ impl BaseMessagesConfig for AnthropicMessagesConfig { request: MessagesRequest, context: &MessagesTransformContext, ) -> Result { - transform_messages_request(request, context) + transform_messages_request(request, &context.keeping_display_updates()) } fn secret_names(&self) -> &'static [&'static str] { @@ -203,6 +204,12 @@ fn feature_betas(request: &MessagesRequest) -> BetaSet { .then_some(AnthropicBeta::FastMode20260201), messages_carry_output_config(&request.messages) .then_some(AnthropicBeta::PerTurnControl20260701), + uses_thinking_display_updates(params) + .then_some(AnthropicBeta::ThinkingDisplayUpdates20260818), + uses_thinking_block_binding(params) + .then_some(AnthropicBeta::ThinkingBindingControls20260801), + messages_carry_tool_changes(&request.messages) + .then_some(AnthropicBeta::MidConversationToolChanges20260701), has_advisor_tool(tools).then_some(AnthropicBeta::AdvisorTool20260301), is_tool_search_used(tools).then_some(AnthropicBeta::AdvancedToolUse20251120), ] @@ -253,6 +260,55 @@ fn messages_carry_output_config(messages: &[Message]) -> bool { .any(|message| message.extra.contains_key("output_config")) } +fn uses_thinking_display_updates(params: &MessagesOptionalParams) -> bool { + let display = match params.thinking.as_ref().and_then(Recognized::known) { + Some(ThinkingConfig::Enabled(thinking)) => thinking.display.as_ref(), + Some(ThinkingConfig::Adaptive(thinking)) => thinking.display.as_ref(), + _ => None, + }; + display.and_then(Recognized::known) == Some(&ThinkingDisplay::Updates) +} + +/// `AnthropicModelInfo.thinking_betas`: any thinking object carrying `block_binding`. +fn uses_thinking_block_binding(params: &MessagesOptionalParams) -> bool { + match params.thinking.as_ref() { + Some(Recognized::Known(ThinkingConfig::Enabled(thinking))) => { + thinking.extra.contains_key("block_binding") + } + Some(Recognized::Known(ThinkingConfig::Adaptive(thinking))) => { + thinking.extra.contains_key("block_binding") + } + Some(Recognized::Known(ThinkingConfig::Disabled(thinking))) => { + thinking.extra.contains_key("block_binding") + } + Some(Recognized::Unrecognized(Value::Object(thinking))) => { + thinking.contains_key("block_binding") + } + Some(Recognized::Unrecognized(_)) | None => false, + } +} + +fn is_tool_reference_change(block: &ContentBlock) -> bool { + matches!( + &block.block_type, + Some(ContentBlockType::Other(block_type)) + if block_type == "tool_addition" || block_type == "tool_removal" + ) && block + .extra + .get("tool") + .and_then(|tool| tool.get("type")) + .and_then(Value::as_str) + == Some("tool_reference") +} + +fn messages_carry_tool_changes(messages: &[Message]) -> bool { + messages + .iter() + .filter(|message| message.role == "system") + .flat_map(Message::blocks) + .any(is_tool_reference_change) +} + fn unsupported_param(model: &str, param: &str, value: &str, hint: &str) -> Error { Error::InvalidRequest(crate::ErrorDetail::UnsupportedParameter { model: model.into(), @@ -262,7 +318,8 @@ fn unsupported_param(model: &str, param: &str, value: &str, hint: &str) -> Error }) } -fn drop_unsupported_params( +/// The speed and sampling gates of Python's `get_requested_anthropic_messages_optional_param`. +pub fn drop_unsupported_params( request: MessagesRequest, context: &MessagesTransformContext, ) -> Result { @@ -803,6 +860,8 @@ mod tests { low: 2000, ..ThinkingBudgets::default() }, + drop_params: true, + keeps_display_updates: false, }, drop_params: true, } @@ -1071,6 +1130,41 @@ mod tests { json!({"safeguards": [{"type": "dangerous_tool_use", "classifier_context": {"v": 1, "permission_mode": "auto"}}]}), &[] )] + #[case::adaptive_display_updates( + json!({"thinking": {"type": "adaptive", "display": "updates"}}), + &["thinking-display-updates-2026-08-18"] + )] + #[case::enabled_display_updates( + json!({"thinking": {"type": "enabled", "budget_tokens": 2048, "display": "updates"}}), + &["thinking-display-updates-2026-08-18"] + )] + #[case::summarized_display(json!({"thinking": {"type": "adaptive", "display": "summarized"}}), &[])] + #[case::disabled_thinking(json!({"thinking": {"type": "disabled"}}), &[])] + #[case::system_tool_addition( + json!({"messages": [ + {"role": "user", "content": "Use the ping tool"}, + {"role": "system", "content": [{"type": "tool_addition", "tool": {"type": "tool_reference", "name": "ping"}}]}, + ]}), + &["mid-conversation-tool-changes-2026-07-01"] + )] + #[case::system_tool_removal( + json!({"messages": [ + {"role": "system", "content": [{"type": "tool_removal", "tool": {"type": "tool_reference", "name": "ping"}}]}, + ]}), + &["mid-conversation-tool-changes-2026-07-01"] + )] + #[case::user_tool_addition( + json!({"messages": [ + {"role": "user", "content": [{"type": "tool_addition", "tool": {"type": "tool_reference", "name": "ping"}}]}, + ]}), + &[] + )] + #[case::tool_addition_without_a_tool_reference( + json!({"messages": [ + {"role": "system", "content": [{"type": "tool_addition", "tool": {"type": "tool_definition"}}]}, + ]}), + &[] + )] fn feature_betas_follow_the_request(#[case] fields: Value, #[case] expected: &[&str]) { assert_eq!(feature_betas(&request(fields)), betas(expected)); } @@ -1093,6 +1187,33 @@ mod tests { ); } + #[rstest] + #[case::vertex_ai( + BetaProvider::VertexAi, + &["mid-conversation-tool-changes-2026-07-01", "thinking-display-updates-2026-08-18"] + )] + #[case::azure_ai( + BetaProvider::AzureAi, + &["mid-conversation-tool-changes-2026-07-01", "thinking-display-updates-2026-08-18"] + )] + #[case::bedrock_converse(BetaProvider::BedrockConverse, &["thinking-display-updates-2026-08-18"])] + #[case::databricks(BetaProvider::Databricks, &["thinking-display-updates-2026-08-18"])] + fn tool_change_and_display_betas_reach_the_providers_that_accept_them( + #[case] provider: BetaProvider, + #[case] expected: &[&str], + ) { + let fields = json!({ + "thinking": {"type": "adaptive", "display": "updates"}, + "messages": [ + {"role": "system", "content": [{"type": "tool_addition", "tool": {"type": "tool_reference", "name": "ping"}}]}, + ], + }); + assert_eq!( + provider_feature_betas(&request(fields), provider), + betas(expected) + ); + } + #[rstest] fn hosts_without_a_beta_map_column_get_safeguards_without_a_derived_beta() { let client = headers(&[("anthropic-beta", "claude-code-20250219")]); diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs index e813d4ac8cc..82119cecfa5 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/analyze_transformation.rs @@ -123,7 +123,7 @@ impl BaseOcrConfig for TextractAnalyzeDocumentConfig { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { error_class(error_message, status_code, headers) } diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs index bc199f40938..e3a4f947433 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/common_utils.rs @@ -17,6 +17,7 @@ const AWS_JSON_CONTENT_TYPE: &str = "application/x-amz-json-1.1"; const TARGET_HEADER: &str = "X-Amz-Target"; const CONTENT_TYPE_HEADER: &str = "Content-Type"; const UNSUPPORTED_DOCUMENT: &str = "UnsupportedDocumentException"; +const SYNC_API_GUIDANCE: &str = "aws_textract uses Textract's synchronous API, which reads a JPEG, PNG, or a single-page PDF or TIFF; other formats and multi-page documents are not supported"; const SYNC_DOCUMENT_MAX_BYTES: usize = 10 * 1024 * 1024; const HEALTH_CHECK_IMAGE_DATA_URI: &str = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"; @@ -321,20 +322,18 @@ struct AwsError { /// Textract answers both an unsupported format and a multi-page PDF or TIFF /// with a bare "unsupported document format", which reads like a corrupt file. /// Say what the synchronous API accepts. -pub(super) fn error_class(body: String, status: u16, headers: Vec<(String, String)>) -> Error { - let unsupported = serde_json::from_str::(&body) - .ok() - .filter(|error| error.kind.ends_with(UNSUPPORTED_DOCUMENT)); - Error::Provider { - status, - body: match unsupported { - Some(error) => format!( - "{UNSUPPORTED_DOCUMENT}: {}. aws_textract uses Textract's synchronous API, which reads a JPEG, PNG, or a single-page PDF or TIFF; other formats and multi-page documents are not supported", - error.message - ), - None => body, +pub(super) fn error_class(body: String, status: u16, headers: reqwest::header::HeaderMap) -> Error { + match serde_json::from_str::(&body) { + Ok(error) if error.kind.ends_with(UNSUPPORTED_DOCUMENT) => Error::RejectedDocument { + provider: "AWS Textract", + message: error.message, + guidance: SYNC_API_GUIDANCE, + }, + Ok(_) | Err(_) => Error::Provider { + status, + body, + headers: Box::new(headers), }, - headers, } } @@ -408,11 +407,11 @@ mod tests { fn a_model_outside_the_operations_is_refused_with_the_supported_names(#[case] model: &str) { let error = TextractOperation::from_model(model).unwrap_err(); - assert_eq!( - error.to_string(), - format!( - "invalid model: aws_textract has no model {model:?} - use one of: detect-document-text, analyze-document" - ) + let message = error.to_string(); + assert!(message.contains(&format!("{model:?}")), "{message}"); + assert!( + message.contains("detect-document-text, analyze-document"), + "{message}" ); assert_eq!(error.http_status_code(), Some(400)); } @@ -542,43 +541,46 @@ mod tests { #[rstest] #[case::bare_type( r#"{"__type":"UnsupportedDocumentException","Message":"Request has unsupported document format"}"#, - Some("Request has unsupported document format") + "Request has unsupported document format" )] #[case::namespaced_type( r#"{"__type":"com.amazonaws.textract#UnsupportedDocumentException","Message":"bad"}"#, - Some("bad") + "bad" )] #[case::lowercase_message( r#"{"__type":"UnsupportedDocumentException","message":"bad"}"#, - Some("bad") + "bad" )] - #[case::other_exception(r#"{"__type":"AccessDeniedException","Message":"no"}"#, None)] - #[case::json_without_a_type(r#"{"Message":"no"}"#, None)] - #[case::not_json("bad gateway", None)] - fn only_an_unsupported_document_gains_the_sync_api_hint( + fn an_unsupported_document_is_the_callers_fault_with_the_sync_api_hint( #[case] body: &str, - #[case] hinted_message: Option<&str>, + #[case] provider_message: &str, ) { - let response_headers = vec![("x-amzn-requestid".to_string(), "abc".to_string())]; + let error = error_class(body.into(), 400, reqwest::header::HeaderMap::new()); - let Error::Provider { - status, - body: reported, - headers, - } = error_class(body.into(), 400, response_headers.clone()) - else { - panic!("expected a provider error"); - }; + let message = error.to_string(); + assert!(error.is_request()); + assert_eq!(error.http_status_code(), Some(400)); + assert!(error.upstream_response().is_none()); + assert!(message.contains(provider_message), "{message}"); + assert!(message.contains(HINT), "{message}"); + } - assert_eq!(status, 400); - assert_eq!(headers, response_headers); - match hinted_message { - Some(message) => { - assert!(reported.contains(message), "{reported}"); - assert!(reported.contains(HINT), "{reported}"); - } - None => assert_eq!(reported, body), - } + #[rstest] + #[case::other_exception(r#"{"__type":"AccessDeniedException","Message":"no"}"#)] + #[case::json_without_a_type(r#"{"Message":"no"}"#)] + #[case::not_json("bad gateway")] + fn any_other_rejection_passes_through_untouched(#[case] body: &str) { + let response_headers = reqwest::header::HeaderMap::from_iter([( + reqwest::header::HeaderName::from_static("x-amzn-requestid"), + reqwest::header::HeaderValue::from_static("abc"), + )]); + + let error = error_class(body.into(), 403, response_headers.clone()); + + assert_eq!( + error.upstream_response(), + Some((403, body, &response_headers)) + ); } #[rstest] diff --git a/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs b/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs index 4b27a191734..34213926942 100644 --- a/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/aws_textract/ocr/transformation.rs @@ -104,7 +104,7 @@ impl BaseOcrConfig for TextractDetectTextConfig { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { error_class(error_message, status_code, headers) } @@ -237,7 +237,7 @@ mod tests { let error = TextractDetectTextConfig.get_error_class( r#"{"__type":"UnsupportedDocumentException","Message":"Request has unsupported document format"}"#.into(), 400, - Vec::new(), + reqwest::header::HeaderMap::new(), ); assert!( diff --git a/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs index bf934fb4d4c..d13cfa81a94 100644 --- a/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/messages/transformation.rs @@ -1,10 +1,10 @@ -use litellm_auth::SecretValue; use litellm_http::request::{has_bearer_auth, has_header}; use litellm_llms_types::{ formats::messages::{MessagesOptionalParams, MessagesRequest}, providers::anthropic::{BetaProvider, DEFAULT_HEADERS, MESSAGES_PATH}, }; use litellm_router_types::LitellmParams; +use litellm_secret_value::SecretValue; use crate::{ Error, @@ -68,7 +68,7 @@ impl BaseMessagesConfig for AzureAnthropicMessagesConfig { .capabilities .supports_mid_conversation_system, )), - context, + &context.keeping_display_updates(), ) } @@ -545,6 +545,11 @@ mod tests { &[("anthropic-beta", "web-search-2025-03-05")], &[("x-api-key", "k"), ("anthropic-beta", "structured-outputs-2025-11-13,web-search-2025-03-05")] )] + #[case::compaction_needs_the_compact_2026_09_04_beta( + json!({"compaction": {"type": "summarize"}}), + &[], + &[("x-api-key", "k"), ("anthropic-beta", "compact-2026-09-04")] + )] #[case::no_feature_needs_a_beta(json!({}), &[], &[("x-api-key", "k")])] #[case::safeguards_need_the_dangerous_tool_use_beta( json!({"safeguards": [{"type": "dangerous_tool_use"}]}), diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs index 72e321547e3..bec8bee3f34 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/cohere_parse_transformation.rs @@ -120,7 +120,8 @@ impl BaseOcrConfig for AzureAICohereParseConfig { impl AzureAICohereParseConfig { fn get_complete_url(&self, base: &str) -> Result { - let mut url = reqwest::Url::parse(base).map_err(|_| invalid_api_base())?; + let mut url = + reqwest::Url::parse(base).map_err(|error| Error::request_field("api_base", error))?; if !matches!(url.scheme(), "http" | "https") { return Err(invalid_api_base()); } @@ -133,13 +134,14 @@ impl AzureAICohereParseConfig { ApiUrl::parse(url.as_str()) .and_then(|url| url.complete_path(&AZURE_COHERE_PARSE_PATH)) .map(|url| url.into_string()) - .map_err(|_| invalid_api_base()) + .map_err(|error| Error::request_field("api_base", error)) } } fn invalid_api_base() -> Error { Error::RequestField { path: "api_base".into(), + cause: None, } } diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs index 8cbf2455e01..7da685febf9 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/common_utils.rs @@ -26,7 +26,7 @@ pub(super) async fn resolve_entra( .get_azure_ad_token(config, env_lookup) .await .or_else(|error| match error { - litellm_auth::Error::EmptyCallerCredential(_) => Ok(None), + litellm_auth::Error::EmptyCallerCredential { .. } => Ok(None), other => Err(other), }) .map(|credential| { @@ -48,7 +48,7 @@ pub(super) fn validate_destination( && credential_source != InputSource::Request { return Err(litellm_auth::Error::InvalidConfiguration( - "host credentials cannot be sent to a request-controlled Azure endpoint".into(), + litellm_auth::ErrorDetail::RequestControlledEndpoint { provider: "Azure" }, ) .into()); } diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs index 2a50989f4e4..870306b1a83 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs @@ -1,4 +1,7 @@ -use std::{collections::BTreeSet, time::Duration}; +use std::{ + collections::{BTreeMap, BTreeSet}, + time::Duration, +}; use base64::{Engine, engine::general_purpose::STANDARD}; use litellm_auth::{InputSource, Sourced}; @@ -8,18 +11,19 @@ use litellm_llms_types::serde_compat::{FiniteF64, LaxI64}; use reqwest::Url; use serde::{Deserialize, Serialize}; use serde_json::{Map, Value}; -use serde_with::serde_as; +use serde_with::{DefaultOnError, serde_as}; use tokio::time::Instant; use crate::base_llm::ocr::{ document::InlineDocument, - error::Error, + error::{Error, PagesFault, PollUrl}, handler::{CallHooks, OcrClient, read_json_response}, settings::OcrSettings, transformation::{ - BaseOcrConfig, DecodedOcrResponse, OCR_INLINE_MAX_BYTES, OCR_POLL_RETRY_SECS, - OcrConnection, OcrCredentialInputs, OcrResponseContext, PreparedOcrRequest, - ResolvedOcrCredentials, decode_and_normalize_response, decode_response, + BaseOcrConfig, DROP_PARAMS_PARAM, DecodedOcrResponse, OCR_INLINE_MAX_BYTES, + OCR_POLL_RETRY_SECS, OcrConnection, OcrCredentialInputs, OcrResponseContext, + PreparedOcrRequest, ResolvedOcrCredentials, decode_and_normalize_response, decode_response, + drops_unsupported_params, response_format, }, }; use litellm_llms_types::formats::ocr::{ @@ -32,6 +36,30 @@ const AZURE_DI_DEFAULT_HEIGHT: f64 = 11.0; const AZURE_DI_API_KEY_ENV: &str = "AZURE_DOCUMENT_INTELLIGENCE_API_KEY"; const AZURE_DI_ENDPOINT_ENV: &str = "AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"; +const OUTPUT_CONTENT_FORMAT_PARAM: &str = "outputContentFormat"; +const OUTPUT_CONTENT_FORMAT_ALIAS: &str = "output_content_format"; +const CODE_POINT_STRING_INDEX: &str = "unicodeCodePoint"; +const STRING_INDEX_TYPE_FIELD: &str = "stringIndexType"; +const MARKDOWN_MODEL_ID: &str = "prebuilt-layout"; + +#[derive( + Clone, + Copy, + Debug, + PartialEq, + strum::EnumString, + strum::IntoStaticStr, + strum::Display, + strum::VariantArray, + serde_with::SerializeDisplay, +)] +#[strum(ascii_case_insensitive)] +pub enum OutputContentFormat { + #[strum(serialize = "text")] + Text, + #[strum(serialize = "markdown")] + Markdown, +} #[derive(Clone, Debug, PartialEq, Serialize)] pub struct DocumentIntelligenceParams { @@ -39,6 +67,13 @@ pub struct DocumentIntelligenceParams { pub pages: Option, #[serde(skip_serializing_if = "Option::is_none")] pub features: Option, + #[serde( + rename = "outputContentFormat", + skip_serializing_if = "Option::is_none" + )] + pub output_content_format: Option, + #[serde(rename = "stringIndexType", skip_serializing_if = "Option::is_none")] + pub string_index_type: Option<&'static str>, } #[derive(Clone, Debug, Serialize, Deserialize)] @@ -80,6 +115,10 @@ pub struct AzureDocumentIntelligenceOperation { #[derive(Clone, Debug, Default, Deserialize)] struct AzureDocumentIntelligenceAnalyzeResult { pub content: Option, + #[serde(rename = "contentFormat")] + pub content_format: Option, + #[serde(rename = "stringIndexType")] + pub string_index_type: Option, #[serde(default)] pub pages: Vec, pub tables: Option>>, @@ -100,6 +139,9 @@ struct AzureDocumentIntelligencePage { pub unit: Option, #[serde(default)] pub lines: Vec, + #[serde(default)] + #[serde_as(deserialize_as = "DefaultOnError")] + pub spans: Option>, } #[derive(Clone, Debug, Deserialize)] @@ -107,6 +149,18 @@ struct AzureDocumentIntelligenceLine { pub content: Option, } +#[derive(Clone, Debug, Deserialize)] +struct AzureDocumentIntelligenceSpan { + pub offset: usize, + pub length: usize, +} + +impl AzureDocumentIntelligenceSpan { + fn end(&self) -> usize { + self.offset.saturating_add(self.length) + } +} + #[derive(Clone, Debug)] pub struct AzureDocumentIntelligenceOcrConfig; @@ -116,7 +170,14 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig { type Environment = Vec<(String, String)>; fn get_supported_ocr_params(&self, _model: &str) -> &'static [&'static str] { - &["pages", "features", "req_format"] + &[ + "pages", + "features", + OUTPUT_CONTENT_FORMAT_PARAM, + OUTPUT_CONTENT_FORMAT_ALIAS, + "req_format", + DROP_PARAMS_PARAM, + ] } fn get_api_key_env_var(&self) -> Option<&'static str> { @@ -154,11 +215,17 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig { fn map_ocr_params( &self, non_default_params: &CallArguments, - _model: &str, + model: &str, ) -> Result { + let output_content_format = requested_output_content_format(non_default_params)?; + let slices_markdown_pages = output_content_format == Some(OutputContentFormat::Markdown) + && response_format(non_default_params)? != OcrResponseFormat::Native + && model_id(model).is_ok_and(|id| id == MARKDOWN_MODEL_ID); Ok(DocumentIntelligenceParams { pages: normalize_pages_param(non_default_params.get("pages"))?, features: normalize_features_param(non_default_params.get("features"))?, + output_content_format, + string_index_type: slices_markdown_pages.then_some(CODE_POINT_STRING_INDEX), }) } @@ -185,7 +252,10 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig { ) -> Result { let endpoint = nonblank(request.connection.api_base.clone()) .or_else(|| nonblank(request.connection.secret(AZURE_DI_ENDPOINT_ENV))) - .ok_or_else(|| Error::Auth(litellm_auth::Error::ProviderAuthentication("Missing Azure Document Intelligence API Base - Set AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT or pass api_base".into())))?; + .ok_or(Error::Auth(litellm_auth::Error::MissingApiBase { + provider: "Azure Document Intelligence", + guidance: "Set AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT or pass api_base", + }))?; self.build_ocr_url( &endpoint, &request.model, @@ -256,14 +326,12 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> Some(Value::Array(pages)) if pages.iter().all(Value::is_number) => pages .iter() .map(|page| { - let page = page - .as_i64() - .ok_or_else(|| Error::Pages("page index is out of range".into()))?; + let page = page.as_i64().ok_or(Error::Pages(PagesFault::OutOfRange))?; if page < 0 { - return Err(Error::Pages("negative page index".into())); + return Err(Error::Pages(PagesFault::Negative)); } page.checked_add(1) - .ok_or_else(|| Error::Pages("page index is out of range".into())) + .ok_or(Error::Pages(PagesFault::OutOfRange)) }) .collect::, _>>()? .into_iter() @@ -276,7 +344,7 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> token .as_str() .map(str::trim) - .ok_or_else(|| Error::Pages("expected only integers or only strings".into())) + .ok_or(Error::Pages(PagesFault::Mixed)) }) .collect::, _>>()? .join(","), @@ -286,13 +354,11 @@ fn normalize_pages_param(pages: Option<&Value>) -> Result, Error> .collect::>() .join(","), Some(_) => { - return Err(Error::Pages( - "expected an array of integers or strings, or a native page range".into(), - )); + return Err(Error::Pages(PagesFault::Shape)); } }; if !normalized.split(',').all(valid_page_token) { - return Err(Error::Pages("invalid native page range".into())); + return Err(Error::Pages(PagesFault::Range)); } Ok(Some(normalized)) } @@ -338,6 +404,36 @@ fn normalize_features_param(features: Option<&Value>) -> Result, Ok(Some(normalized.join(","))) } +fn requested_output_content_format( + params: &CallArguments, +) -> Result, Error> { + let requested = [OUTPUT_CONTENT_FORMAT_PARAM, OUTPUT_CONTENT_FORMAT_ALIAS] + .into_iter() + .filter_map(|name| params.get(name)) + .find(|format| !is_unset(format)); + match requested.map(parse_output_content_format).transpose() { + Err(Error::OutputContentFormat) if drops_unsupported_params(params) => Ok(None), + parsed => parsed, + } +} + +fn is_unset(format: &Value) -> bool { + match format { + Value::Null => true, + Value::String(format) => format.trim().is_empty(), + _ => false, + } +} + +fn parse_output_content_format(format: &Value) -> Result { + format + .as_str() + .ok_or(Error::OutputContentFormat)? + .trim() + .parse() + .map_err(|_| Error::OutputContentFormat) +} + fn build_request(document: OcrDocument) -> Result { let source = document.source(); if source.is_empty() { @@ -368,10 +464,27 @@ fn transform_completed_response( )); } let result = response.analyze_result.unwrap_or_default(); + let markdown_content = result + .content + .as_deref() + .filter(|_| { + result.content_format.as_deref() == Some(<&str>::from(OutputContentFormat::Markdown)) + && result.string_index_type.as_deref() == Some(CODE_POINT_STRING_INDEX) + && result.pages.iter().all(|page| page.spans.is_some()) + }) + .map(|content| { + CodePointText::new( + content, + result + .pages + .iter() + .flat_map(|page| page.spans.iter().flatten()), + ) + }); let pages = result .pages .into_iter() - .map(|page| transform_azure_page(page, dpi)) + .map(|page| transform_azure_page(page, markdown_content.as_ref(), dpi)) .collect::, _>>()?; let pages_processed = i64::try_from(pages.len()).map_err(|_| Error::NumericRange("pages"))?; Ok(LiteLLMOcrResponse { @@ -382,11 +495,20 @@ fn transform_completed_response( pages_processed: Some(pages_processed), ..Default::default() }), + extra_fields: result + .string_index_type + .filter(|unit| unit == CODE_POINT_STRING_INDEX) + .map(|unit| Map::from_iter([(STRING_INDEX_TYPE_FIELD.to_owned(), Value::String(unit))])) + .unwrap_or_default(), ..LiteLLMOcrResponse::new(model, pages) }) } -fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result { +fn transform_azure_page( + page: AzureDocumentIntelligencePage, + markdown_content: Option<&CodePointText<'_>>, + dpi: i64, +) -> Result { let index = page .page_number .unwrap_or(1) @@ -398,12 +520,20 @@ fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result page.unit.as_deref().unwrap_or("inch"), dpi, )?; - let markdown = page - .lines - .iter() - .map(|line| line.content.as_deref().unwrap_or_default()) - .collect::>() - .join("\n"); + let markdown = match markdown_content { + Some(content) => page + .spans + .iter() + .flatten() + .map(|span| content.slice(span)) + .collect::(), + None => page + .lines + .iter() + .map(|line| line.content.as_deref().unwrap_or_default()) + .collect::>() + .join("\n"), + }; Ok(OcrPage { index, markdown, @@ -412,6 +542,43 @@ fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result }) } +struct CodePointText<'a> { + text: &'a str, + byte_offsets: BTreeMap, +} + +impl<'a> CodePointText<'a> { + fn new<'s>( + text: &'a str, + spans: impl Iterator, + ) -> Self { + let span_bounds = spans + .flat_map(|span| [span.offset, span.end()]) + .collect::>(); + Self { + text, + byte_offsets: text + .char_indices() + .map(|(byte_offset, _)| byte_offset) + .enumerate() + .filter(|(code_point, _)| span_bounds.contains(code_point)) + .collect(), + } + } + + fn slice(&self, span: &AzureDocumentIntelligenceSpan) -> &'a str { + let byte_offset = |code_point: usize| { + self.byte_offsets + .get(&code_point) + .copied() + .unwrap_or(self.text.len()) + }; + self.text + .get(byte_offset(span.offset)..byte_offset(span.end())) + .unwrap_or_default() + } +} + fn convert_dimensions( width: f64, height: f64, @@ -458,8 +625,14 @@ async fn read_operation_response( .and_then(|value| value.to_str().ok()) .ok_or(Error::PollLocation)? .to_string(); - let original = Url::parse(original_url).map_err(|_| Error::PollOrigin)?; - let operation = Url::parse(&location).map_err(|_| Error::PollOrigin)?; + let original = Url::parse(original_url).map_err(|source| Error::InvalidPollUrl { + url: PollUrl::Submission, + source, + })?; + let operation = Url::parse(&location).map_err(|source| Error::InvalidPollUrl { + url: PollUrl::OperationLocation, + source, + })?; if original.origin() != operation.origin() || !operation.username().is_empty() || operation.password().is_some() @@ -504,8 +677,7 @@ async fn poll_operation( let response = tokio::time::timeout_at(deadline, litellm_http::request::http_request(builder)) .await - .map_err(|_| Error::PollTimeout)? - .map_err(litellm_http::transport::Error::from)?; + .map_err(|_| Error::PollTimeout)??; let retry = response .headers() .get(reqwest::header::RETRY_AFTER) @@ -574,13 +746,21 @@ impl AzureDocumentIntelligenceOcrConfig { .features .iter() .map(|features| ("features", features.as_str())), + ) + .chain( + params + .output_content_format + .map(|format| (OUTPUT_CONTENT_FORMAT_PARAM, format.into())), + ) + .chain( + params + .string_index_type + .map(|index_type| ("stringIndexType", index_type)), ), ) .into_string() }) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } async fn resolve_headers( @@ -623,7 +803,10 @@ impl AzureDocumentIntelligenceOcrConfig { } let token = super::super::common_utils::resolve_entra(auth, config, env_lookup) .await? - .ok_or(Error::MissingAzureDocumentIntelligenceCredentials)?; + .ok_or(Error::MissingCredential { + credential: "Azure Document Intelligence credentials", + guidance: "set AZURE_DOCUMENT_INTELLIGENCE_API_KEY or configure Entra ID", + })?; super::super::common_utils::validate_destination(connection, token.source())?; Ok( std::iter::once(("Authorization".into(), format!("Bearer {}", token.value()))) @@ -692,8 +875,12 @@ mod tests { } fn map(value: Value) -> Result { + map_for_model(value, "azure_ai/doc-intelligence/prebuilt-layout") + } + + fn map_for_model(value: Value, model: &str) -> Result { let arguments = serde_json::from_value(value).unwrap(); - AzureDocumentIntelligenceOcrConfig.map_ocr_params(&arguments, "model") + AzureDocumentIntelligenceOcrConfig.map_ocr_params(&arguments, model) } #[test] @@ -823,6 +1010,218 @@ mod tests { assert_eq!(map(json!({"features": []})).unwrap().features, None); } + #[rstest] + #[case::markdown( + json!({"outputContentFormat": "markdown"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::markdown_litellm_format( + json!({"outputContentFormat": "markdown", "req_format": "litellm"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::markdown_native_format( + json!({"outputContentFormat": "markdown", "req_format": "native"}), + Some(OutputContentFormat::Markdown), + None + )] + #[case::capitalized_markdown( + json!({"outputContentFormat": "Markdown"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::text(json!({"outputContentFormat": "text"}), Some(OutputContentFormat::Text), None)] + #[case::upper_case_text(json!({"outputContentFormat": "TEXT"}), Some(OutputContentFormat::Text), None)] + #[case::padded_markdown( + json!({"outputContentFormat": " markdown\n"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::snake_case_markdown( + json!({"output_content_format": "markdown"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::camel_case_wins_over_snake_case( + json!({"outputContentFormat": "text", "output_content_format": "markdown"}), + Some(OutputContentFormat::Text), + None + )] + #[case::empty_camel_case_falls_back_to_snake_case( + json!({"outputContentFormat": "", "output_content_format": "markdown"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::null_camel_case_falls_back_to_snake_case( + json!({"outputContentFormat": null, "output_content_format": "markdown"}), + Some(OutputContentFormat::Markdown), + Some("unicodeCodePoint") + )] + #[case::empty(json!({"outputContentFormat": ""}), None, None)] + #[case::blank(json!({"outputContentFormat": " "}), None, None)] + #[case::null(json!({"outputContentFormat": null}), None, None)] + #[case::omitted(json!({}), None, None)] + fn output_content_format_maps_to_query_fields( + #[case] options: Value, + #[case] format: Option, + #[case] string_index_type: Option<&str>, + ) { + let mapped = map(options).unwrap(); + + assert_eq!(mapped.output_content_format, format); + assert_eq!(mapped.string_index_type, string_index_type); + } + + #[rstest] + #[case::layout("azure_ai/doc-intelligence/prebuilt-layout", Some("unicodeCodePoint"))] + #[case::bare_layout("prebuilt-layout", Some("unicodeCodePoint"))] + #[case::read("azure_ai/doc-intelligence/prebuilt-read", None)] + #[case::invoice("azure_ai/doc-intelligence/prebuilt-invoice", None)] + #[case::custom("azure_ai/doc-intelligence/my-extraction-model", None)] + fn only_the_layout_model_asks_for_code_point_offsets( + #[case] model: &str, + #[case] string_index_type: Option<&str>, + ) { + let mapped = map_for_model(json!({"outputContentFormat": "markdown"}), model).unwrap(); + + assert_eq!( + mapped.output_content_format, + Some(OutputContentFormat::Markdown) + ); + assert_eq!(mapped.string_index_type, string_index_type); + } + + #[rstest] + #[case::unknown(json!({"outputContentFormat": "html"}))] + #[case::number(json!({"outputContentFormat": 1}))] + #[case::boolean(json!({"outputContentFormat": true}))] + #[case::list(json!({"outputContentFormat": ["markdown"]}))] + #[case::snake_case_unknown(json!({"output_content_format": "html"}))] + #[case::drop_params_off(json!({"outputContentFormat": "html", "drop_params": false}))] + fn invalid_output_content_format_is_a_request_error(#[case] options: Value) { + let error = map(options).unwrap_err(); + + assert!(matches!(error, Error::OutputContentFormat), "{error:?}"); + assert_eq!(error.http_status_code(), Some(400)); + } + + #[rstest] + #[case::unknown(json!({"outputContentFormat": "html", "drop_params": true, "pages": [0]}))] + #[case::number(json!({"outputContentFormat": 1, "drop_params": true, "pages": [0]}))] + #[case::snake_case_unknown( + json!({"output_content_format": "html", "drop_params": true, "pages": [0]}) + )] + fn drop_params_drops_an_invalid_output_content_format(#[case] options: Value) { + let mapped = map(options).unwrap(); + + assert_eq!(serde_json::to_value(mapped).unwrap(), json!({"pages": "1"})); + } + + #[rstest] + #[case::camel_case(json!({"outputContentFormat": "markdown", "drop_params": true}))] + #[case::snake_case(json!({"output_content_format": "markdown", "drop_params": true}))] + fn drop_params_keeps_a_valid_output_content_format(#[case] options: Value) { + let mapped = map(options).unwrap(); + + assert_eq!( + mapped.output_content_format, + Some(OutputContentFormat::Markdown) + ); + } + + fn markdown_pages(analyze_result: Value) -> Vec { + let raw = + serde_json::to_vec(&json!({"status": "succeeded", "analyzeResult": analyze_result})) + .unwrap(); + AzureDocumentIntelligenceOcrConfig + .transform_ocr_response("model", &raw, OcrResponseFormat::Litellm) + .unwrap() + .pages + .into_iter() + .map(|page| page.markdown) + .collect() + } + + #[rstest] + fn markdown_pages_are_code_point_slices_of_the_content() { + let content = "# Ünïcode क्ष\n\n\n\n## Två 🙂\n\n|a|\n\n> tail"; + let second_page_start = "# Ünïcode क्ष\n\n\n\n".chars().count(); + let pages = markdown_pages(json!({ + "content": content, + "contentFormat": "markdown", + "stringIndexType": "unicodeCodePoint", + "pages": [ + {"pageNumber": 1, "spans": [{"offset": 0, "length": second_page_start}], + "lines": [{"content": "Ünïcode क्ष"}]}, + {"pageNumber": 2, "spans": [ + {"offset": second_page_start, "length": 8}, + {"offset": second_page_start + 10, "length": 3} + ], "lines": [{"content": "Två"}]} + ] + })); + + assert_eq!( + pages, + ["# Ünïcode क्ष\n\n\n\n", "## Två 🙂|a|"] + ); + } + + #[rstest] + fn spans_past_the_content_end_are_clamped() { + let pages = markdown_pages(json!({ + "content": "ab", + "contentFormat": "markdown", + "stringIndexType": "unicodeCodePoint", + "pages": [{"spans": [{"offset": 1, "length": 9}, {"offset": 7, "length": 1}]}] + })); + + assert_eq!(pages, ["b"]); + } + + #[rstest] + #[case::text_content(json!("text"), json!("unicodeCodePoint"))] + #[case::grapheme_offsets(json!("markdown"), json!("textElements"))] + #[case::no_index_type(json!("markdown"), Value::Null)] + #[case::no_content_format(Value::Null, json!("unicodeCodePoint"))] + fn other_responses_keep_line_joined_pages( + #[case] content_format: Value, + #[case] string_index_type: Value, + ) { + let pages = markdown_pages(json!({ + "content": "# A\n\nB", + "contentFormat": content_format, + "stringIndexType": string_index_type, + "pages": [{"spans": [{"offset": 0, "length": 6}], + "lines": [{"content": "A"}, {"content": "B"}]}] + })); + + assert_eq!(pages, ["A\nB"]); + } + + #[rstest] + #[case::null_spans(json!("markdown"), json!(null))] + #[case::fractional_offset(json!("markdown"), json!([{"offset": 0.5, "length": 1}]))] + #[case::missing_length(json!("markdown"), json!([{"offset": 0}]))] + #[case::negative_offset(json!("markdown"), json!([{"offset": -1, "length": 1}]))] + #[case::text_content(json!("text"), json!([{"offset": "0", "length": 1}]))] + fn unusable_page_spans_keep_line_joined_pages( + #[case] content_format: Value, + #[case] broken_spans: Value, + ) { + let pages = markdown_pages(json!({ + "content": "# A\n\nB", + "contentFormat": content_format, + "stringIndexType": "unicodeCodePoint", + "pages": [ + {"spans": [{"offset": 0, "length": 5}], "lines": [{"content": "A"}]}, + {"spans": broken_spans, "lines": [{"content": "B"}]} + ] + })); + + assert_eq!(pages, ["A", "B"]); + } + #[tokio::test] async fn request_endpoint_cannot_receive_environment_key() { let connection = OcrConnection { @@ -842,7 +1241,7 @@ mod tests { .unwrap_err(); assert!( - error + litellm_error::Report::new(&error) .to_string() .contains("request-controlled Azure endpoint") ); @@ -851,7 +1250,7 @@ mod tests { #[tokio::test] async fn request_endpoint_accepts_request_owned_key() { let connection = OcrConnection { - api_key: Some(litellm_auth::SecretValue::new("request-key")), + api_key: Some(litellm_secret_value::SecretValue::new("request-key")), api_key_source: InputSource::Request, api_base: Some("https://request.example".into()), api_base_source: InputSource::Request, diff --git a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs index 83d88bbb3cb..25f93846967 100644 --- a/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/azure_ai/ocr/transformation.rs @@ -169,7 +169,10 @@ impl AzureAiOcrConfig { } let key = super::common_utils::resolve_entra(auth, config, env_lookup) .await? - .ok_or(Error::MissingAzureAiCredentials)?; + .ok_or(Error::MissingCredential { + credential: "Azure AI credentials", + guidance: "set AZURE_AI_API_KEY or configure Entra ID", + })?; super::common_utils::validate_destination(connection, key.source())?; Ok(bearer_headers(connection, key.value())) } @@ -183,9 +186,7 @@ impl AzureAiOcrConfig { ApiUrl::parse(&base) .and_then(|url| url.complete_path(&AZURE_AI_OCR_PATH)) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -210,7 +211,7 @@ mod tests { #[fixture] fn connection() -> OcrConnection { OcrConnection { - api_key: Some(litellm_auth::SecretValue::new("request-key")), + api_key: Some(litellm_secret_value::SecretValue::new("request-key")), api_base: Some("https://example.com".into()), ..Default::default() } @@ -302,7 +303,7 @@ mod tests { .unwrap_err(); assert!( - error + litellm_error::Report::new(&error) .to_string() .contains("request-controlled Azure endpoint") ); @@ -311,7 +312,7 @@ mod tests { #[tokio::test] async fn request_endpoint_accepts_request_owned_key() { let connection = OcrConnection { - api_key: Some(litellm_auth::SecretValue::new("request-key")), + api_key: Some(litellm_secret_value::SecretValue::new("request-key")), api_key_source: InputSource::Request, api_base: Some("https://request.example".into()), api_base_source: InputSource::Request, diff --git a/litellm-rust/crates/llms/src/base_llm/auth.rs b/litellm-rust/crates/llms/src/base_llm/auth.rs index de271a269d3..ab2cf00614e 100644 --- a/litellm-rust/crates/llms/src/base_llm/auth.rs +++ b/litellm-rust/crates/llms/src/base_llm/auth.rs @@ -5,7 +5,9 @@ //! that split: `validate_environment` shapes the forwarded headers and names the credential //! as an [`AuthScheme`], and [`resolve_auth`] turns the scheme into headers and a signer. -use litellm_auth::{AuthServices, CredentialPlacement, SecretValue, TokenProviderHandle}; +use litellm_auth::{AuthServices, CredentialPlacement, TokenProviderHandle}; +use litellm_secret_value::SecretValue; + use litellm_auth_aws::{AwsCredentialSource, SigV4Signer}; use litellm_auth_gcp::VertexConfig; use litellm_http::request::with_header; diff --git a/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs b/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs index a283ca84089..6df321b4b16 100644 --- a/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs +++ b/litellm-rust/crates/llms/src/base_llm/base_model_iterator.rs @@ -5,6 +5,15 @@ use futures_util::{Stream, StreamExt, stream, stream::BoxStream}; pub type ByteStream = BoxStream<'static, Result>; +/// The transport error a body stream carried as an `io::Error`, however many times framing +/// wrapped it on the way. +pub(crate) fn tunnelled_http(error: &io::Error) -> Option<&litellm_http::Error> { + let inner = error.get_ref()?; + inner + .downcast_ref::() + .or_else(|| tunnelled_http(inner.downcast_ref::()?)) +} + pub trait StreamTransformer { type Input; type Output; diff --git a/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs b/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs index 30bff76651e..8ba4030540c 100644 --- a/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs @@ -1,3 +1,4 @@ +use litellm_core_utils::decline::Decline; use litellm_llms_types::formats::chat_completions::{ ChatCompletionsResponse, ChatMessage, ChatMessageContent, }; @@ -29,16 +30,6 @@ const IGNORABLE_MESSAGE_FIELDS: &[&str] = &["name"]; pub use crate::base_llm::auth::{Headers, ValidatedEnvironment}; -/// Why a request cannot be served by the Rust path. -/// -/// The core declines rather than guessing: the host turns this into a -/// transparent fallback to the Python implementation, which covers the full -/// surface. Acceptance is an allowlist, so a parameter or message shape the -/// core has never seen declines by construction instead of being translated -/// wrong. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub struct Unsupported(pub &'static str); - pub trait BaseConfig: Sync { fn secret_names(&self) -> Vec<&'static str>; @@ -97,7 +88,7 @@ pub trait BaseConfig: Sync { &self, messages: &[ChatMessage], optional_params: &Map, - ) -> Option { + ) -> Option { unsupported_param( self.supported_openai_param_mappings(), self.config_params(), @@ -111,13 +102,13 @@ pub fn unsupported_param( supported: &'static [(&'static str, &'static str)], config: &'static [&'static str], optional_params: &Map, -) -> Option { +) -> Option { if optional_params .get(STREAM_PARAM) .and_then(Value::as_bool) .unwrap_or(false) { - return Some(Unsupported("streaming")); + return Some(Decline::new("streaming")); } optional_params .keys() @@ -128,29 +119,29 @@ pub fn unsupported_param( .any(|(_, provider_name)| *provider_name == key) && !config.contains(&key.as_str()) }) - .then_some(Unsupported("unrecognized request parameter")) + .then_some(Decline::new("unrecognized request parameter")) } /// Message shapes the core can translate faithfully: text content, either a /// plain string or a non-empty list of parts that are all /// `{"type": "text", "text": ...}`. Tool calls, tool results, and multimodal /// parts decline so Python's fuller translation handles them. -pub fn unsupported_message(message: &ChatMessage) -> Option { +pub fn unsupported_message(message: &ChatMessage) -> Option { if message .extra .keys() .any(|key| !IGNORABLE_MESSAGE_FIELDS.contains(&key.as_str())) { - return Some(Unsupported("unrecognized message field")); + return Some(Decline::new("unrecognized message field")); } if !matches!(message.role.as_str(), "system" | "user" | "assistant") { - return Some(Unsupported("unrecognized message role")); + return Some(Decline::new("unrecognized message role")); } match &message.content { - None => Some(Unsupported("message without content")), + None => Some(Decline::new("message without content")), Some(ChatMessageContent::Text(_)) => None, Some(ChatMessageContent::Parts(parts)) if parts.is_empty() => { - Some(Unsupported("message without content")) + Some(Decline::new("message without content")) } Some(ChatMessageContent::Parts(parts)) => parts .iter() @@ -159,6 +150,6 @@ pub fn unsupported_message(message: &ChatMessage) -> Option { || part.get("text").and_then(Value::as_str).is_none() || part.as_object().is_some_and(|object| object.len() != 2) }) - .then_some(Unsupported("non-text message content")), + .then_some(Decline::new("non-text message content")), } } diff --git a/litellm-rust/crates/llms/src/base_llm/messages/context.rs b/litellm-rust/crates/llms/src/base_llm/messages/context.rs index e59a63813b0..b514e975aa4 100644 --- a/litellm-rust/crates/llms/src/base_llm/messages/context.rs +++ b/litellm-rust/crates/llms/src/base_llm/messages/context.rs @@ -110,6 +110,11 @@ impl ThinkingBudgets { pub struct ThinkingContext { pub capabilities: MessagesModelCapabilities, pub budgets: ThinkingBudgets, + /// `drop_params` as Python's thinking translation reads it. + pub drop_params: bool, + /// The provider accepts `display: "updates"` on thinking translated from the legacy shape + /// (Python's `LEGACY_THINKING_DISPLAY_UPDATES_PROVIDERS`). + pub keeps_display_updates: bool, } #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub struct MessagesTransformContext { @@ -130,8 +135,22 @@ impl MessagesTransformContext { thinking: ThinkingContext { capabilities, budgets: ThinkingBudgets::from_lookup(env), + drop_params, + keeps_display_updates: false, }, drop_params, } } + + /// The same context for a provider in Python's `LEGACY_THINKING_DISPLAY_UPDATES_PROVIDERS`. + #[must_use] + pub fn keeping_display_updates(&self) -> Self { + Self { + thinking: ThinkingContext { + keeps_display_updates: true, + ..self.thinking + }, + ..*self + } + } } diff --git a/litellm-rust/crates/llms/src/base_llm/messages/transformation.rs b/litellm-rust/crates/llms/src/base_llm/messages/transformation.rs index 8a88af40d6b..744ce89afdb 100644 --- a/litellm-rust/crates/llms/src/base_llm/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/base_llm/messages/transformation.rs @@ -77,6 +77,15 @@ pub trait BaseMessagesConfig: Sync { headers } + fn request_body( + &self, + request: MessagesRequest, + _headers: &Headers, + _drop_params: bool, + ) -> Result { + Ok(request) + } + /// The JSON that goes on the wire, for a host whose body differs from the typed request: /// Python's configs `pop("model")` when the model is addressed by the URL. fn wire_body(&self, body: Value) -> Value { diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/document.rs b/litellm-rust/crates/llms/src/base_llm/ocr/document.rs index b32ea4bf73a..898c24369c3 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/document.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/document.rs @@ -1,9 +1,6 @@ use base64::{Engine, engine::general_purpose::STANDARD}; use data_url::{DataUrl, DataUrlError, forgiving_base64::DecodeError, mime::Mime}; -use litellm_http::{ - media::{DownloadPolicy, Error as MediaError, MediaFetcher}, - transport::Error as TransportError, -}; +use litellm_http::media::{DownloadPolicy, MediaFetcher}; use reqwest::Url; use crate::base_llm::ocr::{ @@ -61,9 +58,7 @@ pub async fn inline_remote_document( validate_inline_document(&document)?; return Ok(document); } - let url = Url::parse(source).map_err(|_| Error::RequestField { - path: "document URL".into(), - })?; + let url = Url::parse(source).map_err(|error| Error::request_field("document URL", error))?; let downloaded = fetcher .fetch( url, @@ -74,7 +69,7 @@ pub async fn inline_remote_document( }, ) .await - .map_err(map_media_error)?; + .map_err(Error::DocumentDownload)?; let result = document.with_source(format!( "data:{};base64,{}", downloaded.content_type, @@ -84,28 +79,6 @@ pub async fn inline_remote_document( Ok(result) } -fn map_media_error(error: MediaError) -> Error { - match error { - MediaError::BlockedUrl => Error::BlockedDocumentUrl, - MediaError::DownloadDisabled => Error::DownloadDisabled, - MediaError::DownloadTooLarge => Error::DownloadTooLarge, - MediaError::TooManyRedirects => Error::TooManyRedirects, - MediaError::MissingRedirectLocation => Error::MissingRedirectLocation, - MediaError::InvalidRedirect => Error::InvalidRedirect, - MediaError::Http(status) => TransportError::Http { - status, - body: "OCR document download failed".into(), - } - .into(), - MediaError::Timeout => TransportError::Http { - status: 408, - body: "OCR document download timed out".into(), - } - .into(), - MediaError::Transport(error) => error.into(), - } -} - #[cfg(test)] mod tests { use std::collections::BTreeMap as Map; @@ -119,20 +92,17 @@ mod tests { } } - #[test] - fn decodes_data_urls_and_limits_decoded_size() { - for (source, expected) in [ - ("data:application/pdf;base64,YWJj", b"abc".as_slice()), - ("DATA:application/pdf;BASE64,YWI", b"ab".as_slice()), - ("data:,a%20b%00%FF", b"a b\0\xff".as_slice()), - ] { - let inline = InlineDocument::parse(source).unwrap().unwrap(); - assert_eq!(inline.decode(expected.len()).unwrap(), expected); - assert!(matches!( - inline.decode(expected.len() - 1), - Err(Error::InlineDocumentTooLarge) - )); - } + #[rstest::rstest] + #[case::base64("data:application/pdf;base64,YWJj", b"abc")] + #[case::uppercase_unpadded("DATA:application/pdf;BASE64,YWI", b"ab")] + #[case::percent_encoded("data:,a%20b%00%FF", b"a b\0\xff")] + fn decodes_data_urls_and_limits_decoded_size(#[case] source: &str, #[case] expected: &[u8]) { + let inline = InlineDocument::parse(source).unwrap().unwrap(); + assert_eq!(inline.decode(expected.len()).unwrap(), expected); + assert!(matches!( + inline.decode(expected.len() - 1), + Err(Error::InlineDocumentTooLarge) + )); } #[test] @@ -150,15 +120,12 @@ mod tests { ); } - #[test] - fn rejects_invalid_inline_documents() { - for source in [ - "https://example.com/document.pdf", - "data:application/pdf;base64", - "data:application/pdf;base64,INVALID!", - ] { - assert!(validate_inline_document(&document(source)).is_err()); - } + #[rstest::rstest] + #[case::remote("https://example.com/document.pdf")] + #[case::no_payload("data:application/pdf;base64")] + #[case::invalid_base64("data:application/pdf;base64,INVALID!")] + fn rejects_invalid_inline_documents(#[case] source: &str) { + assert!(validate_inline_document(&document(source)).is_err()); } #[tokio::test] @@ -190,16 +157,14 @@ mod tests { clippy::disallowed_types, reason = "the pool has no default-header setting to stand in for provider credentials" )] - let provider_http = litellm_http::Client::for_test( + let provider_http = litellm_http::Client::wrap_for_test( reqwest::Client::builder() .default_headers(provider_headers) .build() .unwrap(), ); - let client = crate::base_llm::ocr::handler::OcrClient::for_test( - provider_http, - litellm_http::Client::no_redirect_for_test(), - ); + let client = + crate::base_llm::ocr::handler::OcrClient::for_test().with_provider_http(provider_http); let converted = inline_remote_document( client.document_fetcher(), OcrDocument::ImageUrl { diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/error.rs b/litellm-rust/crates/llms/src/base_llm/ocr/error.rs index 392dd56412c..e5f1c3339cf 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/error.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/error.rs @@ -1,79 +1,96 @@ +use std::{io, path::PathBuf, sync::Arc}; + +use litellm_core_utils::call_arguments::ArgumentError; +use litellm_error::Source; +use reqwest::header::HeaderMap; + +static NO_HEADERS: std::sync::LazyLock = std::sync::LazyLock::new(HeaderMap::new); + #[derive(Clone, Debug, thiserror::Error)] pub enum Error { #[error("upstream OCR error ({status}): {body}")] Provider { status: u16, body: String, - headers: Vec<(String, String)>, + headers: Box, }, #[error("File is empty or could not be read")] EmptyFile, - #[error("Failed to read OCR file {}: {source}", path.display())] + #[error("could not read OCR file {}", path.display())] FileRead { - path: std::path::PathBuf, + path: PathBuf, #[source] - source: std::sync::Arc, + cause: Arc, }, - #[error("OCR document preparation task failed: {0}")] - DocumentTask(#[source] std::sync::Arc), - #[error("Invalid MIME type: {0}")] + #[error("OCR document preparation task failed")] + DocumentTask(#[source] Arc), + #[error("invalid MIME type {0:?}")] InvalidMimeType(String), - #[error( - "Cohere Parse only accepts `image_url` documents; document_url and PDF inputs are not supported" - )] - CohereImageOnly, + #[error("{provider} cannot read this document - {guidance}")] + UnsupportedDocument { + provider: &'static str, + guidance: &'static str, + }, + #[error("{provider} cannot read this document: {message} - {guidance}")] + RejectedDocument { + provider: &'static str, + message: String, + guidance: &'static str, + }, #[error("Invalid `req_format`. Expected 'native' or 'litellm'.")] RequestFormat, #[error("invalid OCR request field: {path}")] - RequestField { path: String }, + RequestField { + path: String, + #[source] + cause: Option, + }, + #[error("invalid OCR optional params")] + Arguments(#[from] ArgumentError), #[error("missing required field: {0}")] MissingField(&'static str), - #[error("Document URL is required")] + #[error("document URL is required")] MissingDocumentUrl, #[error("invalid OCR document data URI")] InvalidDataUri, - #[error( - "Reducto requires a reducto:// id or a data URI; plain HTTP URLs are not supported, upload the file first" - )] - ReductoSource, #[error("inline OCR document exceeds the size limit")] InlineDocumentTooLarge, - #[error("OCR document URL is blocked by network policy")] - BlockedDocumentUrl, - #[error("OCR document downloads are disabled")] - DownloadDisabled, - #[error("OCR document download exceeds the size limit")] - DownloadTooLarge, - #[error("OCR document download exceeded the redirect limit")] - TooManyRedirects, #[error("invalid OCR pages: {0}")] - Pages(String), + Pages(PagesFault), #[error("invalid OCR features")] Features, + #[error( + "invalid OCR outputContentFormat, expected text or markdown. To drop unsupported params, set `litellm.drop_params = True`." + )] + OutputContentFormat, #[error("OCR model cannot be a dot segment")] DotModel, #[error("OCR response exceeds the size limit of {limit} bytes")] TooLarge { limit: usize }, #[error("invalid OCR response field: {path}")] - ResponseField { path: String }, + ResponseField { + path: String, + #[source] + cause: Option, + }, #[error("OCR response is missing non-empty content")] EmptyContent, - #[error("OCR document redirect is missing a location")] - MissingRedirectLocation, - #[error("OCR document redirect location is invalid")] - InvalidRedirect, #[error("OCR operation ended with status {0}")] OperationStatus(String), #[error("OCR response numeric value is out of range: {0}")] NumericRange(&'static str), #[error("OCR accepted response is missing a valid operation-location")] PollLocation, + #[error("OCR {url} is not a valid URL")] + InvalidPollUrl { + url: PollUrl, + #[source] + source: url::ParseError, + }, #[error("OCR operation-location must use the submission origin without credentials")] PollOrigin, #[error("OCR polling timed out")] PollTimeout, - #[error("unsupported by the rust path: {0}")] - Unsupported(&'static str), #[error("invalid provider: {0}")] InvalidProvider(String), #[error("invalid model: {provider} has no model {model:?} - use one of: {}", supported.join(", "))] @@ -82,108 +99,270 @@ pub enum Error { model: String, supported: &'static [&'static str], }, - #[error("invalid request: {0}")] - InvalidRequest(String), - #[error("invalid response: {0}")] - InvalidResponse(String), - #[error( - "invalid authentication configuration: Missing Azure AI credentials - set AZURE_AI_API_KEY or configure Entra ID" - )] - MissingAzureAiCredentials, - #[error( - "invalid authentication configuration: Missing Azure Document Intelligence credentials - set AZURE_DOCUMENT_INTELLIGENCE_API_KEY or configure Entra ID" - )] - MissingAzureDocumentIntelligenceCredentials, - #[error( - "Missing REDUCTO_API_KEY - set it in the environment or pass api_key to litellm.ocr()/litellm.aocr()" - )] - MissingReductoApiKey, - #[error("secret resolution failed: {0}")] - Secret(#[source] std::sync::Arc), + #[error("Missing {credential} - {guidance}")] + MissingCredential { + credential: &'static str, + guidance: &'static str, + }, + #[error("secret resolution failed")] + Secret(#[source] Arc), + #[error("the OCR host driver failed")] + Host(#[source] litellm_host::machine::MachineFault), + #[error("the OCR host call raised an exception")] + HostCall, #[error(transparent)] Auth(#[from] litellm_auth::Error), #[error(transparent)] - Transport(#[from] litellm_http::transport::Error), - #[error(transparent)] Params(#[from] litellm_core_utils::params::Error), #[error(transparent)] Headers(#[from] litellm_http::request::HeaderError), #[error(transparent)] Http(#[from] litellm_http::Error), + #[error(transparent)] + DocumentDownload(litellm_http::Error), +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum PagesFault { + #[error("page index is out of range")] + OutOfRange, + #[error("negative page index")] + Negative, + #[error("expected only integers or only strings")] + Mixed, + #[error("expected an array of integers or strings, or a native page range")] + Shape, + #[error("invalid native page range")] + Range, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum PollUrl { + #[error("submission URL")] + Submission, + #[error("operation-location")] + OperationLocation, } impl From for Error { fn from(fault: litellm_host::machine::MachineFault) -> Self { - use litellm_host::machine::MachineFault; - Self::InvalidRequest(match fault { - MachineFault::Abandoned => "OCR host driver was abandoned".into(), - MachineFault::Protocol(message) => format!("OCR {message}"), - }) - } -} - -impl From for Error { - fn from(error: litellm_core_utils::call_arguments::ArgumentError) -> Self { - Self::RequestField { - path: format!("optional_params.{}", error.path), - } + Self::Host(fault) } } impl Error { - pub fn http_status_code(&self) -> Option { - match self { - Self::Provider { status, .. } - | Self::Transport(litellm_http::transport::Error::Http { status, .. }) => Some(*status), - error if error.is_request() => Some(400), - _ => None, + pub fn request_field( + path: impl Into, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::RequestField { + path: path.into(), + cause: Some(Source::new(cause)), } } - pub fn is_request(&self) -> bool { - matches!( - self, + pub fn response_field( + path: impl Into, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::ResponseField { + path: path.into(), + cause: Some(Source::new(cause)), + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str, &HeaderMap)> { + match self { + Self::Provider { + status, + body, + headers, + } => Some((*status, body, headers)), + Self::DocumentDownload(error) => error.status().map(|status| { + let body = if error.is_timeout() { + "OCR document download timed out" + } else { + "OCR document download failed" + }; + (status, body, &*NO_HEADERS) + }), + Self::Http(error) => error + .upstream_response() + .map(|(status, body)| (status, body, &*NO_HEADERS)), Self::EmptyFile - | Self::InvalidMimeType(_) - | Self::CohereImageOnly - | Self::RequestFormat - | Self::RequestField { .. } - | Self::MissingField(_) - | Self::MissingDocumentUrl - | Self::InvalidDataUri - | Self::ReductoSource - | Self::InlineDocumentTooLarge - | Self::BlockedDocumentUrl - | Self::DownloadDisabled - | Self::DownloadTooLarge - | Self::TooManyRedirects - | Self::Pages(_) - | Self::Features - | Self::DotModel - | Self::InvalidRequest(_) - | Self::InvalidProvider(_) - | Self::InvalidModel { .. } - | Self::Params(_) - | Self::Headers(_) - | Self::Http(_) - | Self::Auth(litellm_auth::Error::MissingParam { .. }) - ) + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::OutputContentFormat + | Self::DotModel + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Auth(_) + | Self::Params(_) + | Self::Headers(_) => None, + } + } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::MissingCredential { .. } => true, + Self::Auth(error) => error.is_missing_credential(), + Self::Provider { .. } + | Self::EmptyFile + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::OutputContentFormat + | Self::DotModel + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Params(_) + | Self::Headers(_) + | Self::Http(_) + | Self::DocumentDownload(_) => false, + } + } + + pub fn http_status_code(&self) -> Option { + if let Some((status, ..)) = self.upstream_response() { + return Some(status); + } + if let Self::Http(error) = self + && let Some(status) = error.status() + { + return Some(status); + } + self.is_request().then_some(400) + } + + pub fn is_request(&self) -> bool { + match self { + Self::EmptyFile + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::OutputContentFormat + | Self::DotModel + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::Params(_) + | Self::Headers(_) => true, + Self::Auth(error) => error.is_request(), + Self::Http(error) | Self::DocumentDownload(error) => error.is_request(), + Self::Provider { .. } + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::TooLarge { .. } + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall => false, + } } pub fn is_response(&self) -> bool { - matches!( - self, + match self { Self::TooLarge { .. } - | Self::ResponseField { .. } - | Self::EmptyContent - | Self::MissingRedirectLocation - | Self::InvalidRedirect - | Self::OperationStatus(_) - | Self::NumericRange(_) - | Self::PollLocation - | Self::PollOrigin - | Self::PollTimeout - | Self::InvalidResponse(_) - ) + | Self::ResponseField { .. } + | Self::EmptyContent + | Self::OperationStatus(_) + | Self::NumericRange(_) + | Self::PollLocation + | Self::InvalidPollUrl { .. } + | Self::PollOrigin + | Self::PollTimeout => true, + Self::Http(error) | Self::DocumentDownload(error) => error.is_response(), + Self::Provider { .. } + | Self::EmptyFile + | Self::FileRead { .. } + | Self::DocumentTask(_) + | Self::InvalidMimeType(_) + | Self::UnsupportedDocument { .. } + | Self::RejectedDocument { .. } + | Self::RequestFormat + | Self::RequestField { .. } + | Self::Arguments(_) + | Self::MissingField(_) + | Self::MissingDocumentUrl + | Self::InvalidDataUri + | Self::InlineDocumentTooLarge + | Self::Pages(_) + | Self::Features + | Self::OutputContentFormat + | Self::DotModel + | Self::InvalidProvider(_) + | Self::InvalidModel { .. } + | Self::MissingCredential { .. } + | Self::Secret(_) + | Self::Host(_) + | Self::HostCall + | Self::Auth(_) + | Self::Params(_) + | Self::Headers(_) => false, + } } } diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs b/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs index 2cf955ff0c6..7bbc751c614 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/handler.rs @@ -7,8 +7,8 @@ use litellm_host::interceptors::WireRequest; use litellm_http::{ Client, ClientVariant, HttpClientConfig, HttpClientPool, media::{MediaFetcher, UrlPolicy}, - outbound::{OutboundRequest, RequestSigner}, - transport, + outbound::{RequestSigner, json, signed}, + request::header_pairs, }; use serde::{Serialize, de::DeserializeOwned}; use serde_json::Value; @@ -86,21 +86,27 @@ impl OcrClient { } #[cfg(any(test, feature = "test-support"))] - pub fn for_test(provider_http: Client, no_redirect_http: Client) -> Self { + pub fn for_test() -> Self { Self { secrets: Arc::new( - litellm_secrets::source::EnvironmentSecrets::python_compatible( - provider_http.clone(), - ), + litellm_secrets::source::EnvironmentSecrets::python_compatible(Client::for_test()), ), - provider_http, - polling_http: no_redirect_http.clone(), - document_fetcher: MediaFetcher::for_test(no_redirect_http), + provider_http: Client::for_test(), + polling_http: Client::for_test(), + document_fetcher: MediaFetcher::for_test(Client::for_test()), auth: Arc::new(AuthServices::default()), settings: OcrSettings::default(), } } + #[cfg(any(test, feature = "test-support"))] + pub fn with_provider_http(self, provider_http: Client) -> Self { + Self { + provider_http, + ..self + } + } + #[cfg(any(test, feature = "test-support"))] pub fn with_settings(self, settings: OcrSettings) -> Self { Self { settings, ..self } @@ -122,29 +128,17 @@ pub async fn ocr( ) -> Result { let http = config.prepare_request(request, client, hooks).await?; let url = http.url().to_string(); - let headers = http.headers().to_vec(); - let response = http - .send(client.provider_http()) + let headers = header_pairs(http.headers()); + let response = client + .provider_http() + .execute(http) .await - .map_err(transport_error)?; + .map_err(litellm_http::Error::from)?; if !response.status().is_success() { - let headers = response - .headers() - .iter() - .filter_map(|(name, value)| { - value - .to_str() - .ok() - .map(|value| (name.to_string(), value.to_string())) - }) - .collect(); - return match read_response_bytes(response, request.connection.max_response_bytes).await { - Err(Error::Transport(transport::Error::Http { status, body })) => { - Err(config.get_error_class(body, status, headers)) - } - Err(error) => Err(error), - Ok(_) => unreachable!("non-success response produces an HTTP error"), - }; + let status = response.status().as_u16(); + let headers = response.headers().clone(); + let body = read_error_body(response, request.connection.max_response_bytes).await?; + return Err(config.get_error_class(body, status, headers)); } let context = OcrResponseContext { client, @@ -172,43 +166,41 @@ pub async fn read_response_bytes( mut response: reqwest::Response, limit: usize, ) -> Result { - let status = response.status(); - if status.is_success() - && response - .content_length() - .is_some_and(|length| length > limit as u64) + if !response.status().is_success() { + let status = response.status().as_u16(); + let headers = response.headers().clone(); + let body = read_error_body(response, limit).await?; + return Err(Error::Provider { + status, + body, + headers: Box::new(headers), + }); + } + if response + .content_length() + .is_some_and(|length| length > limit as u64) { return Err(Error::TooLarge { limit }); } let mut bytes = BytesMut::new(); - while let Some(chunk) = response.chunk().await.map_err(transport_error)? { - let remaining = limit.saturating_sub(bytes.len()); - if status.is_success() && chunk.len() > remaining { + while let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? { + if chunk.len() > limit.saturating_sub(bytes.len()) { return Err(Error::TooLarge { limit }); } - bytes.extend_from_slice(&chunk[..chunk.len().min(remaining)]); - if !status.is_success() && bytes.len() == limit { - break; - } - } - if !status.is_success() { - return Err(transport::Error::Http { - status: status.as_u16(), - body: String::from_utf8_lossy(&bytes).into_owned(), - } - .into()); + bytes.extend_from_slice(&chunk); } Ok(bytes.freeze()) } -pub fn transport_error(error: reqwest::Error) -> Error { - if error.is_timeout() { - return Error::Transport(transport::Error::Http { - status: 408, - body: "OCR request timed out".into(), - }); +async fn read_error_body(mut response: reqwest::Response, limit: usize) -> Result { + let mut bytes = BytesMut::new(); + while bytes.len() < limit + && let Some(chunk) = response.chunk().await.map_err(litellm_http::Error::from)? + { + let remaining = limit - bytes.len(); + bytes.extend_from_slice(&chunk[..chunk.len().min(remaining)]); } - transport::Error::from(error).into() + Ok(String::from_utf8_lossy(&bytes).into_owned()) } pub async fn transform_request_body( @@ -219,7 +211,7 @@ pub async fn transform_request_body( body: B, signer: Option<&dyn RequestSigner>, hooks: &dyn CallHooks, -) -> Result { +) -> Result { let composed = litellm_core_utils::call_arguments::compose_body( &request.optional_params, &body, @@ -232,20 +224,20 @@ pub async fn transform_request_body( if !changed.body.is_object() { return Err(Error::RequestField { path: "guardrail.body".into(), + cause: None, }); } config.validate_request_body(&changed.body)?; - let timeout = Some(request.connection.timeout); + let built = json( + url, + &changed.headers, + &changed.body, + Some(request.connection.timeout), + )?; Ok(match signer { - Some(signer) => OutboundRequest::signed_json( - url.into(), - changed.headers, - &changed.body, - timeout, - signer, - ), - None => OutboundRequest::json(url.into(), changed.headers, &changed.body, timeout), - }?) + Some(signer) => signed(built, signer)?, + None => built, + }) } fn wire_request(url: &str, headers: &[(String, String)], body: Value) -> WireRequest { @@ -261,10 +253,10 @@ pub fn build_http_request( url: String, headers: Vec<(String, String)>, body: &impl Serialize, -) -> Result { - Ok(OutboundRequest::json( - url, - headers, +) -> Result { + Ok(json( + &url, + &headers, body, Some(request.connection.timeout), )?) @@ -276,9 +268,8 @@ pub async fn guardrail_document( headers: &[(String, String)], hooks: &dyn CallHooks, ) -> Result<(OcrDocument, Vec<(String, String)>), Error> { - let body = serde_json::to_value(&request.document).map_err(|_| Error::RequestField { - path: "document".into(), - })?; + let body = serde_json::to_value(&request.document) + .map_err(|error| Error::request_field("document", error))?; let changed = hooks .before_provider_request(wire_request(url, headers, body)) .await?; @@ -292,6 +283,7 @@ pub fn body_document(body: &Value) -> Result { .and_then(Value::as_object) .ok_or_else(|| Error::RequestField { path: "body.document".into(), + cause: None, })?; let source = document .iter() @@ -311,8 +303,6 @@ mod tests { net::TcpListener, }; - use super::*; - #[rstest::rstest] #[tokio::test] async fn request_timeout_has_an_http_408_status() { @@ -322,16 +312,16 @@ mod tests { let _connection = listener.accept().await.unwrap(); tokio::time::sleep(Duration::from_secs(1)).await; }); - let error = litellm_http::Client::plain_for_test() + let error = litellm_http::Client::for_test() .get(format!("http://{address}")) .timeout(Duration::from_millis(10)) .send() .await .unwrap_err(); - assert!(matches!( - transport_error(error), - Error::Transport(transport::Error::Http { status: 408, .. }) - )); + assert_eq!( + Error::from(litellm_http::Error::from(error)).http_status_code(), + Some(408) + ); server.abort(); } @@ -347,7 +337,7 @@ mod tests { socket.write_all(response.as_bytes()).await.unwrap(); std::future::pending::<()>().await; }); - let response = litellm_http::Client::plain_for_test() + let response = litellm_http::Client::for_test() .get(format!("http://{address}")) .send() .await @@ -402,10 +392,7 @@ mod tests { .await .unwrap_err(); - let Error::Transport(litellm_http::transport::Error::Http { status, body }) = error else { - panic!("unexpected error: {error}"); - }; - assert_eq!(status, 429); - assert_eq!(body, prefix); + let (status, body, _) = error.upstream_response().expect("the provider answered"); + assert_eq!((status, body), (429, prefix.as_str())); } } diff --git a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs index 2da0e1abbfc..9574e99d357 100644 --- a/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs @@ -1,15 +1,17 @@ -use litellm_llms_types::formats::ocr::{LiteLLMOcrResponse, OcrDocument, OcrResponseFormat}; -use std::{collections::BTreeMap, future::Future, sync::Arc, time::Duration}; - -use litellm_auth::{InputSource, SecretValue, Sourced, TokenProviderHandle}; -use litellm_core_utils::{call_arguments::CallArguments, settings::ProcessEnvironment}; -use litellm_http::outbound::{OutboundRequest, RequestSigner}; -use litellm_secrets::source::Secrets; use serde::{ Serialize, de::{DeserializeOwned, IntoDeserializer}, }; use serde_json::{Map, Value}; +use std::{collections::BTreeMap, future::Future, sync::Arc, time::Duration}; + +use litellm_auth::{InputSource, Sourced, TokenProviderHandle}; +use litellm_secret_value::SecretValue; + +use litellm_core_utils::{call_arguments::CallArguments, settings::ProcessEnvironment}; +use litellm_http::outbound::RequestSigner; +use litellm_llms_types::formats::ocr::{LiteLLMOcrResponse, OcrDocument, OcrResponseFormat}; +use litellm_secrets::source::Secrets; use crate::base_llm::ocr::{ error::Error, @@ -21,6 +23,7 @@ pub const OCR_RESPONSE_MAX_BYTES: usize = 64 * 1024 * 1024; pub const OCR_INLINE_MAX_BYTES: usize = 50 * 1024 * 1024; pub const OCR_MAX_FETCH_REDIRECTS: usize = 10; pub const OCR_POLL_RETRY_SECS: u64 = 2; +pub const DROP_PARAMS_PARAM: &str = "drop_params"; #[derive(Clone, Default)] pub struct OcrCredentialInputs { @@ -185,6 +188,10 @@ pub fn response_format(optional_params: &CallArguments) -> Result bool { + optional_params.get(DROP_PARAMS_PARAM) == Some(&Value::Bool(true)) +} + #[derive(Debug)] pub struct DecodedOcrResponse { pub data: T, @@ -194,17 +201,13 @@ pub struct DecodedOcrResponse { pub fn decode_request_value(value: Value, prefix: &str) -> Result { serde_path_to_error::deserialize(value.into_deserializer()).map_err(|error| { - Error::RequestField { - path: format!("{prefix}.{}", error.path()), - } + Error::request_field(format!("{prefix}.{}", error.path()), error.into_inner()) }) } pub fn decode_response_value(value: Value, prefix: &str) -> Result { serde_path_to_error::deserialize(value.into_deserializer()).map_err(|error| { - Error::ResponseField { - path: format!("{prefix}.{}", error.path()), - } + Error::response_field(format!("{prefix}.{}", error.path()), error.into_inner()) }) } @@ -213,19 +216,15 @@ pub fn decode_response( native: bool, ) -> Result, Error> { let mut deserializer = serde_json::Deserializer::from_slice(bytes); - let data = serde_path_to_error::deserialize(&mut deserializer).map_err(|error| { - Error::ResponseField { - path: error.path().to_string(), - } - })?; - deserializer.end().map_err(|_| Error::ResponseField { - path: "response".into(), - })?; + let data = serde_path_to_error::deserialize(&mut deserializer) + .map_err(|error| Error::response_field(error.path().to_string(), error.into_inner()))?; + deserializer + .end() + .map_err(|error| Error::response_field("response", error))?; let native = if native { Some( - serde_json::from_slice(bytes).map_err(|_| Error::ResponseField { - path: "response".into(), - })?, + serde_json::from_slice(bytes) + .map_err(|error| Error::response_field("response", error))?, ) } else { None @@ -369,12 +368,12 @@ pub trait BaseOcrConfig: Send + Sync + Sized + 'static { &self, error_message: String, status_code: u16, - headers: Vec<(String, String)>, + headers: reqwest::header::HeaderMap, ) -> Error { Error::Provider { status: status_code, body: error_message, - headers, + headers: Box::new(headers), } } @@ -391,7 +390,7 @@ pub trait BaseOcrConfig: Send + Sync + Sized + 'static { request: &PreparedOcrRequest, client: &OcrClient, hooks: &dyn CallHooks, - ) -> impl Future> + Send { + ) -> impl Future> + Send { async move { let params = self.map_ocr_params(&request.optional_params, &request.model)?; let environment = self.validate_environment(request, client).await?; diff --git a/litellm-rust/crates/llms/src/base_llm/sse.rs b/litellm-rust/crates/llms/src/base_llm/sse.rs index 5ff2e3adddd..23f0c1b9389 100644 --- a/litellm-rust/crates/llms/src/base_llm/sse.rs +++ b/litellm-rust/crates/llms/src/base_llm/sse.rs @@ -15,12 +15,15 @@ use bytes::Bytes; use futures_util::{StreamExt, TryStreamExt, future::ready, stream::BoxStream}; use litellm_framer::{ - frames, + SseError, frames, sse::{SseCodec, SseEvent, decode_all}, }; use serde::{Serialize, de::DeserializeOwned}; -use crate::{Error, ErrorDetail, base_llm::base_model_iterator::ByteStream}; +use crate::{ + Error, ErrorDetail, + base_llm::base_model_iterator::{ByteStream, tunnelled_http}, +}; pub type EventStream = BoxStream<'static, Result>; @@ -70,8 +73,13 @@ pub fn invalid(error: serde_json::Error) -> Error { Error::InvalidResponse(ErrorDetail::invalid("stream event", error)) } -fn framing(error: litellm_framer::SseError) -> Error { - Error::InvalidResponse(ErrorDetail::failed("stream framing", error)) +fn framing(error: SseError) -> Error { + match &error { + SseError::Body(body) if let Some(http) = tunnelled_http(body) => Error::Http(http.clone()), + SseError::Body(_) | SseError::InvalidUtf8(_) => { + Error::InvalidResponse(ErrorDetail::failed("stream framing", error)) + } + } } #[cfg(test)] @@ -94,6 +102,45 @@ mod tests { assert_eq!(is_done(&event), done); } + #[rstest] + #[tokio::test] + async fn a_transport_failure_tunnelled_through_the_body_comes_back_as_itself() { + use futures_util::StreamExt; + + let failure = litellm_http::Error::upstream(502, "gateway went away".into()); + let bytes: ByteStream = futures_util::stream::iter([ + Ok(Bytes::from_static(b"data: {}\n\n")), + Err(std::io::Error::other(failure.clone())), + ]) + .boxed(); + + let mut events = event_stream::(bytes, is_done); + assert!(events.next().await.expect("first event").is_ok()); + let error = events.next().await.expect("failure").unwrap_err(); + assert_eq!(error, Error::Http(failure)); + } + + #[rstest] + #[tokio::test] + async fn a_body_failure_that_is_not_transport_is_an_invalid_response() { + use futures_util::StreamExt; + + let bytes: ByteStream = + futures_util::stream::iter([Err(std::io::Error::other("disk full"))]).boxed(); + + let error = event_stream::(bytes, is_done) + .next() + .await + .expect("failure") + .unwrap_err(); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + assert!( + litellm_error::Report::new(&error) + .to_string() + .contains("disk full") + ); + } + #[rstest] fn the_done_frame_we_send_decodes_as_a_control_frame() { let events: Vec = decode(DONE_FRAME, is_done).unwrap(); diff --git a/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs b/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs index 61dd264336e..1c976c1f73d 100644 --- a/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs +++ b/litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs @@ -6,15 +6,15 @@ use litellm_auth_aws::{ }; use litellm_core_utils::core_helpers::json_type_name; use litellm_llms_types::{ - formats::audio_transcription::AudioTranscriptionResponseData, - providers::bedrock::ConverseResponse, + formats::audio_transcription::{AudioTranscriptionResponseData, AudioTranscriptionUsage}, + providers::bedrock::{ConverseResponse, ConverseUsage}, }; use serde::Deserialize; use serde_json::{Map, Value, json}; use strum::IntoStaticStr; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ audio_transcription::transformation::{ AudioTranscriptionRequestData, BaseAudioTranscriptionConfig, Headers, @@ -59,20 +59,12 @@ fn audio_fields(audio: Value) -> Result { .and_then(Value::as_str) .filter(|value| !value.is_empty()) .ok_or(Error::MissingField("audio.data"))?; - let format = object.get("format").cloned().ok_or_else(|| { - Error::InvalidRequest( - "audio.format must be wav, mp3, flac, or ogg" - .to_string() - .into(), - ) - })?; - let format: AudioFormat = serde_json::from_value(format).map_err(|_| { - Error::InvalidRequest( - "audio.format must be wav, mp3, flac, or ogg" - .to_string() - .into(), - ) - })?; + let format = object + .get("format") + .cloned() + .ok_or(Error::MissingField("audio.format"))?; + let format: AudioFormat = serde_json::from_value(format) + .map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("audio.format", error)))?; Ok(AudioInput { data: data.to_string(), format, @@ -109,10 +101,10 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig { if let Some(prompt) = optional_string(&optional_params, "prompt") { instruction.push_str(&format!(" Additional context: {prompt}")); } - let mut inference_config = Map::from_iter([("maxTokens".to_string(), json!(4096))]); - if let Some(temperature) = optional_params.get("temperature") { - inference_config.insert("temperature".to_string(), temperature.clone()); - } + let temperature = optional_params + .get("temperature") + .cloned() + .unwrap_or(json!(0)); Ok(AudioTranscriptionRequestData { body: json!({ "messages": [{ @@ -122,8 +114,7 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig { {"text": instruction} ] }], - "system": [{"text": "You are a transcription assistant."}], - "inferenceConfig": inference_config, + "inferenceConfig": {"maxTokens": 4096, "temperature": temperature}, }), }) } @@ -133,19 +124,20 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig { _model: &str, response_json: Value, ) -> Result { + let reports_usage = response_json.get("usage").is_some(); let response: ConverseResponse = serde_json::from_value(response_json).map_err(|error| { - Error::InvalidResponse(format!("invalid Converse response: {error}").into()) + Error::InvalidResponse(ErrorDetail::invalid("Converse response", error)) })?; if response.message_content_is_non_text() { - return Err(Error::InvalidResponse( - "Bedrock response contains non-text transcript content" - .to_string() - .into(), - )); + return Err(Error::InvalidResponse(ErrorDetail::Expected { + subject: "Bedrock transcript content", + expected: "text", + })); } Ok(AudioTranscriptionResponseData { text: response.content_text(), + usage: reports_usage.then(|| transcription_usage(&response.usage)), }) } @@ -200,22 +192,41 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig { } } +fn transcription_usage(usage: &ConverseUsage) -> AudioTranscriptionUsage { + AudioTranscriptionUsage { + input_tokens: usage.input_tokens, + output_tokens: usage.output_tokens, + total_tokens: usage + .total_tokens + .unwrap_or(usage.input_tokens + usage.output_tokens), + } +} + #[cfg(test)] mod tests { use super::*; + use rstest::rstest; fn no_env(_: &str) -> Option { None } - #[test] - fn request_matches_python_shape() { - let params = Map::from_iter([ - ("language".to_string(), json!("en")), - ("prompt".to_string(), json!("Speaker names")), - ("temperature".to_string(), json!(0)), - ("timestamp_granularities".to_string(), json!(["word"])), - ]); + #[rstest] + #[case::caller_temperature(Some(json!(0.4)), json!(0.4))] + #[case::default_temperature(None, json!(0))] + fn request_sends_audio_and_instruction_without_system_block( + #[case] temperature: Option, + #[case] expected_temperature: Value, + ) { + let params = Map::from_iter( + [ + ("language".to_string(), json!("en")), + ("prompt".to_string(), json!("Speaker names")), + ("timestamp_granularities".to_string(), json!(["word"])), + ] + .into_iter() + .chain(temperature.map(|value| ("temperature".to_string(), value))), + ); let params = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG.map_transcription_params(¶ms); let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG .transform_audio_transcription_request( @@ -234,22 +245,39 @@ mod tests { {"text": "Transcribe the audio. Respond with only the transcript. The audio language is en. Additional context: Speaker names"} ] }], - "system": [{"text": "You are a transcription assistant."}], - "inferenceConfig": {"maxTokens": 4096, "temperature": 0} + "inferenceConfig": {"maxTokens": 4096, "temperature": expected_temperature} }) ); } - #[test] - fn response_concatenates_content_blocks() { + #[rstest] + #[case::reported_total( + json!({"inputTokens": 390, "outputTokens": 60, "totalTokens": 450}), + json!({"text": "hello world", "usage": {"type": "tokens", "input_tokens": 390, "output_tokens": 60, "total_tokens": 450}}) + )] + #[case::derived_total( + json!({"inputTokens": 1, "outputTokens": 2}), + json!({"text": "hello world", "usage": {"type": "tokens", "input_tokens": 1, "output_tokens": 2, "total_tokens": 3}}) + )] + fn response_carries_text_and_converse_usage(#[case] usage: Value, #[case] expected: Value) { let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG .transform_audio_transcription_response( "model", - json!({"output": {"message": {"content": [{"text": "hello "}, {"text": "world"}]}}, "usage": {"inputTokens": 1, "outputTokens": 2}}), + json!({"output": {"message": {"content": [{"text": "hello "}, {"text": "world"}]}}, "usage": usage}), ) .expect("response"); - assert_eq!(result.text, "hello world"); - assert_eq!(result.into_json(), json!({"text": "hello world"})); + assert_eq!(result.into_json(), expected); + } + + #[rstest] + fn response_without_usage_reports_none() { + let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG + .transform_audio_transcription_response( + "model", + json!({"output": {"message": {"content": [{"text": "hello"}]}}}), + ) + .expect("response"); + assert_eq!(result.into_json(), json!({"text": "hello"})); } #[test] diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs index caa345a7024..6cf0167a30f 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs @@ -1,10 +1,13 @@ use litellm_auth::AwsParams; -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_auth_aws::{ AwsCredentialSource, bedrock_model_id_and_region, constants::{AWS_BEARER_TOKEN_BEDROCK, BEDROCK_RUNTIME_ENDPOINT_TEMPLATE, BEDROCK_SERVICE}, resolve_bedrock_region, }; +use litellm_core_utils::decline::Decline; use litellm_core_utils::{ core_helpers::{finish_reason_for, unix_now, usage_from_parts}, prompt_templates::factory::{Conversation, TurnRole, build_conversation}, @@ -19,14 +22,14 @@ use litellm_llms_types::{ use serde_json::{Map, Value, json}; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ auth::AuthScheme, chat::{ streaming::StreamShape, transformation::{ BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, - Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param, + ValidatedEnvironment, unsupported_message, unsupported_param, }, }, }, @@ -154,25 +157,20 @@ impl BaseConfig for AmazonConverseConfig { response: ProviderChatResponseData, ) -> Result { let body = response.body; - if !body.is_object() { - return Err(Error::InvalidResponse( - "converse response is not an object".into(), - )); - } - for field in ["output", "usage"] { - if body.get(field).is_none() { - return Err(Error::InvalidResponse( - format!("invalid Converse response: missing field `{field}`").into(), - )); - } - } + let has_usage = body.get("usage").is_some(); let response: ConverseResponse = serde_json::from_value(body).map_err(|error| { - Error::InvalidResponse(format!("invalid Converse response: {error}").into()) + Error::InvalidResponse(ErrorDetail::invalid("Converse response", error)) })?; - // The route declines tool requests, so anything other than a text block - // is something this path never asked for. Decline; the host falls back. + if !has_usage { + return Err(Error::InvalidResponse(ErrorDetail::MissingField { + subject: "Converse response", + field: "usage", + })); + } if response.message_content_is_non_text() { - return Err(Error::Unsupported("non-text response content block")); + return Err(Error::InvalidResponse(ErrorDetail::Unhandled( + "non-text response content block", + ))); } let text = response.content_text(); let computed = usage_from_parts( @@ -269,7 +267,7 @@ impl BaseConfig for AmazonConverseConfig { &self, messages: &[ChatMessage], optional_params: &Map, - ) -> Option { + ) -> Option { unsupported_param( self.supported_openai_param_mappings(), CONFIG_PARAMS, @@ -283,7 +281,7 @@ impl BaseConfig for AmazonConverseConfig { messages .iter() .any(has_blank_text) - .then_some(Unsupported("blank message text")) + .then_some(Decline::new("blank message text")) }) // Converse has no assistant prefill: Python inserts a continue turn // when a conversation opens or closes on an assistant message, and @@ -296,7 +294,7 @@ impl BaseConfig for AmazonConverseConfig { .turns .last() .is_some_and(|turn| turn.role == TurnRole::Assistant); - (!conversation.opens_on_user_turn() || ends_on_assistant).then_some(Unsupported( + (!conversation.opens_on_user_turn() || ends_on_assistant).then_some(Decline::new( "conversation does not run user turn to user turn", )) }) @@ -358,11 +356,12 @@ mod tests { Error, base_llm::{ auth::AuthScheme, - chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported}, + chat::transformation::{BaseConfig, ProviderChatResponseData}, }, bedrock::chat::converse_transformation::BEDROCK_CHAT_COMPLETIONS_CONFIG, }; use litellm_auth::CredentialPlacement; + use litellm_core_utils::decline::Decline; use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage}; use rstest::rstest; use serde_json::{Map, Value, json}; @@ -396,7 +395,7 @@ mod tests { ) } - fn reason(msgs: Value, opts: Value) -> Option { + fn reason(msgs: Value, opts: Value) -> Option { BEDROCK_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts)) } @@ -472,7 +471,7 @@ mod tests { json!([{"role": "user", "content": "hi"}]), json!({"stream": true}) ), - Some(Unsupported("streaming")) + Some(Decline::new("streaming")) ); } @@ -483,7 +482,7 @@ mod tests { json!([{"role": "user", "content": "hi"}]), json!({"topK": 40}) ), - Some(Unsupported("unrecognized request parameter")) + Some(Decline::new("unrecognized request parameter")) ); } @@ -497,7 +496,7 @@ mod tests { fn declines_tools_and_other_params_outside_the_allowlist(#[case] param: Value) { assert_eq!( reason(json!([{"role": "user", "content": "hi"}]), param.clone()), - Some(Unsupported("unrecognized request parameter")), + Some(Decline::new("unrecognized request parameter")), "expected {param} to decline" ); } @@ -514,7 +513,7 @@ mod tests { json!([{"role": "user", "content": content}, {"role": "user", "content": "hi"}]), json!({}) ), - Some(Unsupported("blank message text")), + Some(Decline::new("blank message text")), "expected blank content {content} to decline" ); } @@ -526,7 +525,7 @@ mod tests { // host before the call rather than an error after it. assert_eq!( reason(json!([{"role": "user", "content": []}]), json!({})), - Some(Unsupported("message without content")) + Some(Decline::new("message without content")) ); assert_eq!( reason( @@ -547,7 +546,7 @@ mod tests { ]), json!({}) ), - Some(Unsupported( + Some(Decline::new( "conversation does not run user turn to user turn" )) ); @@ -559,7 +558,7 @@ mod tests { ]), json!({}) ), - Some(Unsupported( + Some(Decline::new( "conversation does not run user turn to user turn" )) ); @@ -819,7 +818,7 @@ mod tests { ]}]), json!({}) ), - Some(Unsupported("non-text message content")) + Some(Decline::new("non-text message content")) ); } @@ -846,7 +845,7 @@ mod tests { } #[test] - fn declines_a_response_carrying_a_tool_use_block() { + fn a_response_carrying_a_tool_use_block_is_invalid() { let err = transform_response(json!({ "output": {"message": {"content": [ {"toolUse": {"toolUseId": "t1", "name": "f", "input": {}}} @@ -855,37 +854,32 @@ mod tests { "usage": {"inputTokens": 1, "outputTokens": 1} })) .expect_err("tool use block"); - assert_eq!(err, Error::Unsupported("non-text response content block")); - } - - #[test] - fn errors_on_a_response_missing_required_fields() { assert_eq!( - transform_response(json!("nope")).expect_err("not an object"), - Error::InvalidResponse("converse response is not an object".to_string().into()) - ); - assert_eq!( - transform_response(json!({"usage": {}})).expect_err("no output"), - Error::InvalidResponse("invalid Converse response: missing field `output`".into()) - ); - assert_eq!( - transform_response(json!({"output": {"message": {"content": []}}})) - .expect_err("no usage"), - Error::InvalidResponse("invalid Converse response: missing field `usage`".into()) + err, + Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "non-text response content block" + )) ); } - #[test] - fn rejects_malformed_text_and_token_counts() { - for response in [ - json!({"output": {"message": {"content": [{"text": 123}]}}, "usage": {"inputTokens": 1, "outputTokens": 1}}), - json!({"output": {"message": {"content": [{"text": "x"}]}}, "usage": {"inputTokens": "1", "outputTokens": 1}}), - ] { - assert!(matches!( - transform_response(response), - Err(Error::InvalidResponse(message)) if message.to_string().starts_with("invalid Converse response:") - )); - } + #[rstest::rstest] + #[case::not_an_object(json!("nope"), "invalid type")] + #[case::no_output(json!({"usage": {"inputTokens": 1, "outputTokens": 1}}), "missing field `output`")] + #[case::no_usage(json!({"output": {"message": {"content": []}}}), "missing usage")] + #[case::non_string_text( + json!({"output": {"message": {"content": [{"text": 123}]}}, "usage": {"inputTokens": 1, "outputTokens": 1}}), + "invalid type" + )] + #[case::non_integer_tokens( + json!({"output": {"message": {"content": [{"text": "x"}]}}, "usage": {"inputTokens": "1", "outputTokens": 1}}), + "invalid type" + )] + fn rejects_a_malformed_response(#[case] response: Value, #[case] fact: &str) { + let error = transform_response(response).expect_err("malformed response"); + assert!(matches!(error, Error::InvalidResponse(_)), "{error:?}"); + let report = litellm_error::Report::new(&error).to_string(); + assert!(report.contains("Converse response"), "{report}"); + assert!(report.contains(fact), "{report}"); } #[test] diff --git a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs index 878c424ae2d..15a6ff38822 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs @@ -2,6 +2,7 @@ use base64::Engine; use bytes::Buf; use futures_util::{Stream, StreamExt}; use litellm_framer::{ + EventStreamError, aws_event_stream::{AwsEventStreamCodec, Message}, frames, }; @@ -11,9 +12,8 @@ use serde_json::Value; use crate::{ Error, - anthropic::chat::handler::ModelResponseIterator, base_llm::{ - chat::streaming::{ChatStream, StreamShape}, + base_model_iterator::tunnelled_http, messages::streaming::{ByteStream, EventStream}, }, }; @@ -47,11 +47,21 @@ where B: Buf + Send, E: std::error::Error + Send + Sync + 'static, { - frames(input, AwsEventStreamCodec).map(|message| { - decode_invoke_chunk(message.map_err(|error| { + frames(input, AwsEventStreamCodec).map(|message| decode_invoke_chunk(message.map_err(framing)?)) +} + +fn framing(error: EventStreamError) -> Error { + match &error { + EventStreamError::Body(body) if let Some(http) = tunnelled_http(body) => { + Error::Http(http.clone()) + } + EventStreamError::Body(_) + | EventStreamError::InvalidLength(_) + | EventStreamError::Truncated + | EventStreamError::Malformed(_) => { Error::InvalidResponse(crate::ErrorDetail::failed("stream framing", error)) - })?) - }) + } + } } pub fn decode_invoke_anthropic_chunk(chunk: Value) -> Result { @@ -64,34 +74,6 @@ pub fn invoke_anthropic_event_stream(bytes: ByteStream) -> EventStream { Box::pin(invoke_chunk_stream(bytes).map(|chunk| decode_invoke_anthropic_chunk(chunk?))) } -#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::EnumString)] -enum InvokeProvider { - #[strum(serialize = "anthropic")] - Anthropic, - #[strum(serialize = "deepseek_r1")] - DeepseekR1, - #[strum(serialize = "moonshot")] - Moonshot, - #[strum(disabled)] - Unsupported, -} - -pub fn invoke_chat_stream(invoke_provider: &str, shape: StreamShape) -> Result { - match invoke_provider - .parse() - .unwrap_or(InvokeProvider::Unsupported) - { - InvokeProvider::Anthropic => Ok(ChatStream::new( - invoke_anthropic_event_stream, - ModelResponseIterator::new(shape), - )), - InvokeProvider::DeepseekR1 | InvokeProvider::Moonshot => Err(Error::Unsupported( - "Bedrock invoke streaming for this model family", - )), - InvokeProvider::Unsupported => Err(Error::Unsupported("Bedrock invoke streaming")), - } -} - #[cfg(test)] mod tests { use aws_smithy_eventstream::frame::write_message_to; @@ -112,22 +94,6 @@ mod tests { const TEXT_DELTA: &str = r#"{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"hello"}}"#; - #[rstest::rstest] - #[case::anthropic("anthropic", InvokeProvider::Anthropic)] - #[case::deepseek_r1("deepseek_r1", InvokeProvider::DeepseekR1)] - #[case::moonshot("moonshot", InvokeProvider::Moonshot)] - #[case::unknown("qwen", InvokeProvider::Unsupported)] - #[case::case_sensitive("Anthropic", InvokeProvider::Unsupported)] - fn invoke_provider_maps_each_model_family_name( - #[case] name: &str, - #[case] expected: InvokeProvider, - ) { - assert_eq!( - name.parse().unwrap_or(InvokeProvider::Unsupported), - expected - ); - } - fn aws_wire(chunk: &str) -> Vec { let payload = serde_json::json!({"bytes": STANDARD.encode(chunk)}); let message = Message::new(Bytes::from(serde_json::to_vec(&payload).unwrap())).add_header( diff --git a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs index 72958fd6cb9..5b89345a1c5 100644 --- a/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/claude_platform/messages/transformation.rs @@ -1,6 +1,8 @@ use std::sync::LazyLock; -use litellm_auth::{AwsParams, CredentialPlacement, ParamSpec, SecretValue}; +use litellm_auth::{AwsParams, CredentialPlacement, ParamKind, ParamSpec}; +use litellm_secret_value::SecretValue; + use litellm_auth_aws::{AwsCredentialSource, resolve_aws_region}; use litellm_core_utils::settings::resolve_non_empty; use litellm_http::request::{header_value, with_header, without_headers}; @@ -36,6 +38,7 @@ const WORKSPACE_FIELDS: &[&str] = &[ const WORKSPACE_HEADER: &str = "anthropic-workspace-id"; static WORKSPACE: ParamSpec = ParamSpec { setting: "workspace_id", + kind: ParamKind::Setting, wire: WORKSPACE_FIELDS, module_global: None, env: WORKSPACE_ENVS, diff --git a/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs b/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs index 1f6b6edf28d..1d380c917f6 100644 --- a/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/messages/invoke_transformations/anthropic_claude_transformation.rs @@ -1,3 +1,4 @@ +use litellm_core_utils::decline::Decline; use std::convert::Infallible; use crate::{ @@ -6,7 +7,9 @@ use crate::{ }; use futures_util::StreamExt; use litellm_auth::AwsParams; -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_auth_aws::{ AwsCredentialSource, bedrock_model_id_and_region, constants::{ @@ -130,9 +133,9 @@ impl BaseMessagesConfig for AmazonAnthropicClaudeMessagesConfig { _request: MessagesRequest, _context: &MessagesTransformContext, ) -> Result { - Err(Error::Unsupported( + Err(Error::Declined(Decline::new( "Bedrock invoke messages request shaping", - )) + ))) } fn secret_names(&self) -> &'static [&'static str] { diff --git a/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs b/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs index ae13704f4a0..f139b93e259 100644 --- a/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock_mantle/messages/transformation.rs @@ -1,6 +1,8 @@ use std::sync::LazyLock; -use litellm_auth::{AwsParams, CredentialPlacement, SecretValue}; +use litellm_auth::{AwsParams, CredentialPlacement}; +use litellm_secret_value::SecretValue; + use litellm_auth_aws::AwsCredentialSource; use litellm_auth_aws::bedrock_model_id_and_region; use litellm_core_utils::settings::resolve_non_empty; diff --git a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs index dadab413fe4..5bad06adda2 100644 --- a/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/cohere/ocr/transformation.rs @@ -195,11 +195,10 @@ impl CohereParseConfig { .and_then(env_lookup) .filter(|key| !key.trim().is_empty()) }) - .ok_or_else(|| { - Error::Auth(litellm_auth::Error::ProviderAuthentication( - "Missing COHERE_API_KEY - set it in the environment or pass api_key".into(), - )) - })?; + .ok_or(Error::Auth(litellm_auth::Error::MissingApiKey { + provider: "Cohere", + environment_variable: COHERE_API_KEY_ENV, + }))?; Ok( std::iter::once(("Authorization".into(), format!("Bearer {key}"))) .chain(connection.extra_headers.clone()) @@ -208,27 +207,28 @@ impl CohereParseConfig { } fn build_ocr_url(&self, api_base: &str) -> Result { - let parsed = reqwest::Url::parse(api_base).map_err(|_| invalid_api_base())?; + let parsed = reqwest::Url::parse(api_base) + .map_err(|error| Error::request_field("api_base", error))?; if !matches!(parsed.scheme(), "http" | "https") { return Err(invalid_api_base()); } ApiUrl::parse(api_base) .and_then(|url| url.complete_path(&["v2", "parse"])) .map(|url| url.into_string()) - .map_err(|_| invalid_api_base()) + .map_err(|error| Error::request_field("api_base", error)) } } pub fn validate_document(document: &OcrDocument) -> Result<(), Error> { let OcrDocument::ImageUrl { image_url, .. } = document else { - return Err(Error::CohereImageOnly); + return Err(image_only()); }; if image_url.is_empty() { - return Err(Error::CohereImageOnly); + return Err(image_only()); } if let Some(inline) = InlineDocument::parse(image_url)? { if !inline.mime_type().type_.eq_ignore_ascii_case("image") { - return Err(Error::CohereImageOnly); + return Err(image_only()); } inline.decode(OCR_INLINE_MAX_BYTES)?; } @@ -260,7 +260,7 @@ pub fn normalize_response( fn image_url(document: OcrDocument) -> Result { validate_document(&document)?; let OcrDocument::ImageUrl { image_url, .. } = document else { - return Err(Error::CohereImageOnly); + return Err(image_only()); }; Ok(image_url) } @@ -329,9 +329,17 @@ fn billed_pages(response: &CohereResponse) -> Option { response.meta.as_ref()?.billed_units.as_ref()?.pages } +fn image_only() -> Error { + Error::UnsupportedDocument { + provider: "Cohere Parse", + guidance: "it only accepts `image_url` documents, not document_url or PDF inputs", + } +} + fn invalid_api_base() -> Error { Error::RequestField { path: "api_base".into(), + cause: None, } } @@ -368,11 +376,14 @@ mod tests { #[test] fn options_reject_invalid_output_format() { let invalid = serde_json::from_value(json!({"output_format":"html"})).unwrap(); - assert!(matches!( - CohereParseConfig.map_ocr_params(&invalid, "parse"), - Err(Error::RequestField { path }) - if path == "optional_params.output_format" - )); + let Err(error) = CohereParseConfig.map_ocr_params(&invalid, "parse") else { + panic!("html is not a Cohere output format"); + }; + assert!(error.is_request()); + let argument = + litellm_error::find::(&error) + .expect("the argument failure is kept"); + assert_eq!(argument.path, "output_format"); } #[test] @@ -432,7 +443,7 @@ mod tests { .unwrap(); assert!(matches!( normalize_response("parse", response).unwrap_err(), - Error::ResponseField { path } + Error::ResponseField { path, .. } if path == "pages[0].markdown.images[0].image_base64" )); } @@ -657,7 +668,10 @@ mod tests { fn request_requires_image(#[case] value: Value) { assert!(matches!( validate_document(&serde_json::from_value(value).unwrap()), - Err(Error::CohereImageOnly) + Err(Error::UnsupportedDocument { + provider: "Cohere Parse", + .. + }) )); } @@ -721,7 +735,7 @@ mod tests { assert!(matches!( CohereParseConfig.resolve_headers( &OcrConnection { - api_key: Some(litellm_auth::SecretValue::new(" ")), + api_key: Some(litellm_secret_value::SecretValue::new(" ")), ..Default::default() }, &|_| None, diff --git a/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs b/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs index 168c3e63920..0dbcea01439 100644 --- a/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/deepseek/messages/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_llms_types::{ formats::messages::{MessagesOptionalParams, MessagesRequest, MessagesTool}, diff --git a/litellm-rust/crates/llms/src/edenai/messages/transformation.rs b/litellm-rust/crates/llms/src/edenai/messages/transformation.rs index 2c3a7eaef1d..d5944484522 100644 --- a/litellm-rust/crates/llms/src/edenai/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/edenai/messages/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_llms_types::formats::messages::{ MessagesOptionalParams, MessagesRequest, MessagesResponse, diff --git a/litellm-rust/crates/llms/src/error.rs b/litellm-rust/crates/llms/src/error.rs index 10a4f696406..2844fbf3e01 100644 --- a/litellm-rust/crates/llms/src/error.rs +++ b/litellm-rust/crates/llms/src/error.rs @@ -1,3 +1,6 @@ +use litellm_core_utils::decline::Decline; +use litellm_error::Source; + #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum Error { #[error("expected {expected}, got {actual}")] @@ -7,31 +10,144 @@ pub enum Error { }, #[error("missing required field: {0}")] MissingField(&'static str), - #[error("invalid request: {0}")] + #[error("invalid request")] InvalidRequest(#[source] ErrorDetail), - #[error("invalid response: {0}")] + #[error("invalid response")] InvalidResponse(#[source] ErrorDetail), - #[error("unsupported: {0}")] - Unsupported(&'static str), + #[error(transparent)] + Declined(Decline), #[error(transparent)] Auth(#[from] litellm_auth::Error), + #[error(transparent)] + Http(#[from] litellm_http::Error), +} + +impl Error { + pub fn is_request(&self) -> bool { + match self { + Self::InvalidType { .. } | Self::MissingField(_) | Self::InvalidRequest(_) => true, + Self::Auth(error) => error.is_request(), + Self::Http(error) => error.is_request(), + Self::InvalidResponse(_) | Self::Declined(_) => false, + } + } + + pub fn is_response(&self) -> bool { + match self { + Self::InvalidResponse(_) => true, + Self::Http(error) => error.is_response(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::Declined(_) + | Self::Auth(_) => false, + } + } + + pub fn is_missing_credential(&self) -> bool { + match self { + Self::Auth(error) => error.is_missing_credential(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Declined(_) + | Self::Http(_) => false, + } + } + + /// A transformation runs on both sides of the send, so only the transport and credential + /// resolution, which always runs before it, can tell whether the request reached the provider. + pub fn dispatched(&self) -> bool { + match self { + Self::Http(error) => error.dispatched(), + Self::Auth(_) => false, + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) => true, + Self::Declined(_) => false, + } + } + + /// Why the transformation rejected the caller's request, rendered as Python's own + /// validation renders it, without this layer's "invalid request" prefix. + pub fn rejected_request(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::InvalidRequest(detail) => Some(detail), + Self::InvalidType { .. } | Self::MissingField(_) => Some(self), + Self::InvalidResponse(_) | Self::Declined(_) | Self::Auth(_) | Self::Http(_) => None, + } + } + + pub fn declined(&self) -> Option<&Decline> { + match self { + Self::Declined(decline) => Some(decline), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Auth(_) + | Self::Http(_) => None, + } + } + + pub fn upstream_headers(&self) -> &[(String, String)] { + match self { + Self::Http(error) => error.upstream_headers(), + _ => &[], + } + } + + pub fn is_body_read_timeout(&self) -> bool { + matches!(self, Self::Http(error) if error.is_body_read_timeout()) + } + + pub fn is_timeout(&self) -> Option { + match self { + Self::Http(error) if error.is_timeout() => Some(error.is_connect_timeout()), + _ => None, + } + } + + pub fn upstream_response(&self) -> Option<(u16, &str)> { + match self { + Self::Http(error) => error.upstream_response(), + Self::InvalidType { .. } + | Self::MissingField(_) + | Self::InvalidRequest(_) + | Self::InvalidResponse(_) + | Self::Declined(_) + | Self::Auth(_) => None, + } + } } #[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] pub enum ErrorDetail { - #[error("{0}")] - Message(String), - #[error("invalid {subject}: {source}")] + #[error("invalid {subject}")] Invalid { subject: &'static str, #[source] - source: ErrorSource, + cause: Source, }, - #[error("{operation} failed: {source}")] + #[error("{operation} failed")] Failed { operation: &'static str, #[source] - source: ErrorSource, + cause: Source, + }, + #[error("{route} requires at least one message")] + NoMessages { route: &'static str }, + #[error("{subject} must be {expected}")] + Expected { + subject: &'static str, + expected: &'static str, + }, + #[error("{subject} is missing {field}")] + MissingField { + subject: &'static str, + field: &'static str, }, #[error("{field} must be {expected}, got {actual}")] InvalidValue { @@ -60,12 +176,24 @@ pub enum ErrorDetail { value: String, hint: String, }, - #[error("invalid {subject} on line {line}: {source}")] + #[error( + "thinking.budget_tokens must be a number, got {actual}. To drop it, set `litellm.drop_params=True`, or `drop_params: true` under `litellm_settings` on the proxy" + )] + NonNumericThinkingBudget { actual: &'static str }, + #[error( + "Missing API Base - A call is being made to LLM Provider but no api base is set either in the environment variables ({{LLM_PROVIDER}}_API_KEY) or via params" + )] + MissingApiBase, + #[error("the host call raised an exception")] + HostCall, + #[error("{0} is not handled by the rust path")] + Unhandled(&'static str), + #[error("invalid {subject} on line {line}")] InvalidLine { subject: &'static str, line: usize, #[source] - source: ErrorSource, + cause: Source, }, #[error("{operation} failed: {detail}")] RemoteFailure { @@ -77,64 +205,33 @@ pub enum ErrorDetail { impl ErrorDetail { pub fn invalid( subject: &'static str, - source: impl std::error::Error + Send + Sync + 'static, + cause: impl std::error::Error + Send + Sync + 'static, ) -> Self { Self::Invalid { subject, - source: ErrorSource::new(source), + cause: Source::new(cause), } } pub fn failed( operation: &'static str, - source: impl std::error::Error + Send + Sync + 'static, + cause: impl std::error::Error + Send + Sync + 'static, ) -> Self { Self::Failed { operation, - source: ErrorSource::new(source), + cause: Source::new(cause), + } + } + + pub fn invalid_line( + subject: &'static str, + line: usize, + cause: impl std::error::Error + Send + Sync + 'static, + ) -> Self { + Self::InvalidLine { + subject, + line, + cause: Source::new(cause), } } } - -impl From for ErrorDetail { - fn from(message: String) -> Self { - Self::Message(message) - } -} - -impl From<&str> for ErrorDetail { - fn from(message: &str) -> Self { - Self::Message(message.into()) - } -} - -#[derive(Clone, Debug)] -pub struct ErrorSource(std::sync::Arc); - -impl std::ops::Deref for ErrorSource { - type Target = dyn std::error::Error + Send + Sync; - - fn deref(&self) -> &Self::Target { - self.0.as_ref() - } -} - -impl std::fmt::Display for ErrorSource { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - std::fmt::Display::fmt(&self.0, formatter) - } -} - -impl ErrorSource { - pub fn new(error: impl std::error::Error + Send + Sync + 'static) -> Self { - Self(std::sync::Arc::new(error)) - } -} - -impl PartialEq for ErrorSource { - fn eq(&self, other: &Self) -> bool { - std::sync::Arc::ptr_eq(&self.0, &other.0) - } -} - -impl Eq for ErrorSource {} diff --git a/litellm-rust/crates/llms/src/lib.rs b/litellm-rust/crates/llms/src/lib.rs index eb232404d93..68e6e4010c2 100644 --- a/litellm-rust/crates/llms/src/lib.rs +++ b/litellm-rust/crates/llms/src/lib.rs @@ -17,4 +17,4 @@ pub mod reducto; pub mod tencent; pub mod vertex_ai; -pub use error::{Error, ErrorDetail, ErrorSource}; +pub use error::{Error, ErrorDetail}; diff --git a/litellm-rust/crates/llms/src/minimax/messages/transformation.rs b/litellm-rust/crates/llms/src/minimax/messages/transformation.rs index 012a657fceb..dafe5fbd16b 100644 --- a/litellm-rust/crates/llms/src/minimax/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/minimax/messages/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_llms_types::{ formats::messages::{MessagesOptionalParams, MessagesRequest}, diff --git a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs index 29d4d5c1610..679aa65f7fd 100644 --- a/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/mistral/ocr/transformation.rs @@ -171,9 +171,7 @@ impl MistralOcrConfig { ApiUrl::parse(base) .and_then(|url| url.complete_path(&["v1", "ocr"])) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -186,6 +184,7 @@ pub fn normalize_response( Some(None) => { return Err(Error::ResponseField { path: "model".into(), + cause: None, }); } None => model.to_string(), @@ -220,7 +219,7 @@ mod tests { #[default(vec![])] extra_headers: Vec<(String, String)>, ) -> OcrConnection { OcrConnection { - api_key: api_key.map(litellm_auth::SecretValue::new), + api_key: api_key.map(litellm_secret_value::SecretValue::new), extra_headers, ..OcrConnection::default() } @@ -231,7 +230,7 @@ mod tests { let response = serde_json::from_value(json!({"model":null})).unwrap(); assert!(matches!( normalize_response("fallback", response).unwrap_err(), - Error::ResponseField { path } if path == "model" + Error::ResponseField { path, .. } if path == "model" )); } @@ -263,7 +262,7 @@ mod tests { .unwrap_err(); assert!(matches!( error, - Error::ResponseField { path: actual } if actual == path + Error::ResponseField { path: actual, .. } if actual == path )); } diff --git a/litellm-rust/crates/llms/src/openai/responses/transformation.rs b/litellm-rust/crates/llms/src/openai/responses/transformation.rs index 99cc3559514..deb163daa07 100644 --- a/litellm-rust/crates/llms/src/openai/responses/transformation.rs +++ b/litellm-rust/crates/llms/src/openai/responses/transformation.rs @@ -3,10 +3,11 @@ use litellm_llms_types::formats::responses::{ }; use serde_json::{Map, Value}; -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; use crate::{ - Error, + Error, ErrorDetail, base_llm::{ auth::{AuthScheme, ValidatedEnvironment}, responses::transformation::{ @@ -180,15 +181,19 @@ pub fn responses_request( params: Map, ) -> Result { if !input.is_string() && !input.is_array() { - return Err(Error::InvalidRequest( - "responses input must be a string or an array".into(), - )); + return Err(Error::InvalidRequest(ErrorDetail::Expected { + subject: "responses input", + expected: "a string or an array", + })); } if params .get("stream") .is_some_and(|stream| !stream.is_boolean()) { - return Err(Error::InvalidRequest("stream must be a boolean".into())); + return Err(Error::InvalidRequest(ErrorDetail::Expected { + subject: "stream", + expected: "a boolean", + })); } Ok(Value::Object( params @@ -202,7 +207,9 @@ pub fn responses_request( } pub fn responses_response(body: Value) -> Result { - serde_json::from_value(body).map_err(|error| Error::InvalidResponse(error.to_string().into())) + serde_json::from_value(body).map_err(|error| { + Error::InvalidResponse(ErrorDetail::invalid("Responses API response", error)) + }) } #[cfg(test)] diff --git a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs index 0a89adf90f7..83fb8ac3fa2 100644 --- a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs @@ -4,7 +4,9 @@ //! the `max_completion_tokens` -> `max_tokens` rename and the usage //! `*_tokens` null-to-zero sanitize. -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::core_helpers::unix_now; use litellm_llms_types::formats::chat_completions::{ ChatCompletionsChoice, ChatCompletionsChoiceMessage, ChatCompletionsResponse, @@ -123,9 +125,12 @@ impl BaseConfig for OpenAILikeChatConfig { ) -> Result { let mut body = response.body; sanitize_usage(&mut body); - let body = body - .as_object() - .ok_or_else(|| Error::InvalidResponse("chat response is not an object".into()))?; + let body = + body.as_object() + .ok_or(Error::InvalidResponse(crate::ErrorDetail::Expected { + subject: "chat response", + expected: "an object", + }))?; let choices = body .get("choices") @@ -243,14 +248,20 @@ fn normalize_choice(position: usize, choice: &Value) -> Result Option { + fn reason(msgs: Value, opts: Value) -> Option { OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts)) } @@ -414,9 +426,13 @@ mod tests { #[rstest] fn a_missing_api_base_is_an_error() { assert!(matches!( - OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG - .get_complete_url(None, "my-model", ¶ms(json!({})), &no_env), - Err(Error::InvalidRequest(message)) if message.to_string().starts_with("Missing API Base") + OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG.get_complete_url( + None, + "my-model", + ¶ms(json!({})), + &no_env + ), + Err(Error::InvalidRequest(crate::ErrorDetail::MissingApiBase)) )); } @@ -528,9 +544,7 @@ mod tests { } #[rstest] - fn a_tool_call_response_declines_instead_of_dropping_the_calls() { - // The `json_mode` rewrite needs a request flag the route does not carry, so - // a tool-call answer falls back to Python rather than losing the calls. + fn a_tool_call_response_fails_instead_of_dropping_the_calls() { assert_eq!( transform_response(json!({ "model": "m", @@ -547,12 +561,14 @@ mod tests { "finish_reason": "tool_calls", }], })), - Err(Error::Unsupported("tool call response")) + Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "tool call response" + ))) ); } #[rstest] - fn a_refusal_declines_instead_of_returning_an_empty_reply() { + fn a_refusal_fails_instead_of_returning_an_empty_reply() { assert_eq!( transform_response(json!({ "model": "m", @@ -561,12 +577,14 @@ mod tests { "finish_reason": "stop", }], })), - Err(Error::Unsupported("refusal response")) + Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "refusal response" + ))) ); } #[rstest] - fn a_non_text_response_content_declines() { + fn a_non_text_response_content_is_invalid() { assert_eq!( transform_response(json!({ "model": "m", @@ -575,7 +593,9 @@ mod tests { "finish_reason": "stop", }], })), - Err(Error::Unsupported("non-text response content")) + Err(Error::InvalidResponse(crate::ErrorDetail::Unhandled( + "non-text response content" + ))) ); } @@ -585,7 +605,7 @@ mod tests { fn declines(#[case] opts: Value, #[case] expected: &'static str) { assert_eq!( reason(json!([{"role": "user", "content": "hi"}]), opts), - Some(Unsupported(expected)) + Some(Decline::new(expected)) ); } @@ -615,7 +635,7 @@ mod tests { json!([{"role": "user", "content": "hi"}]), json!({"tools": [{"type": "function", "function": {"name": "f"}}]}), ), - Some(Unsupported("unrecognized request parameter")) + Some(Decline::new("unrecognized request parameter")) ); } } diff --git a/litellm-rust/crates/llms/src/openai_like/common_utils.rs b/litellm-rust/crates/llms/src/openai_like/common_utils.rs index b0f5adafe00..d798f851d79 100644 --- a/litellm-rust/crates/llms/src/openai_like/common_utils.rs +++ b/litellm-rust/crates/llms/src/openai_like/common_utils.rs @@ -1,7 +1,7 @@ //! Shared OpenAI-like credential and endpoint resolution, mirroring //! `litellm/llms/openai_like/common_utils.py`. -use crate::Error; +use crate::{Error, ErrorDetail}; /// `OpenAILikeChatConfig._get_openai_compatible_provider_info`: the deployment's /// `api_base` wins over `OPENAI_LIKE_API_BASE`, and the deployment key over @@ -31,12 +31,7 @@ pub fn complete_openai_like_url( env_lookup: &dyn Fn(&str) -> Option, ) -> Result { let (api_base, _) = openai_compatible_provider_info(api_base, None, env_lookup); - let api_base = api_base.ok_or_else(|| { - Error::InvalidRequest( - "Missing API Base - A call is being made to LLM Provider but no api base is set either in the environment variables ({LLM_PROVIDER}_API_KEY) or via params" - .into(), - ) - })?; + let api_base = api_base.ok_or(Error::InvalidRequest(ErrorDetail::MissingApiBase))?; if custom_endpoint { return Ok(api_base); } diff --git a/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs b/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs index 5b4e4f77f2a..eeb3d1cb7f8 100644 --- a/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/openrouter/messages/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_http::request::{has_bearer_auth, without_headers}; use litellm_llms_types::{ diff --git a/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs b/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs index 900fba9e916..76155ba55e5 100644 --- a/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs +++ b/litellm-rust/crates/llms/src/openrouter/responses/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_llms_types::formats::responses::ResponsesApiResponse; use serde_json::{Map, Value}; diff --git a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs index 1979b442936..beda6899d04 100644 --- a/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/reducto/ocr/transformation.rs @@ -1,11 +1,5 @@ use std::collections::BTreeMap; -use litellm_core_utils::{ - call_arguments::{CallArguments, compose_body}, - params::OpaqueParams, - url_utils::ApiUrl, -}; -use litellm_http::outbound::OutboundRequest; use serde::{Deserialize, Deserializer, Serialize}; use serde_json::{Map, Value, json}; @@ -18,6 +12,11 @@ use crate::base_llm::ocr::{ decode_and_normalize_response, }, }; +use litellm_core_utils::{ + call_arguments::{CallArguments, compose_body}, + params::OpaqueParams, + url_utils::ApiUrl, +}; use litellm_llms_types::formats::ocr::{ LiteLLMOcrResponse, OcrDocument, OcrPage, OcrResponseFormat, OcrUsageInfo, }; @@ -173,7 +172,7 @@ impl BaseOcrConfig for ReductoParseV3Config { request: &PreparedOcrRequest, client: &OcrClient, hooks: &dyn CallHooks, - ) -> Result { + ) -> Result { prepare_upload_request(self, request, client, hooks).await } } @@ -262,7 +261,7 @@ impl BaseOcrConfig for ReductoParseLegacyConfig { request: &PreparedOcrRequest, client: &OcrClient, hooks: &dyn CallHooks, - ) -> Result { + ) -> Result { prepare_upload_request(self, request, client, hooks).await } } @@ -275,7 +274,7 @@ async fn prepare_upload_request, -) -> Result { +) -> Result { let params = config.map_ocr_params(&request.optional_params, &request.model)?; let headers = config.validate_environment(request, client).await?; let url = config.get_complete_url(request, ¶ms, &headers)?; @@ -300,9 +299,16 @@ async fn prepare_upload_request Error { + Error::UnsupportedDocument { + provider: "Reducto", + guidance: "pass a reducto:// id or a data URI, or upload the file first for plain HTTP URLs", + } +} + fn uploaded_file_id(document: OcrDocument) -> Result { if !document.source().starts_with(REDUCTO_ID_PREFIX) { - return Err(Error::ReductoSource); + return Err(unsupported_source()); } if document.source()[REDUCTO_ID_PREFIX.len()..] .trim() @@ -310,6 +316,7 @@ fn uploaded_file_id(document: OcrDocument) -> Result { { return Err(Error::RequestField { path: "document file id".into(), + cause: None, }); } Ok(ReductoFileId(document.source().into())) @@ -393,6 +400,7 @@ fn build_pages_from_reducto(chunks: Vec) -> Result, E Some(Value::String(content)) => Ok(Some(content.as_str())), Some(_) => Err(Error::ResponseField { path: "result.chunks.blocks.content".into(), + cause: None, }), }) .collect::, _>>()?; @@ -437,9 +445,7 @@ fn complete_endpoint_url(api_base: Option<&str>, path: &str) -> Result( response, false, @@ -547,6 +555,7 @@ async fn upload_bytes_async( let Some(file_id) = file_id else { return Err(Error::ResponseField { path: "file_id".into(), + cause: None, }); }; Ok(ReductoFileId(file_id.to_string())) @@ -554,6 +563,8 @@ async fn upload_bytes_async( #[cfg(test)] mod tests { + use rstest::rstest; + use super::*; #[tokio::test] @@ -566,10 +577,7 @@ mod tests { let params = ReductoParseV3Config .map_ocr_params(&overrides, "parse-v3") .unwrap(); - let client = OcrClient::for_test( - litellm_http::Client::plain_for_test(), - litellm_http::Client::no_redirect_for_test(), - ); + let client = OcrClient::for_test(); let connection = OcrConnection::default(); let document = serde_json::from_value( json!({"type":"document_url","document_url":"reducto://ready.pdf"}), @@ -629,15 +637,16 @@ mod tests { ); } + #[rstest] + #[case::fractional_pages(json!({"num_pages":1.5}))] + #[case::non_numeric_pages(json!({"num_pages":[]}))] + #[case::non_numeric_credits(json!({"credits":{}}))] + fn invalid_usage_is_rejected(#[case] usage: Value) { + assert!(serde_json::from_value::(json!({"usage":usage})).is_err()); + } + #[test] - fn usage_uses_shared_validation_while_block_page_numbers_are_best_effort() { - for usage in [ - json!({"num_pages":1.5}), - json!({"num_pages":[]}), - json!({"credits":{}}), - ] { - assert!(serde_json::from_value::(json!({"usage":usage})).is_err()); - } + fn block_page_numbers_are_best_effort() { let response = serde_json::from_value(json!({"result":{"chunks":[{"blocks":[ {"content":"ignored", "bbox":{"page":"invalid"}}, {"content":"kept", "bbox":{"page":2.5}, "extra":null} @@ -681,7 +690,7 @@ mod tests { #[test] fn explicit_key_precedes_environment_key() { let connection = OcrConnection { - api_key: Some(litellm_auth::SecretValue::new("passed-key")), + api_key: Some(litellm_secret_value::SecretValue::new("passed-key")), ..Default::default() }; let headers = resolve_headers(&connection, &|_| Some("env-key".into())).unwrap(); @@ -691,7 +700,7 @@ mod tests { #[test] fn blank_explicit_key_uses_environment_key() { let connection = OcrConnection { - api_key: Some(litellm_auth::SecretValue::new(" ")), + api_key: Some(litellm_secret_value::SecretValue::new(" ")), ..Default::default() }; let headers = resolve_headers(&connection, &|_| Some(" env-key ".into())).unwrap(); diff --git a/litellm-rust/crates/llms/src/tencent/messages/transformation.rs b/litellm-rust/crates/llms/src/tencent/messages/transformation.rs index a70336748ad..621a09f4fb0 100644 --- a/litellm-rust/crates/llms/src/tencent/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/tencent/messages/transformation.rs @@ -1,4 +1,6 @@ -use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_auth::CredentialPlacement; +use litellm_secret_value::SecretValue; + use litellm_core_utils::settings::resolve_non_empty; use litellm_llms_types::{ formats::messages::{MessagesOptionalParams, MessagesRequest}, diff --git a/litellm-rust/crates/llms/src/vertex_ai/messages/AGENTS.md b/litellm-rust/crates/llms/src/vertex_ai/messages/AGENTS.md index b5d637d6bc7..d95da65ac9d 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/messages/AGENTS.md +++ b/litellm-rust/crates/llms/src/vertex_ai/messages/AGENTS.md @@ -1,6 +1,7 @@ # rules -- Claude on Vertex AI: reuses `anthropic/messages` shaping, moves the model into the publisher URL, puts the Vertex `anthropic_version` in the body, and authenticates with a Google access token +- Claude on Vertex AI: reuses `anthropic/messages` shaping, moves the model into the publisher URL, and authenticates with a Google access token +- Vertex reads the API version only from the body's `anthropic_version` and answers 400 when an `anthropic-version` header names a different one, so the caller's header never reaches Vertex. A version Vertex accepts in the body is carried there, `2023-06-01`, `2023-01-01`, or no header get `vertex-2023-10-16`, and any other version drops under `drop_params` and is refused without it - Applies Vertex's beta policy and strips `output_config.effort` for models that do not accept it # references diff --git a/litellm-rust/crates/llms/src/vertex_ai/messages/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/messages/transformation.rs index 2e77dbc1ca8..038617c2e10 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/messages/transformation.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/messages/transformation.rs @@ -3,10 +3,10 @@ use litellm_auth_gcp::{ VertexConfig, get_vertex_ai_location, get_vertex_ai_project, get_vertex_ai_project_from_credentials, secret_names, }; -use litellm_http::request::has_header; +use litellm_http::request::{has_header, header_value, without_headers}; use litellm_llms_types::{ formats::messages::{MessagesOptionalParams, MessagesRequest, MessagesTool, OutputConfig}, - providers::anthropic::{AnthropicBeta, BetaProvider, BetaSet}, + providers::anthropic::{AnthropicBeta, BetaProvider, BetaSet, VERSION_HEADER}, recognized::Recognized, }; use litellm_router_types::LitellmParams; @@ -36,6 +36,8 @@ use litellm_llms_types::providers::vertex_ai::{ }; const WEB_SEARCH_TOOL_PREFIX: &str = "web_search"; +const VERSIONS_SERVED_AS_VERTEX_DEFAULT: [&str; 3] = ["", "2023-01-01", "2023-06-01"]; +const VERSIONS_VERTEX_ACCEPTS_IN_BODY: [&str; 2] = [ANTHROPIC_VERSION, "2026-10-21"]; /// Claude on Vertex AI, Python's `VertexAIPartnerModelsAnthropicMessagesConfig`: the Anthropic /// payload addressed to a project and location, authenticated with a Google access token. @@ -75,7 +77,7 @@ impl BaseMessagesConfig for VertexAiPartnerModelsAnthropicMessagesConfig { request: MessagesRequest, context: &MessagesTransformContext, ) -> Result { - let request = transform_messages_request(request, context)?; + let request = transform_messages_request(request, &context.keeping_display_updates())?; let request = strip_cache_control_scope(normalize_system_role_messages( request, context @@ -90,12 +92,6 @@ impl BaseMessagesConfig for VertexAiPartnerModelsAnthropicMessagesConfig { .params .output_config .and_then(|config| sanitize_output_config(config, accepts_effort)), - extra: request - .params - .extra - .into_iter() - .chain([("anthropic_version".to_string(), json!(ANTHROPIC_VERSION))]) - .collect(), ..request.params }, ..request @@ -133,7 +129,35 @@ impl BaseMessagesConfig for VertexAiPartnerModelsAnthropicMessagesConfig { } fn request_headers(&self, headers: Headers, request: &MessagesRequest) -> Headers { - merge_beta_headers(headers, vertex_feature_betas(request)) + merge_beta_headers( + without_headers(headers, &[VERSION_HEADER]), + vertex_feature_betas(request), + ) + } + + fn request_body( + &self, + request: MessagesRequest, + headers: &Headers, + drop_params: bool, + ) -> Result { + let version = vertex_version( + &request.model, + header_value(headers, VERSION_HEADER), + drop_params, + )?; + Ok(MessagesRequest { + params: MessagesOptionalParams { + extra: request + .params + .extra + .into_iter() + .chain([("anthropic_version".to_string(), json!(version))]) + .collect(), + ..request.params + }, + ..request + }) } /// Vertex addresses the model in the URL and rejects it in the body. @@ -225,6 +249,26 @@ fn vertex_feature_betas(request: &MessagesRequest) -> BetaSet { ) } +fn vertex_version<'a>( + model: &str, + requested: Option<&'a str>, + drop_params: bool, +) -> Result<&'a str, Error> { + match requested.map(str::trim).unwrap_or_default() { + version if VERSIONS_SERVED_AS_VERTEX_DEFAULT.contains(&version) => Ok(ANTHROPIC_VERSION), + version if VERSIONS_VERTEX_ACCEPTS_IN_BODY.contains(&version) => Ok(version), + _ if drop_params => Ok(ANTHROPIC_VERSION), + version => Err(Error::InvalidRequest( + crate::ErrorDetail::UnsupportedParameter { + model: model.into(), + param: VERSION_HEADER.into(), + value: version.into(), + hint: String::new(), + }, + )), + } +} + #[cfg(test)] mod tests { use rstest::rstest; @@ -471,6 +515,28 @@ mod tests { ); } + #[rstest] + #[case::sdk_spelling("anthropic-version")] + #[case::any_case("Anthropic-Version")] + fn a_forwarded_anthropic_version_never_reaches_vertex(#[case] name: &str) { + let request: MessagesRequest = serde_json::from_value(json!({ + "model": "claude-sonnet-4-5", + "max_tokens": 16, + "messages": [{"role": "user", "content": "hi"}], + })) + .unwrap(); + assert_eq!( + VERTEX_ANTHROPIC_MESSAGES_CONFIG.request_headers( + vec![ + (name.to_string(), "2026-10-21".to_string()), + ("x-request-id".to_string(), "req-1".to_string()), + ], + &request, + ), + vec![("x-request-id".to_string(), "req-1".to_string())] + ); + } + fn transformed(value: Value, context: &MessagesTransformContext) -> Value { let request: MessagesRequest = serde_json::from_value(value).unwrap(); serde_json::to_value( @@ -482,16 +548,87 @@ mod tests { } #[rstest] - fn the_body_carries_the_vertex_version_and_the_wire_drops_the_model() { - let body = transformed( - json!({ - "model": "claude-sonnet-4-5@20250929", - "max_tokens": 16, - "messages": [{"role": "user", "content": "hi"}] - }), - &MessagesTransformContext::default(), + #[case::sdk_2(&[("anthropic-version", "2026-10-21")], false, "2026-10-21")] + #[case::any_case(&[("Anthropic-Version", "2026-10-21")], false, "2026-10-21")] + #[case::vertex_name(&[("anthropic-version", "vertex-2023-10-16")], false, "vertex-2023-10-16")] + #[case::sdk_1(&[("anthropic-version", "2023-06-01")], false, "vertex-2023-10-16")] + #[case::first_version(&[("anthropic-version", "2023-01-01")], false, "vertex-2023-10-16")] + #[case::blank(&[("anthropic-version", " ")], false, "vertex-2023-10-16")] + #[case::no_header(&[], false, "vertex-2023-10-16")] + #[case::unknown_under_drop_params(&[("anthropic-version", "latest")], true, "vertex-2023-10-16")] + fn the_requested_version_goes_in_the_body( + #[case] forwarded: &[(&str, &str)], + #[case] drop_params: bool, + #[case] body_version: &str, + ) { + let request: MessagesRequest = serde_json::from_value(json!({ + "model": "claude-sonnet-4-5", + "max_tokens": 16, + "messages": [{"role": "user", "content": "hi"}], + })) + .unwrap(); + let headers: Headers = forwarded + .iter() + .map(|(name, value)| (name.to_string(), value.to_string())) + .collect(); + let body = serde_json::to_value( + VERTEX_ANTHROPIC_MESSAGES_CONFIG + .request_body(request, &headers, drop_params) + .unwrap(), + ) + .unwrap(); + assert_eq!(body["anthropic_version"], json!(body_version)); + } + + #[rstest] + #[case::unreleased("2025-01-01")] + #[case::alias("latest")] + fn an_unsupported_version_is_refused_without_drop_params(#[case] requested: &str) { + let request: MessagesRequest = serde_json::from_value(json!({ + "model": "claude-sonnet-4-5", + "max_tokens": 16, + "messages": [{"role": "user", "content": "hi"}], + })) + .unwrap(); + let headers: Headers = vec![("anthropic-version".to_string(), requested.to_string())]; + let Error::InvalidRequest(detail) = VERTEX_ANTHROPIC_MESSAGES_CONFIG + .request_body(request, &headers, false) + .unwrap_err() + else { + panic!("expected an invalid request"); + }; + assert_eq!( + detail.to_string(), + format!( + "claude-sonnet-4-5 does not support anthropic-version={requested}. \ + To drop unsupported params, set `litellm.drop_params = True`." + ) ); - assert_eq!(body["anthropic_version"], json!("vertex-2023-10-16")); + } + + #[rstest] + fn the_body_carries_the_vertex_version_and_the_wire_drops_the_model() { + let request: MessagesRequest = serde_json::from_value(json!({ + "model": "claude-sonnet-4-5@20250929", + "max_tokens": 16, + "messages": [{"role": "user", "content": "hi"}] + })) + .unwrap(); + let body = serde_json::to_value( + VERTEX_ANTHROPIC_MESSAGES_CONFIG + .request_body( + VERTEX_ANTHROPIC_MESSAGES_CONFIG + .transform_anthropic_messages_request( + request, + &MessagesTransformContext::default(), + ) + .unwrap(), + &Vec::new(), + false, + ) + .unwrap(), + ) + .unwrap(); assert_eq!(body["model"], json!("claude-sonnet-4-5@20250929")); let wire = VERTEX_ANTHROPIC_MESSAGES_CONFIG.wire_body(body); assert_eq!(wire.get("model"), None); @@ -661,6 +798,18 @@ mod tests { ]}}), &["compact-2026-01-12", "context-management-2025-06-27"] )] + #[case::compaction(json!({"compaction": {"type": "summarize"}}), &["compact-2026-09-04"])] + #[case::thinking_display_updates( + json!({"thinking": {"type": "adaptive", "display": "updates"}}), + &["thinking-display-updates-2026-08-18"] + )] + #[case::tool_addition( + json!({"messages": [ + {"role": "user", "content": "Use the ping tool"}, + {"role": "system", "content": [{"type": "tool_addition", "tool": {"type": "tool_reference", "name": "ping"}}]} + ]}), + &["mid-conversation-tool-changes-2026-07-01"] + )] #[case::advisor_tool_is_not_a_vertex_beta( json!({"tools": [{"type": "advisor_20260301", "name": "advisor"}]}), &[] diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs index 810462c57f8..d760da9073e 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/common_utils.rs @@ -16,7 +16,9 @@ pub(super) fn vertex_config(request: &PreparedOcrRequest) -> Result Result<(), Error> { if connection.api_base.is_some() && connection.api_base_source == InputSource::Request { return Err(litellm_auth::Error::InvalidConfiguration( - "credentials cannot be sent to a request-controlled Vertex AI endpoint".into(), + litellm_auth::ErrorDetail::RequestControlledEndpoint { + provider: "Vertex AI", + }, ) .into()); } diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs index 67f9122ed67..5d680588612 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/deepseek_transformation.rs @@ -235,8 +235,9 @@ pub fn normalize_response( &mut output, PythonJsonFormatter, )) - .map_err(|_| response_field("content"))?; - String::from_utf8(output).map_err(|_| response_field("content"))? + .map_err(|error| Error::response_field(content_path("content"), error))?; + String::from_utf8(output) + .map_err(|error| Error::response_field(content_path("content"), error))? }; (data, fallback) } @@ -261,7 +262,12 @@ pub fn normalize_response( }) }) .collect::, Error>>()?, - Some(_) => return Err(response_field("pages")), + Some(_) => { + return Err(Error::ResponseField { + path: content_path("pages"), + cause: None, + }); + } None => Vec::new(), }; let usage = ocr_data @@ -273,7 +279,12 @@ pub fn normalize_response( .transpose()?; let model = match ocr_data.get("model") { Some(Value::String(model)) => model.clone(), - Some(_) => return Err(response_field("model")), + Some(_) => { + return Err(Error::ResponseField { + path: content_path("model"), + cause: None, + }); + } None => model.to_string(), }; Ok(LiteLLMOcrResponse { @@ -369,10 +380,8 @@ impl serde_json::ser::Formatter for PythonJsonFormatter { } } -fn response_field(field: &str) -> Error { - Error::ResponseField { - path: format!("choices[0].message.content.{field}"), - } +fn content_path(field: &str) -> String { + format!("choices[0].message.content.{field}") } pub fn provider_model(model: &str) -> Result { @@ -380,6 +389,7 @@ pub fn provider_model(model: &str) -> Result { if local_model.is_empty() { return Err(Error::RequestField { path: "model".into(), + cause: None, }); } Ok(format!("{MODEL_PREFIX}{local_model}")) @@ -411,9 +421,7 @@ impl VertexAIDeepSeekOCRConfig { ]) }) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -625,20 +633,18 @@ mod tests { assert_eq!(result["future"], "kept"); } - #[test] - fn response_transform_rejects_missing_empty_and_malformed_content() { - for value in [ - json!({"choices":[]}), - json!({"choices":[{"message":{"content":{}}}]}), - json!({"choices":[{"message":{"content":""}}]}), - json!({"choices":[{"message":{"content":"{\"pages\":[{\"markdown\":42}]}"}}]}), - json!({"choices":[{"message":{"content":{"pages":[{"markdown":42}]}}}]}), - ] { - let result = serde_json::from_value::(value) - .map_err(|_| ()) - .and_then(|response| normalize_response("model", response).map_err(|_| ())); - assert!(result.is_err()); - } + #[rstest::rstest] + #[case::no_choices(json!({"choices":[]}))] + #[case::empty_object(json!({"choices":[{"message":{"content":{}}}]}))] + #[case::empty_string(json!({"choices":[{"message":{"content":""}}]}))] + #[case::malformed_string(json!({"choices":[{"message":{"content":"{\"pages\":[{\"markdown\":42}]}"}}]}))] + #[case::malformed_object(json!({"choices":[{"message":{"content":{"pages":[{"markdown":42}]}}}]}))] + fn response_transform_rejects_missing_empty_and_malformed_content(#[case] value: Value) { + let rejected = match serde_json::from_value::(value) { + Ok(response) => normalize_response("model", response).is_err(), + Err(_) => true, + }; + assert!(rejected); } #[test] diff --git a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs index ab28ee8bff4..b902967d9dc 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/ocr/transformation.rs @@ -137,7 +137,7 @@ impl VertexAiOcrConfig { connection .api_key .as_ref() - .map(litellm_auth::SecretValue::expose), + .map(litellm_secret_value::SecretValue::expose), config, &|name: &str| connection.secret(name), ) @@ -174,9 +174,7 @@ impl VertexAiOcrConfig { ]) }) .map(|url| url.into_string()) - .map_err(|_| Error::RequestField { - path: "api_base".into(), - }) + .map_err(|error| Error::request_field("api_base", error)) } } @@ -198,6 +196,7 @@ fn validate_location(location: &str) -> Result<(), Error> { } Err(Error::RequestField { path: "vertex_location".into(), + cause: None, }) } diff --git a/litellm-rust/crates/model-catalog-refresh/Cargo.toml b/litellm-rust/crates/model-catalog-refresh/Cargo.toml new file mode 100644 index 00000000000..27219042c6c --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "litellm-model-catalog-refresh" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +litellm-http.workspace = true +litellm-model-catalog.workspace = true +thiserror.workspace = true +url.workspace = true + +[dev-dependencies] +litellm-error.workspace = true +litellm-http = { workspace = true, features = ["test-support"] } +reqwest.workspace = true +rstest.workspace = true +tokio.workspace = true diff --git a/litellm-rust/crates/model-catalog-refresh/src/error.rs b/litellm-rust/crates/model-catalog-refresh/src/error.rs new file mode 100644 index 00000000000..656f32370cb --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/src/error.rs @@ -0,0 +1,105 @@ +use std::io; + +use thiserror::Error; + +#[derive(Debug, Error)] +#[error(transparent)] +pub struct Error(Box); + +#[derive(Debug, Error)] +enum Repr { + #[error(transparent)] + Http(#[from] litellm_http::Error), + #[error("could not read the catalog")] + Read(#[source] io::Error), + #[error("invalid catalog URL")] + InvalidUrl(#[source] url::ParseError), + #[error("unsupported catalog URL scheme {scheme}")] + UnsupportedScheme { scheme: String }, +} + +impl Error { + pub fn unsupported_scheme(scheme: String) -> Self { + Self(Box::new(Repr::UnsupportedScheme { scheme })) + } +} + +impl From for Error { + fn from(cause: litellm_http::Error) -> Self { + Self(Box::new(Repr::Http(cause))) + } +} + +impl From for Error { + fn from(cause: io::Error) -> Self { + Self(Box::new(Repr::Read(cause))) + } +} + +impl From for Error { + fn from(cause: url::ParseError) -> Self { + Self(Box::new(Repr::InvalidUrl(cause))) + } +} + +#[cfg(test)] +mod tests { + use litellm_error::{Report, find}; + use rstest::rstest; + + use super::*; + use crate::{DoneReason, Reply, Step}; + + #[rstest] + fn exhausted_transport_keeps_its_typed_cause() { + let reply = Reply::Transport { + error: io::Error::from(io::ErrorKind::ConnectionReset).into(), + }; + let Reply::Transport { error } = reply else { + panic!("transport failure") + }; + let step = Step::Done { + reason: DoneReason::Exhausted(error), + }; + let Step::Done { + reason: DoneReason::Exhausted(error), + } = step + else { + panic!("exhausted attempts") + }; + let cause = find::(&error).expect("typed I/O cause"); + assert_eq!(cause.kind(), io::ErrorKind::ConnectionReset); + assert_eq!( + Report::new(&error) + .to_string() + .matches(&cause.to_string()) + .count(), + 1 + ); + assert!(!error.to_string().contains(&cause.to_string())); + } + + #[rstest] + fn invalid_source_keeps_its_typed_cause() { + let error = Error::from(url::Url::parse("http://[invalid").unwrap_err()); + assert!(find::(&error).is_some()); + } + + #[rstest] + #[tokio::test] + async fn http_failures_keep_the_cause_without_the_url() { + let cause = litellm_http::Client::for_test() + .get("http://localhost:invalid/private?api_key=secret") + .send() + .await + .unwrap_err(); + let error = Error::from(litellm_http::Error::from(cause)); + assert!(find::(&error).is_some()); + let report = Report::new(&error).to_string(); + assert!(!report.contains("secret"), "{report}"); + assert!(!report.contains("private"), "{report}"); + let debug = format!("{error:?}"); + assert!(!debug.contains("secret"), "{debug}"); + assert!(!debug.contains("private"), "{debug}"); + } +} diff --git a/litellm-rust/crates/model-catalog-refresh/src/lib.rs b/litellm-rust/crates/model-catalog-refresh/src/lib.rs new file mode 100644 index 00000000000..006c4887fad --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/src/lib.rs @@ -0,0 +1,9 @@ +mod error; +mod policy; +mod source; +mod step; + +pub use error::Error; +pub use policy::*; +pub use source::*; +pub use step::*; diff --git a/litellm-rust/crates/model-catalog-refresh/src/policy.rs b/litellm-rust/crates/model-catalog-refresh/src/policy.rs new file mode 100644 index 00000000000..bc32e40408e --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/src/policy.rs @@ -0,0 +1,20 @@ +use std::time::Duration; + +use litellm_model_catalog::IntegrityLimits; + +/// Main's defaults, 3 attempts, 30 s max wait, 5 s timeout, backoff base 2.0, jitter 0.0 to 1.0 s +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct RetryPolicy { + pub max_attempts: u32, + pub max_wait: Duration, + pub timeout: Duration, + pub backoff_base_seconds: f64, + pub jitter_min_seconds: f64, + pub jitter_max_seconds: f64, + /// Main's 50 model floor and 0.5 shrink ratio + pub limits: IntegrityLimits, +} + +/// Main's default `LITELLM_MODEL_COST_MAP_URL` +pub const DEFAULT_REGISTRY_URL: &str = + "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"; diff --git a/litellm-rust/crates/model-catalog-refresh/src/source.rs b/litellm-rust/crates/model-catalog-refresh/src/source.rs new file mode 100644 index 00000000000..713e0386941 --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/src/source.rs @@ -0,0 +1,17 @@ +use std::path::PathBuf; + +use url::Url; + +/// Where a fetched catalog comes from +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Source { + Http(Url), + File(PathBuf), + S3(S3Object), +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct S3Object { + pub bucket: String, + pub key: String, +} diff --git a/litellm-rust/crates/model-catalog-refresh/src/step.rs b/litellm-rust/crates/model-catalog-refresh/src/step.rs new file mode 100644 index 00000000000..822d1807916 --- /dev/null +++ b/litellm-rust/crates/model-catalog-refresh/src/step.rs @@ -0,0 +1,66 @@ +use std::time::Duration; + +use litellm_model_catalog::Catalog; + +use crate::Error; +use crate::source::Source; + +/// One attempt. Where to fetch from is the `Source`, so the request carries no URL +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FetchRequest { + pub if_none_match: Option, + pub timeout: Duration, +} + +#[derive(Debug)] +pub enum Reply { + /// An HTTP, S3 or file answer. The body is read only on 200 + Response { + status: u16, + etag: Option, + retry_after: Option, + body: Option>, + }, + /// No answer reached us + Transport { error: Error }, +} + +/// What the fetch state machine asks its driver to do next +#[derive(Debug)] +pub enum Step { + Fetch(FetchRequest), + Sleep(Duration), + /// A parsed document to install, with its rejected row errors + Adopt { + catalog: Box, + installed: Installed, + errors: Vec, + }, + /// 304, a final status, or retries exhausted + Done { + reason: DoneReason, + }, +} + +#[derive(Debug)] +pub enum DoneReason { + NotModified, + Status(u16), + Exhausted(Error), +} + +/// What is installed now, the input to the next revalidation +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Installed { + pub origin: Origin, + pub etag: Option, + /// `git_blob_id` of the installed bytes + pub revision: String, + pub model_count: usize, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Origin { + Bundled, + Remote { source: Source }, +} diff --git a/litellm-rust/crates/model-catalog/AGENTS.md b/litellm-rust/crates/model-catalog/AGENTS.md index 0a380788c19..abc5d198729 100644 --- a/litellm-rust/crates/model-catalog/AGENTS.md +++ b/litellm-rust/crates/model-catalog/AGENTS.md @@ -1,5 +1,65 @@ -## Validation +## Concepts -For `model_prices_and_context_window.json` validation, we should eventually: +Every price in a row lowers to one `Price { meter, channel, when }` and its `UnitPrice`. These keys come from the `FlatKey::parse` tests -- Stop skipping this crate's tests +- `input_cost_per_token` + - meter `Input(Token(Text))`, channel `Online`, tier `Standard`, band `Base` +- `output_cost_per_token_above_272k_tokens_priority` + - meter `Output(Token(Text))`, channel `Online` + - `_priority` sets tier `Priority` + - `_above_272k_tokens` sets band `OverPromptTokens(272000)` +- `cache_creation_input_token_cost_above_1hr_above_200k_tokens` + - meter `CacheWrite { Text, OneHour }`, where `_above_1hr` is the cache TTL, not a band + - `_above_200k_tokens` sets band `OverPromptTokens(200000)` + - channel `Online`, tier `Standard` +- `ocr_cost_per_page_batches` + - meter `Feature(OcrPage)`, tier `Standard`, band `Base` + - `_batches` sets channel `Batch` + +- Meter is what gets counted, such as text input tokens, reasoning tokens, an image of a given size or a web search + - A variation only one meter has stays inside it (cache TTL, image size, resolution, `_above_Ns_interval`, `max_results_range`) +- `UnitPrice` is USD for one unit of the meter, such as one token, image, page or second +- Band is a range of prompt sizes, starting where `_above__tokens` or `_above_k_tokens` says + - `Prices::band` picks one band from prompt tokens alone, the highest edge passed wins, and `Band::Base` is below every edge + - The whole call moves, so every token meter, output included, bills all its tokens at that band's rate + - `Billing::band_edge` says whether a prompt exactly at the edge is in the band (exclusive by default, inclusive for xAI) + - A band only another tier or the batch channel prices never moves the call +- `When` is the request's condition, its service tier and band + - A missing price falls back from `(tier, band)` to `(standard, band)`, `(tier, base)`, then `(standard, base)` +- `Channel` is the route whose price list the call reads, and each falls back differently + - `Online` falls back across `When` as above + - `Batch` (`_batches`) uses bands but no service tier, and a missing price is half the online rate + - `Decisions` (`_decisions`, OpenAI `/v1/decisions`) overrides single online fields, and a missing price is the full online rate +- Off-peak and `Adjustments` are never prices + - Off-peak swaps in other token rates while a UTC window is open + - `Adjustments` multiplies a priced call for region, residency, geo or `speed: fast` +- `Billing` on `Prices` holds row attributes that change how its prices apply (text unit, band edge, cache-read fallback, wall clock, web search unit, tier scope) + - `lower.rs` derives it from the row and its `litellm_provider`, so `litellm-cost` holds no provider list + +## Layout + +`wire/` reads one entry as written. `EntryWire::read` makes one pass over the entry's keys and puts every non-null key in exactly one place, which is an attribute, a flat price parsed by `FlatKey::parse`, a price block, a `supports_*` capability or `unknown`. A known key with a bad value becomes an `Error` that names the key, and the rest of the entry is still read + +`lower.rs` is the only code that knows both the wire types and `litellm-cost`. It lowers an `EntryWire` into `Prices`, `Adjustments` and `Model` and checks the rules that span keys, such as one price per meter and condition, contiguous tiers, no suffix inside a block, and off-peak needing hours and prices. Lowering only goes one way. Nothing turns `Prices` back into JSON, since the raw fields are kept + +`catalog.rs` reads the document, indexes aliases and runs the checks across entries. A bad entry keeps its raw fields and its errors are listed in `Loaded.errors`. Deciding what to do with them is the caller's job. `Catalog::written` gives back every top-level key and value as the document wrote them, in document order. `Catalog::served` gives back every model and alias as Python serves it, without `sample_spec` and `fallback_generalizations` + +`store/` is the catalog's in-memory store and the only place that indexes model names. Its rules are in [src/store/AGENTS.md](src/store/AGENTS.md) + +`layers.rs`, `resolve/`, `model_info.rs` and `register.rs` hold types only. The functions over them land with their first caller + +Catalog errors use one opaque `Error` with typed causes and JSON Pointer accessors (`path`, `key`, `field`). `Error::rejects` alone decides whether an error fails the document, rejects its row or neither, and `ErrorKind` is the stable name hosts report it by + +## Prices + +The price vocabulary lives in `litellm-cost`, and only this crate knows JSON key names + +A new `N`, image size or resolution in an existing key shape parses with no Rust change. A new unit, token kind, feature or tier needs a variant in `litellm-cost` plus its spelling in `wire/flat_key.rs` and `FLAT_KEY_PATTERN` in `schema.rs` + +`Prices::band` and `Prices::rate` are pinned to Python's `_get_token_base_cost` by `tests/price_lookup_reference.json`. After a billing change in Python, regenerate it from the repo root with `PYTHONPATH=. python litellm-rust/crates/model-catalog/tests/generate_price_lookup_reference.py` + +## Tests + +`BUNDLED_REGISTRY` embeds the root `model_prices_and_context_window.json`, so the tests over it fail a JSON change with a bad value or a key no type reads + +`Catalog::problems` is the one definition of a mergeable cost map. The bundled test asserts it is empty, and the cost map guard workflow runs it on a pull request's JSON through `cargo run --bin check-registry ` diff --git a/litellm-rust/crates/model-catalog/Cargo.toml b/litellm-rust/crates/model-catalog/Cargo.toml index 570e68ca4fd..63b9ad14768 100644 --- a/litellm-rust/crates/model-catalog/Cargo.toml +++ b/litellm-rust/crates/model-catalog/Cargo.toml @@ -6,15 +6,20 @@ license.workspace = true repository.workspace = true [features] -schema = ["dep:schemars", "litellm-llms-types/schema"] +schema = ["dep:schemars", "litellm-cost/schema"] [dependencies] -litellm-llms-types.workspace = true +litellm-cost = { workspace = true, features = ["serde"] } +litellm-error.workspace = true indexmap = { version = "2.14.0", features = ["serde"] } +jiff = { version = "0.2.37", default-features = false, features = ["std", "tzdb-bundle-always"] } +macro_rules_attribute.workspace = true schemars = { workspace = true, optional = true } serde.workspace = true -serde_json.workspace = true +serde_json = { workspace = true, features = ["preserve_order"] } +serde_with.workspace = true +strum.workspace = true thiserror.workspace = true time.workspace = true diff --git a/litellm-rust/crates/model-catalog/src/bin/check-registry.rs b/litellm-rust/crates/model-catalog/src/bin/check-registry.rs new file mode 100644 index 00000000000..6fdc7621a0a --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/bin/check-registry.rs @@ -0,0 +1,26 @@ +use std::{env, fs, path::PathBuf, process::ExitCode}; + +use litellm_model_catalog::Catalog; + +fn main() -> ExitCode { + let Some(path) = env::args_os().nth(1).map(PathBuf::from) else { + eprintln!("usage: check-registry "); + return ExitCode::from(2); + }; + let body = match fs::read(&path) { + Ok(body) => body, + Err(error) => { + eprintln!("{}: {error}", path.display()); + return ExitCode::FAILURE; + } + }; + let problems = Catalog::problems(&body); + for problem in &problems { + eprintln!("{}", litellm_error::Report::new(problem)); + } + if problems.is_empty() { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + } +} diff --git a/litellm-rust/crates/model-catalog/src/bundled.rs b/litellm-rust/crates/model-catalog/src/bundled.rs new file mode 100644 index 00000000000..d71a2d57ebb --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/bundled.rs @@ -0,0 +1,3 @@ +/// The registry document this build serves when no other one is loaded. +pub const BUNDLED_REGISTRY: &[u8] = + include_bytes!("../../../../model_prices_and_context_window.json"); diff --git a/litellm-rust/crates/model-catalog/src/capabilities.rs b/litellm-rust/crates/model-catalog/src/capabilities.rs deleted file mode 100644 index 30724766aeb..00000000000 --- a/litellm-rust/crates/model-catalog/src/capabilities.rs +++ /dev/null @@ -1,67 +0,0 @@ -use serde::{Deserialize, Serialize}; - -/// Primary API surface / task type of the model. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum Mode { - AudioSpeech, - AudioTranscription, - Chat, - Completion, - Decisions, - Embedding, - Evaluation, - Guardrail, - ImageEdit, - ImageGeneration, - Moderation, - Ocr, - Realtime, - Rerank, - Responses, - Search, - VectorStore, - VideoGeneration, -} - -/// Gemini audio generation API the model is served through. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum VertexAiAudioApi { - LyriaPredict, - LyriaInteractions, -} - -/// Audio container format the model can return. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum AudioFormat { - Mp3, - Wav, -} - -/// Input modality the model accepts. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum InputModality { - Text, - Image, - Audio, - Video, -} - -/// Output modality the model can produce. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum OutputModality { - Text, - Image, - Audio, - Video, - Code, -} diff --git a/litellm-rust/crates/model-catalog/src/catalog.rs b/litellm-rust/crates/model-catalog/src/catalog.rs index 0b113436a5b..98af785b17e 100644 --- a/litellm-rust/crates/model-catalog/src/catalog.rs +++ b/litellm-rust/crates/model-catalog/src/catalog.rs @@ -1,50 +1,29 @@ -use crate::error::Error; -use crate::fallback::{FallbackGeneralizations, FallbackRule}; -use crate::model_info::ModelInfo; +use std::borrow::Cow; +use std::collections::HashSet; + use indexmap::IndexMap; use serde::Deserialize; -use serde_json::{Map, Value}; -use std::collections::HashMap; +use serde_json::Value; -#[derive(Clone, Debug, Default, PartialEq, Eq)] -pub struct Provenance { - pub source: Option, - pub revision: Option, - pub etag: Option, -} +use crate::error::{Error, Loaded, Rejects, Repr, pointer_child, pointer_for_key}; +use crate::fallback::{FallbackGeneralizations, FallbackRule}; +use crate::model::Model; +use crate::store::{Rows, Store}; +use crate::wire::{EntryWire, Fields}; + +const SAMPLE_SPEC: &str = "sample_spec"; +const FALLBACK_GENERALIZATIONS: &str = "fallback_generalizations"; #[derive(Clone, Copy, Debug, PartialEq)] pub struct IntegrityLimits { - pub reference_model_count: usize, - pub min_model_count: usize, - pub min_reference_ratio: f64, -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum AliasIssue { - InvalidList { model: String }, - InvalidName { model: String }, - CanonicalCollision { model: String, alias: String }, - AliasCollision { model: String, alias: String }, + min_model_count: usize, + min_reference_ratio: f64, } #[derive(Clone, Debug)] pub struct ModelEntry { - fields: Map, - info: ModelInfo, -} - -impl ModelEntry { - pub fn field(&self, name: &str) -> Option<&Value> { - self.fields.get(name) - } - pub fn fields(&self) -> &Map { - &self.fields - } - /// The entry deserialized into the typed mirror of the catalog schema. - pub fn info(&self) -> &ModelInfo { - &self.info - } + fields: Fields, + model: Option, } #[derive(Clone, Copy, Debug)] @@ -54,142 +33,342 @@ pub struct ModelMatch<'a> { pub entry: &'a ModelEntry, } -#[derive(Debug)] +#[derive(Clone, Debug)] +struct Section { + index: usize, + key: String, + value: Value, +} + +/// One fetched key kept off the served map because its row failed validation +#[derive(Clone, Debug)] +pub struct Rejection<'a> { + pub key: String, + pub errors: Vec<&'a Error>, + /// The bundled row for the key is served instead. `false` means the key is left out + pub replaced_by_bundled: bool, +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +#[serde(untagged)] +pub enum Written<'a> { + Entry(&'a Fields), + Section(&'a Value), +} + +#[derive(Clone, Debug)] pub struct Catalog { + store: Store, + sections: Vec
, + fallback_generalizations: Option, +} + +/// Models and aliases, an alias served as its owner's row +#[derive(Clone, Debug)] +struct CatalogRows { entries: IndexMap, aliases: IndexMap, - lowercase_keys: HashMap, - sample_spec: Option, - fallback_generalizations: Option, - provenance: Provenance, - alias_issues: Vec, +} + +impl CatalogRows { + fn owner<'a>(&'a self, key: &'a str) -> &'a str { + self.aliases.get(key).map_or(key, String::as_str) + } + + fn fields(&self, key: &str) -> Option<&Fields> { + Some(self.entries.get(self.owner(key))?.fields()) + } +} + +/// Model names match ignoring case, and keys are grouped by their raw `litellm_provider` +impl Rows for CatalogRows { + type Row = Fields; + type Stamp = (); + + fn fold(name: &str) -> String { + name.to_lowercase() + } + + fn stamp(&self) -> Self::Stamp {} + + fn contains(&self, key: &str) -> bool { + self.entries.contains_key(key) || self.aliases.contains_key(key) + } + + fn row(&self, key: &str) -> Option> { + self.fields(key).map(Cow::Borrowed) + } + + fn scan(&self, visit: &mut dyn FnMut(&str, Option<&str>)) { + let keys = self.entries.keys().chain(self.aliases.keys()); + for key in keys { + let provider = self + .fields(key) + .and_then(|fields| fields.get("litellm_provider")?.as_str()); + visit(key, provider); + } + } +} + +impl IntegrityLimits { + pub fn new(min_model_count: usize, min_reference_ratio: f64) -> Option { + (0.0..=1.0).contains(&min_reference_ratio).then_some(Self { + min_model_count, + min_reference_ratio, + }) + } +} + +impl ModelEntry { + pub fn field(&self, name: &str) -> Option<&Value> { + self.fields.get(name) + } + + pub fn fields(&self) -> &Fields { + &self.fields + } + + pub fn model(&self) -> Option<&Model> { + self.model.as_ref() + } } impl Catalog { - pub fn parse(body: &[u8], provenance: Provenance) -> Result { - let root: IndexMap = serde_json::from_slice(body)?; - if root.is_empty() { - return Err(Error::Empty); - } + pub fn parse(body: &[u8]) -> Result, Error> { + let document = serde_json::from_slice::(body) + .map_err(|error| Error::new("", Repr::Json(error)))?; - let mut entries = IndexMap::with_capacity(root.len()); + let mut problems = Vec::new(); + let mut sections = Vec::new(); + let mut entries = IndexMap::with_capacity(document.len()); let mut alias_lists = Vec::new(); - let mut alias_issues = Vec::new(); - let mut sample_spec = None; - let mut fallback_generalizations = None; - for (name, value) in root { - match name.as_str() { - "sample_spec" => { - sample_spec = Some(value); + for (index, (name, value)) in document.into_iter().enumerate() { + let fields = match value { + Value::Object(fields) if !is_section(&name) => fields, + value => { + if !is_section(&name) { + problems.push(Error::from_key(&name, Repr::NotAnObject)); + } + sections.push(Section { + index, + key: name, + value, + }); continue; } - "fallback_generalizations" => { - fallback_generalizations = - Some(serde_json::from_value::(value)?); - continue; - } - _ => {} - } - let Value::Object(mut fields) = value else { - return Err(Error::EntryNotObject { model: name }); }; - let info = ModelInfo::deserialize(&fields)?; - if let Some(aliases) = fields.remove("aliases") { - match aliases { - Value::Array(names) => alias_lists.push((name.clone(), names)), - _ => alias_issues.push(AliasIssue::InvalidList { - model: name.clone(), - }), - } + let (model, aliases, row_problems) = read_row(&fields); + problems.extend( + row_problems + .into_iter() + .map(|problem| problem.prepend(&name)), + ); + if let Some(aliases) = aliases { + alias_lists.push((name.clone(), aliases)); } - entries.insert(name, ModelEntry { fields, info }); + entries.insert(name, ModelEntry { fields, model }); + } + if entries.is_empty() { + return Err(Error::new("", Repr::Empty)); } - let mut aliases = IndexMap::new(); + let mut aliases: IndexMap = IndexMap::new(); for (model, names) in alias_lists { - for name in names { - let Value::String(alias) = name else { - alias_issues.push(AliasIssue::InvalidName { - model: model.clone(), - }); - continue; - }; + for alias in names { if entries.contains_key(&alias) { - alias_issues.push(AliasIssue::CanonicalCollision { - model: model.clone(), - alias, - }); - } else if aliases.contains_key(&alias) { - alias_issues.push(AliasIssue::AliasCollision { - model: model.clone(), - alias, - }); + problems.push(Error::from_key(&model, Repr::AliasShadowsModel { alias })); + } else if let Some(taken_by) = aliases.get(&alias) { + problems.push(Error::from_key( + &model, + Repr::AliasTaken { + alias, + taken_by: taken_by.clone(), + }, + )); } else { aliases.insert(alias, model.clone()); } } } - let lowercase_keys = entries - .keys() - .chain(aliases.keys()) - .map(|key| (key.to_lowercase(), key.clone())) - .collect(); - Ok(Self { - entries, - aliases, - lowercase_keys, - sample_spec, - fallback_generalizations, - provenance, - alias_issues, + let fallback_generalizations = sections + .iter() + .find(|section| section.key == FALLBACK_GENERALIZATIONS) + .and_then(|section| { + FallbackGeneralizations::deserialize(§ion.value) + .map_err(|error| { + problems.push(Error::from_key(§ion.key, Repr::Invalid(error))) + }) + .ok() + }); + Ok(Loaded { + value: Self { + store: Store::new(CatalogRows { entries, aliases }), + sections, + fallback_generalizations, + }, + errors: problems, }) } - pub fn validate(&self, limits: IntegrityLimits) -> Result<(), Error> { - if !limits.min_reference_ratio.is_finite() - || !(0.0..=1.0).contains(&limits.min_reference_ratio) + pub fn problems(body: &[u8]) -> Vec { + let Loaded { + value: catalog, + errors, + } = match Self::parse(body) { + Ok(loaded) => loaded, + Err(fatal) => return vec![fatal], + }; + let unread = catalog.rows().entries.iter().flat_map(|(model, entry)| { + EntryWire::read(entry.fields()) + .unknown + .into_iter() + .map(move |(key, _)| { + Error::new( + pointer_child(&pointer_for_key(model), &key), + Repr::UnreadField, + ) + }) + }); + let no_fallback_rules = catalog + .fallback_rules() + .is_none_or(<[_]>::is_empty) + .then(|| Error::from_key(FALLBACK_GENERALIZATIONS, Repr::NoFallbackRules)); + errors + .into_iter() + .chain(unread) + .chain(no_fallback_rules) + .collect() + } + + pub fn row_errors(model: &str, fields: &Fields) -> Vec { + let (_, _, errors) = read_row(fields); + errors + .into_iter() + .map(|error| error.prepend(model)) + .collect() + } + + pub fn written(&self) -> impl Iterator)> { + let mut sections = self.sections.iter().peekable(); + let mut entries = self.rows().entries.iter(); + (0..self.rows().entries.len() + self.sections.len()).map_while(move |index| match sections + .next_if(|section| section.index == index) { - return Err(Error::InvalidRatio); - } - let actual = self.entries.len(); + Some(section) => Some((section.key.as_str(), Written::Section(§ion.value))), + None => entries + .next() + .map(|(name, entry)| (name.as_str(), Written::Entry(&entry.fields))), + }) + } + + pub fn served(&self) -> impl Iterator { + let rows = self.rows(); + rows.entries + .iter() + .map(|(name, entry)| (name.as_str(), entry.fields())) + .chain( + rows.aliases + .iter() + .map(|(alias, owner)| (alias.as_str(), rows.entries[owner].fields())), + ) + } + + pub fn check_size(&self, reference: usize, limits: IntegrityLimits) -> Result<(), Error> { + let actual = self.model_count(); if actual < limits.min_model_count { - return Err(Error::BelowMinimum { - actual, - minimum: limits.min_model_count, - }); + return Err(Error::new( + "", + Repr::BelowMinimum { + actual, + minimum: limits.min_model_count, + }, + )); } - if limits.reference_model_count > 0 - && (actual as f64) < (limits.reference_model_count as f64) * limits.min_reference_ratio - { - return Err(Error::Shrunk { - actual, - reference: limits.reference_model_count, - ratio: limits.min_reference_ratio, - }); + if (actual as f64) < (reference as f64) * limits.min_reference_ratio { + return Err(Error::new( + "", + Repr::Shrunk { + actual, + reference, + ratio: limits.min_reference_ratio, + }, + )); } Ok(()) } - pub fn lookup(&self, key: &str) -> Option> { - let matched_key = if self.entries.contains_key(key) || self.aliases.contains_key(key) { - key - } else { - self.lowercase_keys.get(&key.to_lowercase())?.as_str() - }; - let canonical_key = self - .aliases - .get(matched_key) - .map(String::as_str) - .unwrap_or(matched_key); - let (canonical_key, entry) = self.entries.get_key_value(canonical_key)?; - let matched_key = self + pub fn without_rejected(self, errors: &[Error]) -> (Self, Vec<&Error>) { + let row_names: HashSet = self + .rows() .entries - .get_key_value(matched_key) + .keys() + .cloned() + .chain( + self.sections + .iter() + .filter(|section| !is_section(§ion.key)) + .map(|section| section.key.clone()), + ) + .collect(); + let rejected: HashSet = errors + .iter() + .filter(|error| error.rejects() == Rejects::Row) + .filter_map(|error| crate::error::pointer_root(error.path())) + .filter(|root| row_names.contains(root)) + .collect(); + let removed_errors = errors + .iter() + .filter(|error| { + crate::error::pointer_root(error.path()) + .is_some_and(|root| rejected.contains(&root)) + }) + .collect(); + let removed_positions: HashSet = self + .written() + .enumerate() + .filter(|(_, (key, _))| rejected.contains(*key)) + .map(|(index, _)| index) + .collect(); + let mut catalog = self; + catalog.store.edit(|rows| { + rows.entries.retain(|name, _| !rejected.contains(name)); + rows.aliases.retain(|_, owner| !rejected.contains(owner)); + }); + catalog + .sections + .retain(|section| is_section(§ion.key) || !rejected.contains(§ion.key)); + for section in &mut catalog.sections { + let removed_before = removed_positions + .iter() + .filter(|index| **index < section.index) + .count(); + section.index -= removed_before; + } + (catalog, removed_errors) + } + + pub fn unrecognized(&self) -> impl Iterator { + self.rows().entries.iter().flat_map(|(name, entry)| { + entry + .model + .iter() + .flat_map(|model| &model.unrecognized) + .map(move |key| (name.as_str(), key.as_str())) + }) + } + + pub fn lookup(&self, key: &str) -> Option> { + let rows = self.rows(); + let found = self.store.find_key(key)?; + let (canonical_key, entry) = rows.entries.get_key_value(rows.owner(&found))?; + let matched_key = rows + .entries + .get_key_value(found.as_str()) .map(|(key, _)| key.as_str()) .or_else(|| { - self.aliases - .get_key_value(matched_key) + rows.aliases + .get_key_value(found.as_str()) .map(|(key, _)| key.as_str()) })?; Some(ModelMatch { @@ -200,30 +379,140 @@ impl Catalog { } pub fn model_count(&self) -> usize { - self.entries.len() + self.rows().entries.len() } + pub fn model_names(&self) -> impl Iterator { - self.entries.keys().map(String::as_str) - } - pub fn alias_count(&self) -> usize { - self.aliases.len() + self.rows().entries.keys().map(String::as_str) } + pub fn aliases(&self) -> &IndexMap { - &self.aliases + &self.rows().aliases } - pub fn alias_issues(&self) -> &[AliasIssue] { - &self.alias_issues + + fn rows(&self) -> &CatalogRows { + self.store.rows() } + pub fn sample_spec(&self) -> Option<&Value> { - self.sample_spec.as_ref() - } - pub fn fallback_generalizations(&self) -> Option<&FallbackGeneralizations> { - self.fallback_generalizations.as_ref() + self.sections + .iter() + .find(|section| section.key == SAMPLE_SPEC) + .map(|section| §ion.value) } + pub fn fallback_rules(&self) -> Option<&[FallbackRule]> { Some(self.fallback_generalizations.as_ref()?.rules.as_slice()) } - pub fn provenance(&self) -> &Provenance { - &self.provenance +} + +fn read_row(fields: &Fields) -> (Option, Option>, Vec) { + let wire = EntryWire::read(fields); + let aliases = wire.attributes.aliases.clone(); + let timezone_error = unknown_timezone(&wire); + match Model::try_from(wire) { + Ok(model) => (Some(model), aliases, timezone_error.into_iter().collect()), + Err(mut errors) => { + errors.extend(timezone_error); + (None, aliases, errors) + } + } +} + +fn is_section(key: &str) -> bool { + key == SAMPLE_SPEC || key == FALLBACK_GENERALIZATIONS +} + +fn unknown_timezone(wire: &EntryWire) -> Option { + let zone = wire.off_peak.as_ref()?.weekday_timezone.as_deref()?.trim(); + (!zone.is_empty() && jiff::tz::TimeZone::get(zone).is_err()).then(|| { + Error::from_path( + "off_peak_pricing.weekday_timezone", + Repr::UnknownTimezone { + zone: zone.to_owned(), + }, + ) + }) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + const RULES: &str = r#""fallback_generalizations":{"rules":[{"name":"family","pattern":"^new-","model_info":{"mode":"chat"}}]}"#; + + #[rstest] + #[case::clean(format!(r#"{{{RULES},"m":{{"litellm_provider":"test"}}}}"#), vec![])] + #[case::not_json("{".to_owned(), vec![("", "json")])] + #[case::row_error(format!(r#"{{{RULES},"m":{{}}}}"#), vec![("/m/litellm_provider", "missing_provider")])] + #[case::unread_key( + format!(r#"{{{RULES},"m":{{"litellm_provider":"test","typo_cost":1}}}}"#), + vec![("/m/typo_cost", "unread_field")] + )] + #[case::empty_model_key( + format!(r#"{{{RULES},"":{{"litellm_provider":"test","future_field":true}}}}"#), + vec![("//future_field", "unread_field")] + )] + #[case::unread_key_on_invalid_row( + format!(r#"{{{RULES},"m":{{"typo_cost":1}}}}"#), + vec![("/m/litellm_provider", "missing_provider"), ("/m/typo_cost", "unread_field")] + )] + #[case::escaped_unread_key_on_invalid_row( + format!(r#"{{{RULES},"open/router~v1":{{"typo/cost~":1}}}}"#), + vec![("/open~1router~0v1/litellm_provider", "missing_provider"), ("/open~1router~0v1/typo~1cost~0", "unread_field")] + )] + #[case::no_rules( + r#"{"m":{"litellm_provider":"test"}}"#.to_owned(), + vec![("/fallback_generalizations", "no_fallback_rules")] + )] + #[case::empty_rules( + r#"{"fallback_generalizations":{"rules":[]},"m":{"litellm_provider":"test"}}"#.to_owned(), + vec![("/fallback_generalizations", "no_fallback_rules")] + )] + fn problems_list_what_a_cost_map_change_must_fix( + #[case] body: String, + #[case] expected: Vec<(&str, &str)>, + ) { + let problems: Vec<_> = Catalog::problems(body.as_bytes()) + .iter() + .map(|problem| (problem.path().to_owned(), <&str>::from(problem.kind()))) + .collect(); + let expected: Vec<_> = expected + .into_iter() + .map(|(path, kind)| (path.to_owned(), kind)) + .collect(); + assert_eq!(problems, expected); + } + + #[rstest] + fn runtime_parsing_preserves_custom_fields_and_recovers_valid_rows() { + let body = br#"{ + "good":{"litellm_provider":"test","custom_metadata":true}, + "bad":{"litellm_provider":"test","input_cost_per_token":"wrong"} + }"#; + let loaded = Catalog::parse(body).unwrap(); + assert_eq!(loaded.errors.len(), 1); + assert_eq!(loaded.errors[0].path(), "/bad/input_cost_per_token"); + let (catalog, rejected) = loaded.value.without_rejected(&loaded.errors); + assert_eq!(rejected.len(), 1); + assert!(catalog.lookup("bad").is_none()); + let good = catalog.lookup("good").unwrap(); + assert!(good.entry.model().is_some()); + assert_eq!( + good.entry.field("custom_metadata"), + Some(&Value::Bool(true)) + ); + assert!( + Catalog::problems(body) + .iter() + .any(|problem| problem.kind() == crate::ErrorKind::UnreadField) + ); + assert!( + Catalog::problems(body) + .iter() + .any(|problem| problem.kind() == crate::ErrorKind::NoFallbackRules) + ); } } diff --git a/litellm-rust/crates/model-catalog/src/error.rs b/litellm-rust/crates/model-catalog/src/error.rs index edb6ba1eb11..89e27c05f46 100644 --- a/litellm-rust/crates/model-catalog/src/error.rs +++ b/litellm-rust/crates/model-catalog/src/error.rs @@ -1,27 +1,400 @@ +use std::num::ParseIntError; + +use strum::{EnumDiscriminants, IntoStaticStr, VariantArray}; use thiserror::Error; #[derive(Debug, Error)] -pub enum Error { - /// The body is not valid JSON, or a model entry fails typed deserialization. - #[error("invalid JSON: {0}")] - Json(#[from] serde_json::Error), - /// The catalog has no entries at all. - #[error("catalog is empty")] +#[error(transparent)] +pub struct Error(Box); + +#[derive(Debug, Error)] +enum Context { + #[error(transparent)] + Bare(Repr), + #[error("{}", display_path(path))] + At { + path: String, + #[source] + cause: Repr, + }, +} + +#[derive(Debug, Error)] +pub(crate) enum DateCause { + #[error("invalid date component")] + Number(#[from] ParseIntError), + #[error("calendar component out of range")] + Range(#[from] time::error::ComponentRange), +} + +#[derive(Debug, Error, EnumDiscriminants)] +#[strum_discriminants(name(ErrorKind), derive(IntoStaticStr, VariantArray), vis(pub))] +pub(crate) enum Repr { + #[error("invalid JSON")] + #[strum_discriminants(strum(serialize = "json"))] + Json(#[source] serde_json::Error), + #[error("has no models")] + #[strum_discriminants(strum(serialize = "empty"))] Empty, - /// A non-reserved top level value is not a JSON object. - #[error("model {model:?} must be an object")] - EntryNotObject { model: String }, - /// Canonical entry count is under the configured minimum. - #[error("catalog has {actual} models, below minimum {minimum}")] + #[error("has {actual} models, below minimum {minimum}")] + #[strum_discriminants(strum(serialize = "below_minimum"))] BelowMinimum { actual: usize, minimum: usize }, - /// Canonical entry count is under the configured reference ratio. - #[error("catalog has {actual} models, below {ratio} of reference count {reference}")] + #[error("has {actual} models, below {ratio} of reference count {reference}")] + #[strum_discriminants(strum(serialize = "shrunk"))] Shrunk { actual: usize, reference: usize, ratio: f64, }, - /// The configured minimum reference ratio is not finite or outside `[0, 1]`. - #[error("minimum reference ratio must be finite and between zero and one")] - InvalidRatio, + #[error("must be a JSON object")] + #[strum_discriminants(strum(serialize = "not_an_object"))] + NotAnObject, + #[error("invalid value")] + #[strum_discriminants(strum(serialize = "invalid"))] + Invalid(#[source] serde_json::Error), + #[error("not a field this block reads")] + #[strum_discriminants(strum(serialize = "unknown_block_key"))] + UnknownBlockKey, + #[error("required")] + #[strum_discriminants(strum(serialize = "missing_provider"))] + MissingProvider, + #[error("{zone:?} is not an IANA time zone")] + #[strum_discriminants(strum(serialize = "unknown_timezone"))] + UnknownTimezone { zone: String }, + #[error("alias {alias:?} is already a model name")] + #[strum_discriminants(strum(serialize = "alias_shadows_model"))] + AliasShadowsModel { alias: String }, + #[error("alias {alias:?} already belongs to {taken_by}")] + #[strum_discriminants(strum(serialize = "alias_taken"))] + AliasTaken { alias: String, taken_by: String }, + #[error("{first} and {second} price the same meter under the same conditions")] + #[strum_discriminants(strum(serialize = "duplicate_price"))] + DuplicatePrice { first: String, second: String }, + #[error("a price inside {block} takes no band or tier suffix")] + #[strum_discriminants(strum(serialize = "suffix_inside_block"))] + SuffixInsideBlock { block: &'static str }, + #[error("a tier needs exactly one of range and max_results_range")] + #[strum_discriminants(strum(serialize = "tier_range"))] + TierRange, + #[error("the range ends below its start")] + #[strum_discriminants(strum(serialize = "tier_reversed"))] + TierReversed, + #[error("starts at {start}, but the tiers before it end at {expected}")] + #[strum_discriminants(strum(serialize = "tier_gap"))] + TierGap { start: u64, expected: u64 }, + #[error("tiered_pricing mixes token ranges and result ranges")] + #[strum_discriminants(strum(serialize = "mixed_tier_basis"))] + MixedTierBasis, + #[error("a tier by result count only prices input_cost_per_query")] + #[strum_discriminants(strum(serialize = "not_priced_by_results"))] + NotPricedByResults, + #[error("tiered_pricing and prompt token bands are set in two places")] + #[strum_discriminants(strum(serialize = "bands_from_two_sources"))] + BandsFromTwoSources, + #[error("has no off-peak rate")] + #[strum_discriminants(strum(serialize = "not_an_off_peak_rate"))] + NotAnOffPeakRate, + #[error("off_peak_pricing has no prices")] + #[strum_discriminants(strum(serialize = "no_off_peak_prices"))] + NoOffPeakPrices, + #[error("off_peak_pricing needs hours_utc or windows")] + #[strum_discriminants(strum(serialize = "no_off_peak_hours"))] + NoOffPeakHours, + #[error("must list at least one entry")] + #[strum_discriminants(strum(serialize = "empty_list"))] + EmptyList, + #[error("expected a calendar date YYYY-MM-DD")] + #[strum_discriminants(strum(serialize = "invalid_date"))] + InvalidDate { + #[source] + cause: Option, + }, + #[error("not a field litellm reads")] + #[strum_discriminants(strum(serialize = "unread_field"))] + UnreadField, + #[error("has no rules")] + #[strum_discriminants(strum(serialize = "no_fallback_rules"))] + NoFallbackRules, +} + +/// What a problem takes out of the served map +#[derive(Clone, Copy, Debug, PartialEq, Eq, VariantArray)] +pub enum Rejects { + Document, + Row, + Nothing, +} + +impl Error { + pub fn kind(&self) -> ErrorKind { + ErrorKind::from(self.repr()) + } + + pub fn rejects(&self) -> Rejects { + match self.repr() { + Repr::Json(_) | Repr::Empty | Repr::BelowMinimum { .. } | Repr::Shrunk { .. } => { + Rejects::Document + } + Repr::NotAnObject + | Repr::Invalid(_) + | Repr::UnknownBlockKey + | Repr::UnknownTimezone { .. } + | Repr::AliasShadowsModel { .. } + | Repr::AliasTaken { .. } + | Repr::DuplicatePrice { .. } + | Repr::SuffixInsideBlock { .. } + | Repr::TierRange + | Repr::TierReversed + | Repr::TierGap { .. } + | Repr::MixedTierBasis + | Repr::NotPricedByResults + | Repr::BandsFromTwoSources + | Repr::NotAnOffPeakRate + | Repr::NoOffPeakPrices + | Repr::NoOffPeakHours + | Repr::EmptyList + | Repr::InvalidDate { .. } => Rejects::Row, + Repr::MissingProvider | Repr::UnreadField | Repr::NoFallbackRules => Rejects::Nothing, + } + } +} + +#[derive(Debug)] +pub struct Loaded { + pub value: T, + pub errors: Vec, +} + +impl From for Error { + fn from(cause: Repr) -> Self { + Self(Box::new(Context::Bare(cause))) + } +} + +impl From for Error { + fn from(cause: DateCause) -> Self { + Repr::InvalidDate { cause: Some(cause) }.into() + } +} + +impl Error { + fn repr(&self) -> &Repr { + match &*self.0 { + Context::Bare(cause) | Context::At { cause, .. } => cause, + } + } + + pub(crate) fn new(path: impl Into, cause: Repr) -> Self { + Self(Box::new(Context::At { + path: path.into(), + cause, + })) + } + + pub(crate) fn from_key(key: &str, cause: Repr) -> Self { + Self::new(pointer_for_key(key), cause) + } + + pub(crate) fn from_path(path: &str, cause: Repr) -> Self { + Self::new(path_to_pointer(path), cause) + } + + pub fn path(&self) -> &str { + match &*self.0 { + Context::Bare(_) => "", + Context::At { path, .. } => path, + } + } + + pub fn key(&self) -> Option { + pointer_root(self.path()) + } + + pub fn field(&self) -> Option { + let segments: Vec<_> = self + .path() + .split('/') + .skip(2) + .map(unescape_pointer_segment) + .collect(); + (!segments.is_empty()).then(|| segments.join("/")) + } + + pub(crate) fn prepend(self, key: &str) -> Self { + let (path, cause) = match *self.0 { + Context::Bare(cause) => (String::new(), cause), + Context::At { path, cause } => (path, cause), + }; + Self::new(format!("/{}{path}", escape_pointer_segment(key)), cause) + } +} + +pub(crate) fn pointer_root(path: &str) -> Option { + path.strip_prefix('/') + .and_then(|path| path.split('/').next()) + .map(unescape_pointer_segment) +} + +fn escape_pointer_segment(segment: &str) -> String { + segment.replace('~', "~0").replace('/', "~1") +} + +pub(crate) fn pointer_for_key(key: &str) -> String { + format!("/{}", escape_pointer_segment(key)) +} + +pub(crate) fn pointer_child(parent: &str, key: &str) -> String { + format!("{parent}/{}", escape_pointer_segment(key)) +} + +fn unescape_pointer_segment(segment: &str) -> String { + segment.replace("~1", "/").replace("~0", "~") +} + +fn path_to_pointer(key: &str) -> String { + let mut pointer = String::new(); + let mut segment = String::new(); + let mut chars = key.chars(); + let mut push_segment = |segment: &mut String| { + if !segment.is_empty() { + pointer.push('/'); + pointer.push_str(&escape_pointer_segment(segment)); + segment.clear(); + } + }; + while let Some(character) = chars.next() { + match character { + '.' => push_segment(&mut segment), + '[' => { + push_segment(&mut segment); + let mut index = String::new(); + for character in chars.by_ref() { + if character == ']' { + break; + } + index.push(character); + } + push_segment(&mut index); + } + character => segment.push(character), + } + } + push_segment(&mut segment); + pointer +} + +fn display_path(path: &str) -> String { + if path.is_empty() { + return "catalog".to_owned(); + } + path.split('/') + .skip(1) + .map(unescape_pointer_segment) + .enumerate() + .fold(String::new(), |mut display, (index, segment)| { + if index == 0 { + display.push_str(&segment); + } else if segment.bytes().all(|byte| byte.is_ascii_digit()) { + display.push('['); + display.push_str(&segment); + display.push(']'); + } else { + display.push('.'); + display.push_str(&segment); + } + display + }) +} + +#[cfg(test)] +mod tests { + use litellm_error::{Report, find}; + use rstest::rstest; + use serde_json::Value; + + use super::*; + use crate::{Catalog, wire::IsoDate}; + + #[rstest] + #[case::json(Repr::Json(serde_json::from_str::("{").unwrap_err()), Rejects::Document)] + #[case::empty(Repr::Empty, Rejects::Document)] + #[case::below_minimum(Repr::BelowMinimum { actual: 1, minimum: 2 }, Rejects::Document)] + #[case::shrunk(Repr::Shrunk { actual: 1, reference: 3, ratio: 0.5 }, Rejects::Document)] + #[case::not_an_object(Repr::NotAnObject, Rejects::Row)] + #[case::invalid(Repr::Invalid(serde_json::from_str::("x").unwrap_err()), Rejects::Row)] + #[case::unknown_block_key(Repr::UnknownBlockKey, Rejects::Row)] + #[case::missing_provider(Repr::MissingProvider, Rejects::Nothing)] + #[case::unknown_timezone(Repr::UnknownTimezone { zone: "invalid".into() }, Rejects::Row)] + #[case::alias_shadows_model(Repr::AliasShadowsModel { alias: "model".into() }, Rejects::Row)] + #[case::alias_taken(Repr::AliasTaken { alias: "alias".into(), taken_by: "owner".into() }, Rejects::Row)] + #[case::duplicate_price(Repr::DuplicatePrice { first: "a".into(), second: "b".into() }, Rejects::Row)] + #[case::suffix_inside_block(Repr::SuffixInsideBlock { block: "tiered_pricing" }, Rejects::Row)] + #[case::tier_range(Repr::TierRange, Rejects::Row)] + #[case::tier_reversed(Repr::TierReversed, Rejects::Row)] + #[case::tier_gap(Repr::TierGap { start: 1, expected: 0 }, Rejects::Row)] + #[case::mixed_tier_basis(Repr::MixedTierBasis, Rejects::Row)] + #[case::not_priced_by_results(Repr::NotPricedByResults, Rejects::Row)] + #[case::bands_from_two_sources(Repr::BandsFromTwoSources, Rejects::Row)] + #[case::not_an_off_peak_rate(Repr::NotAnOffPeakRate, Rejects::Row)] + #[case::no_off_peak_prices(Repr::NoOffPeakPrices, Rejects::Row)] + #[case::no_off_peak_hours(Repr::NoOffPeakHours, Rejects::Row)] + #[case::empty_list(Repr::EmptyList, Rejects::Row)] + #[case::invalid_date(Repr::InvalidDate { cause: None }, Rejects::Row)] + #[case::unread_field(Repr::UnreadField, Rejects::Nothing)] + #[case::no_fallback_rules(Repr::NoFallbackRules, Rejects::Nothing)] + fn policy_classifies_each_error(#[case] error: Repr, #[case] expected: Rejects) { + assert_eq!(Error::from(error).rejects(), expected); + } + + #[rstest] + #[case::root("", "x", "/x")] + #[case::empty_model("/", "x", "//x")] + #[case::empty_parent_segment("/m/", "x", "/m//x")] + #[case::escaped("/m~1n", "a~/b", "/m~1n/a~0~1b")] + fn pointers_preserve_segments(#[case] parent: &str, #[case] key: &str, #[case] expected: &str) { + assert_eq!(pointer_child(parent, key), expected); + } + + #[rstest] + #[case::malformed(br#"{"#)] + #[case::bad_price(br#"{"m":{"litellm_provider":"test","input_cost_per_token":"free"}}"#)] + fn json_causes_are_typed_and_rendered_once(#[case] body: &[u8]) { + let error = match Catalog::parse(body) { + Ok(loaded) => loaded.errors.into_iter().next().expect("invalid price"), + Err(error) => error, + }; + let cause = find::(&error) + .expect("typed JSON cause") + .to_string(); + let report = Report::new(&error).to_string(); + assert_eq!(report.matches(&cause).count(), 1, "{report}"); + assert!(!error.to_string().contains(&cause)); + } + + #[rstest] + #[case::year("xxxx-01-01")] + #[case::month("2026-xx-01")] + #[case::day("2026-01-xx")] + fn date_number_errors_keep_the_cause(#[case] value: &str) { + let error = value.parse::().unwrap_err(); + let cause = find::(&error) + .expect("typed integer cause") + .to_string(); + assert_eq!(error.rejects(), Rejects::Row); + assert_eq!(Report::new(&error).to_string().matches(&cause).count(), 1); + assert!(!error.to_string().contains(&cause)); + } + + #[rstest] + #[case::month("2026-13-01")] + #[case::day("2026-02-30")] + fn date_range_errors_keep_the_cause(#[case] value: &str) { + let error = value.parse::().unwrap_err(); + let cause = find::(&error) + .expect("typed calendar cause") + .to_string(); + assert_eq!(error.rejects(), Rejects::Row); + assert_eq!(Report::new(&error).to_string().matches(&cause).count(), 1); + assert!(!error.to_string().contains(&cause)); + } } diff --git a/litellm-rust/crates/model-catalog/src/fallback.rs b/litellm-rust/crates/model-catalog/src/fallback.rs index 62291a84929..f5a0cadb6b9 100644 --- a/litellm-rust/crates/model-catalog/src/fallback.rs +++ b/litellm-rust/crates/model-catalog/src/fallback.rs @@ -1,10 +1,9 @@ -use serde::{Deserialize, Serialize}; use serde_json::Value; use std::collections::BTreeMap; /// One regex rule generalizing unknown model ids to known families. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Eq, Hash)] pub struct FallbackRule { pub name: String, pub pattern: String, @@ -15,8 +14,8 @@ pub struct FallbackRule { } /// Regex rules that generalize unknown model ids to known families; not a model entry. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Eq, Hash)] #[serde(deny_unknown_fields)] pub struct FallbackGeneralizations { pub rules: Vec, diff --git a/litellm-rust/crates/model-catalog/src/layers.rs b/litellm-rust/crates/model-catalog/src/layers.rs new file mode 100644 index 00000000000..881f0f6af9b --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/layers.rs @@ -0,0 +1,32 @@ +use std::sync::Arc; + +use indexmap::IndexMap; +use serde_json::Value; + +use crate::catalog::Catalog; +use crate::register::Registrations; +use crate::wire::Fields; + +/// One field of a fetched row over the layer below, RFC 7386. An absent field inherits, JSON `null` is `Remove` +#[derive(Clone, Debug, PartialEq)] +pub enum Patch { + Set(Value), + Remove, +} + +/// The four layers served rows are composed from, lowest first +#[derive(Clone, Debug)] +#[expect( + dead_code, + reason = "02 and 07 add the functions that read these layers" +)] +pub struct Layers { + /// 0, the wheel's embedded map + bundled: Arc, + /// 1, the last installed fetch, rejected rows left out + fetched: Option>, + /// 2, `register(persist=False)`, cleared by every install and refilled by the router replay + transient: IndexMap, + /// 3, `register(persist=True)`, applied over every install + persisted: Registrations, +} diff --git a/litellm-rust/crates/model-catalog/src/lib.rs b/litellm-rust/crates/model-catalog/src/lib.rs index 066c2c83c6b..fe419dde054 100644 --- a/litellm-rust/crates/model-catalog/src/lib.rs +++ b/litellm-rust/crates/model-catalog/src/lib.rs @@ -1,18 +1,65 @@ -mod capabilities; +//! The model catalog: per-model pricing, context limits and capabilities. +//! +//! This is the Rust form of `model_prices_and_context_window.json`, the map Python reads as +//! `litellm.model_cost` and through `litellm.get_model_info`. It answers "what does this model +//! cost and what can it do"; it knows nothing about configured deployments. The router's view of +//! deployments (groups, aliases, wildcard patterns, team mappings) is the router registry, defined +//! in `litellm-router-core`, which reads this catalog through its dynamic config. +//! +//! # Structure +//! - `Catalog`: the parsed map. `Catalog::parse` takes the JSON body and returns `Loaded`, +//! keeping every entry in file order with its raw fields and collecting row and alias errors. +//! - `ModelEntry`: one model. `ModelEntry::fields` is the raw JSON object, so unknown fields are +//! never lost; `ModelEntry::model` is the optional typed `Model` with attributes, prices, +//! adjustments and capabilities. Invalid rows retain their raw fields without a typed model. +//! - Aliases: an entry may list `aliases`; each alias resolves to its canonical key. Problems +//! found while indexing them (bad lists, collisions) are kept in `Loaded::errors` instead of +//! failing the document. +//! - `FallbackGeneralizations` / `FallbackRule`: regex rules that map unknown model ids onto known +//! families. They are data in the file, not entries. +//! - Validation: `Catalog::check_size` checks a catalog against `IntegrityLimits` (a remote copy +//! that lost most of its models is refused), and `Catalog::row_errors` validates one row. +//! +//! # Lookups +//! `Catalog::lookup` resolves a key exactly first (entry or alias), then case-insensitively, and +//! returns a `ModelMatch` with both the key that matched and the canonical key of the entry. Provider +//! prefixes are not stripped here; callers decide which keys to try. +//! +//! # Readers +//! Cost tracking, the router (through `RouterDynamicConfig`) and the gateway. Nothing in this crate +//! performs I/O; loading and refreshing the file is the caller's job. + +macro_rules_attribute::attribute_alias! { + #[apply(wire_type)] = + #[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] + #[cfg_attr(feature = "schema", derive(schemars::JsonSchema))]; +} + +mod bundled; mod catalog; mod error; mod fallback; +mod layers; +mod lower; +mod model; mod model_info; -mod pricing; -mod validation; +mod register; +mod resolve; +mod store; +mod vocabulary; +pub mod wire; -pub use capabilities::*; +pub use bundled::*; pub use catalog::*; pub use error::*; pub use fallback::*; +pub use layers::*; +pub use model::*; pub use model_info::*; -pub use pricing::*; -pub use validation::*; +pub use register::*; +pub use resolve::*; +pub use store::*; +pub use vocabulary::*; #[cfg(feature = "schema")] mod schema; diff --git a/litellm-rust/crates/model-catalog/src/lower.rs b/litellm-rust/crates/model-catalog/src/lower.rs new file mode 100644 index 00000000000..7acfc781431 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/lower.rs @@ -0,0 +1,863 @@ +use std::collections::BTreeMap; +use std::collections::btree_map::Entry; +use std::num::NonZeroU64; + +use litellm_cost::money::UnitPrice; +use litellm_cost::price::{ + Adjustments, Band, BandEdge, Billing, CacheReadFallback, CacheReadMode, CacheTtl, + GuardrailUnit, InputUnit, Meter, OffPeak, OffPeakRates, OutputUnit, Price, Prices, Residency, + SearchContextSize, TextUnit, TierScope, TokenKind, Uplift, WallClock, WebSearchUnit, When, +}; +use litellm_cost::schedule::{Schedule, Window}; + +use crate::error::{Error, Repr, pointer_child, pointer_for_key}; +use crate::model::Model; +use crate::vocabulary::Mode; +use crate::wire::{EntryWire, FlatPrice, OffPeakBlock, TierRow, WebSearchBillingUnit}; + +impl TryFrom<&EntryWire> for Prices { + type Error = Vec; + + fn try_from(wire: &EntryWire) -> Result { + let mut problems = Vec::new(); + let sourced = flat_prices(&wire.prices) + .chain(tier_prices(wire, &mut problems)) + .chain(search_context_prices(wire)) + .chain(guardrail_prices(wire)); + let rates = dedupe(sourced, &mut problems); + let off_peak = wire + .off_peak + .as_ref() + .and_then(|block| off_peak(block, &mut problems)); + if problems.is_empty() { + let billing = billing(wire, &rates); + Ok(Prices::new(rates, off_peak, billing)) + } else { + Err(problems) + } + } +} + +fn billing(wire: &EntryWire, rates: &BTreeMap) -> Billing { + let provider = wire.attributes.litellm_provider.as_deref(); + let prices_meter = |priced: fn(&Meter) -> bool| rates.keys().any(|price| priced(&price.meter)); + let prices_tokens = prices_meter(|meter| { + matches!( + meter, + Meter::Input(InputUnit::Token(_)) | Meter::Output(OutputUnit::Token(_)) + ) + }); + let prices_characters = prices_meter(|meter| { + matches!( + meter, + Meter::Input(InputUnit::Character) | Meter::Output(OutputUnit::Character) + ) + }); + let prices_cache_read = prices_meter(|meter| { + matches!( + meter, + Meter::CacheRead { + kind: TokenKind::Text, + .. + } + ) + }); + Billing { + text_unit: if prices_characters && !prices_tokens { + TextUnit::Character + } else { + TextUnit::Token + }, + band_edge: if provider == Some("xai") { + BandEdge::Inclusive + } else { + BandEdge::Exclusive + }, + cache_read_fallback: if provider == Some("fireworks_ai") && !prices_cache_read { + CacheReadFallback::HalfInputRate + } else { + CacheReadFallback::InputRate + }, + wall_clock: match wire.attributes.mode { + None | Some(Mode::Chat | Mode::Completion | Mode::Embedding | Mode::Responses) => { + WallClock::ResponseTime + } + Some(Mode::AudioTranscription) => WallClock::AudioFile, + Some(_) => WallClock::Never, + }, + web_search: match wire.attributes.web_search_billing_unit { + Some(WebSearchBillingUnit::PerQuery) => WebSearchUnit::PerQuery, + Some(WebSearchBillingUnit::PerPrompt) | None => WebSearchUnit::PerPrompt, + }, + tier_scope: if provider == Some("dashscope") && wire.tier_rows.is_some() { + TierScope::WholeCall + } else { + TierScope::PerMeter + }, + } +} + +impl From<&EntryWire> for Adjustments { + fn from(wire: &EntryWire) -> Self { + Adjustments::new( + [ + (Uplift::NonGlobalEndpoint, wire.regional_endpoint_uplift), + (Uplift::Residency(Residency::Eu), wire.residency_uplift_eu), + (Uplift::Residency(Residency::Us), wire.residency_uplift_us), + (Uplift::FedrampEndpoint, wire.fedramp_uplift), + ] + .into_iter() + .filter_map(|(uplift, factor)| Some((uplift, factor?))), + ) + } +} + +impl TryFrom for Model { + type Error = Vec; + + fn try_from(wire: EntryWire) -> Result { + let prices = Prices::try_from(&wire); + let adjustments = Adjustments::from(&wire); + let EntryWire { + attributes, + capabilities, + unknown, + problems, + .. + } = wire; + let missing_provider = attributes.litellm_provider.is_none().then_some(Error::new( + pointer_for_key("litellm_provider"), + Repr::MissingProvider, + )); + match prices { + Ok(prices) if problems.is_empty() && missing_provider.is_none() => Ok(Model { + attributes, + prices, + adjustments, + capabilities, + unrecognized: unknown.into_iter().map(|(key, _)| key).collect(), + }), + prices => Err(problems + .into_iter() + .chain(missing_provider) + .chain(prices.err().into_iter().flatten()) + .collect()), + } + } +} + +struct Sourced { + price: Price, + rate: UnitPrice, + source: String, + path: String, +} + +fn flat_prices(prices: &[FlatPrice]) -> impl Iterator + '_ { + prices.iter().map(|flat| { + let (channel, when) = flat.flat.channel_and_when(); + Sourced { + price: Price { + meter: flat.flat.meter.clone(), + channel, + when, + }, + rate: flat.price, + source: flat.key.clone(), + path: flat.path.clone(), + } + }) +} + +fn dedupe( + sourced: impl Iterator, + problems: &mut Vec, +) -> BTreeMap { + let mut seen: BTreeMap = BTreeMap::new(); + for Sourced { + price, + rate, + source, + path, + } in sourced + { + match seen.entry(price) { + Entry::Occupied(first) if first.get().0 == rate => {} + Entry::Occupied(first) => problems.push(Error::new( + path, + Repr::DuplicatePrice { + first: first.get().1.clone(), + second: source.clone(), + }, + )), + Entry::Vacant(slot) => { + slot.insert((rate, source, path)); + } + } + } + seen.into_iter() + .map(|(price, (rate, _, _))| (price, rate)) + .collect() +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum TierBasis { + PromptTokens, + Results, +} + +fn tier_basis(row: &TierRow) -> Result<(TierBasis, [u64; 2]), Error> { + match (row.range, row.max_results_range) { + (Some(range), None) => Ok((TierBasis::PromptTokens, range)), + (None, Some(range)) => Ok((TierBasis::Results, range)), + _ => Err(Error::new(row.path.clone(), Repr::TierRange)), + } +} + +fn tier_prices(wire: &EntryWire, problems: &mut Vec) -> Vec { + let Some(rows) = wire.tier_rows.as_deref().filter(|rows| !rows.is_empty()) else { + return Vec::new(); + }; + let based: Vec<_> = rows + .iter() + .filter_map(|row| match tier_basis(row) { + Ok((basis, range)) => Some((basis, range, row)), + Err(problem) => { + problems.push(problem); + None + } + }) + .collect(); + let Some(&(basis, ..)) = based.first() else { + return Vec::new(); + }; + if based.iter().any(|(other, ..)| *other != basis) { + problems.push(Error::from_key("tiered_pricing", Repr::MixedTierBasis)); + return Vec::new(); + } + if basis == TierBasis::PromptTokens { + problems.extend( + wire.prices + .iter() + .filter(|flat| flat.flat.band != Band::Base) + .map(|flat| Error::new(flat.path.clone(), Repr::BandsFromTwoSources)), + ); + } + let mut ordered = based; + ordered.sort_by_key(|(_, [start, _], _)| *start); + problems.extend(ladder_problems(basis, &ordered)); + ordered + .iter() + .flat_map(|(basis, [start, _], row)| { + row.prices.iter().map(move |flat| (*basis, *start, flat)) + }) + .filter_map( + |(basis, start, flat)| match tier_price(basis, start, flat) { + Ok(sourced) => Some(sourced), + Err(problem) => { + problems.push(problem); + None + } + }, + ) + .collect() +} + +fn ladder_problems(basis: TierBasis, ordered: &[(TierBasis, [u64; 2], &TierRow)]) -> Vec { + let step = match basis { + TierBasis::PromptTokens => 0, + TierBasis::Results => 1, + }; + let reversed = ordered + .iter() + .filter(|(_, [start, end], _)| end < start) + .map(|(.., row)| Error::new(row.path.clone(), Repr::TierReversed)); + let first = ordered + .first() + .filter(|(_, [start, _], _)| *start > step) + .map(|(_, [start, _], row)| { + Error::new( + row.path.clone(), + Repr::TierGap { + start: *start, + expected: 0, + }, + ) + }); + let gaps = ordered.windows(2).filter_map(|pair| { + let (_, [_, end], _) = pair[0]; + let (_, [start, _], row) = pair[1]; + (end.checked_add(step) != Some(start)).then(|| { + Error::new( + row.path.clone(), + Repr::TierGap { + start, + expected: end, + }, + ) + }) + }); + reversed.chain(first).chain(gaps).collect() +} + +fn tier_price(basis: TierBasis, start: u64, flat: &FlatPrice) -> Result { + if flat.flat.has_suffix() { + return Err(Error::new( + flat.path.clone(), + Repr::SuffixInsideBlock { + block: "tiered_pricing", + }, + )); + } + let (meter, band) = match (basis, &flat.flat.meter) { + (TierBasis::PromptTokens, meter) => (meter.clone(), Band::over_prompt_tokens(start)), + (TierBasis::Results, Meter::Input(InputUnit::Query { over_results: None })) => ( + Meter::Input(InputUnit::Query { + over_results: NonZeroU64::new(start.saturating_sub(1)), + }), + Band::Base, + ), + (TierBasis::Results, _) => { + return Err(Error::new(flat.path.clone(), Repr::NotPricedByResults)); + } + }; + Ok(Sourced { + price: Price::online( + meter, + When { + band, + ..When::default() + }, + ), + rate: flat.price, + source: flat.key.clone(), + path: flat.path.clone(), + }) +} + +fn search_context_prices(wire: &EntryWire) -> impl Iterator + '_ { + wire.search_context.iter().flat_map(|costs| { + [ + (SearchContextSize::Low, "low", costs.search_context_size_low), + ( + SearchContextSize::Medium, + "medium", + costs.search_context_size_medium, + ), + ( + SearchContextSize::High, + "high", + costs.search_context_size_high, + ), + ] + .into_iter() + .filter_map(|(size, name, rate)| { + Some(Sourced { + price: Price::online(Meter::WebSearch(size), When::default()), + rate: rate?, + source: format!("search_context_cost_per_query.search_context_size_{name}"), + path: pointer_child( + &pointer_for_key("search_context_cost_per_query"), + &format!("search_context_size_{name}"), + ), + }) + }) + }) +} + +fn guardrail_prices(wire: &EntryWire) -> impl Iterator + '_ { + wire.guardrail.iter().flatten().map(|(unit, rate)| Sourced { + price: Price::online(Meter::Guardrail(GuardrailUnit::new(unit)), When::default()), + rate: *rate, + source: format!("guardrail_cost_per_unit.{unit}"), + path: pointer_child(&pointer_for_key("guardrail_cost_per_unit"), unit), + }) +} + +fn off_peak(block: &OffPeakBlock, problems: &mut Vec) -> Option { + let rates = off_peak_rates(&block.prices, problems); + let schedule = schedule(block, problems)?; + let off_peak = OffPeak::new(schedule, rates); + if off_peak.is_none() { + problems.push(Error::from_key("off_peak_pricing", Repr::NoOffPeakPrices)); + } + off_peak +} + +fn off_peak_rates(prices: &[FlatPrice], problems: &mut Vec) -> OffPeakRates { + let mut rates = OffPeakRates::default(); + for flat in prices { + if flat.flat.has_suffix() { + problems.push(Error::new( + flat.path.clone(), + Repr::SuffixInsideBlock { + block: "off_peak_pricing", + }, + )); + continue; + } + let slot = match flat.flat.meter { + Meter::Input(InputUnit::Token(TokenKind::Text)) => &mut rates.input, + Meter::Output(OutputUnit::Token(TokenKind::Text)) => &mut rates.output, + Meter::Output(OutputUnit::ReasoningToken) => &mut rates.reasoning, + Meter::CacheRead { + kind: TokenKind::Text, + mode: CacheReadMode::Implicit, + } => &mut rates.cache_read, + Meter::CacheRead { + kind: TokenKind::Text, + mode: CacheReadMode::Explicit, + } => &mut rates.explicit_cache_read, + Meter::CacheWrite { + kind: TokenKind::Text, + ttl: CacheTtl::Default, + } => &mut rates.cache_write, + _ => { + problems.push(Error::new(flat.path.clone(), Repr::NotAnOffPeakRate)); + continue; + } + }; + *slot = Some(flat.price); + } + rates +} + +fn schedule(block: &OffPeakBlock, problems: &mut Vec) -> Option { + let empty = |key: String| Error::from_path(&key, Repr::EmptyList); + let every_day = block + .hours_utc + .iter() + .map(|hours| ("off_peak_pricing.hours_utc".to_owned(), hours, None)); + let windows = block + .windows + .iter() + .flatten() + .enumerate() + .map(|(index, window)| { + ( + format!("off_peak_pricing.windows[{index}].hours_utc"), + &window.hours_utc, + Some((index, window.weekdays.as_ref())), + ) + }); + let windows: Vec = every_day + .chain(windows) + .map(|(key, hours, weekdays)| { + if hours.windows().is_empty() { + problems.push(empty(key)); + } + if let Some((index, Some(days))) = weekdays + && days.is_empty() + { + problems.push(empty(format!("off_peak_pricing.windows[{index}].weekdays"))); + } + Window { + hours: hours.windows().to_vec(), + weekdays: weekdays.and_then(|(_, days)| days.cloned()), + } + }) + .collect(); + if block.windows.as_ref().is_some_and(Vec::is_empty) { + problems.push(empty("off_peak_pricing.windows".to_owned())); + } + if block.hours_utc.is_none() && block.windows.is_none() { + problems.push(Error::from_key("off_peak_pricing", Repr::NoOffPeakHours)); + return None; + } + Some(Schedule { + windows, + weekday_timezone: block.weekday_timezone.clone(), + }) +} + +#[cfg(test)] +mod tests { + use litellm_cost::money::Multiplier; + use litellm_cost::price::{Channel, ServiceTier}; + use litellm_cost::schedule::{UtcWindow, Weekday}; + use rstest::rstest; + use serde_json::{Value, json}; + + use super::*; + + const TEXT_INPUT: Meter = Meter::Input(InputUnit::Token(TokenKind::Text)); + + fn wire(entry: Value) -> EntryWire { + let Value::Object(fields) = entry else { + panic!("entry must be an object") + }; + let wire = EntryWire::read(&fields); + assert!(wire.problems.is_empty(), "{:?}", wire.problems); + wire + } + + fn prices(entry: Value) -> Result> { + Prices::try_from(&wire(entry)) + } + + fn online(meter: Meter, tier: ServiceTier, band: Band) -> Price { + Price::online(meter, When { tier, band }) + } + + #[rstest] + #[case::flat_key(json!({"input_cost_per_token": 1e-6}), online(TEXT_INPUT, ServiceTier::Standard, Band::Base))] + #[case::band_and_tier( + json!({"input_cost_per_token_above_272k_tokens_priority": 1e-6}), + online(TEXT_INPUT, ServiceTier::Priority, Band::over_prompt_tokens(272_000)) + )] + #[case::batch( + json!({"input_cost_per_token_batches": 1e-6}), + Price { meter: TEXT_INPUT, channel: Channel::Batch, when: When::default() } + )] + #[case::first_token_tier_is_the_base_band( + json!({"tiered_pricing": [{"range": [0.0, 256000.0], "input_cost_per_token": 1e-6}]}), + online(TEXT_INPUT, ServiceTier::Standard, Band::Base) + )] + #[case::result_tier_stays_in_its_meter( + json!({"tiered_pricing": [ + {"max_results_range": [0, 25], "input_cost_per_query": 2e-6}, + {"max_results_range": [26, 100], "input_cost_per_query": 1e-6} + ]}), + online( + Meter::Input(InputUnit::Query { over_results: NonZeroU64::new(25) }), + ServiceTier::Standard, + Band::Base + ) + )] + #[case::search_context( + json!({"search_context_cost_per_query": {"search_context_size_low": 1e-6}}), + online(Meter::WebSearch(SearchContextSize::Low), ServiceTier::Standard, Band::Base) + )] + #[case::character_band_is_a_prompt_token_band( + json!({"input_cost_per_character_above_128k_tokens": 1e-6}), + online(Meter::Input(InputUnit::Character), ServiceTier::Standard, Band::over_prompt_tokens(128_000)) + )] + #[case::guardrail( + json!({"guardrail_cost_per_unit": {"contentPolicyUnits": 1e-6}}), + online(Meter::Guardrail(GuardrailUnit::new("contentPolicyUnits")), ServiceTier::Standard, Band::Base) + )] + fn every_spelling_lowers_to_one_price(#[case] entry: Value, #[case] expected: Price) { + assert_eq!(prices(entry).unwrap().get(&expected), UnitPrice::new(1e-6)); + } + + #[rstest] + fn two_spellings_that_agree_are_one_price() { + let prices = prices(json!({ + "input_cost_per_token": 1e-6, + "tiered_pricing": [ + {"range": [0, 10], "input_cost_per_token": 1e-6}, + {"range": [10, 20], "input_cost_per_token": 2e-6} + ] + })) + .unwrap(); + assert_eq!( + prices.get(&Price::online(TEXT_INPUT, When::default())), + UnitPrice::new(1e-6) + ); + } + + #[rstest] + fn token_tiers_lower_to_prompt_bands_at_each_start() { + let prices = prices(json!({"tiered_pricing": [ + {"range": [0, 256000], "input_cost_per_token": 1e-6}, + {"range": [256000, 1000000], "input_cost_per_token": 2e-6} + ]})) + .unwrap(); + assert_eq!( + prices.band(ServiceTier::Standard, 256_001), + Band::over_prompt_tokens(256_000) + ); + assert_eq!( + prices + .rate( + &TEXT_INPUT, + ServiceTier::Standard, + Band::over_prompt_tokens(256_000) + ) + .map(UnitPrice::get), + Some(2e-6) + ); + } + + #[rstest] + #[case::two_spellings_of_one_price( + json!({ + "input_cost_per_token": 1e-6, + "tiered_pricing": [{"range": [0, 10], "input_cost_per_token": 2e-6}] + }), + "/tiered_pricing/0/input_cost_per_token", + "duplicate_price" + )] + #[case::zero_band_is_the_base_price( + json!({"input_cost_per_token": 1e-6, "input_cost_per_token_above_0k_tokens": 2e-6}), + "/input_cost_per_token_above_0k_tokens", + "duplicate_price" + )] + #[case::tier_without_range( + json!({"tiered_pricing": [{"input_cost_per_token": 1e-6}]}), + "/tiered_pricing/0", + "tier_range" + )] + #[case::tier_with_suffix( + json!({"tiered_pricing": [{"range": [0, 10], "input_cost_per_token_priority": 1e-6}]}), + "/tiered_pricing/0/input_cost_per_token_priority", + "suffix_inside_block" + )] + #[case::reversed_range( + json!({"tiered_pricing": [{"range": [0, 10], "input_cost_per_token": 1e-6}, {"range": [10, 5], "input_cost_per_token": 1e-6}]}), + "/tiered_pricing/1", + "tier_reversed" + )] + #[case::gap_between_tiers( + json!({"tiered_pricing": [ + {"range": [0, 10], "input_cost_per_token": 1e-6}, + {"range": [20, 30], "input_cost_per_token": 2e-6} + ]}), + "/tiered_pricing/1", + "tier_gap" + )] + #[case::ladder_not_from_zero( + json!({"tiered_pricing": [{"range": [5, 10], "input_cost_per_token": 1e-6}]}), + "/tiered_pricing/0", + "tier_gap" + )] + #[case::result_ladder_overlaps( + json!({"tiered_pricing": [ + {"max_results_range": [1, 10], "input_cost_per_query": 1e-6}, + {"max_results_range": [10, 20], "input_cost_per_query": 2e-6} + ]}), + "/tiered_pricing/1", + "tier_gap" + )] + #[case::mixed_basis( + json!({"tiered_pricing": [ + {"range": [0, 10], "input_cost_per_token": 1e-6}, + {"max_results_range": [11, 20], "input_cost_per_query": 2e-6} + ]}), + "/tiered_pricing", + "mixed_tier_basis" + )] + #[case::result_tier_prices_a_token( + json!({"tiered_pricing": [{"max_results_range": [0, 10], "input_cost_per_token": 1e-6}]}), + "/tiered_pricing/0/input_cost_per_token", + "not_priced_by_results" + )] + #[case::two_band_sources( + json!({ + "output_cost_per_token_above_200k_tokens": 1e-6, + "tiered_pricing": [{"range": [0, 10], "input_cost_per_token": 1e-6}] + }), + "/output_cost_per_token_above_200k_tokens", + "bands_from_two_sources" + )] + #[case::off_peak_without_hours( + json!({"off_peak_pricing": {"input_cost_per_token": 1e-6}}), + "/off_peak_pricing", + "no_off_peak_hours" + )] + #[case::off_peak_without_prices( + json!({"off_peak_pricing": {"hours_utc": "00:00-01:00"}}), + "/off_peak_pricing", + "no_off_peak_prices" + )] + #[case::off_peak_empty_windows( + json!({"off_peak_pricing": {"windows": [], "input_cost_per_token": 1e-6}}), + "/off_peak_pricing/windows", + "empty_list" + )] + #[case::off_peak_empty_hours( + json!({"off_peak_pricing": {"hours_utc": [], "input_cost_per_token": 1e-6}}), + "/off_peak_pricing/hours_utc", + "empty_list" + )] + #[case::off_peak_empty_weekdays( + json!({"off_peak_pricing": { + "windows": [{"hours_utc": "00:00-01:00", "weekdays": []}], + "input_cost_per_token": 1e-6 + }}), + "/off_peak_pricing/windows/0/weekdays", + "empty_list" + )] + #[case::off_peak_image_price( + json!({"off_peak_pricing": {"hours_utc": "00:00-01:00", "output_cost_per_image": 1}}), + "/off_peak_pricing/output_cost_per_image", + "not_an_off_peak_rate" + )] + #[case::off_peak_band( + json!({"off_peak_pricing": {"hours_utc": "00:00-01:00", "input_cost_per_token_above_200k_tokens": 1}}), + "/off_peak_pricing/input_cost_per_token_above_200k_tokens", + "suffix_inside_block" + )] + fn rules_across_keys_name_where_each_price_came_from( + #[case] entry: Value, + #[case] expected_path: &str, + #[case] expected_variant: &str, + ) { + let problems = prices(entry).unwrap_err(); + assert!( + problems + .iter() + .any(|problem| problem.path() == expected_path + && <&str>::from(problem.kind()) == expected_variant), + "{problems:?}" + ); + } + + #[rstest] + fn off_peak_hours_and_windows_form_one_schedule_with_its_rates() { + let prices = prices(json!({"off_peak_pricing": { + "hours_utc": ["00:00-01:00"], + "windows": [{"hours_utc": "10:00-12:00", "weekdays": [6, " sun "]}], + "weekday_timezone": "Asia/Shanghai", + "output_cost_per_token": 1e-6, + "explicit_cache_read_input_token_cost": 2e-6 + }})) + .unwrap(); + let off_peak = prices.off_peak().unwrap(); + let window = |text: &str| text.parse::().unwrap(); + + assert_eq!( + off_peak.schedule(), + &Schedule { + windows: vec![ + Window { + hours: vec![window("00:00-01:00")], + weekdays: None, + }, + Window { + hours: vec![window("10:00-12:00")], + weekdays: Some(vec![Weekday::Saturday, Weekday::Sunday]), + }, + ], + weekday_timezone: Some("Asia/Shanghai".into()), + } + ); + assert_eq!( + off_peak.rates(), + &OffPeakRates { + output: UnitPrice::new(1e-6), + explicit_cache_read: UnitPrice::new(2e-6), + ..OffPeakRates::default() + } + ); + assert_eq!(prices.iter().count(), 0); + } + + #[rstest] + fn uplifts_are_adjustments_not_prices() { + let wire = wire(json!({ + "regional_endpoint_uplift_multiplier": 1.1, + "regional_processing_uplift_multiplier_eu": "1.2", + "fedramp_uplift_multiplier": 1.1 + })); + let adjustments = Adjustments::from(&wire); + + assert_eq!(Prices::try_from(&wire).unwrap().iter().count(), 0); + assert_eq!( + adjustments + .get(&Uplift::NonGlobalEndpoint) + .map(Multiplier::get), + Some(1.1) + ); + assert_eq!( + adjustments + .get(&Uplift::Residency(Residency::Eu)) + .map(Multiplier::get), + Some(1.2) + ); + assert_eq!(adjustments.get(&Uplift::Residency(Residency::Us)), None); + assert_eq!( + adjustments + .get(&Uplift::FedrampEndpoint) + .map(Multiplier::get), + Some(1.1) + ); + } + + #[rstest] + fn a_model_reports_field_provider_and_price_problems_together() { + let Value::Object(fields) = json!({ + "mode": "chatt", + "tiered_pricing": [{"input_cost_per_token": 1e-6}] + }) else { + unreachable!() + }; + let problems: Vec = Model::from_fields(&fields) + .unwrap_err() + .iter() + .map(|problem| litellm_error::Report::new(problem).to_string()) + .collect(); + + assert_eq!(problems.len(), 3, "{problems:?}"); + assert!(problems[0].starts_with("mode: invalid value: unknown variant `chatt`")); + assert_eq!(problems[1], "litellm_provider: required"); + assert_eq!( + problems[2], + "tiered_pricing[0]: a tier needs exactly one of range and max_results_range" + ); + } + + #[rstest] + fn a_null_value_is_absent() { + let Value::Object(fields) = json!({ + "litellm_provider": "openai", + "input_cost_per_token": null, + "mode": null + }) else { + unreachable!() + }; + let model = Model::from_fields(&fields).unwrap(); + + assert_eq!(model.prices.iter().count(), 0); + assert_eq!(model.attributes.mode, None); + assert!(model.unrecognized.is_empty()); + } + + #[rstest] + #[case::tokens_win_over_characters( + json!({"litellm_provider": "vertex_ai-language-models", "input_cost_per_character": 1e-6, "input_cost_per_token": 1e-6}), + Billing::default() + )] + #[case::characters_only( + json!({"litellm_provider": "vertex_ai-language-models", "input_cost_per_character": 1e-6}), + Billing { text_unit: TextUnit::Character, ..Billing::default() } + )] + #[case::xai_band_is_inclusive( + json!({"litellm_provider": "xai", "input_cost_per_token": 1e-6}), + Billing { band_edge: BandEdge::Inclusive, ..Billing::default() } + )] + #[case::fireworks_without_cache_read_borrows_half( + json!({"litellm_provider": "fireworks_ai", "input_cost_per_token": 1e-6}), + Billing { cache_read_fallback: CacheReadFallback::HalfInputRate, ..Billing::default() } + )] + #[case::fireworks_with_cache_read_keeps_it( + json!({"litellm_provider": "fireworks_ai", "input_cost_per_token": 1e-6, "cache_read_input_token_cost": 1e-7}), + Billing::default() + )] + #[case::transcription_prices_the_audio_file( + json!({"litellm_provider": "openai", "mode": "audio_transcription", "input_cost_per_second": 1e-4}), + Billing { wall_clock: WallClock::AudioFile, ..Billing::default() } + )] + #[case::provisioned_chat_prices_response_time( + json!({"litellm_provider": "bedrock", "mode": "chat", "cost_per_second": 1e-2}), + Billing::default() + )] + #[case::image_generation_is_never_wall_clock( + json!({"litellm_provider": "openai", "mode": "image_generation"}), + Billing { wall_clock: WallClock::Never, ..Billing::default() } + )] + #[case::gemini_per_query_search( + json!({"litellm_provider": "gemini", "web_search_billing_unit": "per_query"}), + Billing { web_search: WebSearchUnit::PerQuery, ..Billing::default() } + )] + #[case::dashscope_tiers_bill_the_whole_call( + json!({"litellm_provider": "dashscope", "tiered_pricing": [{"range": [0, 32000], "input_cost_per_token": 1e-6}]}), + Billing { tier_scope: TierScope::WholeCall, ..Billing::default() } + )] + fn billing_is_lowered_from_the_row_and_its_provider( + #[case] entry: Value, + #[case] expected: Billing, + ) { + assert_eq!(prices(entry).unwrap().billing(), expected); + } +} diff --git a/litellm-rust/crates/model-catalog/src/model.rs b/litellm-rust/crates/model-catalog/src/model.rs new file mode 100644 index 00000000000..c1acd1c14cb --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/model.rs @@ -0,0 +1,22 @@ +use std::collections::BTreeMap; + +use litellm_cost::price::{Adjustments, Prices}; + +use crate::error::Error; +use crate::wire::{Attributes, EntryWire, Fields}; + +/// One catalog entry read into types. +#[derive(Clone, Debug, PartialEq)] +pub struct Model { + pub attributes: Attributes, + pub prices: Prices, + pub adjustments: Adjustments, + pub capabilities: BTreeMap, + pub unrecognized: Vec, +} + +impl Model { + pub fn from_fields(fields: &Fields) -> Result> { + Self::try_from(EntryWire::read(fields)) + } +} diff --git a/litellm-rust/crates/model-catalog/src/model_info.rs b/litellm-rust/crates/model-catalog/src/model_info.rs index ebee2c64140..afbbfec59e6 100644 --- a/litellm-rust/crates/model-catalog/src/model_info.rs +++ b/litellm-rust/crates/model-catalog/src/model_info.rs @@ -1,582 +1,198 @@ -use crate::capabilities::{AudioFormat, InputModality, Mode, OutputModality, VertexAiAudioApi}; -use crate::pricing::{OffPeakPricing, SearchContextCostPerQuery, TieredRate, WebSearchBillingUnit}; -use litellm_llms_types::formats::chat_completions::ReasoningEffort; -use serde::{Deserialize, Serialize}; +use indexmap::IndexMap; use serde_json::Value; -use std::collections::BTreeMap; -/// Typed mirror of one catalog model entry. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -pub struct ModelInfo { - #[serde(skip_serializing_if = "Option::is_none")] - pub annotation_cost_per_page: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub annotation_cost_per_page_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub audio_transcription_config: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub bedrock_converse_supports_strict_tools: Option, - /// Highest reasoning effort the Bedrock output_config accepts for this model. - #[serde(skip_serializing_if = "Option::is_none")] - pub bedrock_output_config_effort_ceiling: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_audio_token_cost: Option, - /// USD per token written to the provider's prompt cache. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_32k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_100k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_100k_tokens_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_128k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_1hr: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_1hr_above_100k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_1hr_above_200k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_200k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_200k_tokens_batches: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_256k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_272k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_272k_tokens_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_272k_tokens_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_272k_tokens_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_above_272k_tokens_ultrafast: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost_ultrafast: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_audio_token_cost: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_image_token_cost: Option, - /// USD per prompt token served from the provider's prompt cache. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_32k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_100k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_100k_tokens_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_128k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_200k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_200k_tokens_batches: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_200k_tokens_priority: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_256k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_272k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_272k_tokens_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_272k_tokens_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_272k_tokens_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_272k_tokens_ultrafast: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_above_512k_tokens: Option, - /// Balanced service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_balanced: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost_ultrafast: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub citation_cost_per_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub code_interpreter_cost_per_session: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub comment: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub computer_use_input_cost_per_1k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub computer_use_output_cost_per_1k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cost_per_second: Option, - /// Reasoning effort the provider applies when the request omits reasoning_effort. Gates whether a non-default temperature or the top_p/logprobs sampling params are accepted, which hold only when the effort resolves to 'none'. - #[serde(skip_serializing_if = "Option::is_none")] - pub default_reasoning_effort: Option, - /// Date the provider deprecates the model, YYYY-MM-DD. - #[serde(skip_serializing_if = "Option::is_none")] - pub deprecation_date: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub file_search_cost_per_1k_calls: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub file_search_cost_per_gb_per_day: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub gemini_audio_only_live: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub gemini_native_audio: Option, - /// USD per Grounding with Google Maps request; billed per query or per prompt per web_search_billing_unit. - #[serde(skip_serializing_if = "Option::is_none")] - pub google_maps_grounding_cost_per_query: Option, - /// USD cost per billable guardrail unit, keyed by the provider's usage counter name (e.g. Bedrock's contentPolicyUnits). - #[serde(skip_serializing_if = "Option::is_none")] - pub guardrail_cost_per_unit: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_audio_per_second: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_audio_per_second_above_128k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_audio_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_audio_token_batches: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_audio_token_priority: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_character: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_character_above_128k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_image: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_image_above_128k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_image_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_image_token_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_pixel: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_query: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_request: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_second: Option, - /// USD per prompt token. - #[serde(skip_serializing_if = "Option::is_none")] +/// The fields main's `ModelInfoBase(...)` copies (`litellm/utils.py` at c7c895db45), one typed field each, +/// read from the raw row without lowering it, so building one never fails +#[derive(Clone, Debug, Default, PartialEq)] +pub struct ModelInfoFields { + pub key: String, + pub max_tokens: Option, + pub max_input_tokens: Option, + pub max_output_tokens: Option, pub input_cost_per_token: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_32k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_100k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_100k_tokens_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_128k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_200k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_200k_tokens_batches: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_200k_tokens_priority: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_256k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_272k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_272k_tokens_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_272k_tokens_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_272k_tokens_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_272k_tokens_ultrafast: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_above_512k_tokens: Option, - /// Balanced service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_balanced: Option, - /// USD per prompt token via the provider's batch API. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token_cache_hit: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] pub input_cost_per_token_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] pub input_cost_per_token_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] + pub input_cost_per_token_balanced: Option, pub input_cost_per_token_ultrafast: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_video_per_second: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_video_per_second_above_128k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_video_per_second_above_15s_interval: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_video_per_second_above_8s_interval: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub cache_creation_input_token_cost: Option, + pub cache_creation_input_token_cost_above_200k_tokens: Option, + pub cache_creation_input_token_cost_above_272k_tokens: Option, + pub cache_creation_input_token_cost_above_272k_tokens_priority: Option, + pub cache_creation_input_token_cost_above_272k_tokens_flex: Option, + pub cache_creation_input_token_cost_above_272k_tokens_ultrafast: Option, + pub cache_creation_input_token_cost_flex: Option, + pub cache_creation_input_token_cost_priority: Option, + pub cache_creation_input_token_cost_ultrafast: Option, + pub cache_read_input_token_cost: Option, + pub cache_read_input_audio_token_cost: Option, + pub cache_read_input_audio_token_cost_flex: Option, + pub cache_read_input_audio_token_cost_priority: Option, + pub cache_read_input_audio_token_cost_batches: Option, + pub explicit_cache_read_input_token_cost: Option, + pub prompt_cache_min_tokens: Option, + pub cache_read_input_token_cost_above_200k_tokens: Option, + pub cache_read_input_token_cost_above_200k_tokens_priority: Option, + pub cache_read_input_token_cost_above_272k_tokens: Option, + pub cache_read_input_token_cost_above_272k_tokens_priority: Option, + pub cache_read_input_token_cost_above_272k_tokens_flex: Option, + pub cache_read_input_token_cost_above_272k_tokens_ultrafast: Option, + pub cache_read_input_token_cost_above_512k_tokens: Option, + pub cache_read_input_token_cost_flex: Option, + pub cache_read_input_token_cost_priority: Option, + pub cache_read_input_token_cost_balanced: Option, + pub cache_read_input_token_cost_ultrafast: Option, + pub cache_read_input_token_cost_batches: Option, + pub cache_read_input_token_cost_above_200k_tokens_batches: Option, + pub cache_read_input_token_cost_above_272k_tokens_batches: Option, + pub cache_creation_input_token_cost_batches: Option, + pub cache_creation_input_token_cost_above_200k_tokens_batches: Option, + pub cache_creation_input_token_cost_above_272k_tokens_batches: Option, + pub cache_creation_input_token_cost_above_1hr: Option, + pub cache_creation_input_token_cost_above_12hr: Option, + pub cache_creation_input_token_cost_above_12hr_flex: Option, + pub cache_creation_input_token_cost_above_12hr_priority: Option, + pub cache_creation_input_token_cost_above_12hr_above_272k_tokens: Option, + pub cache_creation_input_token_cost_above_12hr_above_272k_tokens_flex: Option, + pub cache_creation_input_token_cost_above_12hr_above_272k_tokens_priority: Option, + pub cache_creation_input_token_cost_above_12hr_batches: Option, + pub cache_creation_input_token_cost_above_12hr_ultrafast: Option, + pub cache_creation_input_token_cost_above_12hr_above_272k_tokens_batches: Option, + pub cache_creation_input_token_cost_above_12hr_above_272k_tokens_ultrafast: Option, + pub off_peak_pricing: Option, + pub input_cost_per_character: Option, + pub input_cost_per_token_above_128k_tokens: Option, + pub input_cost_per_token_above_200k_tokens: Option, + pub input_cost_per_token_above_200k_tokens_priority: Option, + pub input_cost_per_token_above_272k_tokens: Option, + pub input_cost_per_token_above_272k_tokens_priority: Option, + pub input_cost_per_token_above_272k_tokens_flex: Option, + pub input_cost_per_token_above_272k_tokens_ultrafast: Option, + pub input_cost_per_token_above_512k_tokens: Option, + pub input_cost_per_query: Option, + pub cost_per_second: Option, + pub input_cost_per_second: Option, + pub input_cost_per_text_message: Option, + pub input_cost_per_audio_token: Option, + pub input_cost_per_audio_token_flex: Option, + pub input_cost_per_audio_token_priority: Option, + pub input_cost_per_image_token: Option, pub input_cost_per_video_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub input_cost_per_audio_token_batches: Option, + pub input_cost_per_image_token_batches: Option, + pub input_cost_per_image: Option, + pub input_cost_per_audio_per_second: Option, + pub input_cost_per_video_per_second: Option, + pub input_cost_per_token_batches: Option, pub input_cost_per_video_token_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_dbu_cost_per_token: Option, - /// LiteLLM provider slug; one of https://docs.litellm.ai/docs/providers. - #[serde(skip_serializing_if = "Option::is_none")] - pub litellm_provider: Option, - /// Maximum prompt/context tokens the model accepts. - #[serde(skip_serializing_if = "Option::is_none")] - pub max_input_tokens: Option, - /// Maximum tokens the model can generate in one response. - #[serde(skip_serializing_if = "Option::is_none")] - pub max_output_tokens: Option, - /// Legacy field: max output tokens if the provider specifies it, else max input tokens. - #[serde(skip_serializing_if = "Option::is_none")] - pub max_tokens: Option, - /// Free-form notes about the entry (e.g. pricing derivation). - #[serde(skip_serializing_if = "Option::is_none")] - pub metadata: Option>, - /// Primary API surface / task type of the model. - #[serde(skip_serializing_if = "Option::is_none")] - pub mode: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub ocr_cost_per_credit: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub ocr_cost_per_page: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub ocr_cost_per_page_batches: Option, - /// Rates that replace the same-named base fields while the request falls inside the stated UTC windows. - #[serde(skip_serializing_if = "Option::is_none")] - pub off_peak_pricing: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_audio_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_character: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_character_above_128k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_image: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_image_1024: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_image_1536: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_image_512: Option, - #[serde( - rename = "output_cost_per_image_0.5K", - skip_serializing_if = "Option::is_none" - )] - pub output_cost_per_image_0_5k: Option, - #[serde( - rename = "output_cost_per_image_1K", - skip_serializing_if = "Option::is_none" - )] - pub output_cost_per_image_1k: Option, - #[serde( - rename = "output_cost_per_image_2K", - skip_serializing_if = "Option::is_none" - )] - pub output_cost_per_image_2k: Option, - #[serde( - rename = "output_cost_per_image_4K", - skip_serializing_if = "Option::is_none" - )] - pub output_cost_per_image_4k: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_image_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_pixel: Option, - /// USD per reasoning/thinking token, when billed separately. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_reasoning_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_1080p: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_2k: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_480p: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_4k: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_720p: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_second_768p: Option, - /// USD per generated token. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_32k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_100k_tokens: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_100k_tokens_batches: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_128k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_200k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_200k_tokens_batches: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_200k_tokens_priority: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_256k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_272k_tokens: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_272k_tokens_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_272k_tokens_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_272k_tokens_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_272k_tokens_ultrafast: Option, - /// Rate applied once the prompt exceeds the token threshold in the field name. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_above_512k_tokens: Option, - /// Balanced service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token_balanced: Option, - /// USD per generated token via the provider's batch API. - #[serde(skip_serializing_if = "Option::is_none")] + pub input_cost_per_token_above_200k_tokens_batches: Option, + pub input_cost_per_token_above_272k_tokens_batches: Option, pub output_cost_per_token_batches: Option, - /// Flex service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] + pub output_cost_per_token_above_200k_tokens_batches: Option, + pub output_cost_per_token_above_272k_tokens_batches: Option, + pub output_cost_per_token: Option, pub output_cost_per_token_flex: Option, - /// Priority service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] pub output_cost_per_token_priority: Option, - /// Ultrafast service-tier rate for the same-named base field. - #[serde(skip_serializing_if = "Option::is_none")] + pub output_cost_per_token_balanced: Option, pub output_cost_per_token_ultrafast: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_video_per_second: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_video_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_dbu_cost_per_token: Option, - /// Embedding dimension for embedding models. - #[serde(skip_serializing_if = "Option::is_none")] - pub output_vector_size: Option, - /// Smallest prefix the provider will actually cache; absent means the provider default applies. - #[serde(skip_serializing_if = "Option::is_none")] - pub prompt_cache_min_tokens: Option, - /// Provider-internal routing hints (e.g. bedrock_invocation_schema). - #[serde(skip_serializing_if = "Option::is_none")] - pub provider_specific_entry: Option>, - /// Exact reasoning_effort levels this deployment accepts; wins over supports_* flags. - #[serde(skip_serializing_if = "Option::is_none")] - pub reasoning_effort_levels: Option>, - /// Multiplier applied to all token costs when served from a non-global Vertex AI endpoint (e.g. 1.10 = +10%). - #[serde(skip_serializing_if = "Option::is_none")] - pub regional_endpoint_uplift_multiplier: Option, - /// Multiplier applied to all token costs for EU data residency (e.g. 1.10 = +10%). - #[serde(skip_serializing_if = "Option::is_none")] pub regional_processing_uplift_multiplier_eu: Option, - /// Multiplier applied to all token costs for US data residency (e.g. 1.10 = +10%). - #[serde(skip_serializing_if = "Option::is_none")] pub regional_processing_uplift_multiplier_us: Option, - /// Provider default requests-per-minute limit. - #[serde(skip_serializing_if = "Option::is_none")] - pub rpm: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub rules: Option>, - /// USD cost per web search query, keyed by search context size. - #[serde(skip_serializing_if = "Option::is_none")] - pub search_context_cost_per_query: Option, - /// URL of the provider pricing/model page this entry was taken from. - #[serde(skip_serializing_if = "Option::is_none")] - pub source: Option, - /// Audio container formats the model can return. - #[serde(skip_serializing_if = "Option::is_none")] - pub supported_audio_formats: Option>, - /// OpenAI-style API routes this model can be called through, e.g. /v1/chat/completions. - #[serde(skip_serializing_if = "Option::is_none")] + pub fedramp_uplift_multiplier: Option, + pub regional_endpoint_uplift_multiplier: Option, + pub output_cost_per_audio_token: Option, + pub output_cost_per_character: Option, + pub output_cost_per_reasoning_token: Option, + pub output_cost_per_reasoning_token_flex: Option, + pub output_cost_per_reasoning_token_priority: Option, + pub output_cost_per_token_above_128k_tokens: Option, + pub output_cost_per_character_above_128k_tokens: Option, + pub output_cost_per_token_above_200k_tokens: Option, + pub output_cost_per_token_above_200k_tokens_priority: Option, + pub output_cost_per_token_above_272k_tokens: Option, + pub output_cost_per_token_above_272k_tokens_priority: Option, + pub output_cost_per_token_above_272k_tokens_flex: Option, + pub output_cost_per_token_above_272k_tokens_ultrafast: Option, + pub output_cost_per_token_above_512k_tokens: Option, + pub output_cost_per_second: Option, + pub output_cost_per_second_1080p: Option, + pub output_cost_per_second_480p: Option, + pub output_cost_per_second_720p: Option, + pub output_cost_per_second_768p: Option, + pub output_cost_per_second_2k: Option, + pub output_cost_per_second_4k: Option, + pub output_cost_per_video_per_second: Option, + pub output_cost_per_image: Option, + pub output_cost_per_pixel: Option, + pub output_cost_per_image_token: Option, + pub output_cost_per_image_token_batches: Option, + pub output_cost_per_video_token: Option, + pub output_vector_size: Option, + pub citation_cost_per_token: Option, + pub tiered_pricing: Option, + pub litellm_provider: Option, + pub mode: Option, pub supported_endpoints: Option>, - /// Input modalities the model accepts. - #[serde(skip_serializing_if = "Option::is_none")] - pub supported_modalities: Option>, - /// Output modalities the model can produce. - #[serde(skip_serializing_if = "Option::is_none")] - pub supported_output_modalities: Option>, - /// Cloud regions the model is available in ('global' or region ids). - #[serde(skip_serializing_if = "Option::is_none")] - pub supported_regions: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_adaptive_thinking: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_anthropic_compaction: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_anthropic_thinking_payload: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_assistant_prefill: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_audio_input: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_audio_output: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_bedrock_runtime_chat_completions_response_format: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_bedrock_runtime_chat_completions_tools_with_reasoning: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_computer_use: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_embedding_image_input: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_fast_mode: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_forced_tool_use: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_function_calling: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_image_input: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_image_size: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_legacy_thinking: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_low_reasoning_effort: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_max_reasoning_effort: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_mid_conversation_system: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_minimal_reasoning_effort: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_multimodal: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_native_streaming: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_native_structured_output: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_none_reasoning_effort: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_nova_canvas_image_edit: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_output_config: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_parallel_function_calling: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_parallel_tool_use_config: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_pdf_input: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_prompt_cache_breakpoint: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_prompt_caching: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_reasoning: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_response_schema: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_sampling_params: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_speed: Option, - #[serde(skip_serializing_if = "Option::is_none")] pub supports_system_messages: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_thinking_cache_preservation: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_tool_choice: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_tool_search: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_url_context: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_video_input: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub supports_response_schema: Option, pub supports_vision: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub supports_function_calling: Option, + pub supports_parallel_function_calling: Option, + pub supports_tool_choice: Option, + pub supports_assistant_prefill: Option, + pub supports_prompt_caching: Option, + pub supports_prompt_cache_breakpoint: Option, + pub supports_thinking_cache_preservation: Option, + pub supports_audio_input: Option, + pub supports_video_input: Option, + pub supports_box_questions: Option, + pub supports_audio_output: Option, + pub supports_pdf_input: Option, + pub supports_embedding_image_input: Option, + pub supports_native_streaming: Option, + pub supports_native_structured_output: Option, + pub supports_bedrock_native_structured_output: Option, pub supports_web_search: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub supports_xhigh_reasoning_effort: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub supports_url_context: Option, + pub supports_reasoning: Option, + pub supports_adaptive_thinking: Option, + pub supports_legacy_thinking: Option, pub thinking_always_on: Option, - /// Context-length or result-count tiered rates; each tier's costs apply within its range. - #[serde(skip_serializing_if = "Option::is_none")] - pub tiered_pricing: Option>, - /// Provider default tokens-per-minute limit. - #[serde(skip_serializing_if = "Option::is_none")] - pub tpm: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub use_openai_responses_path: Option, - #[serde(skip_serializing_if = "Option::is_none")] + pub supports_tool_search: Option, + pub supports_mid_conversation_system: Option, + pub supports_anthropic_thinking_payload: Option, + pub supports_anthropic_compaction: Option, + pub supports_none_reasoning_effort: Option, + pub supports_minimal_reasoning_effort: Option, + pub supports_low_reasoning_effort: Option, + pub supports_xhigh_reasoning_effort: Option, + pub supports_max_reasoning_effort: Option, + pub reasoning_effort_levels: Option, + pub default_reasoning_effort: Option, + pub bedrock_output_config_effort_ceiling: Option, + pub bedrock_converse_supports_strict_tools: Option, + pub supports_regex_lookaround: Option, + pub supports_computer_use: Option, + pub search_context_cost_per_query: Option, + pub web_search_billing_unit: Option, + pub search_free_requests: Option, + pub google_maps_grounding_cost_per_query: Option, + pub tpm: Option, + pub rpm: Option, + pub ocr_cost_per_page: Option, + pub ocr_cost_per_page_batches: Option, + pub ocr_cost_per_credit: Option, + pub annotation_cost_per_page: Option, + pub annotation_cost_per_page_batches: Option, + pub provider_specific_entry: Option, pub uses_embed_content: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub vector_store_cost_per_gb_per_day: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub vertex_ai_audio_api: Option, - /// Whether web search is billed per query or per prompt. - #[serde(skip_serializing_if = "Option::is_none")] - pub web_search_billing_unit: Option, + pub supports_image_size: Option, + pub supported_audio_formats: Option, + pub token_count_framing: Option, + pub token_count_tool_framing: Option, + pub token_count_responses_framing: Option, + pub token_count_responses_tool_framing: Option, + pub token_count_model: Option, + pub token_count_tokenizer: Option, + pub vertex_ai_audio_api: Option, + /// Every key matching `ABOVE_THRESHOLD_COST_KEY_PATTERN`, copied as written + pub above_threshold_costs: IndexMap, } diff --git a/litellm-rust/crates/model-catalog/src/pricing.rs b/litellm-rust/crates/model-catalog/src/pricing.rs deleted file mode 100644 index b8ed652451c..00000000000 --- a/litellm-rust/crates/model-catalog/src/pricing.rs +++ /dev/null @@ -1,97 +0,0 @@ -use serde::{Deserialize, Serialize}; - -/// Whether web search is billed per query or per prompt. -#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(rename_all = "snake_case")] -pub enum WebSearchBillingUnit { - PerQuery, - PerPrompt, -} - -/// UTC "HH:MM-HH:MM" window, or a list of them; a window may wrap past midnight. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(untagged)] -pub enum UtcHours { - Single(String), - Multiple(Vec), -} - -/// ISO-8601 weekday number (1 = Monday .. 7 = Sunday) or English day name. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(untagged)] -pub enum Weekday { - Number(u8), - Name(String), -} - -/// One off-peak window entry inside `windows`. -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(deny_unknown_fields)] -pub struct OffPeakWindow { - pub hours_utc: UtcHours, - #[serde(skip_serializing_if = "Option::is_none")] - pub weekdays: Option>, -} - -/// Rates that replace the same-named base fields inside the stated UTC windows. -#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(deny_unknown_fields)] -pub struct OffPeakPricing { - #[serde(skip_serializing_if = "Option::is_none")] - pub hours_utc: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub windows: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub weekday_timezone: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_reasoning_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost: Option, -} - -/// USD cost per web search query, keyed by search context size. -#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(deny_unknown_fields)] -pub struct SearchContextCostPerQuery { - #[serde(skip_serializing_if = "Option::is_none")] - pub search_context_size_low: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub search_context_size_medium: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub search_context_size_high: Option, -} - -/// One tier of a context-length or result-count tiered rate. -#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize)] -#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] -#[serde(deny_unknown_fields)] -pub struct TieredRate { - #[serde(skip_serializing_if = "Option::is_none")] - pub range: Option<[f64; 2]>, - #[serde(skip_serializing_if = "Option::is_none")] - pub max_results_range: Option<[f64; 2]>, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub output_cost_per_reasoning_token: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_read_input_token_cost: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_creation_input_token_cost: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub input_cost_per_query: Option, -} diff --git a/litellm-rust/crates/model-catalog/src/register.rs b/litellm-rust/crates/model-catalog/src/register.rs new file mode 100644 index 00000000000..ed34082ae57 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/register.rs @@ -0,0 +1,26 @@ +use std::collections::{BTreeSet, HashMap}; + +use indexmap::IndexMap; + +use crate::wire::Fields; + +/// Where one `register` call writes and which row it inherits from +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RegistrationTarget { + /// Always the caller's key, never a case variant of it + pub write: String, + /// The first candidate that is an exact key whose row passes `provider_matches` + pub inherit_from: Option, +} + +/// Python's inheritance candidates in main's order, `get_model_info`'s then `_resolve_builtin_model_cost_entry`'s +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RegistrationCandidates(pub Vec); + +/// Layer 3, and the fields `register` last wrote, which litellm owns again +#[derive(Clone, Debug, Default)] +#[expect(dead_code, reason = "07 adds the functions that read registrations")] +pub struct Registrations { + persisted: IndexMap, + registered_fields: HashMap>, +} diff --git a/litellm-rust/crates/model-catalog/src/resolve/image.rs b/litellm-rust/crates/model-catalog/src/resolve/image.rs new file mode 100644 index 00000000000..d24e170b32c --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/resolve/image.rs @@ -0,0 +1,18 @@ +/// What an image call asks for. `resolve_image` builds main's eight key spellings from it +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ImageCandidates { + pub model: String, + pub provider: Option, + /// Width and height, `1024 x 1024` when the request names none, as on main + pub size: Option<(u32, u32)>, + pub quality: Option, +} + +/// A size and quality keyed row, `{quality}/{W}-x-{H}/{model}`, parsed once per index build for fal's nearest match +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ImageKey { + pub quality: Option, + pub width: u32, + pub height: u32, + pub model: String, +} diff --git a/litellm-rust/crates/model-catalog/src/resolve/mod.rs b/litellm-rust/crates/model-catalog/src/resolve/mod.rs new file mode 100644 index 00000000000..fbaead622aa --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/resolve/mod.rs @@ -0,0 +1,46 @@ +mod image; + +pub use image::*; + +/// The seven names main's `get_model_info` tries, built by Python, walked in this order by Rust +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Candidates { + pub combined_model_name: Option, + pub model: String, + pub region_free_combined_model_name: Option, + pub split_model: Option, + pub combined_stripped_model_name: Option, + pub stripped_model_name: Option, + pub provider_prefixed_model_name: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CandidateSlot { + CombinedModelName, + Model, + RegionFreeCombinedModelName, + SplitModel, + CombinedStrippedModelName, + StrippedModelName, + ProviderPrefixedModelName, +} + +/// Which step matched, so a test can name the step that changed +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Via { + pub slot: CandidateSlot, + /// Matched through the case-insensitive index rather than an exact key + pub case_insensitive: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Resolution { + Found { + key: String, + via: Via, + }, + /// Every candidate tried, in order + Missing { + tried: Vec, + }, +} diff --git a/litellm-rust/crates/model-catalog/src/schema.rs b/litellm-rust/crates/model-catalog/src/schema.rs index 7de988b3ff3..6fba985a8bd 100644 --- a/litellm-rust/crates/model-catalog/src/schema.rs +++ b/litellm-rust/crates/model-catalog/src/schema.rs @@ -1,9 +1,82 @@ -use schemars::Schema; +use std::collections::BTreeMap; + +use litellm_cost::money::{Multiplier, UnitPrice}; +use schemars::{JsonSchema, Schema}; use serde_json::{Map, Value, json}; +use crate::fallback::FallbackGeneralizations; +use crate::wire::{Attributes, OffPeakWindow, SearchContextCostPerQuery, UtcHours}; + +#[derive(JsonSchema)] +#[schemars(rename = "ModelEntry")] +struct ModelEntry { + #[serde(flatten)] + _attributes: Attributes, + #[serde(flatten)] + _blocks: PriceBlocks, +} + +#[derive(JsonSchema)] +#[allow(dead_code)] +struct PriceBlocks { + /// USD cost per billable guardrail unit, keyed by the provider's usage counter name (e.g. Bedrock's contentPolicyUnits). + guardrail_cost_per_unit: Option>, + /// Rates that replace the same-named base fields while the request falls inside the stated UTC windows. + off_peak_pricing: Option, + /// Multiplier applied to all token costs when served from a non-global Vertex AI endpoint (e.g. 1.10 = +10%). + regional_endpoint_uplift_multiplier: Option, + /// Multiplier applied to all token costs for EU data residency (e.g. 1.10 = +10%). + regional_processing_uplift_multiplier_eu: Option, + /// Multiplier applied to all token costs for US data residency (e.g. 1.10 = +10%). + regional_processing_uplift_multiplier_us: Option, + /// Multiplier applied to all token costs when served from the OpenAI FedRAMP endpoint (e.g. 1.10 = +10%). + fedramp_uplift_multiplier: Option, + /// USD cost per web search query, keyed by search context size. + search_context_cost_per_query: Option, + /// Context-length or result-count tiered rates; each tier's costs apply within its range. + tiered_pricing: Option>, +} + +/// Rates that replace the same-named base fields inside the stated UTC windows. +#[derive(JsonSchema)] +#[allow(dead_code)] +struct OffPeakPricing { + hours_utc: Option, + windows: Option>, + weekday_timezone: Option, +} + +/// One tier of a context-length or result-count tiered rate. +#[derive(JsonSchema)] +#[allow(dead_code)] +struct TieredRate { + range: Option<[u64; 2]>, + max_results_range: Option<[u64; 2]>, +} + +#[derive(JsonSchema)] +#[allow(dead_code)] +struct RegistryDocument { + sample_spec: Option>, + fallback_generalizations: Option, +} + +pub(crate) const FLAT_KEY_PATTERN: &str = concat!( + r"^(", + r"input_cost_per_((audio_|image_|video_)?token|character|image|pixel|query|request|second|text_message|audio_per_second|video_per_second(_above_\d+s_interval)?)", + r"|output_cost_per_((audio_|image_|video_|reasoning_)?token|character|image(_\d+(\.\d+)?K?)?|pixel|second(_\d+[pk])?|video_per_second)", + r"|cache_read_input_(audio_|image_|video_)?token_cost", + r"|explicit_cache_read_input_token_cost", + r"|cache_creation_input_(audio_|image_|video_)?token_cost(_above_1hr|_above_12hr)?", + r"|annotation_cost_per_page|citation_cost_per_token|code_interpreter_cost_per_session", + r"|google_maps_grounding_cost_per_query|ocr_cost_per_credit|ocr_cost_per_page|cost_per_second", + r"|browser_open_cost_per_request|cost_per_additional_result", + r")(_above_\d+k?_tokens)?(_(priority|flex|ultrafast|balanced|batches|decisions))?$", +); + /// JSON Schema for one model entry, including registry validation constraints. pub fn model_entry_json_schema() -> Schema { - let mut schema = serde_json::to_value(schemars::schema_for!(crate::ModelInfo)) + let mut schema = serde_json::to_value(schemars::schema_for!(ModelEntry)) .expect("derived model schema serializes"); remove_nullable_optional_fields(&mut schema); decorate_model_entry(&mut schema); @@ -22,24 +95,21 @@ pub fn registry_json_schema() -> Schema { entry.as_object_mut().unwrap().remove("$schema"); definitions.insert("modelEntry".into(), entry); - let mut fallback = serde_json::to_value(schemars::schema_for!(crate::FallbackGeneralizations)) - .expect("derived fallback schema serializes"); - remove_nullable_optional_fields(&mut fallback); - definitions.extend(take_definitions(&mut fallback)); - fallback.as_object_mut().unwrap().remove("$schema"); - - let root = json!({ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "title": "LiteLLM model prices and context window registry", - "type": "object", - "properties": { - "sample_spec": {"type": "object"}, - "fallback_generalizations": fallback - }, - "additionalProperties": {"$ref": "#/$defs/modelEntry"}, - "$defs": definitions - }); - Schema::from(root.as_object().unwrap().clone()) + let mut registry = serde_json::to_value(schemars::schema_for!(RegistryDocument)) + .expect("derived registry schema serializes"); + remove_nullable_optional_fields(&mut registry); + definitions.extend(take_definitions(&mut registry)); + let root = registry.as_object_mut().unwrap(); + root.insert( + "title".into(), + "LiteLLM model prices and context window registry".into(), + ); + root.insert( + "additionalProperties".into(), + json!({"$ref": "#/$defs/modelEntry"}), + ); + root.insert("$defs".into(), Value::Object(definitions)); + Schema::from(root.clone()) } fn take_definitions(schema: &mut Value) -> Map { @@ -80,42 +150,21 @@ fn remove_nullable_optional_fields(value: &mut Value) { } fn decorate_model_entry(schema: &mut Value) { + let rates = json!({FLAT_KEY_PATTERN: {"type": "number", "minimum": 0}}); let object = schema.as_object_mut().unwrap(); object.insert("required".into(), json!(["litellm_provider"])); object.insert("additionalProperties".into(), Value::Bool(true)); - let properties = object - .get_mut("properties") - .unwrap() - .as_object_mut() - .unwrap(); - properties.insert( - "aliases".into(), - json!({"type": "array", "items": {"type": "string"}}), + object.insert( + "patternProperties".into(), + json!({ + FLAT_KEY_PATTERN: {"type": "number", "minimum": 0}, + "^supports_": {"type": "boolean"}, + }), ); - properties.get_mut("deprecation_date").unwrap()["format"] = json!("date"); - properties.get_mut("deprecation_date").unwrap()["pattern"] = - json!(r"^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$"); - - properties.iter_mut().for_each(|(name, property)| { - if name.contains("cost") { - property["minimum"] = json!(0); - } else if name.contains("uplift_multiplier") { - property["minimum"] = json!(1); - } - }); - properties.get_mut("guardrail_cost_per_unit").unwrap()["additionalProperties"]["minimum"] = - json!(0); - let definitions = object.get_mut("$defs").unwrap().as_object_mut().unwrap(); - for definition in ["OffPeakPricing", "TieredRate", "SearchContextCostPerQuery"] { - let properties = definitions[definition]["properties"] - .as_object_mut() - .unwrap(); - properties.iter_mut().for_each(|(name, property)| { - if name.contains("cost") || definition == "SearchContextCostPerQuery" { - property["minimum"] = json!(0); - } - }); + for definition in ["OffPeakPricing", "TieredRate"] { + definitions[definition]["patternProperties"] = rates.clone(); + definitions[definition]["additionalProperties"] = Value::Bool(false); } definitions["OffPeakPricing"]["anyOf"] = json!([ {"required": ["hours_utc"]}, @@ -123,15 +172,43 @@ fn decorate_model_entry(schema: &mut Value) { ]); definitions["OffPeakPricing"]["properties"]["windows"]["minItems"] = json!(1); definitions["OffPeakWindow"]["properties"]["weekdays"]["minItems"] = json!(1); - definitions["TieredRate"]["properties"]["range"]["items"]["minimum"] = json!(0); - definitions["TieredRate"]["properties"]["max_results_range"]["items"]["minimum"] = json!(0); - definitions["Weekday"]["anyOf"][0]["minimum"] = json!(1); - definitions["Weekday"]["anyOf"][0]["maximum"] = json!(7); - definitions["Weekday"]["anyOf"][1]["pattern"] = json!( - r"(?i)^(mon|monday|tue|tues|tuesday|wed|wednesday|thu|thur|thurs|thursday|fri|friday|sat|saturday|sun|sunday)$" - ); - let window_pattern = json!(r"^([01]\d|2[0-3]):[0-5]\d-([01]\d|2[0-3]):[0-5]\d$"); - definitions["UtcHours"]["anyOf"][0]["pattern"] = window_pattern.clone(); - definitions["UtcHours"]["anyOf"][1]["items"]["pattern"] = window_pattern; definitions["UtcHours"]["anyOf"][1]["minItems"] = json!(1); } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::FLAT_KEY_PATTERN; + use crate::wire::FlatKey; + + #[rstest] + #[case::input_token("input_cost_per_token")] + #[case::cache_write_one_hour_band( + "cache_creation_input_token_cost_above_1hr_above_200k_tokens" + )] + #[case::image_label("output_cost_per_image_0.5K")] + #[case::resolution_tier("output_cost_per_second_4k_flex")] + #[case::feature_batch("annotation_cost_per_page_batches")] + #[case::video_interval("input_cost_per_video_per_second_above_8s_interval")] + #[case::explicit_cache_read("explicit_cache_read_input_token_cost")] + #[case::wall_clock("cost_per_second")] + #[case::unused_meter("cost_per_token")] + #[case::other_currency("input_dbu_cost_per_token")] + #[case::typo("input_cost_per_tokne")] + #[case::read_has_no_ttl("cache_read_input_token_cost_above_1hr")] + #[case::interval_on_a_token_meter("input_cost_per_token_above_8s_interval")] + fn flat_key_pattern_agrees_with_the_parser(#[case] key: &str) { + let schema = serde_json::json!({ + "type": "object", + "patternProperties": {FLAT_KEY_PATTERN: {}}, + "additionalProperties": false, + }); + let validator = jsonschema::validator_for(&schema).unwrap(); + + assert_eq!( + validator.is_valid(&serde_json::json!({key: 1})), + FlatKey::parse(key).is_some() + ); + } +} diff --git a/litellm-rust/crates/model-catalog/src/store/AGENTS.md b/litellm-rust/crates/model-catalog/src/store/AGENTS.md new file mode 100644 index 00000000000..f1dacd2e217 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/store/AGENTS.md @@ -0,0 +1,35 @@ +# Store + +`Store` is an in-memory store, a small database built for exactly three queries. It knows nothing about the catalog. A source picks its own `Rows::Row` type, how names fold for matching, and the group each key is listed under. The catalog's sources fold to lowercase and group by raw `litellm_provider` + +| Query | Method | Real caller | +|---|---|---| +| row by exact key | `Store::get` | reading `litellm.model_cost["gpt-4o"]` | +| actual key for a name that folds to the same form | `Store::find_key` | `get_model_info("GPT-4o")` finding `gpt-4o` | +| keys in a group | `Store::keys_in` | the provider sets, `litellm.open_ai_chat_completion_models` | + +## Rules + +- nothing in `store/` imports from the rest of the crate. Wire types, case rules and field names such as `litellm_provider` belong to the source that implements `Rows` +- the index in `name_index.rs` is private to the store. Never build a folded-name map or a provider grouping anywhere else in the workspace. A new lookup pattern is a new `Store` method backed by `NameIndex` +- the index holds keys, never rows. Rows are always read from `Rows`, so a price edit is seen on the next read and never rebuilds the index +- `find_key` and `keys_in` never return a key the rows lack. A hit on a removed key rebuilds the index once and asks again +- `keys_in` matches the group exactly as the source wrote it. Provider aliases such as `vertex_ai` over `vertex_ai-language-models` are Python's job +- when two keys fold to the same name the later key in row order wins, like main's `_model_cost_lowercase_map` + +## Stamps + +`Rows::stamp` is how the store learns that the index may be stale. The store keeps the stamp it built the index at and rebuilds when the source returns a different one. A source must change its stamp whenever a key or a key's group changes, since those are the two things the index holds + +| Source | Stamp | Blind spot | +|---|---|---| +| `Catalog`'s `CatalogRows` | `()`, every change goes through `Store::edit`, which drops the index | none | +| python-bridge's bound `litellm.model_cost` | the dict object and its `len` | a delete plus an add between two reads, or a `litellm_provider` rewrite on an existing key | + +A source with a blind spot calls `Store::invalidate` when it knows keys changed. Until then a key added in the blind spot is not found, and a key moved to another provider stays listed under the old one. Removed keys are still never returned + +## Tests + +Tests live in `mod.rs`'s `tests` module and drive the store through `ModelCost`, a fake `litellm.model_cost` with real bundled keys whose row is just its `litellm_provider`. Each Python write is a method (`set`, `del`, `rebind`) and its stamp is what python-bridge can see. Add a scenario as a named test that reads like the Python user code it stands for + +The two property tests compare every answer with a linear scan of the same dict. With a stamp that sees every key change, answers must match exactly. With the dict stamp, every answer must be sound (present, same folded name), and `invalidate` must make them match again. A store change that fails either is a bug in the store, not in the test diff --git a/litellm-rust/crates/model-catalog/src/store/mod.rs b/litellm-rust/crates/model-catalog/src/store/mod.rs new file mode 100644 index 00000000000..2220526f8b6 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/store/mod.rs @@ -0,0 +1,460 @@ +mod name_index; + +use std::borrow::Cow; +use std::fmt; +use std::sync::{PoisonError, RwLock}; + +use name_index::NameIndex; + +/// Where a `Store`'s rows live. The store rebuilds its index whenever `stamp` changes, so a source must change its +/// stamp whenever a key or a key's group changes, or call `Store::invalidate` when it cannot +pub trait Rows { + type Row: ToOwned + ?Sized; + type Stamp: PartialEq; + + /// The name two spellings of one key share, which `Store::find_key` matches on + fn fold(name: &str) -> String; + fn stamp(&self) -> Self::Stamp; + fn contains(&self, key: &str) -> bool; + fn row(&self, key: &str) -> Option>; + /// Every key in row order with the group it is listed under + fn scan(&self, visit: &mut dyn FnMut(&str, Option<&str>)); +} + +/// An in-memory store for three queries (row by exact key, actual key for a name by its folded form, keys in a group) +/// that owns the only index behind them +pub struct Store { + rows: R, + index: RwLock>>, +} + +struct Built { + stamp: S, + names: NameIndex, +} + +impl Store { + pub fn new(rows: R) -> Self { + Self { + rows, + index: RwLock::new(None), + } + } + + pub fn rows(&self) -> &R { + &self.rows + } + + /// Changes the rows in place. The index is rebuilt on the next query + pub fn edit(&mut self, edit: impl FnOnce(&mut R) -> T) -> T { + let edited = edit(&mut self.rows); + *self.index.get_mut().unwrap_or_else(PoisonError::into_inner) = None; + edited + } + + pub fn get(&self, key: &str) -> Option> { + self.rows.row(key) + } + + /// The key itself when present, otherwise the actual key that folds to the same name. Never a key the rows lack + pub fn find_key(&self, name: &str) -> Option { + if self.rows.contains(name) { + return Some(name.to_owned()); + } + let folded = R::fold(name); + let hit = self.query(|names| names.key_for(&folded).map(str::to_owned))?; + if self.rows.contains(&hit) { + return Some(hit); + } + self.invalidate(); + self.query(|names| names.key_for(&folded).map(str::to_owned)) + .filter(|key| self.rows.contains(key)) + } + + /// Keys whose group is exactly `group`, in row order. Never a key the rows lack + pub fn keys_in(&self, group: &str) -> Vec { + self.query(|names| names.keys_in(group).to_vec()) + .into_iter() + .filter(|key| self.rows.contains(key)) + .collect() + } + + /// Drops the index, for a source whose stamp cannot see every key change + pub fn invalidate(&self) { + *self.index.write().unwrap_or_else(PoisonError::into_inner) = None; + } + + fn query(&self, read: impl Fn(&NameIndex) -> T) -> T { + let stamp = self.rows.stamp(); + if let Some(built) = self + .index + .read() + .unwrap_or_else(PoisonError::into_inner) + .as_ref() + .filter(|built| built.stamp == stamp) + { + return read(&built.names); + } + let names = NameIndex::scan(&self.rows); + let answer = read(&names); + *self.index.write().unwrap_or_else(PoisonError::into_inner) = Some(Built { stamp, names }); + answer + } +} + +impl Clone for Store { + fn clone(&self) -> Self { + Self::new(self.rows.clone()) + } +} + +impl fmt::Debug for Store { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Store") + .field("rows", &self.rows) + .finish_non_exhaustive() + } +} + +#[cfg(test)] +mod tests { + use std::cell::{Cell, RefCell}; + + use indexmap::IndexMap; + use rstest::{fixture, rstest}; + + use super::*; + + const BUNDLED: &[(&str, Option<&str>)] = &[ + ("gpt-4o", Some("openai")), + ("o3-pro", Some("openai")), + ("azure/gpt-4o", Some("azure")), + ("claude-sonnet-4-5", Some("anthropic")), + ("gemini-2.5-pro", Some("vertex_ai-language-models")), + ( + "fireworks_ai/accounts/fireworks/models/llama-v3p1-8b-instruct", + Some("fireworks_ai"), + ), + ( + "anyscale/meta-llama/Meta-Llama-3-8B-Instruct", + Some("anyscale"), + ), + ("sample_free_row", None), + ]; + + /// `litellm.model_cost` as Python sees it. The stamp is what python-bridge can observe for free, the dict's + /// identity and its length, so a write that keeps both hides a key change from the store + struct ModelCost { + dict: RefCell>>, + identity: Cell, + index_changes: Cell, + honest_stamp: bool, + scans: Cell, + } + + impl ModelCost { + fn new(rows: &[(&str, Option<&str>)], honest_stamp: bool) -> Self { + Self { + dict: RefCell::new( + rows.iter() + .map(|(key, provider)| ((*key).to_owned(), provider.map(str::to_owned))) + .collect(), + ), + identity: Cell::new(0), + index_changes: Cell::new(0), + honest_stamp, + scans: Cell::new(0), + } + } + + /// `model_cost[key] = {"litellm_provider": provider, ...}` + fn set(&self, key: &str, provider: Option<&str>) { + let previous = self + .dict + .borrow_mut() + .insert(key.to_owned(), provider.map(str::to_owned)); + if previous.as_ref().map(Option::as_deref) != Some(provider) { + self.index_changes.set(self.index_changes.get() + 1); + } + } + + /// `del model_cost[key]` + fn del(&self, key: &str) { + if self.dict.borrow_mut().shift_remove(key).is_some() { + self.index_changes.set(self.index_changes.get() + 1); + } + } + + /// `litellm.model_cost = {...}`, a new dict object + fn rebind(&self, rows: &[(&str, Option<&str>)]) { + *self.dict.borrow_mut() = rows + .iter() + .map(|(key, provider)| ((*key).to_owned(), provider.map(str::to_owned))) + .collect(); + self.identity.set(self.identity.get() + 1); + self.index_changes.set(self.index_changes.get() + 1); + } + } + + /// A row is its `litellm_provider`, which is also the group the store lists its key under + impl Rows for &ModelCost { + type Row = Option; + type Stamp = (u64, u64); + + fn fold(name: &str) -> String { + name.to_lowercase() + } + + fn stamp(&self) -> Self::Stamp { + if self.honest_stamp { + (self.identity.get(), self.index_changes.get()) + } else { + (self.identity.get(), self.dict.borrow().len() as u64) + } + } + + fn contains(&self, key: &str) -> bool { + self.dict.borrow().contains_key(key) + } + + fn row(&self, key: &str) -> Option>> { + Some(Cow::Owned(self.dict.borrow().get(key)?.clone())) + } + + fn scan(&self, visit: &mut dyn FnMut(&str, Option<&str>)) { + self.scans.set(self.scans.get() + 1); + for (key, provider) in self.dict.borrow().iter() { + visit(key, provider.as_deref()); + } + } + } + + #[fixture] + fn model_cost() -> ModelCost { + ModelCost::new(BUNDLED, false) + } + + /// Rows only `Store::edit` can change, so the stamp never moves + #[derive(Debug)] + struct Owned(IndexMap>); + + impl Rows for Owned { + type Row = Option; + type Stamp = (); + + fn fold(name: &str) -> String { + name.to_lowercase() + } + + fn stamp(&self) -> Self::Stamp {} + + fn contains(&self, key: &str) -> bool { + self.0.contains_key(key) + } + + fn row(&self, key: &str) -> Option>> { + self.0.get(key).map(Cow::Borrowed) + } + + fn scan(&self, visit: &mut dyn FnMut(&str, Option<&str>)) { + for (key, provider) in &self.0 { + visit(key, provider.as_deref()); + } + } + } + + fn owned(rows: &[(&str, Option<&str>)]) -> Store { + Store::new(Owned( + rows.iter() + .map(|(key, provider)| ((*key).to_owned(), provider.map(str::to_owned))) + .collect(), + )) + } + + #[rstest] + #[case::bundled_key_as_written("gpt-4o", Some("gpt-4o"))] + #[case::user_typed_upper_case("GPT-4o", Some("gpt-4o"))] + #[case::provider_prefixed("Azure/GPT-4o", Some("azure/gpt-4o"))] + #[case::fireworks_path_in_another_case( + "fireworks_ai/accounts/fireworks/models/Llama-v3p1-8B-Instruct", + Some("fireworks_ai/accounts/fireworks/models/llama-v3p1-8b-instruct") + )] + #[case::mixed_case_bundled_key( + "anyscale/meta-llama/meta-llama-3-8b-instruct", + Some("anyscale/meta-llama/Meta-Llama-3-8B-Instruct") + )] + #[case::unmapped_model("my-finetune", None)] + #[case::prefix_is_not_a_match("gpt-4", None)] + fn find_key_maps_a_requested_model_to_its_key( + model_cost: ModelCost, + #[case] model: &str, + #[case] expected: Option<&str>, + ) { + assert_eq!(Store::new(&model_cost).find_key(model).as_deref(), expected); + } + + #[rstest] + #[case::the_bundled_spelling("gpt-4o", "gpt-4o")] + #[case::the_registered_spelling("GPT-4o", "GPT-4o")] + #[case::another_spelling_takes_the_later_key("Gpt-4O", "GPT-4o")] + fn a_registered_case_variant_is_its_own_row( + model_cost: ModelCost, + #[case] model: &str, + #[case] expected: &str, + ) { + model_cost.set("GPT-4o", Some("openai")); + assert_eq!( + Store::new(&model_cost).find_key(model).as_deref(), + Some(expected) + ); + } + + #[rstest] + #[case::openai_in_map_order("openai", &["gpt-4o", "o3-pro"])] + #[case::azure("azure", &["azure/gpt-4o"])] + #[case::raw_vertex_family("vertex_ai-language-models", &["gemini-2.5-pro"])] + #[case::provider_alias_is_python_s_job("vertex_ai", &[])] + #[case::matched_as_written("OpenAI", &[])] + #[case::unknown_provider("acme", &[])] + fn keys_in_lists_the_keys_a_raw_provider_names( + model_cost: ModelCost, + #[case] provider: &str, + #[case] expected: &[&str], + ) { + assert_eq!(Store::new(&model_cost).keys_in(provider), expected); + } + + #[rstest] + fn a_model_written_straight_into_the_dict_is_found_in_any_case(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("mycorp/custom-model"), None); + + model_cost.set("MyCorp/Custom-Model", Some("openai")); + + assert_eq!( + store.find_key("mycorp/custom-model").as_deref(), + Some("MyCorp/Custom-Model") + ); + assert_eq!( + store.keys_in("openai"), + ["gpt-4o", "o3-pro", "MyCorp/Custom-Model"] + ); + } + + #[rstest] + fn a_price_edit_reuses_the_index(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("GPT-4o").as_deref(), Some("gpt-4o")); + + model_cost.set("gpt-4o", Some("openai")); + model_cost.set("o3-pro", Some("openai")); + model_cost.set("gpt-4o", Some("openai")); + + assert_eq!(store.find_key("O3-PRO").as_deref(), Some("o3-pro")); + assert_eq!(store.keys_in("openai"), ["gpt-4o", "o3-pro"]); + assert_eq!(model_cost.scans.get(), 1); + } + + #[rstest] + fn a_reload_that_rebinds_the_dict_is_seen_even_at_the_same_length(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("GPT-4o").as_deref(), Some("gpt-4o")); + + let mut reloaded = BUNDLED.to_vec(); + reloaded[0] = ("gpt-4o-2024-11-20", Some("openai")); + model_cost.rebind(&reloaded); + + assert_eq!(store.find_key("GPT-4o"), None); + assert_eq!( + store.find_key("GPT-4o-2024-11-20").as_deref(), + Some("gpt-4o-2024-11-20") + ); + } + + #[rstest] + fn a_deleted_model_is_never_returned(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("O3-PRO").as_deref(), Some("o3-pro")); + + model_cost.del("o3-pro"); + + assert_eq!(store.find_key("O3-PRO"), None); + assert_eq!(store.keys_in("openai"), ["gpt-4o"]); + } + + #[rstest] + fn a_respelled_key_at_the_same_length_is_found_after_one_rebuild(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("o3-PRO").as_deref(), Some("o3-pro")); + + model_cost.del("o3-pro"); + model_cost.set("O3-Pro", Some("openai")); + + assert_eq!(store.find_key("o3-PRO").as_deref(), Some("O3-Pro")); + assert_eq!(model_cost.scans.get(), 2); + } + + #[rstest] + fn a_swap_at_the_same_length_hides_the_new_key_until_invalidate(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.find_key("O3-PRO").as_deref(), Some("o3-pro")); + + model_cost.del("o3-pro"); + model_cost.set("MyCorp/Custom-Model", Some("openai")); + + assert_eq!(store.find_key("mycorp/custom-model"), None); + assert_eq!(store.keys_in("openai"), ["gpt-4o"]); + + store.invalidate(); + + assert_eq!( + store.find_key("mycorp/custom-model").as_deref(), + Some("MyCorp/Custom-Model") + ); + assert_eq!(store.keys_in("openai"), ["gpt-4o", "MyCorp/Custom-Model"]); + } + + #[rstest] + fn a_provider_rewrite_on_an_existing_key_needs_invalidate(model_cost: ModelCost) { + let store = Store::new(&model_cost); + assert_eq!(store.keys_in("openai"), ["gpt-4o", "o3-pro"]); + + model_cost.set("o3-pro", Some("azure")); + + assert_eq!(store.keys_in("openai"), ["gpt-4o", "o3-pro"]); + + store.invalidate(); + + assert_eq!(store.keys_in("openai"), ["gpt-4o"]); + assert_eq!(store.keys_in("azure"), ["o3-pro", "azure/gpt-4o"]); + } + + #[rstest] + fn an_edit_on_owned_rows_is_seen_by_the_next_query() { + let mut store = owned(BUNDLED); + assert_eq!(store.find_key("GPT-4o").as_deref(), Some("gpt-4o")); + + store.edit(|rows| { + rows.0.shift_remove("gpt-4o"); + rows.0 + .insert("MyCorp/Custom-Model".to_owned(), Some("openai".to_owned())); + }); + + assert_eq!(store.find_key("GPT-4o"), None); + assert_eq!( + store.find_key("mycorp/custom-model").as_deref(), + Some("MyCorp/Custom-Model") + ); + assert_eq!(store.keys_in("openai"), ["o3-pro", "MyCorp/Custom-Model"]); + } + + #[rstest] + fn get_reads_the_row_by_exact_key_only() { + let store = owned(BUNDLED); + assert_eq!( + store.get("azure/gpt-4o").as_deref(), + Some(&Some("azure".to_owned())) + ); + assert_eq!(store.get("Azure/GPT-4o"), None); + } +} diff --git a/litellm-rust/crates/model-catalog/src/store/name_index.rs b/litellm-rust/crates/model-catalog/src/store/name_index.rs new file mode 100644 index 00000000000..86f05cb31af --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/store/name_index.rs @@ -0,0 +1,35 @@ +use std::collections::HashMap; + +use super::Rows; + +/// Keys only, never rows, so a value write never makes it stale. A later key wins when two keys fold to one name +#[derive(Debug, Default)] +pub(super) struct NameIndex { + by_folded: HashMap, + by_group: HashMap>, +} + +impl NameIndex { + pub(super) fn scan(rows: &R) -> Self { + let mut index = Self::default(); + rows.scan(&mut |key, group| { + index.by_folded.insert(R::fold(key), key.to_owned()); + if let Some(group) = group { + index + .by_group + .entry(group.to_owned()) + .or_default() + .push(key.to_owned()); + } + }); + index + } + + pub(super) fn key_for(&self, folded: &str) -> Option<&str> { + self.by_folded.get(folded).map(String::as_str) + } + + pub(super) fn keys_in(&self, group: &str) -> &[String] { + self.by_group.get(group).map_or(&[], Vec::as_slice) + } +} diff --git a/litellm-rust/crates/model-catalog/src/validation.rs b/litellm-rust/crates/model-catalog/src/validation.rs deleted file mode 100644 index 73f08a865c1..00000000000 --- a/litellm-rust/crates/model-catalog/src/validation.rs +++ /dev/null @@ -1,218 +0,0 @@ -use std::collections::BTreeSet; - -use serde_json::{Map, Value}; -use thiserror::Error; - -use crate::{AliasIssue, Catalog, ModelInfo, UtcHours, Weekday}; - -/// A registry entry violates the checked-in catalog contract. -#[derive(Debug, Error)] -pub enum RegistryValidationError { - #[error("{reason}")] - Entry { model: String, reason: String }, - #[error("alias issue: {0:?}")] - Alias(AliasIssue), -} - -/// Validate one registry entry without restricting the tolerant catalog reader. -pub fn validate_model_entry(model: &str, value: &Value) -> Result<(), RegistryValidationError> { - validate_entry_inner(model, value).map_err(|reason| RegistryValidationError::Entry { - model: model.to_owned(), - reason, - }) -} - -/// Check every model and alias in a parsed catalog against registry rules. -pub fn validate_registry(catalog: &Catalog) -> Result<(), RegistryValidationError> { - if let Some(issue) = catalog.alias_issues().first() { - return Err(RegistryValidationError::Alias(issue.clone())); - } - catalog.model_names().try_for_each(|name| { - let entry = catalog.lookup(name).expect("catalog name must resolve"); - validate_model_entry(name, &Value::Object(entry.entry.fields().clone())) - }) -} - -fn json_eq(left: &Value, right: &Value) -> bool { - match (left, right) { - (Value::Number(left), Value::Number(right)) => left.as_f64() == right.as_f64(), - (Value::Array(left), Value::Array(right)) => { - left.len() == right.len() && left.iter().zip(right).all(|(a, b)| json_eq(a, b)) - } - (Value::Object(left), Value::Object(right)) => { - left.len() == right.len() - && left - .iter() - .all(|(key, value)| right.get(key).is_some_and(|other| json_eq(value, other))) - } - _ => left == right, - } -} - -fn keys(value: &Map) -> BTreeSet { - value.keys().cloned().collect() -} - -fn symmetric_difference(left: &BTreeSet, right: &BTreeSet) -> BTreeSet { - left.symmetric_difference(right).cloned().collect() -} - -fn validate_entry_inner(model_name: &str, value: &Value) -> Result<(), String> { - let object = value - .as_object() - .ok_or_else(|| format!("{model_name} must be an object"))?; - if let Some(aliases) = object.get("aliases") { - let names = aliases - .as_array() - .ok_or_else(|| format!("{model_name}.aliases must be an array"))?; - if names.iter().any(|name| !name.is_string()) { - return Err(format!("{model_name}.aliases must contain strings")); - } - } - let info: ModelInfo = - serde_json::from_value(value.clone()).map_err(|error| format!("{model_name}: {error}"))?; - if info.litellm_provider.is_none() { - return Err(format!("{model_name}.litellm_provider is required")); - } - validate_dates_and_windows(model_name, &info)?; - let serialized = serde_json::to_value(info).map_err(|error| error.to_string())?; - let mut expected = object.clone(); - expected.remove("aliases"); - if !json_eq(&Value::Object(expected.clone()), &serialized) { - let actual = serialized - .as_object() - .expect("ModelInfo serializes as an object"); - return Err(format!( - "{model_name} has an unknown field, null, or changed value: {:?}", - symmetric_difference(&keys(&expected), &keys(actual)) - )); - } - check_prices(model_name, value) -} - -fn validate_dates_and_windows(model_name: &str, info: &ModelInfo) -> Result<(), String> { - if let Some(date) = &info.deprecation_date { - let format = time::format_description::parse_borrowed::<2>("[year]-[month]-[day]").unwrap(); - time::Date::parse(date, &format) - .map_err(|error| format!("{model_name}.deprecation_date: {error}"))?; - } - let Some(pricing) = &info.off_peak_pricing else { - return Ok(()); - }; - if pricing.hours_utc.is_none() && pricing.windows.is_none() { - return Err(format!( - "{model_name}.off_peak_pricing needs hours or windows" - )); - } - if let Some(hours) = &pricing.hours_utc { - validate_hours(hours)?; - } - if let Some(windows) = &pricing.windows { - if windows.is_empty() { - return Err(format!("{model_name}.off_peak_pricing.windows is empty")); - } - windows.iter().try_for_each(|window| { - validate_hours(&window.hours_utc)?; - if let Some(days) = &window.weekdays - && (days.is_empty() || days.iter().any(|day| !valid_weekday(day))) - { - return Err(format!("{model_name}.off_peak_pricing.weekdays is invalid")); - } - Ok(()) - })?; - } - Ok(()) -} - -fn validate_hours(hours: &UtcHours) -> Result<(), String> { - let values = match hours { - UtcHours::Single(value) => std::slice::from_ref(value), - UtcHours::Multiple(values) => values.as_slice(), - }; - if values.is_empty() || values.iter().any(|value| !valid_utc_window(value)) { - return Err("off_peak_pricing.hours_utc is invalid".into()); - } - Ok(()) -} - -fn valid_utc_window(value: &str) -> bool { - let Some((start, end)) = value.split_once('-') else { - return false; - }; - [start, end].into_iter().all(|clock| { - let Some((hour, minute)) = clock.split_once(':') else { - return false; - }; - hour.len() == 2 - && minute.len() == 2 - && hour.parse::().is_ok_and(|hour| hour < 24) - && minute.parse::().is_ok_and(|minute| minute < 60) - }) -} - -fn valid_weekday(day: &Weekday) -> bool { - match day { - Weekday::Number(number) => (1..=7).contains(number), - Weekday::Name(name) => matches!( - name.to_ascii_lowercase().as_str(), - "mon" - | "monday" - | "tue" - | "tues" - | "tuesday" - | "wed" - | "wednesday" - | "thu" - | "thur" - | "thurs" - | "thursday" - | "fri" - | "friday" - | "sat" - | "saturday" - | "sun" - | "sunday" - ), - } -} - -fn check_prices(path: &str, value: &Value) -> Result<(), String> { - let Some(object) = value.as_object() else { - return Ok(()); - }; - object.iter().try_for_each(|(key, field)| { - let field_path = format!("{path}.{key}"); - if (key.contains("cost") - || path.ends_with(".guardrail_cost_per_unit") - || path.ends_with(".search_context_cost_per_query")) - && let Some(number) = field.as_f64() - && number < 0.0 - { - return Err(format!("{field_path} must be nonnegative")); - } - if key.contains("uplift_multiplier") - && let Some(number) = field.as_f64() - && number < 1.0 - { - return Err(format!("{field_path} must be at least one")); - } - if matches!(key.as_str(), "range" | "max_results_range") - && field.as_array().is_some_and(|values| { - values - .iter() - .any(|value| value.as_f64().is_some_and(|n| n < 0.0)) - }) - { - return Err(format!("{field_path} must be nonnegative")); - } - if matches!(key.as_str(), "metadata" | "provider_specific_entry") { - return Ok(()); - } - match field.as_array() { - Some(items) => items.iter().enumerate().try_for_each(|(index, item)| { - check_prices(&format!("{field_path}[{index}]"), item) - }), - None => check_prices(&field_path, field), - } - }) -} diff --git a/litellm-rust/crates/model-catalog/src/vocabulary.rs b/litellm-rust/crates/model-catalog/src/vocabulary.rs new file mode 100644 index 00000000000..6bf2cc55224 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/vocabulary.rs @@ -0,0 +1,132 @@ +/// Primary API surface / task type of the model. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum Mode { + AudioSpeech, + AudioTranscription, + Chat, + Completion, + Decisions, + Embedding, + Evaluation, + Extraction, + Guardrail, + ImageEdit, + ImageGeneration, + Moderation, + Ocr, + Realtime, + Rerank, + Responses, + Search, + VectorStore, + VideoGeneration, +} + +/// Gemini audio generation API the model is served through. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum VertexAiAudioApi { + LyriaPredict, + LyriaInteractions, +} + +/// Audio container format the model can return. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum AudioFormat { + Mp3, + Wav, +} + +/// Input modality the model accepts. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum InputModality { + Text, + Image, + Audio, + Video, +} + +/// Output modality the model can produce. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum OutputModality { + Text, + Image, + Audio, + Video, + Code, +} + +/// API route a model can be called through. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +pub enum Endpoint { + #[serde(rename = "/v1/chat/completions")] + ChatCompletions, + #[serde(rename = "/v1/completions")] + Completions, + #[serde(rename = "/v1/responses")] + Responses, + #[serde(rename = "/v1/messages")] + Messages, + #[serde(rename = "/v1/batch")] + Batch, + #[serde(rename = "/v1/embeddings")] + Embeddings, + #[serde(rename = "/v1/images/generations")] + ImageGenerations, + #[serde(rename = "/v1/images/edits")] + ImageEdits, + #[serde(rename = "/v1/audio/speech")] + AudioSpeech, + #[serde(rename = "/v1/audio/transcriptions")] + AudioTranscriptions, + #[serde(rename = "/v1/realtime")] + Realtime, + #[serde(rename = "/v1/realtime/transcription_sessions")] + RealtimeTranscriptionSessions, + #[serde(rename = "/v1/realtime/translations")] + RealtimeTranslations, + #[serde(rename = "/v1/realtime/translations/calls")] + RealtimeTranslationCalls, + #[serde(rename = "/v1/realtime/translations/client_secrets")] + RealtimeTranslationClientSecrets, + #[serde(rename = "/v1/videos")] + Videos, + #[serde(rename = "/v1/ocr")] + Ocr, + #[serde(rename = "/v1/listen")] + Listen, + #[serde(rename = "/v1/systemone")] + SystemOne, + #[serde(rename = "/v1/decisions")] + Decisions, + #[serde(rename = "/v1beta/interactions")] + Interactions, + #[serde(rename = "/v1beta/models")] + Models, + #[serde(rename = "/vertex_ai/live")] + VertexAiLive, +} + +/// Reasoning effort level a model accepts or applies. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum EffortLevel { + None, + Minimal, + Low, + Medium, + High, + Xhigh, + Max, +} diff --git a/litellm-rust/crates/model-catalog/src/wire/attributes.rs b/litellm-rust/crates/model-catalog/src/wire/attributes.rs new file mode 100644 index 00000000000..401634f8c62 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/attributes.rs @@ -0,0 +1,371 @@ +use std::collections::BTreeMap; +use std::fmt; +use std::str::FromStr; + +use litellm_cost::money::UnitPrice; +use serde::de::{self, Deserializer}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use crate::error::{DateCause, Error, Repr}; +use crate::vocabulary::{ + AudioFormat, EffortLevel, Endpoint, InputModality, Mode, OutputModality, VertexAiAudioApi, +}; +use crate::wire::lax::{Count, NumericCount}; + +macro_rules! attributes { + ($($(#[$doc:meta])* $name:ident: $ty:ty $(as $via:ty)?),* $(,)?) => { + /// The named, non-price fields of a catalog entry. + #[derive(Clone, Debug, Default, PartialEq)] + #[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] + pub struct Attributes { + $($(#[$doc])* pub $name: Option<$ty>,)* + } + + impl Attributes { + pub(crate) fn read( + &mut self, + key: &str, + value: &Value, + ) -> Option> { + match key { + $(stringify!($name) => Some( + attributes!(@read value, $ty $(, $via)?).map(|read| self.$name = Some(read)), + ),)* + _ => None, + } + } + } + }; + (@read $value:ident, $ty:ty) => { + <$ty as Deserialize>::deserialize($value) + }; + (@read $value:ident, $ty:ty, $via:ty) => { + serde_with::de::DeserializeAsWrap::<$ty, $via>::deserialize($value) + .map(serde_with::de::DeserializeAsWrap::into_inner) + }; +} + +attributes! { + /// Other names this entry is served under. + aliases: Vec, + audio_transcription_config: String, + bedrock_converse_supports_strict_tools: bool, + /// Highest reasoning effort the Bedrock output_config accepts for this model. + bedrock_output_config_effort_ceiling: EffortLevel, + comment: String, + /// Reasoning effort the provider applies when the request omits reasoning_effort. Gates whether a non-default temperature or the top_p/logprobs sampling params are accepted, which hold only when the effort resolves to 'none'. + default_reasoning_effort: EffortLevel, + /// Date the provider deprecates the model, YYYY-MM-DD. + deprecation_date: IsoDate, + gemini_audio_only_live: bool, + gemini_native_audio: bool, + input_cost_per_token_cache_hit: UnitPrice, + /// Databricks units, not USD. + input_dbu_cost_per_token: Dbu, + /// LiteLLM provider slug; one of https://docs.litellm.ai/docs/providers. + litellm_provider: String, + /// Maximum prompt/context tokens the model accepts. + max_input_tokens: u64 as Count, + /// Maximum tokens the model can generate in one response. + max_output_tokens: u64 as Count, + /// Legacy field: max output tokens if the provider specifies it, else max input tokens. + max_tokens: u64 as Count, + /// Free-form notes about the entry (e.g. pricing derivation). + metadata: BTreeMap, + /// Primary API surface / task type of the model. + mode: Mode, + /// Databricks units, not USD. + output_dbu_cost_per_token: Dbu, + /// Embedding dimension for embedding models. + output_vector_size: u64 as Count, + /// Smallest prefix the provider will actually cache; absent means the provider default applies. + prompt_cache_min_tokens: u64 as Count, + /// Provider-internal routing hints (e.g. bedrock_invocation_schema). + provider_specific_entry: BTreeMap, + reasoning_contexts: Vec, + /// Exact reasoning_effort levels this deployment accepts; wins over supports_* flags. + reasoning_effort_levels: Vec, + reasoning_modes: Vec, + /// Provider default requests-per-minute limit. + rpm: u64 as Count, + /// Grounded search requests the provider includes for free before search is billed. + search_free_requests: SearchFreeRequests, + /// URL of the provider pricing/model page this entry was taken from. + source: String, + /// Model the provider recommends migrating to before `deprecation_date`, as a catalog key when one exists. + successor_model: String, + /// URL of the provider page or feed `deprecation_date` was read from. + deprecation_source: String, + /// Audio container formats the model can return. + supported_audio_formats: Vec, + /// OpenAI-style API routes this model can be called through, e.g. /v1/chat/completions. + supported_endpoints: Vec, + /// Input modalities the model accepts. + supported_modalities: Vec, + /// Output modalities the model can produce. + supported_output_modalities: Vec, + /// Cloud regions the model is available in ('global' or region ids). + supported_regions: Vec, + thinking_always_on: bool, + /// Whether the model thinks when the request carries no thinking param. + thinking_on_by_default: bool, + token_count_framing: u64 as NumericCount, + token_count_model: TokenCountModel, + token_count_responses_framing: u64 as NumericCount, + token_count_responses_tool_framing: u64 as NumericCount, + token_count_tokenizer: TokenCountTokenizer, + token_count_tool_framing: u64 as NumericCount, + /// Provider default tokens-per-minute limit. + tpm: u64 as Count, + use_openai_responses_path: bool, + uses_embed_content: bool, + vertex_ai_audio_api: VertexAiAudioApi, + /// Whether web search is billed per query or per prompt. + web_search_billing_unit: WebSearchBillingUnit, +} + +/// The model whose provider token counter counts tokens for this entry. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Eq, Hash)] +#[serde(deny_unknown_fields)] +pub struct TokenCountModel { + pub model: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub litellm_provider: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub region: Option, +} + +/// Free grounded search requests per period, shared across the models of one group. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Eq, Hash)] +#[serde(deny_unknown_fields)] +pub struct SearchFreeRequests { + pub count: u64, + pub period: FreeRequestPeriod, + pub shared_group: String, +} + +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum FreeRequestPeriod { + Day, + Month, +} + +/// Local tokenizer that counts tokens for this entry. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum TokenCountTokenizer { + KimiK3, +} + +/// Whether web search is billed per query or per prompt. +#[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Copy, Eq, Hash)] +#[serde(rename_all = "snake_case")] +pub enum WebSearchBillingUnit { + PerQuery, + PerPrompt, +} + +/// A non-negative amount of Databricks units. +#[derive(Clone, Copy, Debug, PartialEq, PartialOrd, Serialize)] +#[serde(transparent)] +pub struct Dbu(f64); + +impl Dbu { + pub fn new(units: f64) -> Option { + (units.is_finite() && units >= 0.0).then_some(Self(units)) + } + pub fn get(self) -> f64 { + self.0 + } +} + +impl<'de> Deserialize<'de> for Dbu { + fn deserialize>(deserializer: D) -> Result { + let units = f64::deserialize(deserializer)?; + Self::new(units).ok_or_else(|| { + de::Error::invalid_value( + de::Unexpected::Float(units), + &"a non-negative number of Databricks units", + ) + }) + } +} + +/// A calendar date written YYYY-MM-DD. +#[derive( + Clone, + Copy, + Debug, + PartialEq, + Eq, + PartialOrd, + Ord, + Hash, + serde_with::DeserializeFromStr, + serde_with::SerializeDisplay, +)] +pub struct IsoDate(pub time::Date); + +impl FromStr for IsoDate { + type Err = Error; + + fn from_str(date: &str) -> Result { + let mut parts = date.split('-'); + let (Some(year), Some(month), Some(day), None) = + (parts.next(), parts.next(), parts.next(), parts.next()) + else { + return Err(Repr::InvalidDate { cause: None }.into()); + }; + if year.len() != 4 || month.len() != 2 || day.len() != 2 { + return Err(Repr::InvalidDate { cause: None }.into()); + } + let number = |part: &str| part.parse::().map_err(DateCause::from); + let month = time::Month::try_from(number(month)?).map_err(DateCause::from)?; + time::Date::from_calendar_date(year.parse().map_err(DateCause::from)?, month, number(day)?) + .map(Self) + .map_err(DateCause::from) + .map_err(Error::from) + } +} + +impl fmt::Display for IsoDate { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{:04}-{:02}-{:02}", + self.0.year(), + u8::from(self.0.month()), + self.0.day() + ) + } +} + +#[cfg(feature = "schema")] +mod schema { + use std::borrow::Cow; + + use schemars::{JsonSchema, Schema, SchemaGenerator, json_schema}; + + use super::{Dbu, IsoDate}; + + impl JsonSchema for IsoDate { + fn inline_schema() -> bool { + true + } + fn schema_name() -> Cow<'static, str> { + "IsoDate".into() + } + fn json_schema(_: &mut SchemaGenerator) -> Schema { + json_schema!({ + "type": "string", + "format": "date", + "pattern": r"^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$" + }) + } + } + + impl JsonSchema for Dbu { + fn inline_schema() -> bool { + true + } + fn schema_name() -> Cow<'static, str> { + "Dbu".into() + } + fn json_schema(_: &mut SchemaGenerator) -> Schema { + json_schema!({"type": "number", "minimum": 0}) + } + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + #[rstest] + #[case::leap_day("2028-02-29", true)] + #[case::not_a_leap_year("2026-02-29", false)] + #[case::impossible_day("2026-02-31", false)] + #[case::one_digit_month("2026-2-01", false)] + #[case::extra_part("2026-02-01-01", false)] + fn iso_dates_are_real_calendar_dates(#[case] text: &str, #[case] valid: bool) { + assert_eq!(text.parse::().is_ok(), valid); + } + + #[rstest] + #[case::count_written_as_text("max_tokens", json!("8192"), true)] + #[case::negative_count("max_tokens", json!(-1), false)] + #[case::known_mode("mode", json!("chat"), true)] + #[case::unknown_mode("mode", json!("chatt"), false)] + #[case::negative_dbu("input_dbu_cost_per_token", json!(-1), false)] + #[case::successor_model("successor_model", json!("gpt-6-luna"), true)] + #[case::successor_model_not_a_name("successor_model", json!(["gpt-6-luna"]), false)] + #[case::deprecation_source("deprecation_source", json!("https://developers.openai.com/api/docs/deprecations"), true)] + #[case::deprecation_source_not_a_url_string("deprecation_source", json!(7), false)] + #[case::thinking_on_by_default("thinking_on_by_default", json!(true), true)] + #[case::thinking_on_by_default_not_a_flag("thinking_on_by_default", json!("yes"), false)] + #[case::reasoning_contexts("reasoning_contexts", json!(["reasoning", "non_reasoning"]), true)] + #[case::reasoning_modes("reasoning_modes", json!(["none", "adaptive"]), true)] + #[case::reasoning_context_not_a_list("reasoning_contexts", json!("reasoning"), false)] + #[case::reasoning_mode_not_a_string("reasoning_modes", json!([1]), false)] + #[case::framing("token_count_framing", json!(12), true)] + #[case::framing_written_as_text("token_count_framing", json!("12"), false)] + #[case::framing_written_as_whole_float("token_count_framing", json!(12.0), true)] + #[case::fractional_tool_framing("token_count_tool_framing", json!(1.5), false)] + #[case::tool_framing("token_count_tool_framing", json!(214), true)] + #[case::tool_framing_written_as_text("token_count_tool_framing", json!("214"), false)] + #[case::tool_framing_as_a_flag("token_count_tool_framing", json!(true), false)] + #[case::negative_tool_framing("token_count_tool_framing", json!(-1), false)] + #[case::responses_framing("token_count_responses_framing", json!(38), true)] + #[case::responses_framing_as_a_flag("token_count_responses_framing", json!(false), false)] + #[case::responses_tool_framing("token_count_responses_tool_framing", json!(267), true)] + #[case::responses_tool_framing_written_as_text("token_count_responses_tool_framing", json!("267"), false)] + fn each_attribute_reads_its_own_type( + #[case] key: &str, + #[case] value: Value, + #[case] valid: bool, + ) { + assert_eq!( + Attributes::default() + .read(key, &value) + .map(|read| read.is_ok()), + Some(valid) + ); + } + + #[rstest] + fn keys_that_are_not_attributes_are_left_to_the_caller() { + assert!( + Attributes::default() + .read("input_cost_per_token", &json!(1)) + .is_none() + ); + } + + #[rstest] + fn reasoning_metadata_is_retained() { + let mut attributes = Attributes::default(); + attributes + .read("reasoning_contexts", &json!(["reasoning", "non_reasoning"])) + .unwrap() + .unwrap(); + attributes + .read("reasoning_modes", &json!(["none", "adaptive"])) + .unwrap() + .unwrap(); + assert_eq!( + attributes.reasoning_contexts.unwrap(), + ["reasoning", "non_reasoning"] + ); + assert_eq!(attributes.reasoning_modes.unwrap(), ["none", "adaptive"]); + } +} diff --git a/litellm-rust/crates/model-catalog/src/wire/blocks.rs b/litellm-rust/crates/model-catalog/src/wire/blocks.rs new file mode 100644 index 00000000000..3a355686b7c --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/blocks.rs @@ -0,0 +1,182 @@ +use litellm_cost::money::UnitPrice; +use litellm_cost::schedule::{UtcWindow, Weekday}; +use serde::Deserialize; +use serde::de::DeserializeOwned; +use serde_json::{Map, Value}; +use serde_with::de::DeserializeAsWrap; + +use crate::error::{Error, Repr, pointer_child, pointer_for_key}; +use crate::wire::flat_key::FlatKey; +use crate::wire::lax::Count; + +/// A flat price key with its spelling, kept so problems can name it. +#[derive(Clone, Debug, PartialEq)] +pub struct FlatPrice { + pub key: String, + pub path: String, + pub flat: FlatKey, + pub price: UnitPrice, +} + +#[derive(Clone, Debug, Default, PartialEq)] +pub struct TierRow { + pub path: String, + pub range: Option<[u64; 2]>, + pub max_results_range: Option<[u64; 2]>, + pub prices: Vec, +} + +#[derive(Clone, Debug, Default, PartialEq)] +pub struct OffPeakBlock { + pub hours_utc: Option, + pub windows: Option>, + pub weekday_timezone: Option, + pub prices: Vec, +} + +/// One UTC window or a list of them. +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[serde(untagged)] +pub enum UtcHours { + Single(UtcWindow), + Multiple(Vec), +} + +/// One off-peak window entry inside `windows`. +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[serde(deny_unknown_fields)] +pub struct OffPeakWindow { + pub hours_utc: UtcHours, + pub weekdays: Option>, +} + +/// USD cost per web search query, keyed by search context size. +#[derive(Clone, Debug, Default, PartialEq, Deserialize)] +#[cfg_attr(feature = "schema", derive(schemars::JsonSchema))] +#[serde(deny_unknown_fields)] +pub struct SearchContextCostPerQuery { + pub search_context_size_low: Option, + pub search_context_size_medium: Option, + pub search_context_size_high: Option, +} + +impl UtcHours { + pub fn windows(&self) -> &[UtcWindow] { + match self { + Self::Single(window) => std::slice::from_ref(window), + Self::Multiple(windows) => windows, + } + } +} + +pub(crate) struct Reader<'a> { + pub(crate) problems: &'a mut Vec, +} + +impl Reader<'_> { + pub(crate) fn value(&mut self, path: String, value: &Value) -> Option { + T::deserialize(value) + .map_err(|error| self.problems.push(Error::new(path, Repr::Invalid(error)))) + .ok() + } + + pub(crate) fn price( + &mut self, + key: String, + path: String, + flat: FlatKey, + value: &Value, + ) -> Option { + let price = self.value(path.clone(), value)?; + Some(FlatPrice { + key, + path, + flat, + price, + }) + } + + fn object<'v>(&mut self, path: &str, value: &'v Value) -> Option<&'v Map> { + match value { + Value::Object(fields) => Some(fields), + other => self + .value::>(path.to_owned(), other) + .and(None), + } + } + + fn block_price( + &mut self, + name: &str, + key: String, + path: String, + value: &Value, + ) -> Option { + let Some(flat) = FlatKey::parse(name) else { + self.problems.push(Error::new(path, Repr::UnknownBlockKey)); + return None; + }; + self.price(key, path, flat, value) + } + + pub(crate) fn tier_rows(&mut self, value: &Value) -> Option> { + let path = pointer_for_key("tiered_pricing"); + let rows: Vec = self.value(path.clone(), value)?; + Some( + rows.iter() + .enumerate() + .filter_map(|(index, row)| self.tier_row(path.clone(), index, row)) + .collect(), + ) + } + + fn tier_row(&mut self, parent: String, index: usize, value: &Value) -> Option { + let path = format!("{parent}/{index}"); + let display_path = format!("tiered_pricing[{index}]"); + let fields = self.object(&path, value)?; + let mut row = TierRow { + path: path.clone(), + ..TierRow::default() + }; + for (name, value) in fields.iter().filter(|(_, value)| !value.is_null()) { + let key = format!("{display_path}.{name}"); + let child_path = pointer_child(&path, name); + match name.as_str() { + "range" => row.range = self.bounds(child_path, value), + "max_results_range" => row.max_results_range = self.bounds(child_path, value), + _ => row + .prices + .extend(self.block_price(name, key, child_path, value)), + } + } + Some(row) + } + + fn bounds(&mut self, path: String, value: &Value) -> Option<[u64; 2]> { + DeserializeAsWrap::<[u64; 2], [Count; 2]>::deserialize(value) + .map(DeserializeAsWrap::into_inner) + .map_err(|error| self.problems.push(Error::new(path, Repr::Invalid(error)))) + .ok() + } + + pub(crate) fn off_peak(&mut self, value: &Value) -> Option { + let path = pointer_for_key("off_peak_pricing"); + let fields = self.object(&path, value)?; + let mut block = OffPeakBlock::default(); + for (name, value) in fields.iter().filter(|(_, value)| !value.is_null()) { + let key = format!("off_peak_pricing.{name}"); + let child_path = pointer_child(&path, name); + match name.as_str() { + "hours_utc" => block.hours_utc = self.value(child_path, value), + "windows" => block.windows = self.value(child_path, value), + "weekday_timezone" => block.weekday_timezone = self.value(child_path, value), + _ => block + .prices + .extend(self.block_price(name, key, child_path, value)), + } + } + Some(block) + } +} diff --git a/litellm-rust/crates/model-catalog/src/wire/entry.rs b/litellm-rust/crates/model-catalog/src/wire/entry.rs new file mode 100644 index 00000000000..d0bdb394720 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/entry.rs @@ -0,0 +1,195 @@ +use std::collections::BTreeMap; + +use litellm_cost::money::{Multiplier, UnitPrice}; +use serde_json::{Map, Value}; + +use crate::error::{Error, Repr, pointer_for_key}; +use crate::wire::attributes::Attributes; +use crate::wire::blocks::{FlatPrice, OffPeakBlock, Reader, SearchContextCostPerQuery, TierRow}; +use crate::wire::flat_key::FlatKey; + +pub type Fields = Map; + +/// One catalog entry as written. Every non-null key lands in exactly one field, and a known key +/// with a bad value is kept out and listed in `problems`. +#[derive(Debug, Default)] +pub struct EntryWire { + pub attributes: Attributes, + pub prices: Vec, + pub capabilities: BTreeMap, + pub tier_rows: Option>, + pub off_peak: Option, + pub search_context: Option, + pub guardrail: Option>, + pub regional_endpoint_uplift: Option, + pub residency_uplift_eu: Option, + pub residency_uplift_us: Option, + pub fedramp_uplift: Option, + pub unknown: Fields, + pub problems: Vec, +} + +impl EntryWire { + pub fn read(fields: &Fields) -> Self { + let mut problems = Vec::new(); + let mut wire = Self::default(); + let mut reader = Reader { + problems: &mut problems, + }; + for (key, value) in fields.iter().filter(|(_, value)| !value.is_null()) { + wire.read_key(&mut reader, key, value); + } + Self { problems, ..wire } + } + + fn read_key(&mut self, reader: &mut Reader<'_>, key: &str, value: &Value) { + let path = || pointer_for_key(key); + match key { + "tiered_pricing" => self.tier_rows = reader.tier_rows(value), + "off_peak_pricing" => self.off_peak = reader.off_peak(value), + "search_context_cost_per_query" => self.search_context = reader.value(path(), value), + "guardrail_cost_per_unit" => self.guardrail = reader.value(path(), value), + "regional_endpoint_uplift_multiplier" => { + self.regional_endpoint_uplift = reader.value(path(), value) + } + "regional_processing_uplift_multiplier_eu" => { + self.residency_uplift_eu = reader.value(path(), value) + } + "regional_processing_uplift_multiplier_us" => { + self.residency_uplift_us = reader.value(path(), value) + } + "fedramp_uplift_multiplier" => self.fedramp_uplift = reader.value(path(), value), + _ => self.read_open_key(reader, key, value), + } + } + + fn read_open_key(&mut self, reader: &mut Reader<'_>, key: &str, value: &Value) { + if let Some(read) = self.attributes.read(key, value) { + if let Err(error) = read { + reader + .problems + .push(Error::from_key(key, Repr::Invalid(error))); + } + return; + } + if let Some(flat) = FlatKey::parse(key) { + self.prices + .extend(reader.price(key.to_owned(), pointer_for_key(key), flat, value)); + return; + } + if key.starts_with("supports_") { + if let Some(flag) = reader.value(pointer_for_key(key), value) { + self.capabilities.insert(key.to_owned(), flag); + } + return; + } + self.unknown.insert(key.to_owned(), value.clone()); + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn read(entry: Value) -> EntryWire { + let Value::Object(fields) = entry else { + panic!("entry must be an object") + }; + EntryWire::read(&fields) + } + + fn problem_keys(wire: &EntryWire) -> Vec { + wire.problems + .iter() + .map(|problem| problem.path().to_owned()) + .collect() + } + + #[rstest] + fn every_key_lands_in_exactly_one_place() { + let wire = read(json!({ + "litellm_provider": "openai", + "max_tokens": "8192", + "input_cost_per_token": "0.000001", + "tiered_pricing": [{"range": [0, 10], "output_cost_per_token": 2e-6}], + "supports_vision": true, + "id": "deployment-1", + "mode": null + })); + + assert_eq!(wire.attributes.litellm_provider.as_deref(), Some("openai")); + assert_eq!(wire.attributes.max_tokens, Some(8192)); + assert_eq!(wire.attributes.mode, None); + assert_eq!( + wire.prices + .iter() + .map(|price| price.key.as_str()) + .collect::>(), + ["input_cost_per_token"] + ); + assert_eq!(wire.tier_rows.unwrap()[0].prices.len(), 1); + assert_eq!( + wire.capabilities, + BTreeMap::from([("supports_vision".to_owned(), true)]) + ); + assert_eq!(wire.unknown.keys().collect::>(), ["id"]); + assert!(wire.problems.is_empty()); + } + + #[rstest] + fn every_bad_value_is_reported_by_its_key_at_once() { + let wire = read(json!({ + "mode": "chatt", + "max_tokens": -1, + "input_cost_per_token": "x", + "output_cost_per_image_0.5K": "x", + "supports_vision": "yes", + "regional_endpoint_uplift_multiplier": 0.5, + "tiered_pricing": [ + {"range": [0, 10], "input_cost_per_token": 1e-6}, + {"range": [10.5, 20], "typo_cost": 1, "output_cost_per_token": -1} + ], + "off_peak_pricing": {"hours_utc": "25:00-01:00", "input_cost_per_token": 1e-6} + })); + + assert_eq!( + problem_keys(&wire) + .into_iter() + .collect::>(), + [ + "/input_cost_per_token", + "/max_tokens", + "/mode", + "/off_peak_pricing/hours_utc", + "/output_cost_per_image_0.5K", + "/regional_endpoint_uplift_multiplier", + "/supports_vision", + "/tiered_pricing/1/output_cost_per_token", + "/tiered_pricing/1/range", + "/tiered_pricing/1/typo_cost", + ] + .map(str::to_owned) + .into_iter() + .collect() + ); + } + + #[rstest] + #[case::tiers_not_a_list(json!({"tiered_pricing": {"range": [0, 1]}}), "/tiered_pricing")] + #[case::row_not_an_object(json!({"tiered_pricing": [3]}), "/tiered_pricing/0")] + #[case::off_peak_not_an_object(json!({"off_peak_pricing": true}), "/off_peak_pricing")] + #[case::unknown_search_size( + json!({"search_context_cost_per_query": {"search_context_size_huge": 1}}), + "/search_context_cost_per_query" + )] + #[case::negative_guardrail( + json!({"guardrail_cost_per_unit": {"contentPolicyUnits": -1}}), + "/guardrail_cost_per_unit" + )] + fn a_bad_block_is_reported_at_the_block(#[case] entry: Value, #[case] expected: &str) { + assert_eq!(problem_keys(&read(entry)), [expected]); + } +} diff --git a/litellm-rust/crates/model-catalog/src/wire/flat_key.rs b/litellm-rust/crates/model-catalog/src/wire/flat_key.rs new file mode 100644 index 00000000000..906d917b487 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/flat_key.rs @@ -0,0 +1,359 @@ +use std::num::NonZeroU64; + +use litellm_cost::price::{ + Band, CacheReadMode, CacheTtl, Channel, Feature, ImageSize, InputUnit, Meter, OutputUnit, + Resolution, ServiceTier, TokenKind, When, +}; + +/// A flat price key as the grammar reads it: `{meter}[_above_{N}k_tokens][_{tier}]`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FlatKey { + pub meter: Meter, + pub band: Band, + pub tier: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum TierSuffix { + Service(ServiceTier), + Batches, + Decisions, +} + +impl FlatKey { + pub fn parse(key: &str) -> Option { + let (rest, tier) = split_tier(key); + let (base, suffix) = split_above(rest)?; + let (meter, band) = match suffix { + Above::None => (meter(base)?, Band::Base), + Above::PromptTokens(tokens) => (meter(base)?, Band::over_prompt_tokens(tokens)), + Above::Seconds(seconds) => match meter(base)? { + Meter::Input(InputUnit::VideoSecond { over_seconds: None }) => ( + Meter::Input(InputUnit::VideoSecond { + over_seconds: NonZeroU64::new(seconds), + }), + Band::Base, + ), + _ => return None, + }, + }; + Some(Self { meter, band, tier }) + } + + pub fn has_suffix(&self) -> bool { + self.band != Band::Base || self.tier.is_some() + } + + pub fn channel_and_when(&self) -> (Channel, When) { + let (channel, tier) = match self.tier { + None => (Channel::Online, ServiceTier::Standard), + Some(TierSuffix::Service(tier)) => (Channel::Online, tier), + Some(TierSuffix::Batches) => (Channel::Batch, ServiceTier::Standard), + Some(TierSuffix::Decisions) => (Channel::Decisions, ServiceTier::Standard), + }; + ( + channel, + When { + tier, + band: self.band, + }, + ) + } +} + +enum Above { + None, + PromptTokens(u64), + Seconds(u64), +} + +fn split_tier(key: &str) -> (&str, Option) { + let Some((head, tail)) = key.rsplit_once('_') else { + return (key, None); + }; + let tier = match tail { + "priority" => TierSuffix::Service(ServiceTier::Priority), + "flex" => TierSuffix::Service(ServiceTier::Flex), + "ultrafast" => TierSuffix::Service(ServiceTier::Ultrafast), + "balanced" => TierSuffix::Service(ServiceTier::Balanced), + "batches" => TierSuffix::Batches, + "decisions" => TierSuffix::Decisions, + _ => return (key, None), + }; + (head, Some(tier)) +} + +fn split_above(key: &str) -> Option<(&str, Above)> { + let Some((head, tail)) = key.rsplit_once("_above_") else { + return Some((key, Above::None)); + }; + if tail == "1hr" || tail == "12hr" { + return Some((key, Above::None)); + } + if let Some(thousands) = tail.strip_suffix("k_tokens") { + return Some(( + head, + Above::PromptTokens(number(thousands)?.checked_mul(1000)?), + )); + } + if let Some(tokens) = tail.strip_suffix("_tokens") { + return Some((head, Above::PromptTokens(number(tokens)?))); + } + let seconds = tail.strip_suffix("s_interval")?; + Some((head, Above::Seconds(number(seconds)?))) +} + +fn number(digits: &str) -> Option { + (!digits.is_empty() && digits.bytes().all(|byte| byte.is_ascii_digit())) + .then(|| digits.parse().ok()) + .flatten() +} + +fn meter(base: &str) -> Option { + let fixed = match base { + "cost_per_second" => Some(Meter::WallClockSecond), + "explicit_cache_read_input_token_cost" => Some(Meter::CacheRead { + kind: TokenKind::Text, + mode: CacheReadMode::Explicit, + }), + "annotation_cost_per_page" => Some(Meter::Feature(Feature::AnnotationPage)), + "citation_cost_per_token" => Some(Meter::Feature(Feature::CitationToken)), + "code_interpreter_cost_per_session" => { + Some(Meter::Feature(Feature::CodeInterpreterSession)) + } + "google_maps_grounding_cost_per_query" => { + Some(Meter::Feature(Feature::GoogleMapsGroundingQuery)) + } + "ocr_cost_per_credit" => Some(Meter::Feature(Feature::OcrCredit)), + "ocr_cost_per_page" => Some(Meter::Feature(Feature::OcrPage)), + "browser_open_cost_per_request" => Some(Meter::Feature(Feature::BrowserOpen)), + "cost_per_additional_result" => Some(Meter::Feature(Feature::AdditionalResult)), + _ => None, + }; + if fixed.is_some() { + return fixed; + } + if let Some(unit) = base.strip_prefix("input_cost_per_") { + return input_unit(unit).map(Meter::Input); + } + if let Some(unit) = base.strip_prefix("output_cost_per_") { + return output_unit(unit).map(Meter::Output); + } + if let Some(kind) = base + .strip_prefix("cache_read_input_") + .and_then(|rest| rest.strip_suffix("_cost")) + { + return Some(Meter::CacheRead { + kind: token_kind(kind)?, + mode: CacheReadMode::Implicit, + }); + } + let rest = base.strip_prefix("cache_creation_input_")?; + let (kind, ttl) = if let Some(kind) = rest.strip_suffix("_cost_above_1hr") { + (kind, CacheTtl::OneHour) + } else if let Some(kind) = rest.strip_suffix("_cost_above_12hr") { + (kind, CacheTtl::TwelveHours) + } else { + (rest.strip_suffix("_cost")?, CacheTtl::Default) + }; + Some(Meter::CacheWrite { + kind: token_kind(kind)?, + ttl, + }) +} + +fn token_kind(unit: &str) -> Option { + match unit { + "token" => Some(TokenKind::Text), + "audio_token" => Some(TokenKind::Audio), + "image_token" => Some(TokenKind::Image), + "video_token" => Some(TokenKind::Video), + _ => None, + } +} + +fn input_unit(unit: &str) -> Option { + if let Some(kind) = token_kind(unit) { + return Some(InputUnit::Token(kind)); + } + match unit { + "character" => Some(InputUnit::Character), + "image" => Some(InputUnit::Image), + "pixel" => Some(InputUnit::Pixel), + "query" => Some(InputUnit::Query { over_results: None }), + "request" => Some(InputUnit::Request), + "second" => Some(InputUnit::Second), + "text_message" => Some(InputUnit::TextMessage), + "audio_per_second" => Some(InputUnit::AudioSecond), + "video_per_second" => Some(InputUnit::VideoSecond { over_seconds: None }), + _ => None, + } +} + +fn output_unit(unit: &str) -> Option { + if let Some(kind) = token_kind(unit) { + return Some(OutputUnit::Token(kind)); + } + match unit.split_once('_') { + Some(("image", label)) => { + return ImageSize::new(label).map(|size| OutputUnit::Image(Some(size))); + } + Some(("second", label)) => { + return Resolution::new(label).map(|resolution| OutputUnit::Second(Some(resolution))); + } + _ => {} + } + match unit { + "reasoning_token" => Some(OutputUnit::ReasoningToken), + "character" => Some(OutputUnit::Character), + "image" => Some(OutputUnit::Image(None)), + "pixel" => Some(OutputUnit::Pixel), + "second" => Some(OutputUnit::Second(None)), + "video_per_second" => Some(OutputUnit::VideoSecond), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + const TEXT_INPUT: Meter = Meter::Input(InputUnit::Token(TokenKind::Text)); + const TEXT_OUTPUT: Meter = Meter::Output(OutputUnit::Token(TokenKind::Text)); + + fn key(meter: Meter, band: Band, tier: Option) -> FlatKey { + FlatKey { meter, band, tier } + } + + #[rstest] + #[case::input_token("input_cost_per_token", key(TEXT_INPUT, Band::Base, None))] + #[case::band_and_tier( + "output_cost_per_token_above_272k_tokens_priority", + key( + TEXT_OUTPUT, + Band::over_prompt_tokens(272_000), + Some(TierSuffix::Service(ServiceTier::Priority)) + ) + )] + #[case::band_without_k( + "input_cost_per_token_above_128_tokens", + key(TEXT_INPUT, Band::over_prompt_tokens(128), None) + )] + #[case::zero_band_is_base( + "input_cost_per_token_above_0k_tokens", + key(TEXT_INPUT, Band::Base, None) + )] + #[case::batch( + "input_cost_per_token_batches", + key(TEXT_INPUT, Band::Base, Some(TierSuffix::Batches)) + )] + #[case::reasoning( + "output_cost_per_reasoning_token", + key(Meter::Output(OutputUnit::ReasoningToken), Band::Base, None) + )] + #[case::cache_read_audio( + "cache_read_input_audio_token_cost", + key(Meter::CacheRead { kind: TokenKind::Audio, mode: CacheReadMode::Implicit }, Band::Base, None) + )] + #[case::explicit_cache_read( + "explicit_cache_read_input_token_cost", + key(Meter::CacheRead { kind: TokenKind::Text, mode: CacheReadMode::Explicit }, Band::Base, None) + )] + #[case::cache_write_one_hour_band( + "cache_creation_input_token_cost_above_1hr_above_200k_tokens", + key( + Meter::CacheWrite { kind: TokenKind::Text, ttl: CacheTtl::OneHour }, + Band::over_prompt_tokens(200_000), + None + ) + )] + #[case::cache_write_twelve_hours_priority( + "cache_creation_input_token_cost_above_12hr_above_272k_tokens_priority", + key( + Meter::CacheWrite { kind: TokenKind::Text, ttl: CacheTtl::TwelveHours }, + Band::over_prompt_tokens(272_000), + Some(TierSuffix::Service(ServiceTier::Priority)) + ) + )] + #[case::image_label( + "output_cost_per_image_0.5K", + key( + Meter::Output(OutputUnit::Image(ImageSize::new("0.5K"))), + Band::Base, + None + ) + )] + #[case::image_token_is_a_token( + "output_cost_per_image_token", + key(Meter::Output(OutputUnit::Token(TokenKind::Image)), Band::Base, None) + )] + #[case::resolution( + "output_cost_per_second_720p", + key( + Meter::Output(OutputUnit::Second(Resolution::new("720p"))), + Band::Base, + None + ) + )] + #[case::video_interval( + "input_cost_per_video_per_second_above_8s_interval", + key(Meter::Input(InputUnit::VideoSecond { over_seconds: NonZeroU64::new(8) }), Band::Base, None) + )] + #[case::feature_batch( + "ocr_cost_per_page_batches", + key( + Meter::Feature(Feature::OcrPage), + Band::Base, + Some(TierSuffix::Batches) + ) + )] + #[case::wall_clock("cost_per_second", key(Meter::WallClockSecond, Band::Base, None))] + #[case::browser_open( + "browser_open_cost_per_request", + key(Meter::Feature(Feature::BrowserOpen), Band::Base, None) + )] + #[case::additional_result( + "cost_per_additional_result", + key(Meter::Feature(Feature::AdditionalResult), Band::Base, None) + )] + fn parses_every_key_shape(#[case] text: &str, #[case] expected: FlatKey) { + assert_eq!(FlatKey::parse(text), Some(expected)); + } + + #[rstest] + #[case::typo("input_cost_per_tokne")] + #[case::unused_meter("cost_per_token")] + #[case::other_currency("input_dbu_cost_per_token")] + #[case::cache_hit_alias("input_cost_per_token_cache_hit")] + #[case::non_numeric_band("input_cost_per_token_above_xk_tokens")] + #[case::interval_on_a_token_meter("input_cost_per_token_above_8s_interval")] + #[case::read_has_no_ttl("cache_read_input_token_cost_above_1hr")] + #[case::cache_unit_must_be_token("cache_read_input_character_cost")] + #[case::bad_image_size("output_cost_per_image_big")] + #[case::bad_resolution("output_cost_per_second_hd")] + #[case::unknown_feature("telepathy_cost_per_page")] + #[case::empty("")] + fn rejects_keys_outside_the_grammar(#[case] text: &str) { + assert_eq!(FlatKey::parse(text), None); + } + + #[rstest] + #[case::plain("input_cost_per_token", (Channel::Online, When::default()))] + #[case::service_tier( + "input_cost_per_token_flex", + (Channel::Online, When { tier: ServiceTier::Flex, band: Band::Base }) + )] + #[case::batch( + "input_cost_per_token_above_200k_tokens_batches", + (Channel::Batch, When { tier: ServiceTier::Standard, band: Band::over_prompt_tokens(200_000) }) + )] + #[case::decisions("input_cost_per_token_decisions", (Channel::Decisions, When::default()))] + fn batch_and_decisions_pick_a_channel_not_a_tier( + #[case] text: &str, + #[case] expected: (Channel, When), + ) { + assert_eq!(FlatKey::parse(text).unwrap().channel_and_when(), expected); + } +} diff --git a/litellm-rust/crates/model-catalog/src/wire/lax.rs b/litellm-rust/crates/model-catalog/src/wire/lax.rs new file mode 100644 index 00000000000..d98a601e8c1 --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/lax.rs @@ -0,0 +1,183 @@ +use std::fmt; + +use serde::Deserializer; +use serde::de::{self, Visitor}; +use serde_with::DeserializeAs; + +pub(crate) struct LaxI64; + +pub(crate) struct Count; + +pub(crate) struct NumericCount; + +impl<'de> DeserializeAs<'de, i64> for LaxI64 { + fn deserialize_as>(deserializer: D) -> Result { + deserializer.deserialize_any(Self) + } +} + +impl Visitor<'_> for LaxI64 { + type Value = i64; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("an integer in the i64 range") + } + + fn visit_i64(self, value: i64) -> Result { + Ok(value) + } + + fn visit_u64(self, value: u64) -> Result { + i64::try_from(value).map_err(E::custom) + } + + fn visit_f64(self, value: f64) -> Result { + integral_float(value).ok_or_else(|| E::custom("expected an integer in the i64 range")) + } + + fn visit_str(self, value: &str) -> Result { + integer_string(value.trim()) + .ok_or_else(|| E::custom("expected an integer in the i64 range")) + } + + fn visit_bool(self, value: bool) -> Result { + Ok(i64::from(value)) + } +} + +impl<'de> DeserializeAs<'de, u64> for Count { + fn deserialize_as>(deserializer: D) -> Result { + let count = LaxI64::deserialize_as(deserializer)?; + u64::try_from(count).map_err(|_| { + de::Error::invalid_value(de::Unexpected::Signed(count), &"a count of zero or more") + }) + } +} + +impl<'de> DeserializeAs<'de, u64> for NumericCount { + fn deserialize_as>(deserializer: D) -> Result { + deserializer.deserialize_any(Self) + } +} + +impl Visitor<'_> for NumericCount { + type Value = u64; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("a whole number of zero or more") + } + + fn visit_u64(self, value: u64) -> Result { + Ok(value) + } + + fn visit_i64(self, value: i64) -> Result { + u64::try_from(value).map_err(|_| E::invalid_value(de::Unexpected::Signed(value), &self)) + } + + fn visit_f64(self, value: f64) -> Result { + integral_float(value) + .and_then(|whole| u64::try_from(whole).ok()) + .ok_or_else(|| E::invalid_value(de::Unexpected::Float(value), &self)) + } +} + +fn integer_string(value: &str) -> Option { + let integer = match value.split_once('.') { + Some((integer, fraction)) => { + if fraction.is_empty() || !fraction.bytes().all(|byte| byte == b'0') { + return None; + } + integer + } + None => value, + }; + if integer.starts_with('_') || integer.ends_with('_') || integer.contains("__") { + return None; + } + let digits = integer.strip_prefix(['+', '-']).unwrap_or(integer); + if digits.is_empty() + || digits.starts_with('_') + || !digits + .bytes() + .all(|byte| byte.is_ascii_digit() || byte == b'_') + { + return None; + } + integer.replace('_', "").parse().ok() +} + +fn integral_float(value: f64) -> Option { + (value.is_finite() + && value.fract() == 0.0 + && value >= i64::MIN as f64 + && value < -(i64::MIN as f64)) + .then_some(value as i64) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::{Value, json}; + use serde_with::de::DeserializeAsWrap; + + use super::*; + + fn count(value: Value) -> Option { + DeserializeAsWrap::::deserialize(value) + .ok() + .map(DeserializeAsWrap::into_inner) + } + + fn numeric_count(value: Value) -> Option { + DeserializeAsWrap::::deserialize(value) + .ok() + .map(DeserializeAsWrap::into_inner) + } + + use serde::Deserialize; + + #[rstest] + #[case::integer(json!(128000), Some(128_000))] + #[case::whole_float(json!(128000.0), Some(128_000))] + #[case::string(json!(" 8192 "), Some(8192))] + #[case::separated(json!("1_000"), Some(1000))] + #[case::trailing_zeros(json!("2.000"), Some(2))] + #[case::boolean(json!(true), Some(1))] + #[case::maximum(json!(i64::MAX.to_string()), Some(i64::MAX as u64))] + #[case::negative(json!(-5), None)] + #[case::fraction(json!(2.5), None)] + #[case::precise_fraction(json!("1.0000000000000001"), None)] + #[case::exponent(json!("1e3"), None)] + #[case::missing_fraction(json!("2."), None)] + #[case::repeated_separator(json!("2__0"), None)] + #[case::above_i64(json!(u64::MAX), None)] + #[case::float_above_i64(json!(9_223_372_036_854_775_808.0), None)] + #[case::word(json!("8k"), None)] + #[case::object(json!({}), None)] + fn counts_accept_whole_numbers_however_they_are_written( + #[case] written: Value, + #[case] expected: Option, + ) { + assert_eq!(count(written), expected); + } + + #[rstest] + #[case::integer(json!(214), Some(214))] + #[case::zero(json!(0), Some(0))] + #[case::whole_float(json!(214.0), Some(214))] + #[case::negative_zero(json!(-0.0), Some(0))] + #[case::above_i64(json!(u64::MAX), Some(u64::MAX))] + #[case::negative(json!(-1), None)] + #[case::negative_whole_float(json!(-1.0), None)] + #[case::fraction(json!(1.5), None)] + #[case::string(json!("214"), None)] + #[case::boolean(json!(true), None)] + #[case::list(json!([214]), None)] + fn numeric_counts_accept_only_whole_json_numbers( + #[case] written: Value, + #[case] expected: Option, + ) { + assert_eq!(numeric_count(written), expected); + } +} diff --git a/litellm-rust/crates/model-catalog/src/wire/mod.rs b/litellm-rust/crates/model-catalog/src/wire/mod.rs new file mode 100644 index 00000000000..e8f23b7bd0d --- /dev/null +++ b/litellm-rust/crates/model-catalog/src/wire/mod.rs @@ -0,0 +1,10 @@ +mod attributes; +mod blocks; +mod entry; +mod flat_key; +mod lax; + +pub use attributes::*; +pub use blocks::*; +pub use entry::*; +pub use flat_key::*; diff --git a/litellm-rust/crates/model-catalog/tests/catalog.rs b/litellm-rust/crates/model-catalog/tests/catalog.rs index d384dfea9a8..31f907689d3 100644 --- a/litellm-rust/crates/model-catalog/tests/catalog.rs +++ b/litellm-rust/crates/model-catalog/tests/catalog.rs @@ -1,303 +1,401 @@ -use std::path::{Path, PathBuf}; +use std::collections::BTreeSet; -use litellm_model_catalog::{AliasIssue, Catalog, Error, IntegrityLimits, Mode, Provenance}; +use litellm_model_catalog::{ + BUNDLED_REGISTRY, Catalog, Error, ErrorKind, IntegrityLimits, Mode, Rejects, Written, +}; use rstest::{fixture, rstest}; -use serde_json::json; +use serde_json::{Map, Value, json}; const ALPHA_FIXTURE: &[u8] = br#"{ - "sample_spec":{"explanation":"example"}, + "sample_spec":{"explanation":"example","input_cost_per_token":"not checked"}, "fallback_generalizations":{"rules":[{"name":"family","pattern":"^new-","model_info":{"mode":"chat"}}]}, "Alpha":{"litellm_provider":"test","aliases":["short"],"price":0,"enabled":false, "optional":null,"unknown":{"nested":[1,{"x":true}]}} }"#; -#[fixture] -fn repo_root() -> PathBuf { - Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") -} - -#[fixture] -fn current_catalog(repo_root: PathBuf) -> Catalog { - let body = std::fs::read(repo_root.join("model_prices_and_context_window.json")).unwrap(); - Catalog::parse(&body, Provenance::default()).unwrap() -} - -#[fixture] -fn backup_catalog(repo_root: PathBuf) -> Catalog { - let body = std::fs::read(repo_root.join("litellm/model_prices_and_context_window_backup.json")) - .unwrap(); - Catalog::parse(&body, Provenance::default()).unwrap() -} - #[fixture] fn fixture_catalog() -> Catalog { - Catalog::parse( - ALPHA_FIXTURE, - Provenance { - source: Some("fixture".into()), - revision: Some("rev".into()), - etag: None, - }, - ) - .unwrap() + Catalog::parse(ALPHA_FIXTURE).unwrap().value +} + +fn loaded(document: Value) -> litellm_model_catalog::Loaded { + Catalog::parse(document.to_string().as_bytes()).unwrap() +} + +fn catalog(document: Value) -> Catalog { + loaded(document).value +} + +fn written(catalog: &Catalog) -> String { + let document: Map = catalog + .written() + .map(|(key, value)| { + let value = match value { + Written::Entry(fields) => serde_json::to_value(fields).unwrap(), + Written::Section(value) => value.clone(), + }; + (key.to_owned(), value) + }) + .collect(); + serde_json::to_string(&document).unwrap() +} + +fn report(error: &Error) -> String { + litellm_error::Report::new(error).to_string() +} + +fn error_text(document: Value) -> Vec { + loaded(document).errors.iter().map(report).collect() } #[rstest] #[case::decisions("decisions", Mode::Decisions)] #[case::legacy_evaluation("evaluation", Mode::Evaluation)] fn parses_decision_modes(#[case] mode: &str, #[case] expected: Mode) { - let source = json!({"Test":{"litellm_provider":"test","mode":mode}}).to_string(); - let catalog = Catalog::parse(source.as_bytes(), Provenance::default()).unwrap(); - + let catalog = catalog(json!({"Test": {"litellm_provider": "test", "mode": mode}})); assert_eq!( - catalog.lookup("Test").unwrap().entry.info().mode, + catalog + .lookup("Test") + .unwrap() + .entry + .model() + .unwrap() + .attributes + .mode, Some(expected) ); - assert_eq!(serde_json::to_value(expected).unwrap(), json!(mode)); } #[rstest] -#[ignore] -fn preserves_fields_and_metadata(fixture_catalog: Catalog) { - let catalog = fixture_catalog; - let entry = catalog.lookup("SHORT").unwrap(); - assert_eq!(entry.canonical_key, "Alpha"); - assert_eq!(entry.matched_key, "short"); - assert_eq!(entry.entry.field("price"), Some(&json!(0))); - assert_eq!(entry.entry.field("enabled"), Some(&json!(false))); - assert_eq!(entry.entry.field("optional"), Some(&json!(null))); - assert_eq!(entry.entry.field("missing"), None); +fn unknown_keys_are_kept_raw_and_listed_as_unrecognized(fixture_catalog: Catalog) { + let entry = fixture_catalog.lookup("Alpha").unwrap().entry; assert_eq!( - entry.entry.field("unknown"), + entry.field("unknown"), Some(&json!({"nested":[1,{"x":true}]})) ); - assert_eq!(entry.entry.field("aliases"), None); - assert_eq!(entry.entry.info().litellm_provider.as_deref(), Some("test")); assert_eq!( - catalog.sample_spec(), - Some(&json!({"explanation":"example"})) - ); - assert_eq!(catalog.fallback_rules().unwrap().len(), 1); - assert_eq!(catalog.provenance().revision.as_deref(), Some("rev")); - assert_eq!(catalog.model_count(), 1); -} - -#[rstest] -#[ignore] -fn snapshot_does_not_borrow_source() { - let mut source = ALPHA_FIXTURE.to_vec(); - let catalog = Catalog::parse(&source, Provenance::default()).unwrap(); - source.fill(b' '); - - let entry = catalog.lookup("short").unwrap(); - assert_eq!(entry.canonical_key, "Alpha"); - assert_eq!(entry.entry.field("price"), Some(&json!(0))); -} - -#[rstest] -#[case("Shared", "First")] -#[case("Second", "Second")] -#[case("shared", "Second")] -#[case("FIRST", "First")] -#[case("sHaReD", "Second")] -#[ignore] -fn alias_collisions_and_case_fallback_follow_entry_order( - #[case] lookup: &str, - #[case] expected: &str, -) { - let catalog = Catalog::parse( - br#"{ - "First":{"aliases":["Shared","Second","first"],"value":1}, - "Second":{"aliases":["Shared","sHaReD"],"value":2}, - "SHARED":{"value":3} - }"#, - Provenance::default(), - ) - .unwrap(); - assert_eq!(catalog.lookup(lookup).unwrap().canonical_key, expected); - assert_eq!(catalog.alias_count(), 3); - assert!( - catalog - .alias_issues() - .contains(&AliasIssue::CanonicalCollision { - model: "First".into(), - alias: "Second".into(), - }) - ); - assert!( - catalog - .alias_issues() - .contains(&AliasIssue::AliasCollision { - model: "Second".into(), - alias: "Shared".into(), - }) + fixture_catalog.unrecognized().collect::>(), + BTreeSet::from([ + ("Alpha", "enabled"), + ("Alpha", "price"), + ("Alpha", "unknown") + ]) ); } -#[test] -#[ignore] -fn json_entry_order_controls_alias_ownership_and_case_fallback() { - let forward = Catalog::parse( - br#"{ - "Alpha":{"aliases":["shared"]}, - "Beta":{"aliases":["shared"]}, - "Foo":{}, - "fOO":{} - }"#, - Provenance::default(), - ) - .unwrap(); - let reversed = Catalog::parse( - br#"{ - "fOO":{}, - "Foo":{}, - "Beta":{"aliases":["shared"]}, - "Alpha":{"aliases":["shared"]} - }"#, - Provenance::default(), - ) - .unwrap(); - - assert_eq!(forward.lookup("shared").unwrap().canonical_key, "Alpha"); - assert_eq!(reversed.lookup("shared").unwrap().canonical_key, "Beta"); - assert_eq!(forward.lookup("foo").unwrap().canonical_key, "fOO"); - assert_eq!(reversed.lookup("foo").unwrap().canonical_key, "Foo"); -} - -#[derive(Debug)] -enum ValidationOutcome { - Ok, - Shrunk, - BelowMinimum, - InvalidRatio, -} - #[rstest] -#[case( - IntegrityLimits { - reference_model_count: 2, - min_model_count: 1, - min_reference_ratio: 0.5, - }, - ValidationOutcome::Ok -)] -#[case( - IntegrityLimits { - reference_model_count: 3, - min_model_count: 1, - min_reference_ratio: 0.5, - }, - ValidationOutcome::Shrunk -)] -#[case( - IntegrityLimits { - reference_model_count: 0, - min_model_count: 2, - min_reference_ratio: 0.5, - }, - ValidationOutcome::BelowMinimum -)] -#[case( - IntegrityLimits { - reference_model_count: 0, - min_model_count: 0, - min_reference_ratio: f64::NAN, - }, - ValidationOutcome::InvalidRatio -)] -#[ignore] -fn integrity_uses_canonical_count_and_strict_shrink_boundary( - #[case] limits: IntegrityLimits, - #[case] expected: ValidationOutcome, -) { - let catalog = Catalog::parse( - br#"{"sample_spec":{},"fallback_generalizations":{"rules":[]},"a":{"aliases":["b","c"]}}"#, - Provenance::default(), - ) - .unwrap(); - let actual = catalog.validate(limits); - match expected { - ValidationOutcome::Ok => assert!(actual.is_ok()), - ValidationOutcome::Shrunk => { - assert!(matches!(actual, Err(Error::Shrunk { actual: 1, .. }))) - } - ValidationOutcome::BelowMinimum => { - assert!(matches!(actual, Err(Error::BelowMinimum { actual: 1, .. }))) - } - ValidationOutcome::InvalidRatio => assert!(matches!(actual, Err(Error::InvalidRatio))), - } -} - -#[derive(Debug)] -enum MalformedOutcome { - Empty, - Json, - EntryNotObject, -} - -#[rstest] -#[case::empty(b"{}", MalformedOutcome::Empty)] -#[case::invalid_json(b"{", MalformedOutcome::Json)] -#[case::entry_not_object(br#"{"a":1}"#, MalformedOutcome::EntryNotObject)] -#[case::fallback_rules_missing( - br#"{"fallback_generalizations":{},"a":{}}"#, - MalformedOutcome::Json -)] -#[ignore] -fn malformed_input_and_aliases_have_typed_outcomes( - #[case] body: &[u8], - #[case] expected: MalformedOutcome, -) { - let actual = Catalog::parse(body, Provenance::default()); - match expected { - MalformedOutcome::Empty => assert!(matches!(actual, Err(Error::Empty))), - MalformedOutcome::Json => assert!(matches!(actual, Err(Error::Json(_)))), - MalformedOutcome::EntryNotObject => { - assert!(matches!(actual, Err(Error::EntryNotObject { .. }))) - } - } -} - -#[rstest] -#[ignore] -fn invalid_aliases_are_reported_not_fatal() { - let catalog = Catalog::parse( - br#"{"a":{"aliases":"bad"},"b":{"aliases":[9,"ok"]}}"#, - Provenance::default(), - ) - .unwrap(); +fn sections_aliases_and_raw_fields_are_read_in_their_document_views(fixture_catalog: Catalog) { + let found = fixture_catalog.lookup("SHORT").unwrap(); + assert_eq!(found.canonical_key, "Alpha"); + assert_eq!(found.matched_key, "short"); + assert_eq!(found.entry.field("optional"), Some(&json!(null))); + assert_eq!(found.entry.field("aliases"), Some(&json!(["short"]))); assert_eq!( - catalog.alias_issues(), - &[ - AliasIssue::InvalidList { model: "a".into() }, - AliasIssue::InvalidName { model: "b".into() }, - ] + fixture_catalog.sample_spec(), + Some(&json!({"explanation":"example","input_cost_per_token":"not checked"})) + ); + assert_eq!(fixture_catalog.fallback_rules().unwrap()[0].name, "family"); + assert_eq!(fixture_catalog.model_count(), 1); + assert_eq!( + fixture_catalog + .served() + .map(|(key, _)| key) + .collect::>(), + ["Alpha", "short"] ); - assert_eq!(catalog.lookup("ok").unwrap().canonical_key, "b"); - assert!(catalog.lookup("missing").is_none()); } #[rstest] -#[ignore] -fn parses_current_and_packaged_catalogs_against_independent_baseline( - current_catalog: Catalog, - backup_catalog: Catalog, -) { - assert!(current_catalog.model_count() > 0); - assert!(backup_catalog.model_count() > 0); - assert!(current_catalog.sample_spec().is_some()); - assert!(backup_catalog.sample_spec().is_some()); - // Snapshot from 2026-09-23; the backup file mirrors the current file and cannot detect shrinkage. - const REFERENCE_MODEL_COUNT: usize = 4303; - current_catalog - .validate(IntegrityLimits { - reference_model_count: REFERENCE_MODEL_COUNT, - min_model_count: 50, - min_reference_ratio: 0.9, - }) - .unwrap(); - assert!(current_catalog.model_names().all(|name| { - let entry = current_catalog.lookup(name).unwrap().entry; - entry.info().litellm_provider.is_some() == entry.field("litellm_provider").is_some() +fn a_bad_entry_is_reported_by_model_and_key_and_kept_raw() { + let parsed = loaded(json!({ + "Good": {"litellm_provider": "test"}, + "Bad": {"litellm_provider": "test", "mode": "chatt", "input_cost_per_token": "free"}, + "Worse": {"mode": "chat"} })); + let catalog = parsed.value; + let problems: Vec = parsed.errors.iter().map(report).collect(); + assert_eq!(problems.len(), 3, "{problems:?}"); + assert!(problems[0].starts_with("Bad.mode: invalid value: unknown variant `chatt`")); + assert!(problems[1].starts_with("Bad.input_cost_per_token: invalid value")); + assert_eq!(problems[2], "Worse.litellm_provider: required"); + assert!(catalog.lookup("Good").unwrap().entry.model().is_some()); + assert!(catalog.lookup("Bad").unwrap().entry.model().is_none()); + assert_eq!(catalog.model_count(), 3); + assert!( + parsed + .errors + .iter() + .find(|error| error.rejects() == Rejects::Document) + .is_none() + ); +} + +#[rstest] +#[case::alias_names_a_model( + json!({"First": {"litellm_provider": "t", "aliases": ["Second"]}, "Second": {"litellm_provider": "t"}}), + r#"First: alias "Second" is already a model name"# +)] +#[case::alias_claimed_twice( + json!({"First": {"litellm_provider": "t", "aliases": ["shared"]}, "Second": {"litellm_provider": "t", "aliases": ["shared"]}}), + r#"Second: alias "shared" already belongs to First"# +)] +#[case::aliases_not_a_list( + json!({"a": {"litellm_provider": "t", "aliases": "bad"}}), + "a.aliases: invalid value: invalid type: string \"bad\", expected a sequence" +)] +#[case::entry_not_object( + json!({"a": 1, "valid": {"litellm_provider": "test"}}), + "a: must be a JSON object" +)] +#[case::fallback_rules_missing( + json!({"fallback_generalizations": {}, "a": {"litellm_provider": "t"}}), + "fallback_generalizations: invalid value: missing field `rules`" +)] +#[case::unknown_timezone( + json!({"a": {"litellm_provider": "t", "off_peak_pricing": { + "windows": [{"hours_utc": "00:00-01:00"}], + "weekday_timezone": "Not/AZone", + "input_cost_per_token": 1e-6 + }}}), + r#"a.off_peak_pricing.weekday_timezone: "Not/AZone" is not an IANA time zone"# +)] +fn document_level_problems_are_reported(#[case] document: Value, #[case] expected: &str) { + assert_eq!(error_text(document), [expected]); +} + +#[rstest] +fn escaped_row_paths_and_row_validation_share_the_parse_rules() { + let document = json!({ + "open/router~v1": {"litellm_provider": "test", "input_cost_per_token": -1} + }); + let parsed = loaded(document); + let fields = parsed + .value + .lookup("open/router~v1") + .unwrap() + .entry + .fields(); + let row_errors = Catalog::row_errors("open/router~v1", fields); + assert_eq!( + row_errors + .iter() + .map(|error| (error.path(), report(error))) + .collect::>(), + parsed + .errors + .iter() + .map(|error| (error.path(), report(error))) + .collect::>() + ); + assert_eq!( + row_errors[0].path(), + "/open~1router~0v1/input_cost_per_token" + ); + assert!(row_errors[0].kind() == ErrorKind::Invalid); +} + +#[rstest] +fn malformed_fallback_is_collected_while_models_are_read() { + let parsed = loaded(json!({ + "fallback_generalizations": {"rules": "bad"}, + "a": {"litellm_provider": "test"} + })); + assert_eq!(parsed.value.model_count(), 1); + assert_eq!(parsed.errors.len(), 1); + assert_eq!(parsed.errors[0].path(), "/fallback_generalizations"); +} + +#[rstest] +fn rejected_rows_and_their_aliases_are_removed_without_disturbing_written_order() { + let body = r#"{"before":{"litellm_provider":"test","aliases":["before-alias"]},"bad":{"litellm_provider":"test","aliases":["bad-alias"],"mode":"chatt"},"not-object":4,"sample_spec":{"keep":true},"after":{"litellm_provider":"test"}}"#; + let parsed = Catalog::parse(body.as_bytes()).unwrap(); + let (catalog, removed) = parsed.value.without_rejected(&parsed.errors); + + assert_eq!(removed.len(), 2); + assert_eq!( + catalog.lookup("before-alias").unwrap().canonical_key, + "before" + ); + assert_eq!( + catalog.lookup("BEFORE-ALIAS").unwrap().matched_key, + "before-alias" + ); + assert!(catalog.lookup("bad").is_none()); + assert!(catalog.lookup("BAD").is_none()); + assert!(catalog.lookup("Bad-Alias").is_none()); + assert_eq!(catalog.model_count(), 2); + assert_eq!( + written(&catalog), + r#"{"before":{"litellm_provider":"test","aliases":["before-alias"]},"sample_spec":{"keep":true},"after":{"litellm_provider":"test"}}"# + ); +} + +#[rstest] +fn invalid_empty_key_rows_are_removed() { + let parsed = loaded(json!({ + "": {"litellm_provider": "test", "mode": "chatt"}, + "good": {"litellm_provider": "test"} + })); + assert_eq!(parsed.errors[0].path(), "//mode"); + + let (catalog, removed) = parsed.value.without_rejected(&parsed.errors); + assert_eq!(catalog.model_count(), 1); + assert!(catalog.lookup("").is_none()); + assert!(catalog.lookup("good").is_some()); + assert_eq!(removed.len(), 1); +} + +#[rstest] +fn timezone_errors_reject_providerless_rows_without_rejecting_other_providerless_rows() { + let parsed = loaded(json!({ + "good": {"mode": "chat"}, + "bad": {"off_peak_pricing": { + "windows": [{"hours_utc": "00:00-01:00"}], + "weekday_timezone": "Not/AZone", + "input_cost_per_token": 1e-6 + }} + })); + assert_eq!(parsed.errors.len(), 3); + assert!( + parsed + .errors + .iter() + .any(|error| error.kind() == ErrorKind::UnknownTimezone) + ); + + let (catalog, removed) = parsed.value.without_rejected(&parsed.errors); + assert_eq!(catalog.model_count(), 1); + assert!(catalog.lookup("good").is_some()); + assert!(catalog.lookup("bad").is_none()); + assert_eq!(removed.len(), 2); +} + +#[rstest] +fn bundled_per_row_errors_match_document_errors() { + let parsed = Catalog::parse(BUNDLED_REGISTRY).unwrap(); + let row_errors: Vec<_> = parsed + .value + .written() + .flat_map(|(key, written)| match written { + Written::Entry(fields) => Catalog::row_errors(key, fields), + Written::Section(_) => Vec::new(), + }) + .map(|error| (error.path().to_owned(), report(&error))) + .collect(); + let document_errors: Vec<_> = parsed + .errors + .iter() + .map(|error| (error.path().to_owned(), report(error))) + .collect(); + + assert_eq!(row_errors, document_errors); +} + +#[rstest] +fn three_hundred_rows_filter_one_row_with_two_invalid_prices() { + let valid_rows: Map = (0..299) + .map(|index| { + ( + format!("valid-{index:03}"), + json!({"litellm_provider": "test"}), + ) + }) + .collect(); + let mut document = valid_rows; + document.insert( + "bad".to_owned(), + json!({ + "litellm_provider": "test", + "input_cost_per_token": -1, + "output_cost_per_token": -1 + }), + ); + let parsed = Catalog::parse(serde_json::to_vec(&document).unwrap().as_slice()).unwrap(); + assert_eq!(parsed.value.model_count(), 300); + assert_eq!(parsed.errors.len(), 2); + + let (catalog, removed) = parsed.value.without_rejected(&parsed.errors); + assert_eq!(catalog.model_count(), 299); + assert_eq!(removed.len(), 2); +} + +#[rstest] +fn integrity_limits_count_canonical_models_not_aliases() { + let fetched = catalog(json!({"a": {"litellm_provider": "t", "aliases": ["b", "c"]}})); + assert!( + fetched + .check_size(2, IntegrityLimits::new(1, 0.5).unwrap()) + .is_ok() + ); + assert_eq!( + fetched + .check_size(3, IntegrityLimits::new(1, 0.5).unwrap()) + .unwrap_err() + .kind(), + ErrorKind::Shrunk + ); + assert_eq!( + fetched + .check_size(2, IntegrityLimits::new(2, 0.5).unwrap()) + .unwrap_err() + .kind(), + ErrorKind::BelowMinimum + ); +} + +#[rstest] +#[case::empty(b"{}", "empty")] +#[case::only_sections(br#"{"sample_spec":{}}"#, "empty")] +#[case::invalid_json(b"{", "json")] +#[case::array_root_is_json_error(b"[]", "json")] +fn malformed_documents_are_rejected_whole(#[case] body: &[u8], #[case] kind: &str) { + let error = Catalog::parse(body).unwrap_err(); + assert_eq!(error.rejects(), Rejects::Document); + assert_eq!(<&str>::from(error.kind()), kind); +} + +#[rstest] +fn written_is_the_document_in_order_including_sections_and_bad_entries() { + let body = concat!( + r#"{"b":{"litellm_provider":"t","z":1,"a":null},"sample_spec":{"x":"y"},"n":4,"#, + r#""fallback_generalizations":{"rules":"bad"},"a":{"mode":"chatt","price":1.0}}"# + ); + let catalog = Catalog::parse(body.as_bytes()).unwrap().value; + assert_eq!(written(&catalog), body); + assert_eq!(catalog.model_count(), 2); +} + +#[rstest] +fn the_bundled_registry_has_no_problems() { + let problems: Vec<_> = Catalog::problems(BUNDLED_REGISTRY) + .iter() + .map(report) + .collect(); + assert!(problems.is_empty(), "{problems:#?}"); + const REFERENCE_MODEL_COUNT: usize = 4303; + let catalog = Catalog::parse(BUNDLED_REGISTRY).unwrap().value; + assert!(catalog.model_count() as f64 >= REFERENCE_MODEL_COUNT as f64 * 0.9); +} + +#[rstest] +fn the_bundled_registry_is_written_back_as_it_was_read() { + let document: Value = serde_json::from_slice(BUNDLED_REGISTRY).unwrap(); + assert_eq!( + written(&Catalog::parse(BUNDLED_REGISTRY).unwrap().value), + serde_json::to_string(&document).unwrap() + ); +} + +#[rstest] +fn a_bad_value_keeps_its_json_error_as_the_source_and_names_its_key_and_field() { + let parsed = + loaded(json!({"open/router": {"litellm_provider": "t", "input_cost_per_token": "free"}})); + let error = &parsed.errors[0]; + let cause = litellm_error::find::(error) + .expect("the JSON error is in the chain") + .to_string(); + assert_eq!(report(error).matches(&cause).count(), 1); + assert!(!error.to_string().contains(&cause)); + assert_eq!(error.key().as_deref(), Some("open/router")); + assert_eq!(error.field().as_deref(), Some("input_cost_per_token")); } diff --git a/litellm-rust/crates/model-catalog/tests/generate_price_lookup_reference.py b/litellm-rust/crates/model-catalog/tests/generate_price_lookup_reference.py new file mode 100644 index 00000000000..e49b635ff4a --- /dev/null +++ b/litellm-rust/crates/model-catalog/tests/generate_price_lookup_reference.py @@ -0,0 +1,91 @@ +import itertools +import json +import re +import subprocess +from pathlib import Path +from typing import Final + +from litellm.litellm_core_utils.llm_cost_calc.utils import _get_token_base_cost +from litellm.types.utils import Usage + +REPO_ROOT: Final = Path(__file__).resolve().parents[4] +TOKEN_RATE_KEY: Final = re.compile( + r"^(input_cost_per_token|output_cost_per_token|cache_read_input_token_cost" + r"|cache_creation_input_token_cost(_above_1hr)?)" + r"(_above_(\d+)(k?)_tokens)?(_(priority|flex|ultrafast|balanced|batches|decisions))?$" +) +SERVICE_TIERS: Final = ("priority", "flex", "ultrafast", "balanced") +LARGEST_PROMPT: Final = 10_000_000 +INCLUSIVE_PROVIDERS: Final = frozenset({"xai"}) + + +def token_rates(entry: dict[str, object]) -> dict[str, object]: + return { + key: value + for key, value in entry.items() + if value is not None and (key == "tiered_pricing" or TOKEN_RATE_KEY.match(key)) + } + + +def band_floors(info: dict[str, object]) -> frozenset[int]: + matches: Final = (TOKEN_RATE_KEY.match(key) for key in info) + flat: Final = frozenset( + int(match.group(4)) * (1000 if match.group(5) else 1) + for match in matches + if match is not None and match.group(4) is not None + ) + rows: Final = info.get("tiered_pricing") + tiers: Final = ( + frozenset(int(row["range"][0]) for row in rows if isinstance(row, dict) and "range" in row) + if isinstance(rows, list) + else frozenset() + ) + return flat | tiers + + +def tiers_in(info: dict[str, object]) -> tuple[str | None, ...]: + return (None, *(tier for tier in SERVICE_TIERS if any(key.endswith(f"_{tier}") for key in info))) + + +def prompts(info: dict[str, object]) -> tuple[int, ...]: + floors: Final = band_floors(info) + return tuple(sorted(({1, LARGEST_PROMPT} | floors | {floor + 1 for floor in floors}) - {0})) + + +def check(info: dict[str, object], tier: str | None, prompt: int, inclusive: bool) -> list[object]: + usage: Final = Usage(prompt_tokens=prompt, completion_tokens=1, total_tokens=prompt + 1) + rates: Final = _get_token_base_cost(info, usage, service_tier=tier, threshold_is_inclusive=inclusive) + one_hour: Final = rates.cache_creation_rates_by_ttl.get("1hr", rates.cache_creation_rate) + return [tier, prompt, [rates.input_rate, rates.output_rate, rates.cache_creation_rate, one_hour, rates.cache_read_rate]] + + +def shape(info: dict[str, object], inclusive: bool) -> str: + rows: Final = info.get("tiered_pricing") + row_keys: Final = [sorted(row) for row in rows if isinstance(row, dict)] if isinstance(rows, list) else [] + return json.dumps([sorted(info), row_keys, inclusive]) + + +def checks(info: dict[str, object], inclusive: bool) -> list[list[object]]: + return [check(info, tier, prompt, inclusive) for tier, prompt in itertools.product(tiers_in(info), prompts(info))] + + +def main() -> None: + registry: Final = json.loads((REPO_ROOT / "model_prices_and_context_window.json").read_text()) + entries: Final = tuple( + (token_rates(entry), entry.get("litellm_provider") in INCLUSIVE_PROVIDERS) + for name, entry in registry.items() + if name not in ("sample_spec", "fallback_generalizations") and isinstance(entry, dict) + ) + by_shape: Final = {shape(info, inclusive): (info, inclusive) for info, inclusive in reversed(entries) if info} + models: Final = tuple( + json.dumps({"info": info, "inclusive": inclusive, "checks": checks(info, inclusive)}) + for info, inclusive in by_shape.values() + ) + revision: Final = subprocess.check_output(("git", "rev-parse", "HEAD"), text=True, cwd=REPO_ROOT).strip() + Path(__file__).with_name("price_lookup_reference.json").write_text( + f'{{"python_reference_commit": "{revision}", "models": [\n' + ",\n".join(models) + "\n]}\n" + ) + + +if __name__ == "__main__": + main() diff --git a/litellm-rust/crates/model-catalog/tests/price_lookup.rs b/litellm-rust/crates/model-catalog/tests/price_lookup.rs new file mode 100644 index 00000000000..cc16e9f23e9 --- /dev/null +++ b/litellm-rust/crates/model-catalog/tests/price_lookup.rs @@ -0,0 +1,89 @@ +use litellm_cost::money::UnitPrice; +use litellm_cost::price::{ + CacheReadMode, CacheTtl, InputUnit, Meter, OutputUnit, Prices, ServiceTier, TokenKind, +}; +use litellm_model_catalog::wire::{EntryWire, Fields}; +use rstest::rstest; +use serde::Deserialize; + +#[derive(Deserialize)] +struct Reference { + models: Vec, +} + +#[derive(Deserialize)] +struct ReferenceModel { + info: Fields, + inclusive: bool, + checks: Vec<(Option, u64, [f64; 5])>, +} + +const METERS: [Meter; 5] = [ + Meter::Input(InputUnit::Token(TokenKind::Text)), + Meter::Output(OutputUnit::Token(TokenKind::Text)), + Meter::CacheWrite { + kind: TokenKind::Text, + ttl: CacheTtl::Default, + }, + Meter::CacheWrite { + kind: TokenKind::Text, + ttl: CacheTtl::OneHour, + }, + Meter::CacheRead { + kind: TokenKind::Text, + mode: CacheReadMode::Implicit, + }, +]; + +fn service_tier(name: Option<&str>) -> ServiceTier { + match name { + None => ServiceTier::Standard, + Some("priority") => ServiceTier::Priority, + Some("flex") => ServiceTier::Flex, + Some("ultrafast") => ServiceTier::Ultrafast, + Some("balanced") => ServiceTier::Balanced, + Some(other) => panic!("the reference uses an unknown service tier {other}"), + } +} + +fn mismatch( + prices: &Prices, + (tier_name, prompt, python): &(Option, u64, [f64; 5]), +) -> Option { + let tier = service_tier(tier_name.as_deref()); + let band = prices.band(tier, *prompt); + let rust = METERS + .each_ref() + .map(|meter| prices.rate(meter, tier, band).map_or(0.0, UnitPrice::get)); + (rust != *python) + .then(|| format!("tier={tier_name:?} prompt={prompt}: rust {rust:?}, python {python:?}")) +} + +#[rstest] +fn band_and_rate_match_python_token_base_cost() { + let reference: Reference = + serde_json::from_str(include_str!("price_lookup_reference.json")).unwrap(); + + let mismatches: Vec = reference + .models + .iter() + .flat_map(|model| { + let provider = if model.inclusive { "xai" } else { "openai" }; + let fields: Fields = model + .info + .clone() + .into_iter() + .chain([("litellm_provider".to_owned(), provider.into())]) + .collect(); + let prices = Prices::try_from(&EntryWire::read(&fields)).unwrap(); + model + .checks + .iter() + .filter_map(move |check| mismatch(&prices, check)) + .map(move |found| format!("{:?} {found}", model.info)) + }) + .collect(); + + assert!(reference.models.len() > 50); + assert!(mismatches.is_empty(), "{mismatches:#?}"); +} diff --git a/litellm-rust/crates/model-catalog/tests/price_lookup_reference.json b/litellm-rust/crates/model-catalog/tests/price_lookup_reference.json new file mode 100644 index 00000000000..4b87b696723 --- /dev/null +++ b/litellm-rust/crates/model-catalog/tests/price_lookup_reference.json @@ -0,0 +1,89 @@ +{"python_reference_commit": "c7c895db456eda349a7be9661b4b21f578f9b589", "models": [ +{"info": {"cache_creation_input_token_cost": 1.25e-05, "cache_creation_input_token_cost_above_272k_tokens": 2.5e-05, "cache_read_input_token_cost": 1e-06, "cache_read_input_token_cost_above_272k_tokens": 2e-06, "input_cost_per_token": 1e-05, "input_cost_per_token_above_272k_tokens": 2e-05, "output_cost_per_token": 5e-05, "output_cost_per_token_above_272k_tokens": 7.5e-05}, "inclusive": false, "checks": [[null, 1, [1e-05, 5e-05, 1.25e-05, 1.25e-05, 1e-06]], [null, 272000, [1e-05, 5e-05, 1.25e-05, 1.25e-05, 1e-06]], [null, 272001, [2e-05, 7.5e-05, 2.5e-05, 2.5e-05, 2e-06]], [null, 10000000, [2e-05, 7.5e-05, 2.5e-05, 2.5e-05, 2e-06]]]}, +{"info": {"cache_read_input_token_cost": 1e-07, "input_cost_per_token": 1e-06, "output_cost_per_token": 3e-06}, "inclusive": false, "checks": [[null, 1, [1e-06, 3e-06, 1e-06, 1e-06, 1e-07]], [null, 10000000, [1e-06, 3e-06, 1e-06, 1e-06, 1e-07]]]}, +{"info": {"input_cost_per_token": 1.25e-05, "output_cost_per_token": 1.25e-05}, "inclusive": false, "checks": [[null, 1, [1.25e-05, 1.25e-05, 1.25e-05, 1.25e-05, 1.25e-05]], [null, 10000000, [1.25e-05, 1.25e-05, 1.25e-05, 1.25e-05, 1.25e-05]]]}, +{"info": {"cache_creation_input_token_cost": 2.5e-06, "cache_creation_input_token_cost_above_272k_tokens": 5e-06, "cache_creation_input_token_cost_above_272k_tokens_flex": 2.5e-06, "cache_creation_input_token_cost_above_272k_tokens_priority": 1e-05, "cache_creation_input_token_cost_flex": 1.25e-06, "cache_creation_input_token_cost_priority": 5e-06, "cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_272k_tokens": 4e-07, "cache_read_input_token_cost_above_272k_tokens_flex": 2e-07, "cache_read_input_token_cost_above_272k_tokens_priority": 8e-07, "cache_read_input_token_cost_batches": 1e-07, "cache_read_input_token_cost_above_272k_tokens_batches": 2e-07, "cache_creation_input_token_cost_batches": 1.25e-06, "cache_creation_input_token_cost_above_272k_tokens_batches": 2.5e-06, "cache_read_input_token_cost_flex": 1e-07, "cache_read_input_token_cost_priority": 4e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_272k_tokens": 4e-06, "input_cost_per_token_above_272k_tokens_flex": 2e-06, "input_cost_per_token_above_272k_tokens_priority": 8e-06, "input_cost_per_token_batches": 1e-06, "input_cost_per_token_above_272k_tokens_batches": 2e-06, "input_cost_per_token_flex": 1e-06, "input_cost_per_token_priority": 4e-06, "output_cost_per_token": 1e-05, "output_cost_per_token_above_272k_tokens": 1.5e-05, "output_cost_per_token_above_272k_tokens_flex": 7.5e-06, "output_cost_per_token_above_272k_tokens_priority": 3e-05, "output_cost_per_token_batches": 5e-06, "output_cost_per_token_above_272k_tokens_batches": 7.5e-06, "output_cost_per_token_flex": 5e-06, "output_cost_per_token_priority": 2e-05}, "inclusive": false, "checks": [[null, 1, [2e-06, 1e-05, 2.5e-06, 2.5e-06, 2e-07]], [null, 272000, [2e-06, 1e-05, 2.5e-06, 2.5e-06, 2e-07]], [null, 272001, [4e-06, 1.5e-05, 5e-06, 5e-06, 4e-07]], [null, 10000000, [4e-06, 1.5e-05, 5e-06, 5e-06, 4e-07]], ["priority", 1, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], ["priority", 272000, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], ["priority", 272001, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], ["priority", 10000000, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], ["flex", 1, [1e-06, 5e-06, 1.25e-06, 1.25e-06, 1e-07]], ["flex", 272000, [1e-06, 5e-06, 1.25e-06, 1.25e-06, 1e-07]], ["flex", 272001, [2e-06, 7.5e-06, 2.5e-06, 2.5e-06, 2e-07]], ["flex", 10000000, [2e-06, 7.5e-06, 2.5e-06, 2.5e-06, 2e-07]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-06, "cache_creation_input_token_cost_above_1hr": 2e-06, "cache_creation_input_token_cost_batches": 6.25e-07, "cache_read_input_token_cost": 1e-07, "cache_read_input_token_cost_batches": 5e-08, "input_cost_per_token": 1e-06, "input_cost_per_token_batches": 5e-07, "output_cost_per_token": 5e-06, "output_cost_per_token_batches": 2.5e-06}, "inclusive": false, "checks": [[null, 1, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]], [null, 10000000, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]]]}, +{"info": {"input_cost_per_token_above_100k_tokens_batches": 2.5e-07, "output_cost_per_token_above_100k_tokens_batches": 1.25e-06, "cache_creation_input_token_cost_above_100k_tokens_batches": 3.125e-07, "cache_read_input_token_cost_above_100k_tokens_batches": 2.5e-08, "cache_creation_input_token_cost": 1.25e-07, "cache_creation_input_token_cost_above_1hr": 2e-07, "cache_creation_input_token_cost_batches": 6.25e-08, "cache_read_input_token_cost": 1e-08, "cache_read_input_token_cost_batches": 5e-09, "input_cost_per_token": 1e-07, "input_cost_per_token_batches": 5e-08, "output_cost_per_token": 5e-07, "output_cost_per_token_batches": 2.5e-07, "input_cost_per_token_above_100k_tokens": 5e-07, "output_cost_per_token_above_100k_tokens": 2.5e-06, "cache_creation_input_token_cost_above_100k_tokens": 6.25e-07, "cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06, "cache_read_input_token_cost_above_100k_tokens": 5e-08}, "inclusive": false, "checks": [[null, 1, [1e-07, 5e-07, 1.25e-07, 2e-07, 1e-08]], [null, 100000, [1e-07, 5e-07, 1.25e-07, 2e-07, 1e-08]], [null, 100001, [5e-07, 2.5e-06, 6.25e-07, 1e-06, 5e-08]], [null, 10000000, [5e-07, 2.5e-06, 6.25e-07, 1e-06, 5e-08]]]}, +{"info": {"input_cost_per_token": 1.0000000000000001e-07, "output_cost_per_token": 5e-07, "cache_read_input_token_cost": 1e-08, "cache_creation_input_token_cost": 1.25e-07, "tiered_pricing": [{"range": [0, 272000], "input_cost_per_token": 1.0000000000000001e-07, "output_cost_per_token": 5e-07, "cache_read_input_token_cost": 1e-08, "cache_creation_input_token_cost": 1.25e-07}, {"range": [272000, 1050000], "input_cost_per_token": 2.0000000000000002e-07, "output_cost_per_token": 7.5e-07, "cache_read_input_token_cost": 2e-08, "cache_creation_input_token_cost": 2.5e-07}]}, "inclusive": false, "checks": [[null, 1, [1.0000000000000001e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 272000, [1.0000000000000001e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 272001, [2.0000000000000002e-07, 7.5e-07, 2.5e-07, 2.5e-07, 2e-08]], [null, 10000000, [2.0000000000000002e-07, 7.5e-07, 2.5e-07, 2.5e-07, 2e-08]]]}, +{"info": {"cache_creation_input_token_cost": 1e-06, "cache_read_input_token_cost": 8e-08, "input_cost_per_token": 8e-07, "output_cost_per_token": 4e-06}, "inclusive": false, "checks": [[null, 1, [8e-07, 4e-06, 1e-06, 1e-06, 8e-08]], [null, 10000000, [8e-07, 4e-06, 1e-06, 1e-06, 8e-08]]]}, +{"info": {"input_cost_per_token": 2e-06, "output_cost_per_token": 6e-06, "cache_read_input_token_cost": 5e-07, "tiered_pricing": [{"range": [0, 200000], "input_cost_per_token": 2e-06, "output_cost_per_token": 6e-06, "cache_read_input_token_cost": 5e-07}, {"range": [200000, 500000], "input_cost_per_token": 4e-06, "output_cost_per_token": 1.2e-05, "cache_read_input_token_cost": 1e-06}]}, "inclusive": false, "checks": [[null, 1, [2e-06, 6e-06, 2e-06, 2e-06, 5e-07]], [null, 200000, [2e-06, 6e-06, 2e-06, 2e-06, 5e-07]], [null, 200001, [4e-06, 1.2e-05, 4e-06, 4e-06, 1e-06]], [null, 10000000, [4e-06, 1.2e-05, 4e-06, 4e-06, 1e-06]]]}, +{"info": {"cache_read_input_token_cost": 7.5e-08, "cache_read_input_token_cost_priority": 1.5e-07, "input_cost_per_token": 7.5e-07, "input_cost_per_token_priority": 1.5e-06, "output_cost_per_token": 4.5e-06, "output_cost_per_token_priority": 9e-06}, "inclusive": false, "checks": [[null, 1, [7.5e-07, 4.5e-06, 7.5e-07, 7.5e-07, 7.5e-08]], [null, 10000000, [7.5e-07, 4.5e-06, 7.5e-07, 7.5e-07, 7.5e-08]], ["priority", 1, [1.5e-06, 9e-06, 1.5e-06, 1.5e-06, 1.5e-07]], ["priority", 10000000, [1.5e-06, 9e-06, 1.5e-06, 1.5e-06, 1.5e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "input_cost_per_token": 1.25e-06, "input_cost_per_token_above_200k_tokens": 2.5e-06, "output_cost_per_token": 2.5e-06, "output_cost_per_token_above_200k_tokens": 5e-06}, "inclusive": false, "checks": [[null, 1, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200000, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200001, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]], [null, 10000000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-06, "cache_creation_input_token_cost_above_1hr": 2e-06, "cache_read_input_token_cost": 1e-07, "input_cost_per_token": 1e-06, "output_cost_per_token": 5e-06}, "inclusive": false, "checks": [[null, 1, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]], [null, 10000000, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]]]}, +{"info": {"cache_read_input_token_cost": 8.25e-08, "cache_read_input_token_cost_flex": 4.125e-08, "cache_read_input_token_cost_priority": 1.44375e-07, "input_cost_per_token": 3.3e-07, "input_cost_per_token_batches": 1.65e-07, "input_cost_per_token_flex": 1.65e-07, "input_cost_per_token_priority": 5.775e-07, "output_cost_per_token": 2.75e-06, "output_cost_per_token_batches": 1.375e-06, "output_cost_per_token_flex": 1.375e-06, "output_cost_per_token_priority": 4.8125e-06}, "inclusive": false, "checks": [[null, 1, [3.3e-07, 2.75e-06, 3.3e-07, 3.3e-07, 8.25e-08]], [null, 10000000, [3.3e-07, 2.75e-06, 3.3e-07, 3.3e-07, 8.25e-08]], ["priority", 1, [5.775e-07, 4.8125e-06, 5.775e-07, 5.775e-07, 1.44375e-07]], ["priority", 10000000, [5.775e-07, 4.8125e-06, 5.775e-07, 5.775e-07, 1.44375e-07]], ["flex", 1, [1.65e-07, 1.375e-06, 1.65e-07, 1.65e-07, 4.125e-08]], ["flex", 10000000, [1.65e-07, 1.375e-06, 1.65e-07, 1.65e-07, 4.125e-08]]]}, +{"info": {"cache_read_input_token_cost": 7.5e-08, "cache_read_input_token_cost_batches": 3.75e-08, "cache_read_input_token_cost_flex": 3.75e-08, "cache_read_input_token_cost_priority": 1.5e-07, "input_cost_per_token": 7.5e-07, "output_cost_per_token": 4.5e-06, "input_cost_per_token_batches": 3.75e-07, "input_cost_per_token_flex": 3.75e-07, "input_cost_per_token_priority": 1.5e-06, "output_cost_per_token_batches": 2.25e-06, "output_cost_per_token_flex": 2.25e-06, "output_cost_per_token_priority": 9e-06}, "inclusive": false, "checks": [[null, 1, [7.5e-07, 4.5e-06, 7.5e-07, 7.5e-07, 7.5e-08]], [null, 10000000, [7.5e-07, 4.5e-06, 7.5e-07, 7.5e-07, 7.5e-08]], ["priority", 1, [1.5e-06, 9e-06, 1.5e-06, 1.5e-06, 1.5e-07]], ["priority", 10000000, [1.5e-06, 9e-06, 1.5e-06, 1.5e-06, 1.5e-07]], ["flex", 1, [3.75e-07, 2.25e-06, 3.75e-07, 3.75e-07, 3.75e-08]], ["flex", 10000000, [3.75e-07, 2.25e-06, 3.75e-07, 3.75e-07, 3.75e-08]]]}, +{"info": {"cache_read_input_token_cost": 5.5e-09, "input_cost_per_token": 5.5e-08, "input_cost_per_token_batches": 2.75e-08, "output_cost_per_token": 4.4e-07, "output_cost_per_token_batches": 2.2e-07, "cache_read_input_token_cost_batches": 2.75e-09}, "inclusive": false, "checks": [[null, 1, [5.5e-08, 4.4e-07, 5.5e-08, 5.5e-08, 5.5e-09]], [null, 10000000, [5.5e-08, 4.4e-07, 5.5e-08, 5.5e-08, 5.5e-09]]]}, +{"info": {"output_cost_per_token": 0.0}, "inclusive": false, "checks": [[null, 1, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 10000000, [0.0, 0.0, 0.0, 0.0, 0.0]]]}, +{"info": {"input_cost_per_token_above_100k_tokens": 5e-07, "output_cost_per_token_above_100k_tokens": 2.5e-06, "cache_creation_input_token_cost_above_100k_tokens": 6.25e-07, "cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1e-06, "cache_read_input_token_cost_above_100k_tokens": 5e-08, "cache_creation_input_token_cost": 1.25e-07, "cache_creation_input_token_cost_above_1hr": 2e-07, "cache_creation_input_token_cost_batches": 6.25e-08, "cache_read_input_token_cost": 1e-08, "cache_read_input_token_cost_batches": 5e-09, "input_cost_per_token": 1e-07, "input_cost_per_token_batches": 5e-08, "output_cost_per_token": 5e-07, "output_cost_per_token_batches": 2.5e-07}, "inclusive": false, "checks": [[null, 1, [1e-07, 5e-07, 1.25e-07, 2e-07, 1e-08]], [null, 100000, [1e-07, 5e-07, 1.25e-07, 2e-07, 1e-08]], [null, 100001, [5e-07, 2.5e-06, 6.25e-07, 1e-06, 5e-08]], [null, 10000000, [5e-07, 2.5e-06, 6.25e-07, 1e-06, 5e-08]]]}, +{"info": {"input_cost_per_token_above_100k_tokens": 5.5e-07, "output_cost_per_token_above_100k_tokens": 2.75e-06, "cache_creation_input_token_cost_above_100k_tokens": 6.875e-07, "cache_creation_input_token_cost_above_1hr_above_100k_tokens": 1.1e-06, "cache_read_input_token_cost_above_100k_tokens": 5.5e-08, "cache_creation_input_token_cost": 1.375e-07, "cache_creation_input_token_cost_above_1hr": 2.2e-07, "cache_read_input_token_cost": 1.1e-08, "input_cost_per_token": 1.1e-07, "output_cost_per_token": 5.5e-07}, "inclusive": false, "checks": [[null, 1, [1.1e-07, 5.5e-07, 1.375e-07, 2.2e-07, 1.1e-08]], [null, 100000, [1.1e-07, 5.5e-07, 1.375e-07, 2.2e-07, 1.1e-08]], [null, 100001, [5.5e-07, 2.75e-06, 6.875e-07, 1.1e-06, 5.5e-08]], [null, 10000000, [5.5e-07, 2.75e-06, 6.875e-07, 1.1e-06, 5.5e-08]]]}, +{"info": {"cache_creation_input_token_cost": 4.125e-06, "cache_creation_input_token_cost_flex": 2.0625e-06, "cache_creation_input_token_cost_priority": 7.21875e-06, "cache_read_input_token_cost": 3.3e-07, "cache_read_input_token_cost_flex": 1.65e-07, "cache_read_input_token_cost_priority": 5.775e-07, "input_cost_per_token": 3.3e-06, "input_cost_per_token_flex": 1.65e-06, "input_cost_per_token_priority": 5.775e-06, "output_cost_per_token": 1.65e-05, "output_cost_per_token_flex": 8.25e-06, "output_cost_per_token_priority": 2.8875e-05}, "inclusive": false, "checks": [[null, 1, [3.3e-06, 1.65e-05, 4.125e-06, 4.125e-06, 3.3e-07]], [null, 10000000, [3.3e-06, 1.65e-05, 4.125e-06, 4.125e-06, 3.3e-07]], ["priority", 1, [5.775e-06, 2.8875e-05, 7.21875e-06, 7.21875e-06, 5.775e-07]], ["priority", 10000000, [5.775e-06, 2.8875e-05, 7.21875e-06, 7.21875e-06, 5.775e-07]], ["flex", 1, [1.65e-06, 8.25e-06, 2.0625e-06, 2.0625e-06, 1.65e-07]], ["flex", 10000000, [1.65e-06, 8.25e-06, 2.0625e-06, 2.0625e-06, 1.65e-07]]]}, +{"info": {"cache_read_input_token_cost": 1.375e-07, "cache_read_input_token_cost_priority": 2.75e-07, "input_cost_per_token": 1.375e-06, "input_cost_per_token_batches": 6.875e-07, "input_cost_per_token_priority": 2.75e-06, "output_cost_per_token": 1.1e-05, "output_cost_per_token_batches": 5.5e-06, "output_cost_per_token_priority": 2.2e-05, "cache_read_input_token_cost_batches": 6.875e-08}, "inclusive": false, "checks": [[null, 1, [1.375e-06, 1.1e-05, 1.375e-06, 1.375e-06, 1.375e-07]], [null, 10000000, [1.375e-06, 1.1e-05, 1.375e-06, 1.375e-06, 1.375e-07]], ["priority", 1, [2.75e-06, 2.2e-05, 2.75e-06, 2.75e-06, 2.75e-07]], ["priority", 10000000, [2.75e-06, 2.2e-05, 2.75e-06, 2.75e-06, 2.75e-07]]]}, +{"info": {"input_cost_per_token": 5e-06, "input_cost_per_token_batches": 2.5e-06, "output_cost_per_token": 1.5e-05, "output_cost_per_token_batches": 7.5e-06}, "inclusive": false, "checks": [[null, 1, [5e-06, 1.5e-05, 5e-06, 5e-06, 5e-06]], [null, 10000000, [5e-06, 1.5e-05, 5e-06, 5e-06, 5e-06]]]}, +{"info": {"cache_creation_input_token_cost": 2.1e-06, "cache_creation_input_token_cost_batches": 1.05e-06, "cache_creation_input_token_cost_flex": 1.05e-06, "cache_creation_input_token_cost_priority": 3.675e-06, "cache_read_input_token_cost": 3.12e-07, "cache_read_input_token_cost_batches": 1.56e-07, "cache_read_input_token_cost_flex": 1.56e-07, "cache_read_input_token_cost_priority": 5.46e-07, "input_cost_per_token": 1.68e-06, "input_cost_per_token_batches": 8.4e-07, "input_cost_per_token_flex": 8.4e-07, "input_cost_per_token_priority": 2.94e-06, "output_cost_per_token": 5.28e-06, "output_cost_per_token_batches": 2.64e-06, "output_cost_per_token_flex": 2.64e-06, "output_cost_per_token_priority": 9.24e-06}, "inclusive": false, "checks": [[null, 1, [1.68e-06, 5.28e-06, 2.1e-06, 2.1e-06, 3.12e-07]], [null, 10000000, [1.68e-06, 5.28e-06, 2.1e-06, 2.1e-06, 3.12e-07]], ["priority", 1, [2.94e-06, 9.24e-06, 3.675e-06, 3.675e-06, 5.46e-07]], ["priority", 10000000, [2.94e-06, 9.24e-06, 3.675e-06, 3.675e-06, 5.46e-07]], ["flex", 1, [8.4e-07, 2.64e-06, 1.05e-06, 1.05e-06, 1.56e-07]], ["flex", 10000000, [8.4e-07, 2.64e-06, 1.05e-06, 1.05e-06, 1.56e-07]]]}, +{"info": {"input_cost_per_token": 1.1e-05, "input_cost_per_token_above_272k_tokens": 2.2e-05, "cache_creation_input_token_cost": 1.375e-05, "cache_creation_input_token_cost_above_272k_tokens": 2.75e-05, "cache_creation_input_token_cost_above_272k_tokens_ultrafast": 0.000165, "cache_creation_input_token_cost_ultrafast": 8.25e-05, "cache_read_input_token_cost": 1.1e-06, "cache_read_input_token_cost_above_272k_tokens": 2.2e-06, "cache_read_input_token_cost_above_272k_tokens_ultrafast": 1.32e-05, "cache_read_input_token_cost_ultrafast": 6.6e-06, "input_cost_per_token_above_272k_tokens_ultrafast": 0.000132, "input_cost_per_token_ultrafast": 6.6e-05, "output_cost_per_token": 5.5e-05, "output_cost_per_token_above_272k_tokens": 8.25e-05, "output_cost_per_token_above_272k_tokens_ultrafast": 0.000495, "output_cost_per_token_ultrafast": 0.00033}, "inclusive": false, "checks": [[null, 1, [1.1e-05, 5.5e-05, 1.375e-05, 1.375e-05, 1.1e-06]], [null, 272000, [1.1e-05, 5.5e-05, 1.375e-05, 1.375e-05, 1.1e-06]], [null, 272001, [2.2e-05, 8.25e-05, 2.75e-05, 2.75e-05, 2.2e-06]], [null, 10000000, [2.2e-05, 8.25e-05, 2.75e-05, 2.75e-05, 2.2e-06]], ["ultrafast", 1, [6.6e-05, 0.00033, 8.25e-05, 8.25e-05, 6.6e-06]], ["ultrafast", 272000, [6.6e-05, 0.00033, 8.25e-05, 8.25e-05, 6.6e-06]], ["ultrafast", 272001, [0.000132, 0.000495, 0.000165, 0.000165, 1.32e-05]], ["ultrafast", 10000000, [0.000132, 0.000495, 0.000165, 0.000165, 1.32e-05]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-05, "cache_creation_input_token_cost_above_272k_tokens": 2.5e-05, "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-05, "cache_creation_input_token_cost_above_272k_tokens_priority": 5e-05, "cache_creation_input_token_cost_flex": 6.25e-06, "cache_creation_input_token_cost_priority": 2.5e-05, "cache_read_input_token_cost": 1e-06, "cache_read_input_token_cost_above_272k_tokens": 2e-06, "cache_read_input_token_cost_above_272k_tokens_flex": 1e-06, "cache_read_input_token_cost_above_272k_tokens_priority": 4e-06, "cache_read_input_token_cost_batches": 5e-07, "cache_read_input_token_cost_above_272k_tokens_batches": 1e-06, "cache_creation_input_token_cost_batches": 6.25e-06, "cache_creation_input_token_cost_above_272k_tokens_batches": 1.25e-05, "cache_creation_input_token_cost_above_272k_tokens_ultrafast": 0.00015, "cache_creation_input_token_cost_ultrafast": 7.5e-05, "cache_read_input_token_cost_above_272k_tokens_ultrafast": 1.2e-05, "cache_read_input_token_cost_flex": 5e-07, "cache_read_input_token_cost_priority": 2e-06, "cache_read_input_token_cost_ultrafast": 6e-06, "input_cost_per_token": 1e-05, "input_cost_per_token_above_272k_tokens": 2e-05, "input_cost_per_token_above_272k_tokens_flex": 1e-05, "input_cost_per_token_above_272k_tokens_priority": 4e-05, "input_cost_per_token_batches": 5e-06, "input_cost_per_token_above_272k_tokens_batches": 1e-05, "input_cost_per_token_above_272k_tokens_ultrafast": 0.00012, "input_cost_per_token_flex": 5e-06, "input_cost_per_token_priority": 2e-05, "input_cost_per_token_ultrafast": 6e-05, "output_cost_per_token": 5e-05, "output_cost_per_token_above_272k_tokens": 7.5e-05, "output_cost_per_token_above_272k_tokens_flex": 3.75e-05, "output_cost_per_token_above_272k_tokens_priority": 0.00015, "output_cost_per_token_batches": 2.5e-05, "output_cost_per_token_above_272k_tokens_batches": 3.75e-05, "output_cost_per_token_above_272k_tokens_ultrafast": 0.00045, "output_cost_per_token_flex": 2.5e-05, "output_cost_per_token_priority": 0.0001, "output_cost_per_token_ultrafast": 0.0003}, "inclusive": false, "checks": [[null, 1, [1e-05, 5e-05, 1.25e-05, 1.25e-05, 1e-06]], [null, 272000, [1e-05, 5e-05, 1.25e-05, 1.25e-05, 1e-06]], [null, 272001, [2e-05, 7.5e-05, 2.5e-05, 2.5e-05, 2e-06]], [null, 10000000, [2e-05, 7.5e-05, 2.5e-05, 2.5e-05, 2e-06]], ["priority", 1, [2e-05, 0.0001, 2.5e-05, 2.5e-05, 2e-06]], ["priority", 272000, [2e-05, 0.0001, 2.5e-05, 2.5e-05, 2e-06]], ["priority", 272001, [4e-05, 0.00015, 5e-05, 5e-05, 4e-06]], ["priority", 10000000, [4e-05, 0.00015, 5e-05, 5e-05, 4e-06]], ["flex", 1, [5e-06, 2.5e-05, 6.25e-06, 6.25e-06, 5e-07]], ["flex", 272000, [5e-06, 2.5e-05, 6.25e-06, 6.25e-06, 5e-07]], ["flex", 272001, [1e-05, 3.75e-05, 1.25e-05, 1.25e-05, 1e-06]], ["flex", 10000000, [1e-05, 3.75e-05, 1.25e-05, 1.25e-05, 1e-06]], ["ultrafast", 1, [6e-05, 0.0003, 7.5e-05, 7.5e-05, 6e-06]], ["ultrafast", 272000, [6e-05, 0.0003, 7.5e-05, 7.5e-05, 6e-06]], ["ultrafast", 272001, [0.00012, 0.00045, 0.00015, 0.00015, 1.2e-05]], ["ultrafast", 10000000, [0.00012, 0.00045, 0.00015, 0.00015, 1.2e-05]]]}, +{"info": {"input_cost_per_token": 2.2e-06, "output_cost_per_token": 6.6e-06, "cache_read_input_token_cost": 5.5e-07, "cache_read_input_token_cost_flex": 2.75e-07, "cache_read_input_token_cost_priority": 9.625e-07, "input_cost_per_token_flex": 1.1e-06, "input_cost_per_token_priority": 3.85e-06, "output_cost_per_token_flex": 3.3e-06, "output_cost_per_token_priority": 1.155e-05}, "inclusive": false, "checks": [[null, 1, [2.2e-06, 6.6e-06, 2.2e-06, 2.2e-06, 5.5e-07]], [null, 10000000, [2.2e-06, 6.6e-06, 2.2e-06, 2.2e-06, 5.5e-07]], ["priority", 1, [3.85e-06, 1.155e-05, 3.85e-06, 3.85e-06, 9.625e-07]], ["priority", 10000000, [3.85e-06, 1.155e-05, 3.85e-06, 3.85e-06, 9.625e-07]], ["flex", 1, [1.1e-06, 3.3e-06, 1.1e-06, 1.1e-06, 2.75e-07]], ["flex", 10000000, [1.1e-06, 3.3e-06, 1.1e-06, 1.1e-06, 2.75e-07]]]}, +{"info": {"cache_read_input_token_cost": 3e-06, "cache_read_input_token_cost_above_272k_tokens": 6e-06, "input_cost_per_token": 3e-05, "input_cost_per_token_above_272k_tokens": 6e-05, "output_cost_per_token": 0.00018, "output_cost_per_token_above_272k_tokens": 0.00027}, "inclusive": false, "checks": [[null, 1, [3e-05, 0.00018, 3e-05, 3e-05, 3e-06]], [null, 272000, [3e-05, 0.00018, 3e-05, 3e-05, 3e-06]], [null, 272001, [6e-05, 0.00027, 6e-05, 6e-05, 6e-06]], [null, 10000000, [6e-05, 0.00027, 6e-05, 6e-05, 6e-06]]]}, +{"info": {"input_cost_per_token": 1.5e-07, "input_cost_per_token_batches": 1.2e-07, "output_cost_per_token": 0}, "inclusive": false, "checks": [[null, 1, [1.5e-07, 0.0, 1.5e-07, 1.5e-07, 1.5e-07]], [null, 10000000, [1.5e-07, 0.0, 1.5e-07, 1.5e-07, 1.5e-07]]]}, +{"info": {"cache_read_input_token_cost_priority": 6e-07, "input_cost_per_token_priority": 1.2e-06, "output_cost_per_token_priority": 1.2e-06}, "inclusive": false, "checks": [[null, 1, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 10000000, [0.0, 0.0, 0.0, 0.0, 0.0]], ["priority", 1, [1.2e-06, 1.2e-06, 1.2e-06, 1.2e-06, 6e-07]], ["priority", 10000000, [1.2e-06, 1.2e-06, 1.2e-06, 1.2e-06, 6e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "input_cost_per_token": 1.25e-06, "output_cost_per_token": 2.5e-06, "input_cost_per_token_above_200k_tokens": 2.5e-06, "output_cost_per_token_above_200k_tokens": 5e-06, "cache_read_input_token_cost_above_200k_tokens": 4e-07}, "inclusive": true, "checks": [[null, 1, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]], [null, 200001, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]], [null, 10000000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_batches": 1.6e-07, "input_cost_per_token": 1.25e-06, "input_cost_per_token_batches": 1e-06, "output_cost_per_token": 2.5e-06, "output_cost_per_token_batches": 2e-06, "input_cost_per_token_above_200k_tokens": 2.5e-06, "input_cost_per_token_above_200k_tokens_batches": 2e-06, "output_cost_per_token_above_200k_tokens": 5e-06, "output_cost_per_token_above_200k_tokens_batches": 4e-06, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_read_input_token_cost_above_200k_tokens_batches": 3.2e-07}, "inclusive": true, "checks": [[null, 1, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]], [null, 200001, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]], [null, 10000000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 4e-07]]]}, +{"info": {"input_cost_per_token": 5e-06}, "inclusive": false, "checks": [[null, 1, [5e-06, 0.0, 5e-06, 5e-06, 5e-06]], [null, 10000000, [5e-06, 0.0, 5e-06, 5e-06, 5e-06]]]}, +{"info": {"cache_read_input_token_cost": 1.5e-08, "input_cost_per_token": 6e-08, "input_cost_per_token_batches": 3e-08, "output_cost_per_token": 2.4e-07, "output_cost_per_token_batches": 1.2e-07}, "inclusive": false, "checks": [[null, 1, [6e-08, 2.4e-07, 6e-08, 6e-08, 1.5e-08]], [null, 10000000, [6e-08, 2.4e-07, 6e-08, 6e-08, 1.5e-08]]]}, +{"info": {"input_cost_per_token": 3.3e-05, "input_cost_per_token_above_272k_tokens": 6.6e-05, "input_cost_per_token_above_272k_tokens_batches": 3.3e-05, "input_cost_per_token_batches": 1.65e-05, "output_cost_per_token": 0.000198, "output_cost_per_token_above_272k_tokens": 0.000297, "output_cost_per_token_above_272k_tokens_batches": 0.0001485, "output_cost_per_token_batches": 9.9e-05}, "inclusive": false, "checks": [[null, 1, [3.3e-05, 0.000198, 3.3e-05, 3.3e-05, 3.3e-05]], [null, 272000, [3.3e-05, 0.000198, 3.3e-05, 3.3e-05, 3.3e-05]], [null, 272001, [6.6e-05, 0.000297, 6.6e-05, 6.6e-05, 6.6e-05]], [null, 10000000, [6.6e-05, 0.000297, 6.6e-05, 6.6e-05, 6.6e-05]]]}, +{"info": {"cache_read_input_token_cost": 7.5e-08, "cache_read_input_token_cost_priority": 1.3125e-07, "input_cost_per_token": 3e-07, "input_cost_per_token_batches": 1.5e-07, "input_cost_per_token_priority": 5.25e-07, "output_cost_per_token": 2.5e-06, "output_cost_per_token_batches": 1.25e-06, "output_cost_per_token_priority": 4.375e-06}, "inclusive": false, "checks": [[null, 1, [3e-07, 2.5e-06, 3e-07, 3e-07, 7.5e-08]], [null, 10000000, [3e-07, 2.5e-06, 3e-07, 3e-07, 7.5e-08]], ["priority", 1, [5.25e-07, 4.375e-06, 5.25e-07, 5.25e-07, 1.3125e-07]], ["priority", 10000000, [5.25e-07, 4.375e-06, 5.25e-07, 5.25e-07, 1.3125e-07]]]}, +{"info": {"input_cost_per_token": 1.25e-06, "output_cost_per_token": 2.5e-06, "cache_read_input_token_cost": 2e-07, "input_cost_per_token_above_200k_tokens": 2.5e-06, "output_cost_per_token_above_200k_tokens": 5e-06}, "inclusive": false, "checks": [[null, 1, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200000, [1.25e-06, 2.5e-06, 1.25e-06, 1.25e-06, 2e-07]], [null, 200001, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 2e-07]], [null, 10000000, [2.5e-06, 5e-06, 2.5e-06, 2.5e-06, 2e-07]]]}, +{"info": {"input_cost_per_token": 5e-06, "output_cost_per_token": 3e-05, "cache_read_input_token_cost": 5e-07, "input_cost_per_token_above_272k_tokens": 1e-05, "output_cost_per_token_above_272k_tokens": 4.5e-05}, "inclusive": false, "checks": [[null, 1, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272000, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 5e-07]], [null, 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 5e-07]]]}, +{"info": {"input_cost_per_token": 1.5e-07, "input_cost_per_token_batches": 7.5e-08, "input_cost_per_token_flex": 7.5e-08, "input_cost_per_token_priority": 2.625e-07, "output_cost_per_token": 6e-07, "output_cost_per_token_batches": 3e-07, "output_cost_per_token_flex": 3e-07, "output_cost_per_token_priority": 1.05e-06}, "inclusive": false, "checks": [[null, 1, [1.5e-07, 6e-07, 1.5e-07, 1.5e-07, 1.5e-07]], [null, 10000000, [1.5e-07, 6e-07, 1.5e-07, 1.5e-07, 1.5e-07]], ["priority", 1, [2.625e-07, 1.05e-06, 2.625e-07, 2.625e-07, 2.625e-07]], ["priority", 10000000, [2.625e-07, 1.05e-06, 2.625e-07, 2.625e-07, 2.625e-07]], ["flex", 1, [7.5e-08, 3e-07, 7.5e-08, 7.5e-08, 7.5e-08]], ["flex", 10000000, [7.5e-08, 3e-07, 7.5e-08, 7.5e-08, 7.5e-08]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_token": 1.2e-06, "output_cost_per_token": 6e-06, "range": [0, 32000.0]}, {"input_cost_per_token": 2.4e-06, "output_cost_per_token": 1.2e-05, "range": [32000.0, 128000.0]}, {"input_cost_per_token": 3e-06, "output_cost_per_token": 1.5e-05, "range": [128000.0, 252000.0]}]}, "inclusive": false, "checks": [[null, 1, [1.2e-06, 6e-06, 1.2e-06, 1.2e-06, 1.2e-06]], [null, 32000, [1.2e-06, 6e-06, 1.2e-06, 1.2e-06, 1.2e-06]], [null, 32001, [2.4e-06, 1.2e-05, 2.4e-06, 2.4e-06, 2.4e-06]], [null, 128000, [2.4e-06, 1.2e-05, 2.4e-06, 2.4e-06, 2.4e-06]], [null, 128001, [3e-06, 1.5e-05, 3e-06, 3e-06, 3e-06]], [null, 10000000, [3e-06, 1.5e-05, 3e-06, 3e-06, 3e-06]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-06, "cache_creation_input_token_cost_above_1hr": 2e-06, "cache_read_input_token_cost": 1e-07, "input_cost_per_token": 1e-06, "output_cost_per_token": 5e-06, "input_cost_per_token_batches": 5e-07, "output_cost_per_token_batches": 2.5e-06}, "inclusive": false, "checks": [[null, 1, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]], [null, 10000000, [1e-06, 5e-06, 1.25e-06, 2e-06, 1e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07, "cache_read_input_token_cost_priority": 3.6e-07, "input_cost_per_token_above_200k_tokens_priority": 7.2e-06, "input_cost_per_token_batches": 1e-06, "input_cost_per_token_priority": 3.6e-06, "output_cost_per_token_above_200k_tokens_priority": 3.24e-05, "output_cost_per_token_batches": 6e-06, "output_cost_per_token_priority": 2.16e-05, "cache_read_input_token_cost_flex": 2e-07, "input_cost_per_token_flex": 1e-06, "output_cost_per_token_flex": 6e-06}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], ["priority", 1, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200000, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200001, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]], ["priority", 10000000, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]], ["flex", 1, [1e-06, 6e-06, 1e-06, 1e-06, 2e-07]], ["flex", 200000, [1e-06, 6e-06, 1e-06, 1e-06, 2e-07]], ["flex", 200001, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], ["flex", 10000000, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]]]}, +{"info": {"input_cost_per_token": 5e-06, "input_cost_per_token_batches": 2.5e-06, "input_cost_per_token_priority": 8.75e-06, "output_cost_per_token": 1.5e-05, "output_cost_per_token_batches": 7.5e-06, "output_cost_per_token_priority": 2.625e-05}, "inclusive": false, "checks": [[null, 1, [5e-06, 1.5e-05, 5e-06, 5e-06, 5e-06]], [null, 10000000, [5e-06, 1.5e-05, 5e-06, 5e-06, 5e-06]], ["priority", 1, [8.75e-06, 2.625e-05, 8.75e-06, 8.75e-06, 8.75e-06]], ["priority", 10000000, [8.75e-06, 2.625e-05, 8.75e-06, 8.75e-06, 8.75e-06]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_priority": 3.6e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07, "cache_read_input_token_cost_batches": 1e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_priority": 3.6e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "input_cost_per_token_above_200k_tokens_priority": 7.2e-06, "input_cost_per_token_batches": 1e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_priority": 2.16e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "output_cost_per_token_above_200k_tokens_priority": 3.24e-05, "output_cost_per_token_batches": 6e-06}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], ["priority", 1, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200000, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200001, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]], ["priority", 10000000, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07, "cache_read_input_token_cost_batches": 1e-07, "cache_read_input_token_cost_flex": 1e-07, "cache_read_input_token_cost_priority": 3.6e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "input_cost_per_token_above_200k_tokens_priority": 7.2e-06, "input_cost_per_token_batches": 1e-06, "input_cost_per_token_flex": 1e-06, "input_cost_per_token_priority": 3.6e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "output_cost_per_token_above_200k_tokens_priority": 3.24e-05, "output_cost_per_token_batches": 6e-06, "output_cost_per_token_flex": 6e-06, "output_cost_per_token_priority": 2.16e-05}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], ["priority", 1, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200000, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200001, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]], ["priority", 10000000, [7.2e-06, 3.24e-05, 7.2e-06, 7.2e-06, 7.2e-07]], ["flex", 1, [1e-06, 6e-06, 1e-06, 1e-06, 1e-07]], ["flex", 200000, [1e-06, 6e-06, 1e-06, 1e-06, 1e-07]], ["flex", 200001, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]], ["flex", 10000000, [4e-06, 1.8e-05, 4e-06, 4e-06, 4e-07]]]}, +{"info": {"input_cost_per_token": 3e-07, "input_cost_per_token_batches": 1.5e-07, "input_cost_per_token_flex": 1.5e-07, "output_cost_per_token": 2.5e-06, "output_cost_per_token_batches": 1.25e-06, "output_cost_per_token_flex": 1.25e-06}, "inclusive": false, "checks": [[null, 1, [3e-07, 2.5e-06, 3e-07, 3e-07, 3e-07]], [null, 10000000, [3e-07, 2.5e-06, 3e-07, 3e-07, 3e-07]], ["flex", 1, [1.5e-07, 1.25e-06, 1.5e-07, 1.5e-07, 1.5e-07]], ["flex", 10000000, [1.5e-07, 1.25e-06, 1.5e-07, 1.5e-07, 1.5e-07]]]}, +{"info": {"cache_creation_input_token_cost": 3.75e-06, "cache_creation_input_token_cost_above_1hr": 6e-06, "cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05, "cache_read_input_token_cost": 3e-07, "input_cost_per_token": 3e-06, "input_cost_per_token_above_200k_tokens": 6e-06, "output_cost_per_token_above_200k_tokens": 2.25e-05, "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, "cache_read_input_token_cost_above_200k_tokens": 6e-07, "cache_read_input_token_cost_batches": 1.5e-07, "input_cost_per_token_batches": 1.5e-06, "output_cost_per_token": 1.5e-05, "output_cost_per_token_batches": 7.5e-06, "cache_creation_input_token_cost_batches": 1.88e-06}, "inclusive": false, "checks": [[null, 1, [3e-06, 1.5e-05, 3.75e-06, 6e-06, 3e-07]], [null, 200000, [3e-06, 1.5e-05, 3.75e-06, 6e-06, 3e-07]], [null, 200001, [6e-06, 2.25e-05, 7.5e-06, 1.2e-05, 6e-07]], [null, 10000000, [6e-06, 2.25e-05, 7.5e-06, 1.2e-05, 6e-07]]]}, +{"info": {"input_cost_per_token": 7.4e-07, "input_cost_per_token_flex": 3.7e-07, "output_cost_per_token": 2.22e-06, "output_cost_per_token_flex": 1.11e-06}, "inclusive": false, "checks": [[null, 1, [7.4e-07, 2.22e-06, 7.4e-07, 7.4e-07, 7.4e-07]], [null, 10000000, [7.4e-07, 2.22e-06, 7.4e-07, 7.4e-07, 7.4e-07]], ["flex", 1, [3.7e-07, 1.11e-06, 3.7e-07, 3.7e-07, 3.7e-07]], ["flex", 10000000, [3.7e-07, 1.11e-06, 3.7e-07, 3.7e-07, 3.7e-07]]]}, +{"info": {"cache_creation_input_token_cost": 3.75e-06, "cache_read_input_token_cost": 3e-07, "input_cost_per_token": 3e-06, "input_cost_per_token_above_200k_tokens": 6e-06, "output_cost_per_token_above_200k_tokens": 2.25e-05, "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, "cache_read_input_token_cost_above_200k_tokens": 6e-07, "output_cost_per_token": 1.5e-05}, "inclusive": false, "checks": [[null, 1, [3e-06, 1.5e-05, 3.75e-06, 3.75e-06, 3e-07]], [null, 200000, [3e-06, 1.5e-05, 3.75e-06, 3.75e-06, 3e-07]], [null, 200001, [6e-06, 2.25e-05, 7.5e-06, 7.5e-06, 6e-07]], [null, 10000000, [6e-06, 2.25e-05, 7.5e-06, 7.5e-06, 6e-07]]]}, +{"info": {"cache_creation_input_token_cost": 4.5e-06, "cache_creation_input_token_cost_above_1hr": 7.2e-06, "cache_read_input_token_cost": 3.6e-07, "input_cost_per_token": 3.6e-06, "input_cost_per_token_above_200k_tokens": 7.2e-06, "output_cost_per_token_above_200k_tokens": 2.7e-05, "cache_creation_input_token_cost_above_200k_tokens": 9e-06, "cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.44e-05, "cache_read_input_token_cost_above_200k_tokens": 7.2e-07, "output_cost_per_token": 1.8e-05}, "inclusive": false, "checks": [[null, 1, [3.6e-06, 1.8e-05, 4.5e-06, 7.2e-06, 3.6e-07]], [null, 200000, [3.6e-06, 1.8e-05, 4.5e-06, 7.2e-06, 3.6e-07]], [null, 200001, [7.2e-06, 2.7e-05, 9e-06, 1.44e-05, 7.2e-07]], [null, 10000000, [7.2e-06, 2.7e-05, 9e-06, 1.44e-05, 7.2e-07]]]}, +{"info": {"cache_creation_input_token_cost": 3.75e-06, "cache_creation_input_token_cost_above_1hr": 6e-06, "cache_creation_input_token_cost_above_1hr_above_200k_tokens": 1.2e-05, "cache_read_input_token_cost": 3e-07, "input_cost_per_token": 3e-06, "input_cost_per_token_above_200k_tokens": 6e-06, "output_cost_per_token_above_200k_tokens": 2.25e-05, "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, "cache_read_input_token_cost_above_200k_tokens": 6e-07, "output_cost_per_token": 1.5e-05, "input_cost_per_token_batches": 1.5e-06, "output_cost_per_token_batches": 7.5e-06}, "inclusive": false, "checks": [[null, 1, [3e-06, 1.5e-05, 3.75e-06, 6e-06, 3e-07]], [null, 200000, [3e-06, 1.5e-05, 3.75e-06, 6e-06, 3e-07]], [null, 200001, [6e-06, 2.25e-05, 7.5e-06, 1.2e-05, 6e-07]], [null, 10000000, [6e-06, 2.25e-05, 7.5e-06, 1.2e-05, 6e-07]]]}, +{"info": {"input_cost_per_token": 1.25e-06, "input_cost_per_token_above_200k_tokens": 2.5e-06, "output_cost_per_token": 1e-05, "output_cost_per_token_above_200k_tokens": 1.5e-05}, "inclusive": false, "checks": [[null, 1, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 1.25e-06]], [null, 200000, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 1.25e-06]], [null, 200001, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-06]], [null, 10000000, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-06]]]}, +{"info": {"cache_read_input_token_cost": 1.25e-06, "input_cost_per_token": 5e-06}, "inclusive": false, "checks": [[null, 1, [5e-06, 0.0, 5e-06, 5e-06, 1.25e-06]], [null, 10000000, [5e-06, 0.0, 5e-06, 5e-06, 1.25e-06]]]}, +{"info": {"input_cost_per_token": 3e-07, "input_cost_per_token_above_512k_tokens": 6e-07, "output_cost_per_token": 1.2e-06, "output_cost_per_token_above_512k_tokens": 2.4e-06, "cache_read_input_token_cost": 6e-08, "cache_read_input_token_cost_above_512k_tokens": 1.2e-07}, "inclusive": false, "checks": [[null, 1, [3e-07, 1.2e-06, 3e-07, 3e-07, 6e-08]], [null, 512000, [3e-07, 1.2e-06, 3e-07, 3e-07, 6e-08]], [null, 512001, [6e-07, 2.4e-06, 6e-07, 6e-07, 1.2e-07]], [null, 10000000, [6e-07, 2.4e-06, 6e-07, 6e-07, 1.2e-07]]]}, +{"info": {"cache_read_input_token_cost": 1.25e-06, "cache_read_input_token_cost_batches": 6.25e-07, "input_cost_per_token": 5e-06, "input_cost_per_token_batches": 2.5e-06}, "inclusive": false, "checks": [[null, 1, [5e-06, 0.0, 5e-06, 5e-06, 1.25e-06]], [null, 10000000, [5e-06, 0.0, 5e-06, 5e-06, 1.25e-06]]]}, +{"info": {"cache_read_input_token_cost": 5e-09, "cache_read_input_token_cost_batches": 2.5e-09, "cache_read_input_token_cost_flex": 2.5e-09, "input_cost_per_token": 5e-08, "input_cost_per_token_batches": 2.5e-08, "input_cost_per_token_flex": 2.5e-08, "input_cost_per_token_priority": 2.5e-06, "output_cost_per_token": 4e-07, "output_cost_per_token_batches": 2e-07, "output_cost_per_token_flex": 2e-07}, "inclusive": false, "checks": [[null, 1, [5e-08, 4e-07, 5e-08, 5e-08, 5e-09]], [null, 10000000, [5e-08, 4e-07, 5e-08, 5e-08, 5e-09]], ["priority", 1, [2.5e-06, 4e-07, 2.5e-06, 2.5e-06, 5e-09]], ["priority", 10000000, [2.5e-06, 4e-07, 2.5e-06, 2.5e-06, 5e-09]], ["flex", 1, [2.5e-08, 2e-07, 2.5e-08, 2.5e-08, 2.5e-09]], ["flex", 10000000, [2.5e-08, 2e-07, 2.5e-08, 2.5e-08, 2.5e-09]]]}, +{"info": {"cache_read_input_token_cost": 2e-08, "cache_read_input_token_cost_batches": 1e-08, "cache_read_input_token_cost_flex": 1e-08, "input_cost_per_token": 2e-07, "output_cost_per_token": 1.25e-06, "input_cost_per_token_batches": 1e-07, "input_cost_per_token_flex": 1e-07, "output_cost_per_token_batches": 6.25e-07, "output_cost_per_token_flex": 6.25e-07}, "inclusive": false, "checks": [[null, 1, [2e-07, 1.25e-06, 2e-07, 2e-07, 2e-08]], [null, 10000000, [2e-07, 1.25e-06, 2e-07, 2e-07, 2e-08]], ["flex", 1, [1e-07, 6.25e-07, 1e-07, 1e-07, 1e-08]], ["flex", 10000000, [1e-07, 6.25e-07, 1e-07, 1e-07, 1e-08]]]}, +{"info": {"input_cost_per_token": 3e-05, "input_cost_per_token_above_272k_tokens": 6e-05, "input_cost_per_token_flex": 1.5e-05, "input_cost_per_token_batches": 1.5e-05, "input_cost_per_token_above_272k_tokens_batches": 3e-05, "output_cost_per_token": 0.00018, "output_cost_per_token_above_272k_tokens": 0.00027, "output_cost_per_token_flex": 9e-05, "output_cost_per_token_batches": 9e-05, "output_cost_per_token_above_272k_tokens_batches": 0.000135, "input_cost_per_token_above_272k_tokens_flex": 3e-05, "output_cost_per_token_above_272k_tokens_flex": 0.000135}, "inclusive": false, "checks": [[null, 1, [3e-05, 0.00018, 3e-05, 3e-05, 3e-05]], [null, 272000, [3e-05, 0.00018, 3e-05, 3e-05, 3e-05]], [null, 272001, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]], [null, 10000000, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]], ["flex", 1, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 1.5e-05]], ["flex", 272000, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 1.5e-05]], ["flex", 272001, [3e-05, 0.000135, 3e-05, 3e-05, 3e-05]], ["flex", 10000000, [3e-05, 0.000135, 3e-05, 3e-05, 3e-05]]]}, +{"info": {"cache_read_input_token_cost": 5e-07, "cache_read_input_token_cost_above_272k_tokens": 1e-06, "cache_read_input_token_cost_batches": 2.5e-07, "cache_read_input_token_cost_above_272k_tokens_batches": 5e-07, "cache_read_input_token_cost_flex": 2.5e-07, "cache_read_input_token_cost_priority": 1.25e-06, "input_cost_per_token": 5e-06, "input_cost_per_token_above_272k_tokens": 1e-05, "input_cost_per_token_flex": 2.5e-06, "input_cost_per_token_batches": 2.5e-06, "input_cost_per_token_above_272k_tokens_batches": 5e-06, "input_cost_per_token_priority": 1.25e-05, "output_cost_per_token": 3e-05, "output_cost_per_token_above_272k_tokens": 4.5e-05, "output_cost_per_token_flex": 1.5e-05, "output_cost_per_token_batches": 1.5e-05, "output_cost_per_token_above_272k_tokens_batches": 2.25e-05, "output_cost_per_token_priority": 7.5e-05, "input_cost_per_token_above_272k_tokens_flex": 5e-06, "output_cost_per_token_above_272k_tokens_flex": 2.25e-05, "cache_read_input_token_cost_above_272k_tokens_flex": 5e-07}, "inclusive": false, "checks": [[null, 1, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272000, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], [null, 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["priority", 1, [1.25e-05, 7.5e-05, 1.25e-05, 1.25e-05, 1.25e-06]], ["priority", 272000, [1.25e-05, 7.5e-05, 1.25e-05, 1.25e-05, 1.25e-06]], ["priority", 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["priority", 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["flex", 1, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 272000, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 272001, [5e-06, 2.25e-05, 5e-06, 5e-06, 5e-07]], ["flex", 10000000, [5e-06, 2.25e-05, 5e-06, 5e-06, 5e-07]]]}, +{"info": {"input_cost_per_token": 3e-05, "input_cost_per_token_above_272k_tokens": 6e-05, "input_cost_per_token_flex": 1.5e-05, "input_cost_per_token_batches": 1.5e-05, "input_cost_per_token_above_272k_tokens_batches": 3e-05, "output_cost_per_token": 0.00018, "output_cost_per_token_above_272k_tokens": 0.00027, "output_cost_per_token_flex": 9e-05, "output_cost_per_token_batches": 9e-05, "output_cost_per_token_above_272k_tokens_batches": 0.000135}, "inclusive": false, "checks": [[null, 1, [3e-05, 0.00018, 3e-05, 3e-05, 3e-05]], [null, 272000, [3e-05, 0.00018, 3e-05, 3e-05, 3e-05]], [null, 272001, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]], [null, 10000000, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]], ["flex", 1, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 1.5e-05]], ["flex", 272000, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 1.5e-05]], ["flex", 272001, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]], ["flex", 10000000, [6e-05, 0.00027, 6e-05, 6e-05, 6e-05]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-07, "cache_creation_input_token_cost_decisions": 1.25e-07, "cache_creation_input_token_cost_above_272k_tokens": 2.5e-07, "cache_creation_input_token_cost_above_272k_tokens_decisions": 2.5e-07, "cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-07, "cache_creation_input_token_cost_above_272k_tokens_priority": 5e-07, "cache_creation_input_token_cost_flex": 6.25e-08, "cache_creation_input_token_cost_priority": 2.5e-07, "cache_read_input_token_cost": 1e-08, "cache_read_input_token_cost_decisions": 1e-08, "cache_read_input_token_cost_above_272k_tokens": 2e-08, "cache_read_input_token_cost_above_272k_tokens_decisions": 2e-08, "cache_read_input_token_cost_above_272k_tokens_flex": 1e-08, "cache_read_input_token_cost_above_272k_tokens_priority": 4e-08, "cache_read_input_token_cost_batches": 5e-09, "cache_read_input_token_cost_above_272k_tokens_batches": 1e-08, "cache_creation_input_token_cost_batches": 6.25e-08, "cache_creation_input_token_cost_above_272k_tokens_batches": 1.25e-07, "cache_read_input_token_cost_flex": 5e-09, "cache_read_input_token_cost_priority": 2e-08, "input_cost_per_token": 1e-07, "input_cost_per_token_decisions": 1e-07, "input_cost_per_token_above_272k_tokens": 2e-07, "input_cost_per_token_above_272k_tokens_decisions": 2e-07, "input_cost_per_token_above_272k_tokens_flex": 1e-07, "input_cost_per_token_above_272k_tokens_priority": 4e-07, "input_cost_per_token_batches": 5e-08, "input_cost_per_token_above_272k_tokens_batches": 1e-07, "input_cost_per_token_flex": 5e-08, "input_cost_per_token_priority": 2e-07, "output_cost_per_token": 5e-07, "output_cost_per_token_decisions": 0.0, "output_cost_per_token_above_272k_tokens": 7.5e-07, "output_cost_per_token_above_272k_tokens_decisions": 0.0, "output_cost_per_token_above_272k_tokens_flex": 3.75e-07, "output_cost_per_token_above_272k_tokens_priority": 1.5e-06, "output_cost_per_token_batches": 2.5e-07, "output_cost_per_token_above_272k_tokens_batches": 3.75e-07, "output_cost_per_token_flex": 2.5e-07, "output_cost_per_token_priority": 1e-06}, "inclusive": false, "checks": [[null, 1, [1e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 272000, [1e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 272001, [2e-07, 7.5e-07, 2.5e-07, 2.5e-07, 2e-08]], [null, 10000000, [2e-07, 7.5e-07, 2.5e-07, 2.5e-07, 2e-08]], ["priority", 1, [2e-07, 1e-06, 2.5e-07, 2.5e-07, 2e-08]], ["priority", 272000, [2e-07, 1e-06, 2.5e-07, 2.5e-07, 2e-08]], ["priority", 272001, [4e-07, 1.5e-06, 5e-07, 5e-07, 4e-08]], ["priority", 10000000, [4e-07, 1.5e-06, 5e-07, 5e-07, 4e-08]], ["flex", 1, [5e-08, 2.5e-07, 6.25e-08, 6.25e-08, 5e-09]], ["flex", 272000, [5e-08, 2.5e-07, 6.25e-08, 6.25e-08, 5e-09]], ["flex", 272001, [1e-07, 3.75e-07, 1.25e-07, 1.25e-07, 1e-08]], ["flex", 10000000, [1e-07, 3.75e-07, 1.25e-07, 1.25e-07, 1e-08]]]}, +{"info": {"cache_creation_input_token_cost": 1.25e-07, "cache_creation_input_token_cost_above_100k_tokens": 6.25e-07, "cache_read_input_token_cost": 1e-08, "cache_read_input_token_cost_above_100k_tokens": 5e-08, "input_cost_per_token": 1e-07, "input_cost_per_token_above_100k_tokens": 5e-07, "output_cost_per_token": 5e-07, "output_cost_per_token_above_100k_tokens": 2.5e-06}, "inclusive": false, "checks": [[null, 1, [1e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 100000, [1e-07, 5e-07, 1.25e-07, 1.25e-07, 1e-08]], [null, 100001, [5e-07, 2.5e-06, 6.25e-07, 6.25e-07, 5e-08]], [null, 10000000, [5e-07, 2.5e-06, 6.25e-07, 6.25e-07, 5e-08]]]}, +{"info": {"input_cost_per_token": 0, "input_cost_per_token_above_128k_tokens": 0, "output_cost_per_token": 0, "output_cost_per_token_above_128k_tokens": 0}, "inclusive": false, "checks": [[null, 1, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 128000, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 128001, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 10000000, [0.0, 0.0, 0.0, 0.0, 0.0]]]}, +{"info": {"cache_read_input_token_cost": 5e-08, "cache_read_input_token_cost_priority": 9e-08, "input_cost_per_token": 5e-07, "input_cost_per_token_batches": 2.5e-07, "input_cost_per_token_flex": 2.5e-07, "input_cost_per_token_priority": 9e-07, "output_cost_per_token": 3e-06, "output_cost_per_token_batches": 1.5e-06, "output_cost_per_token_flex": 1.5e-06, "output_cost_per_token_priority": 5.4e-06}, "inclusive": false, "checks": [[null, 1, [5e-07, 3e-06, 5e-07, 5e-07, 5e-08]], [null, 10000000, [5e-07, 3e-06, 5e-07, 5e-07, 5e-08]], ["priority", 1, [9e-07, 5.4e-06, 9e-07, 9e-07, 9e-08]], ["priority", 10000000, [9e-07, 5.4e-06, 9e-07, 9e-07, 9e-08]], ["flex", 1, [2.5e-07, 1.5e-06, 2.5e-07, 2.5e-07, 5e-08]], ["flex", 10000000, [2.5e-07, 1.5e-06, 2.5e-07, 2.5e-07, 5e-08]]]}, +{"info": {"cache_read_input_token_cost": 3e-08, "input_cost_per_token": 3e-07, "input_cost_per_token_batches": 1.5e-07, "input_cost_per_token_flex": 1.5e-07, "input_cost_per_token_priority": 5.4e-07, "output_cost_per_token": 2.5e-06}, "inclusive": false, "checks": [[null, 1, [3e-07, 2.5e-06, 3e-07, 3e-07, 3e-08]], [null, 10000000, [3e-07, 2.5e-06, 3e-07, 3e-07, 3e-08]], ["priority", 1, [5.4e-07, 2.5e-06, 5.4e-07, 5.4e-07, 3e-08]], ["priority", 10000000, [5.4e-07, 2.5e-06, 5.4e-07, 5.4e-07, 3e-08]], ["flex", 1, [1.5e-07, 2.5e-06, 1.5e-07, 1.5e-07, 3e-08]], ["flex", 10000000, [1.5e-07, 2.5e-06, 1.5e-07, 1.5e-07, 3e-08]]]}, +{"info": {"cache_read_input_token_cost": 8.3e-08, "input_cost_per_token": 1.65e-07, "input_cost_per_token_batches": 8.3e-08, "output_cost_per_token": 6.6e-07}, "inclusive": false, "checks": [[null, 1, [1.65e-07, 6.6e-07, 1.65e-07, 1.65e-07, 8.3e-08]], [null, 10000000, [1.65e-07, 6.6e-07, 1.65e-07, 1.65e-07, 8.3e-08]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_creation_input_token_cost_above_200k_tokens": 2.5e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "input_cost_per_token_batches": 1e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "output_cost_per_token_batches": 6e-06, "input_cost_per_token_priority": 3.6e-06, "input_cost_per_token_above_200k_tokens_priority": 7.2e-06, "output_cost_per_token_priority": 2.16e-05, "output_cost_per_token_above_200k_tokens_priority": 3.24e-05, "cache_read_input_token_cost_priority": 3.6e-07, "cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], ["priority", 1, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200000, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200001, [7.2e-06, 3.24e-05, 2.5e-07, 2.5e-07, 7.2e-07]], ["priority", 10000000, [7.2e-06, 3.24e-05, 2.5e-07, 2.5e-07, 7.2e-07]]]}, +{"info": {"cache_creation_input_token_cost_above_200k_tokens": 2.5e-07, "cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_read_input_token_cost_above_200k_tokens_priority": 7.2e-07, "cache_read_input_token_cost_priority": 3.6e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "input_cost_per_token_above_200k_tokens_priority": 7.2e-06, "input_cost_per_token_batches": 1e-06, "input_cost_per_token_flex": 1e-06, "input_cost_per_token_priority": 3.6e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "output_cost_per_token_above_200k_tokens_priority": 3.24e-05, "output_cost_per_token_batches": 6e-06, "output_cost_per_token_flex": 6e-06, "output_cost_per_token_priority": 2.16e-05}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], ["priority", 1, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200000, [3.6e-06, 2.16e-05, 3.6e-06, 3.6e-06, 3.6e-07]], ["priority", 200001, [7.2e-06, 3.24e-05, 2.5e-07, 2.5e-07, 7.2e-07]], ["priority", 10000000, [7.2e-06, 3.24e-05, 2.5e-07, 2.5e-07, 7.2e-07]], ["flex", 1, [1e-06, 6e-06, 1e-06, 1e-06, 2e-07]], ["flex", 200000, [1e-06, 6e-06, 1e-06, 1e-06, 2e-07]], ["flex", 200001, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], ["flex", 10000000, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]]]}, +{"info": {"cache_read_input_token_cost": 2e-07, "cache_read_input_token_cost_above_200k_tokens": 4e-07, "cache_creation_input_token_cost_above_200k_tokens": 2.5e-07, "input_cost_per_token": 2e-06, "input_cost_per_token_above_200k_tokens": 4e-06, "input_cost_per_token_batches": 1e-06, "output_cost_per_token": 1.2e-05, "output_cost_per_token_above_200k_tokens": 1.8e-05, "output_cost_per_token_batches": 6e-06}, "inclusive": false, "checks": [[null, 1, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200000, [2e-06, 1.2e-05, 2e-06, 2e-06, 2e-07]], [null, 200001, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]], [null, 10000000, [4e-06, 1.8e-05, 2.5e-07, 2.5e-07, 4e-07]]]}, +{"info": {"cache_read_input_token_cost": 1.25e-07, "cache_read_input_token_cost_above_200k_tokens": 2.5e-07, "cache_creation_input_token_cost_above_200k_tokens": 2.5e-07, "cache_read_input_token_cost_above_200k_tokens_priority": 4.5e-07, "cache_read_input_token_cost_flex": 1.25e-07, "cache_read_input_token_cost_priority": 2.25e-07, "input_cost_per_token": 1.25e-06, "input_cost_per_token_above_200k_tokens": 2.5e-06, "output_cost_per_token": 1e-05, "output_cost_per_token_above_200k_tokens": 1.5e-05, "input_cost_per_token_above_200k_tokens_priority": 4.5e-06, "input_cost_per_token_batches": 6.25e-07, "input_cost_per_token_flex": 6.25e-07, "input_cost_per_token_priority": 2.25e-06, "output_cost_per_token_above_200k_tokens_priority": 2.7e-05, "output_cost_per_token_batches": 5e-06, "output_cost_per_token_flex": 5e-06, "output_cost_per_token_priority": 1.8e-05}, "inclusive": false, "checks": [[null, 1, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 1.25e-07]], [null, 200000, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 1.25e-07]], [null, 200001, [2.5e-06, 1.5e-05, 2.5e-07, 2.5e-07, 2.5e-07]], [null, 10000000, [2.5e-06, 1.5e-05, 2.5e-07, 2.5e-07, 2.5e-07]], ["priority", 1, [2.25e-06, 1.8e-05, 2.25e-06, 2.25e-06, 2.25e-07]], ["priority", 200000, [2.25e-06, 1.8e-05, 2.25e-06, 2.25e-06, 2.25e-07]], ["priority", 200001, [4.5e-06, 2.7e-05, 2.5e-07, 2.5e-07, 4.5e-07]], ["priority", 10000000, [4.5e-06, 2.7e-05, 2.5e-07, 2.5e-07, 4.5e-07]], ["flex", 1, [6.25e-07, 5e-06, 6.25e-07, 6.25e-07, 1.25e-07]], ["flex", 200000, [6.25e-07, 5e-06, 6.25e-07, 6.25e-07, 1.25e-07]], ["flex", 200001, [2.5e-06, 1.5e-05, 2.5e-07, 2.5e-07, 2.5e-07]], ["flex", 10000000, [2.5e-06, 1.5e-05, 2.5e-07, 2.5e-07, 2.5e-07]]]}, +{"info": {"cache_creation_input_token_cost": 1.875e-06, "input_cost_per_token": 3.75e-06, "output_cost_per_token": 1.5e-05}, "inclusive": false, "checks": [[null, 1, [3.75e-06, 1.5e-05, 1.875e-06, 1.875e-06, 3.75e-06]], [null, 10000000, [3.75e-06, 1.5e-05, 1.875e-06, 1.875e-06, 3.75e-06]]]}, +{"info": {"cache_read_input_token_cost": 2.1e-07, "cache_read_input_token_cost_flex": 1.05e-07, "input_cost_per_token": 8.4e-07, "input_cost_per_token_flex": 4.2e-07, "output_cost_per_token": 3.36e-06, "output_cost_per_token_flex": 1.68e-06}, "inclusive": false, "checks": [[null, 1, [8.4e-07, 3.36e-06, 8.4e-07, 8.4e-07, 2.1e-07]], [null, 10000000, [8.4e-07, 3.36e-06, 8.4e-07, 8.4e-07, 2.1e-07]], ["flex", 1, [4.2e-07, 1.68e-06, 4.2e-07, 4.2e-07, 1.05e-07]], ["flex", 10000000, [4.2e-07, 1.68e-06, 4.2e-07, 4.2e-07, 1.05e-07]]]}, +{"info": {"input_cost_per_token": 2.5e-06, "output_cost_per_token": 1.25e-05, "cache_read_input_token_cost": 2.5e-07, "input_cost_per_token_balanced": 2e-06, "output_cost_per_token_balanced": 1e-05, "cache_read_input_token_cost_balanced": 2e-07, "input_cost_per_token_flex": 1.25e-06, "output_cost_per_token_flex": 6.25e-06, "cache_read_input_token_cost_flex": 1.5e-07}, "inclusive": false, "checks": [[null, 1, [2.5e-06, 1.25e-05, 2.5e-06, 2.5e-06, 2.5e-07]], [null, 10000000, [2.5e-06, 1.25e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 1, [1.25e-06, 6.25e-06, 1.25e-06, 1.25e-06, 1.5e-07]], ["flex", 10000000, [1.25e-06, 6.25e-06, 1.25e-06, 1.25e-06, 1.5e-07]], ["balanced", 1, [2e-06, 1e-05, 2e-06, 2e-06, 2e-07]], ["balanced", 10000000, [2e-06, 1e-05, 2e-06, 2e-06, 2e-07]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_query": 0.00166, "max_results_range": [1, 10]}, {"input_cost_per_query": 0.00332, "max_results_range": [11, 20]}, {"input_cost_per_query": 0.00498, "max_results_range": [21, 30]}, {"input_cost_per_query": 0.00664, "max_results_range": [31, 40]}, {"input_cost_per_query": 0.0083, "max_results_range": [41, 50]}, {"input_cost_per_query": 0.00996, "max_results_range": [51, 60]}, {"input_cost_per_query": 0.01162, "max_results_range": [61, 70]}, {"input_cost_per_query": 0.01328, "max_results_range": [71, 80]}, {"input_cost_per_query": 0.01494, "max_results_range": [81, 90]}, {"input_cost_per_query": 0.0166, "max_results_range": [91, 100]}]}, "inclusive": false, "checks": [[null, 1, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 10000000, [0.0, 0.0, 0.0, 0.0, 0.0]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_query": 0.005, "max_results_range": [0, 25]}, {"input_cost_per_query": 0.025, "max_results_range": [26, 100]}]}, "inclusive": false, "checks": [[null, 1, [0.0, 0.0, 0.0, 0.0, 0.0]], [null, 10000000, [0.0, 0.0, 0.0, 0.0, 0.0]]]}, +{"info": {"tiered_pricing": [{"cache_read_input_token_cost": 8e-08, "input_cost_per_token": 4e-07, "output_cost_per_token": 1.6e-06, "range": [0, 256000.0]}, {"cache_read_input_token_cost": 2.4e-07, "input_cost_per_token": 1.2e-06, "output_cost_per_token": 4.8e-06, "range": [256000.0, 1000000.0]}]}, "inclusive": false, "checks": [[null, 1, [4e-07, 1.6e-06, 4e-07, 4e-07, 8e-08]], [null, 256000, [4e-07, 1.6e-06, 4e-07, 4e-07, 8e-08]], [null, 256001, [1.2e-06, 4.8e-06, 1.2e-06, 1.2e-06, 2.4e-07]], [null, 10000000, [1.2e-06, 4.8e-06, 1.2e-06, 1.2e-06, 2.4e-07]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_token": 5e-08, "output_cost_per_token": 4e-07, "range": [0, 256000.0]}, {"input_cost_per_token": 2.5e-07, "output_cost_per_token": 2e-06, "range": [256000.0, 1000000.0]}]}, "inclusive": false, "checks": [[null, 1, [5e-08, 4e-07, 5e-08, 5e-08, 5e-08]], [null, 256000, [5e-08, 4e-07, 5e-08, 5e-08, 5e-08]], [null, 256001, [2.5e-07, 2e-06, 2.5e-07, 2.5e-07, 2.5e-07]], [null, 10000000, [2.5e-07, 2e-06, 2.5e-07, 2.5e-07, 2.5e-07]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_token": 3e-07, "output_cost_per_token": 1.5e-06, "range": [0, 32000.0]}, {"input_cost_per_token": 5e-07, "output_cost_per_token": 2.5e-06, "range": [32000.0, 128000.0]}, {"input_cost_per_token": 8e-07, "output_cost_per_token": 4e-06, "range": [128000.0, 256000.0]}, {"input_cost_per_token": 1.6e-06, "output_cost_per_token": 9.6e-06, "range": [256000.0, 1000000.0]}]}, "inclusive": false, "checks": [[null, 1, [3e-07, 1.5e-06, 3e-07, 3e-07, 3e-07]], [null, 32000, [3e-07, 1.5e-06, 3e-07, 3e-07, 3e-07]], [null, 32001, [5e-07, 2.5e-06, 5e-07, 5e-07, 5e-07]], [null, 128000, [5e-07, 2.5e-06, 5e-07, 5e-07, 5e-07]], [null, 128001, [8e-07, 4e-06, 8e-07, 8e-07, 8e-07]], [null, 256000, [8e-07, 4e-06, 8e-07, 8e-07, 8e-07]], [null, 256001, [1.6e-06, 9.6e-06, 1.6e-06, 1.6e-06, 1.6e-06]], [null, 10000000, [1.6e-06, 9.6e-06, 1.6e-06, 1.6e-06, 1.6e-06]]]}, +{"info": {"tiered_pricing": [{"cache_read_input_token_cost": 8e-08, "input_cost_per_token": 3e-07, "output_cost_per_token": 1.5e-06, "range": [0, 32000.0]}, {"cache_read_input_token_cost": 1.2e-07, "input_cost_per_token": 5e-07, "output_cost_per_token": 2.5e-06, "range": [32000.0, 128000.0]}, {"cache_read_input_token_cost": 2e-07, "input_cost_per_token": 8e-07, "output_cost_per_token": 4e-06, "range": [128000.0, 256000.0]}, {"cache_read_input_token_cost": 4e-07, "input_cost_per_token": 1.6e-06, "output_cost_per_token": 9.6e-06, "range": [256000.0, 1000000.0]}]}, "inclusive": false, "checks": [[null, 1, [3e-07, 1.5e-06, 3e-07, 3e-07, 8e-08]], [null, 32000, [3e-07, 1.5e-06, 3e-07, 3e-07, 8e-08]], [null, 32001, [5e-07, 2.5e-06, 5e-07, 5e-07, 1.2e-07]], [null, 128000, [5e-07, 2.5e-06, 5e-07, 5e-07, 1.2e-07]], [null, 128001, [8e-07, 4e-06, 8e-07, 8e-07, 2e-07]], [null, 256000, [8e-07, 4e-06, 8e-07, 8e-07, 2e-07]], [null, 256001, [1.6e-06, 9.6e-06, 1.6e-06, 1.6e-06, 4e-07]], [null, 10000000, [1.6e-06, 9.6e-06, 1.6e-06, 1.6e-06, 4e-07]]]}, +{"info": {"tiered_pricing": [{"input_cost_per_token": 4e-07, "output_cost_per_reasoning_token": 4e-06, "output_cost_per_token": 1.2e-06, "range": [0, 256000.0]}, {"input_cost_per_token": 1.2e-06, "output_cost_per_reasoning_token": 1.2e-05, "output_cost_per_token": 3.6e-06, "range": [256000.0, 1000000.0]}]}, "inclusive": false, "checks": [[null, 1, [4e-07, 1.2e-06, 4e-07, 4e-07, 4e-07]], [null, 256000, [4e-07, 1.2e-06, 4e-07, 4e-07, 4e-07]], [null, 256001, [1.2e-06, 3.6e-06, 1.2e-06, 1.2e-06, 1.2e-06]], [null, 10000000, [1.2e-06, 3.6e-06, 1.2e-06, 1.2e-06, 1.2e-06]]]}, +{"info": {"cache_creation_input_token_cost": 1.375e-06, "cache_read_input_token_cost": 1.1e-07, "input_cost_per_token": 1.1e-06, "output_cost_per_token": 5.5e-06, "input_cost_per_token_batches": 5.5e-07, "output_cost_per_token_batches": 2.75e-06}, "inclusive": false, "checks": [[null, 1, [1.1e-06, 5.5e-06, 1.375e-06, 1.375e-06, 1.1e-07]], [null, 10000000, [1.1e-06, 5.5e-06, 1.375e-06, 1.375e-06, 1.1e-07]]]}, +{"info": {"cache_read_input_token_cost": 2.75e-07, "cache_read_input_token_cost_above_272k_tokens": 5.5e-07, "cache_read_input_token_cost_above_272k_tokens_batches": 2.75e-07, "cache_read_input_token_cost_batches": 1.43e-07, "cache_read_input_token_cost_priority": 5.5e-07, "input_cost_per_token": 2.75e-06, "input_cost_per_token_priority": 5.5e-06, "output_cost_per_token": 1.65e-05, "output_cost_per_token_priority": 3.3e-05, "input_cost_per_token_above_272k_tokens": 5.5e-06, "input_cost_per_token_above_272k_tokens_batches": 2.75e-06, "input_cost_per_token_batches": 1.375e-06, "output_cost_per_token_above_272k_tokens": 2.475e-05, "output_cost_per_token_above_272k_tokens_batches": 1.2375e-05, "output_cost_per_token_batches": 8.25e-06}, "inclusive": false, "checks": [[null, 1, [2.75e-06, 1.65e-05, 2.75e-06, 2.75e-06, 2.75e-07]], [null, 272000, [2.75e-06, 1.65e-05, 2.75e-06, 2.75e-06, 2.75e-07]], [null, 272001, [5.5e-06, 2.475e-05, 5.5e-06, 5.5e-06, 5.5e-07]], [null, 10000000, [5.5e-06, 2.475e-05, 5.5e-06, 5.5e-06, 5.5e-07]], ["priority", 1, [5.5e-06, 3.3e-05, 5.5e-06, 5.5e-06, 5.5e-07]], ["priority", 272000, [5.5e-06, 3.3e-05, 5.5e-06, 5.5e-06, 5.5e-07]], ["priority", 272001, [5.5e-06, 2.475e-05, 5.5e-06, 5.5e-06, 5.5e-07]], ["priority", 10000000, [5.5e-06, 2.475e-05, 5.5e-06, 5.5e-06, 5.5e-07]]]}, +{"info": {"cache_read_input_token_cost": 5e-07, "cache_read_input_token_cost_above_272k_tokens": 1e-06, "cache_read_input_token_cost_above_272k_tokens_batches": 5e-07, "cache_read_input_token_cost_priority": 1.25e-06, "cache_read_input_token_cost_above_272k_tokens_priority": 2e-06, "cache_read_input_token_cost_batches": 2.5e-07, "cache_read_input_token_cost_flex": 2.5e-07, "input_cost_per_token": 5e-06, "input_cost_per_token_above_272k_tokens": 1e-05, "input_cost_per_token_above_272k_tokens_batches": 5e-06, "input_cost_per_token_above_272k_tokens_flex": 5e-06, "input_cost_per_token_priority": 1.25e-05, "input_cost_per_token_above_272k_tokens_priority": 2e-05, "input_cost_per_token_batches": 2.5e-06, "input_cost_per_token_flex": 2.5e-06, "output_cost_per_token": 3e-05, "output_cost_per_token_above_272k_tokens": 4.5e-05, "output_cost_per_token_above_272k_tokens_batches": 2.25e-05, "output_cost_per_token_priority": 7.5e-05, "output_cost_per_token_above_272k_tokens_priority": 9e-05, "output_cost_per_token_batches": 1.5e-05}, "inclusive": false, "checks": [[null, 1, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272000, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], [null, 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["priority", 1, [1.25e-05, 7.5e-05, 1.25e-05, 1.25e-05, 1.25e-06]], ["priority", 272000, [1.25e-05, 7.5e-05, 1.25e-05, 1.25e-05, 1.25e-06]], ["priority", 272001, [2e-05, 9e-05, 2e-05, 2e-05, 2e-06]], ["priority", 10000000, [2e-05, 9e-05, 2e-05, 2e-05, 2e-06]], ["flex", 1, [2.5e-06, 3e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 272000, [2.5e-06, 3e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 272001, [5e-06, 4.5e-05, 5e-06, 5e-06, 1e-06]], ["flex", 10000000, [5e-06, 4.5e-05, 5e-06, 5e-06, 1e-06]]]}, +{"info": {"cache_creation_input_token_cost": 5e-06, "cache_creation_input_token_cost_above_272k_tokens": 1e-05, "cache_creation_input_token_cost_priority": 1e-05, "cache_creation_input_token_cost_above_272k_tokens_priority": 2e-05, "cache_read_input_token_cost": 4e-07, "cache_read_input_token_cost_above_272k_tokens": 8e-07, "cache_read_input_token_cost_priority": 8e-07, "cache_read_input_token_cost_above_272k_tokens_priority": 1.6e-06, "input_cost_per_token": 4e-06, "input_cost_per_token_above_272k_tokens": 8e-06, "input_cost_per_token_priority": 8e-06, "input_cost_per_token_above_272k_tokens_priority": 1.6e-05, "output_cost_per_token": 2e-05, "output_cost_per_token_above_272k_tokens": 3e-05, "output_cost_per_token_priority": 4e-05, "output_cost_per_token_above_272k_tokens_priority": 6e-05}, "inclusive": false, "checks": [[null, 1, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], [null, 272000, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], [null, 272001, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], [null, 10000000, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], ["priority", 1, [8e-06, 4e-05, 1e-05, 1e-05, 8e-07]], ["priority", 272000, [8e-06, 4e-05, 1e-05, 1e-05, 8e-07]], ["priority", 272001, [1.6e-05, 6e-05, 2e-05, 2e-05, 1.6e-06]], ["priority", 10000000, [1.6e-05, 6e-05, 2e-05, 2e-05, 1.6e-06]]]}, +{"info": {"cache_creation_input_token_cost": 5e-06, "cache_creation_input_token_cost_above_272k_tokens": 1e-05, "cache_creation_input_token_cost_above_272k_tokens_flex": 5e-06, "cache_creation_input_token_cost_priority": 1e-05, "cache_creation_input_token_cost_above_272k_tokens_priority": 2e-05, "cache_creation_input_token_cost_flex": 2.5e-06, "cache_read_input_token_cost": 4e-07, "cache_read_input_token_cost_above_272k_tokens": 8e-07, "cache_read_input_token_cost_above_272k_tokens_flex": 4e-07, "cache_read_input_token_cost_priority": 8e-07, "cache_read_input_token_cost_above_272k_tokens_priority": 1.6e-06, "cache_read_input_token_cost_flex": 2e-07, "input_cost_per_token": 4e-06, "input_cost_per_token_above_272k_tokens": 8e-06, "input_cost_per_token_above_272k_tokens_flex": 4e-06, "input_cost_per_token_priority": 8e-06, "input_cost_per_token_above_272k_tokens_priority": 1.6e-05, "input_cost_per_token_flex": 2e-06, "output_cost_per_token": 2e-05, "output_cost_per_token_above_272k_tokens": 3e-05, "output_cost_per_token_above_272k_tokens_flex": 1.5e-05, "output_cost_per_token_priority": 4e-05, "output_cost_per_token_above_272k_tokens_priority": 6e-05, "output_cost_per_token_flex": 1e-05}, "inclusive": false, "checks": [[null, 1, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], [null, 272000, [4e-06, 2e-05, 5e-06, 5e-06, 4e-07]], [null, 272001, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], [null, 10000000, [8e-06, 3e-05, 1e-05, 1e-05, 8e-07]], ["priority", 1, [8e-06, 4e-05, 1e-05, 1e-05, 8e-07]], ["priority", 272000, [8e-06, 4e-05, 1e-05, 1e-05, 8e-07]], ["priority", 272001, [1.6e-05, 6e-05, 2e-05, 2e-05, 1.6e-06]], ["priority", 10000000, [1.6e-05, 6e-05, 2e-05, 2e-05, 1.6e-06]], ["flex", 1, [2e-06, 1e-05, 2.5e-06, 2.5e-06, 2e-07]], ["flex", 272000, [2e-06, 1e-05, 2.5e-06, 2.5e-06, 2e-07]], ["flex", 272001, [4e-06, 1.5e-05, 5e-06, 5e-06, 4e-07]], ["flex", 10000000, [4e-06, 1.5e-05, 5e-06, 5e-06, 4e-07]]]}, +{"info": {"cache_read_input_token_cost": 3e-06, "cache_read_input_token_cost_above_272k_tokens": 6e-06, "input_cost_per_token": 3e-05, "input_cost_per_token_above_272k_tokens": 6e-05, "input_cost_per_token_above_272k_tokens_batches": 3e-05, "input_cost_per_token_above_272k_tokens_flex": 3e-05, "input_cost_per_token_batches": 1.5e-05, "input_cost_per_token_flex": 1.5e-05, "output_cost_per_token": 0.00018, "output_cost_per_token_above_272k_tokens": 0.00027, "output_cost_per_token_above_272k_tokens_batches": 0.000135, "output_cost_per_token_above_272k_tokens_flex": 0.000135, "output_cost_per_token_batches": 9e-05, "output_cost_per_token_flex": 9e-05}, "inclusive": false, "checks": [[null, 1, [3e-05, 0.00018, 3e-05, 3e-05, 3e-06]], [null, 272000, [3e-05, 0.00018, 3e-05, 3e-05, 3e-06]], [null, 272001, [6e-05, 0.00027, 6e-05, 6e-05, 6e-06]], [null, 10000000, [6e-05, 0.00027, 6e-05, 6e-05, 6e-06]], ["flex", 1, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 3e-06]], ["flex", 272000, [1.5e-05, 9e-05, 1.5e-05, 1.5e-05, 3e-06]], ["flex", 272001, [3e-05, 0.000135, 3e-05, 3e-05, 6e-06]], ["flex", 10000000, [3e-05, 0.000135, 3e-05, 3e-05, 6e-06]]]}, +{"info": {"cache_read_input_token_cost": 2.5e-07, "cache_read_input_token_cost_above_272k_tokens": 5e-07, "cache_read_input_token_cost_above_272k_tokens_batches": 2.5e-07, "cache_read_input_token_cost_above_272k_tokens_flex": 2.5e-07, "cache_read_input_token_cost_priority": 5e-07, "cache_read_input_token_cost_above_272k_tokens_priority": 1e-06, "cache_read_input_token_cost_batches": 1.3e-07, "input_cost_per_token": 2.5e-06, "input_cost_per_token_above_272k_tokens": 5e-06, "input_cost_per_token_priority": 5e-06, "input_cost_per_token_above_272k_tokens_priority": 1e-05, "output_cost_per_token": 1.5e-05, "output_cost_per_token_above_272k_tokens": 2.25e-05, "output_cost_per_token_priority": 3e-05, "output_cost_per_token_above_272k_tokens_priority": 4.5e-05, "input_cost_per_token_above_272k_tokens_batches": 2.5e-06, "input_cost_per_token_above_272k_tokens_flex": 2.5e-06, "input_cost_per_token_batches": 1.25e-06, "input_cost_per_token_flex": 1.25e-06, "output_cost_per_token_above_272k_tokens_batches": 1.125e-05, "output_cost_per_token_above_272k_tokens_flex": 1.125e-05, "output_cost_per_token_batches": 7.5e-06, "output_cost_per_token_flex": 7.5e-06}, "inclusive": false, "checks": [[null, 1, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-07]], [null, 272000, [2.5e-06, 1.5e-05, 2.5e-06, 2.5e-06, 2.5e-07]], [null, 272001, [5e-06, 2.25e-05, 5e-06, 5e-06, 5e-07]], [null, 10000000, [5e-06, 2.25e-05, 5e-06, 5e-06, 5e-07]], ["priority", 1, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], ["priority", 272000, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], ["priority", 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["priority", 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["flex", 1, [1.25e-06, 7.5e-06, 1.25e-06, 1.25e-06, 2.5e-07]], ["flex", 272000, [1.25e-06, 7.5e-06, 1.25e-06, 1.25e-06, 2.5e-07]], ["flex", 272001, [2.5e-06, 1.125e-05, 2.5e-06, 2.5e-06, 2.5e-07]], ["flex", 10000000, [2.5e-06, 1.125e-05, 2.5e-06, 2.5e-06, 2.5e-07]]]}, +{"info": {"cache_read_input_token_cost": 5e-07, "cache_read_input_token_cost_above_272k_tokens": 1e-06, "cache_read_input_token_cost_priority": 1e-06, "cache_read_input_token_cost_above_272k_tokens_priority": 2e-06, "input_cost_per_token": 5e-06, "input_cost_per_token_above_272k_tokens": 1e-05, "input_cost_per_token_priority": 1e-05, "input_cost_per_token_above_272k_tokens_priority": 2e-05, "output_cost_per_token": 3e-05, "output_cost_per_token_above_272k_tokens": 4.5e-05, "output_cost_per_token_priority": 6e-05, "output_cost_per_token_above_272k_tokens_priority": 9e-05}, "inclusive": false, "checks": [[null, 1, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272000, [5e-06, 3e-05, 5e-06, 5e-06, 5e-07]], [null, 272001, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], [null, 10000000, [1e-05, 4.5e-05, 1e-05, 1e-05, 1e-06]], ["priority", 1, [1e-05, 6e-05, 1e-05, 1e-05, 1e-06]], ["priority", 272000, [1e-05, 6e-05, 1e-05, 1e-05, 1e-06]], ["priority", 272001, [2e-05, 9e-05, 2e-05, 2e-05, 2e-06]], ["priority", 10000000, [2e-05, 9e-05, 2e-05, 2e-05, 2e-06]]]}, +{"info": {"cache_read_input_token_cost": 3.125e-07, "input_cost_per_token": 1.25e-06, "input_cost_per_token_batches": 6.25e-07, "input_cost_per_token_priority": 2.1875e-06, "output_cost_per_token": 1e-05, "output_cost_per_token_batches": 5e-06, "output_cost_per_token_priority": 1.75e-05}, "inclusive": false, "checks": [[null, 1, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 3.125e-07]], [null, 10000000, [1.25e-06, 1e-05, 1.25e-06, 1.25e-06, 3.125e-07]], ["priority", 1, [2.1875e-06, 1.75e-05, 2.1875e-06, 2.1875e-06, 3.125e-07]], ["priority", 10000000, [2.1875e-06, 1.75e-05, 2.1875e-06, 2.1875e-06, 3.125e-07]]]} +]} diff --git a/litellm-rust/crates/model-catalog/tests/registry_validation.rs b/litellm-rust/crates/model-catalog/tests/registry_validation.rs deleted file mode 100644 index 67589010aba..00000000000 --- a/litellm-rust/crates/model-catalog/tests/registry_validation.rs +++ /dev/null @@ -1,93 +0,0 @@ -use std::path::{Path, PathBuf}; - -use litellm_model_catalog::{ - Catalog, FallbackGeneralizations, Provenance, validate_model_entry, validate_registry, -}; -use rstest::{fixture, rstest}; -use serde_json::{Map, Value}; - -#[fixture] -fn repo_root() -> PathBuf { - Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") -} - -#[rstest] -#[case("model_prices_and_context_window.json")] -#[case("litellm/model_prices_and_context_window_backup.json")] -#[ignore] -fn checked_in_registry_passes_strict_validation(repo_root: PathBuf, #[case] filename: &str) { - let body = std::fs::read(repo_root.join(filename)).unwrap(); - let catalog = Catalog::parse(&body, Provenance::default()).unwrap(); - validate_registry(&catalog).unwrap(); -} - -#[rstest] -#[ignore] -fn fallback_generalizations_are_typed(repo_root: PathBuf) { - let body = std::fs::read(repo_root.join("model_prices_and_context_window.json")).unwrap(); - let document: Map = serde_json::from_slice(&body).unwrap(); - let Some(raw_rules) = document.get("fallback_generalizations") else { - return; - }; - let _: FallbackGeneralizations = serde_json::from_value(raw_rules.clone()).unwrap(); - let catalog = Catalog::parse(&body, Provenance::default()).unwrap(); - assert!( - catalog - .fallback_rules() - .is_some_and(|rules| !rules.is_empty()) - ); -} - -#[rstest] -#[case::missing_provider(serde_json::json!({"mode": "chat"}), "litellm_provider")] -#[case::unknown_field(serde_json::json!({"litellm_provider": "test", "typo": true}), "unknown field")] -#[case::negative_price(serde_json::json!({"litellm_provider": "test", "input_cost_per_token": -1}), "nonnegative")] -#[case::negative_nested_price(serde_json::json!({"litellm_provider": "test", "guardrail_cost_per_unit": {"unit": -1}}), "nonnegative")] -#[case::invalid_mode(serde_json::json!({"litellm_provider": "test", "mode": "invalid"}), "unknown variant")] -#[case::invalid_date(serde_json::json!({"litellm_provider": "test", "deprecation_date": "2026-02-31"}), "deprecation_date")] -#[case::invalid_hours(serde_json::json!({"litellm_provider": "test", "off_peak_pricing": {"hours_utc": "25:00-01:00"}}), "hours_utc")] -#[case::empty_windows(serde_json::json!({"litellm_provider": "test", "off_peak_pricing": {"windows": []}}), "windows is empty")] -#[case::invalid_weekday(serde_json::json!({"litellm_provider": "test", "off_peak_pricing": {"windows": [{"hours_utc": "00:00-01:00", "weekdays": [0]}]}}), "weekdays is invalid")] -#[case::invalid_aliases(serde_json::json!({"litellm_provider": "test", "aliases": ["good", 7]}), "aliases must contain strings")] -#[case::null_aliases(serde_json::json!({"litellm_provider": "test", "aliases": null}), "aliases must be an array")] -#[ignore] -fn registry_validation_rejects_malformed_entries(#[case] entry: Value, #[case] expected: &str) { - assert!( - validate_model_entry("test", &entry) - .unwrap_err() - .to_string() - .contains(expected) - ); -} - -#[test] -#[ignore] -fn checked_in_catalog_and_backup_match() { - let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); - let current = std::fs::read(root.join("model_prices_and_context_window.json")).unwrap(); - let backup = - std::fs::read(root.join("litellm/model_prices_and_context_window_backup.json")).unwrap(); - assert_eq!(current, backup); - let catalog = Catalog::parse(¤t, Provenance::default()).unwrap(); - assert!( - catalog.alias_issues().is_empty(), - "invalid registry aliases" - ); -} - -#[rstest] -#[case::input("input_cost_per_token_above_100k_tokens")] -#[case::input_batches("input_cost_per_token_above_100k_tokens_batches")] -#[case::output("output_cost_per_token_above_100k_tokens")] -#[case::output_batches("output_cost_per_token_above_100k_tokens_batches")] -#[case::cache_creation("cache_creation_input_token_cost_above_100k_tokens")] -#[case::cache_creation_batches("cache_creation_input_token_cost_above_100k_tokens_batches")] -#[case::cache_creation_1hr("cache_creation_input_token_cost_above_1hr_above_100k_tokens")] -#[case::cache_read("cache_read_input_token_cost_above_100k_tokens")] -#[case::cache_read_batches("cache_read_input_token_cost_above_100k_tokens_batches")] -fn registry_validation_keeps_above_100k_tier_rates(#[case] field: &str) { - let mut entry = Map::new(); - entry.insert("litellm_provider".into(), "anthropic".into()); - entry.insert(field.into(), 5e-7.into()); - validate_model_entry("test", &Value::Object(entry)).unwrap(); -} diff --git a/litellm-rust/crates/model-catalog/tests/schema.rs b/litellm-rust/crates/model-catalog/tests/schema.rs index f9a028a78ab..464cf069cac 100644 --- a/litellm-rust/crates/model-catalog/tests/schema.rs +++ b/litellm-rust/crates/model-catalog/tests/schema.rs @@ -1,16 +1,17 @@ #![cfg(feature = "schema")] -use std::collections::BTreeSet; -use std::path::Path; - -use litellm_model_catalog::{model_entry_json_schema, registry_json_schema}; +use litellm_model_catalog::{BUNDLED_REGISTRY, model_entry_json_schema, registry_json_schema}; use rstest::rstest; -use serde_json::{Value, json}; +use serde_json::{Map, Value, json}; fn schema() -> Value { serde_json::to_value(model_entry_json_schema()).expect("generated schema serializes") } +fn checked_in_catalog() -> Value { + serde_json::from_slice(BUNDLED_REGISTRY).unwrap() +} + fn registry_validator() -> jsonschema::Validator { let schema = serde_json::to_value(registry_json_schema()).unwrap(); jsonschema::options() @@ -19,26 +20,21 @@ fn registry_validator() -> jsonschema::Validator { .expect("generated registry schema is valid") } -#[rstest] -#[case("model_prices_and_context_window.json")] -#[case("litellm/model_prices_and_context_window_backup.json")] -#[ignore] -fn generated_registry_schema_validates_checked_in_catalog(#[case] path: &str) { - let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); - let catalog: Value = serde_json::from_slice(&std::fs::read(root.join(path)).unwrap()).unwrap(); +#[test] +fn generated_registry_schema_validates_checked_in_catalog() { + let catalog = checked_in_catalog(); let validator = registry_validator(); let errors: Vec<_> = validator .iter_errors(&catalog) .map(|error| error.to_string()) .collect(); - assert!(errors.is_empty(), "{path}: {errors:?}"); + assert!(errors.is_empty(), "{errors:?}"); } #[rstest] #[case(json!({"example": {"litellm_provider": "test"}}))] #[case(json!({"example": {"litellm_provider": "test", "future_field": true}}))] #[case(json!({"sample_spec": {"litellm_provider": "placeholder"}}))] -#[ignore] fn generated_registry_schema_keeps_reader_compatibility(#[case] document: Value) { assert!(registry_validator().is_valid(&document)); } @@ -59,47 +55,55 @@ fn generated_registry_schema_keeps_reader_compatibility(#[case] document: Value) #[case::invalid_weekday(json!({"litellm_provider": "test", "off_peak_pricing": {"windows": [{"hours_utc": "00:00-01:00", "weekdays": [0]}]}}))] #[case::invalid_aliases(json!({"litellm_provider": "test", "aliases": "wrong"}))] #[case::non_object_model(json!(4))] -#[ignore] fn generated_registry_schema_rejects_invalid_entries(#[case] entry: Value) { assert!(!registry_validator().is_valid(&json!({"example": entry}))); } -#[rstest] -#[case("model_prices_and_context_window.json")] -#[case("litellm/model_prices_and_context_window_backup.json")] -#[ignore] -fn generated_schema_covers_catalog_fields(#[case] path: &str) { - let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); - let catalog: Value = serde_json::from_slice(&std::fs::read(root.join(path)).unwrap()).unwrap(); +#[test] +fn generated_schema_covers_catalog_fields() { + let catalog = checked_in_catalog(); let schema = schema(); - let properties = schema["properties"] + let any_value = |keys: &Value| -> Map { + keys.as_object() + .expect("schema keys are an object") + .keys() + .map(|key| (key.clone(), json!({}))) + .collect() + }; + let keys_only = json!({ + "type": "object", + "properties": any_value(&schema["properties"]), + "patternProperties": any_value(&schema["patternProperties"]), + "additionalProperties": false, + }); + let registry = serde_json::to_value(registry_json_schema()).unwrap(); + let sections = registry["properties"] .as_object() - .expect("ModelInfo schema has properties"); - let fields: BTreeSet<&str> = catalog + .expect("registry has sections"); + let fields: Map = catalog .as_object() .expect("catalog is an object") .iter() - .filter(|(name, _)| *name != "sample_spec" && *name != "fallback_generalizations") + .filter(|(name, _)| !sections.contains_key(*name)) .flat_map(|(_, entry)| entry.as_object().expect("model entry is an object").keys()) - .map(String::as_str) - .filter(|name| *name != "aliases") + .map(|name| (name.clone(), Value::Null)) .collect(); - let missing: Vec<_> = fields - .into_iter() - .filter(|name| !properties.contains_key(*name)) + let missing: Vec<_> = jsonschema::validator_for(&keys_only) + .unwrap() + .iter_errors(&Value::Object(fields)) + .map(|error| error.to_string()) .collect(); assert!( missing.is_empty(), - "{path}: fields missing from schema: {missing:?}" + "fields missing from schema: {missing:?}" ); } #[rstest] #[case("Mode", "chat")] -#[case("ReasoningEffort", "high")] +#[case("EffortLevel", "high")] #[case("InputModality", "image")] -#[ignore] fn generated_schema_includes_enum_values(#[case] definition: &str, #[case] value: &str) { let schema = schema(); let variants = schema["$defs"][definition]["enum"] @@ -110,7 +114,6 @@ fn generated_schema_includes_enum_values(#[case] definition: &str, #[case] value } #[test] -#[ignore] fn generated_schema_includes_nested_pricing_types() { let schema = schema(); let definitions = schema["$defs"].as_object().expect("schema has definitions"); diff --git a/litellm-rust/crates/python-bridge/AGENTS.md b/litellm-rust/crates/python-bridge/AGENTS.md index c435ba37e78..26858b2afe2 100644 --- a/litellm-rust/crates/python-bridge/AGENTS.md +++ b/litellm-rust/crates/python-bridge/AGENTS.md @@ -106,13 +106,28 @@ GIL handling to `litellm-host-python`. ## Data Handling +- Reuse large immutable bundled data and parsed catalogs across calls, including in + replacement APIs. Share Rust storage and cache repeated Python projections instead + of reparsing, serializing or copying the full payload on every access +- Cache owned Python objects with `PyOnceLock>`, then bind and clone references + while attached. Do not use blocking `OnceLock::get_or_init` while attached + ([PyO3 initialization](https://pyo3.rs/v0.29.2/faq.html), + [reference ownership](https://pyo3.rs/v0.29.2/types.html)) +- Never reuse Python objects across interpreters or interpreter reinitialization. + Once-only caches are for immutable data. Reloadable/custom data needs versioned + entries or invalidation, and caller-mutable Python containers must not be shared +- Test unchanged contents and allocation reuse, not just equal values. Keep this + coverage when replacing a bundled-data accessor with a catalog API - OCR payloads can contain personal data and large base64 images. Do not log payloads or provider responses. - Avoid copying large payloads more than needed. The current JSON round-trip is acceptable for the first scaffold, but future performance work should evaluate direct PyO3 conversion before expanding Rust coverage to image-heavy paths. -- Do not expose raw Rust errors that include document contents or upstream - bodies. +- Render a Rust error for Python only in its mapping function, with + `litellm_error::Report`, never `Debug` or a document's contents. An upstream + body reaches Python intact as the `RustUpstreamError` args for provider error + mapping. The 256-character cap applies only to HTTP error diagnostics, not + response bodies. Do not log those args. ## Tests diff --git a/litellm-rust/crates/python-bridge/Cargo.toml b/litellm-rust/crates/python-bridge/Cargo.toml index 8afde8362f6..6ecfebf0096 100644 --- a/litellm-rust/crates/python-bridge/Cargo.toml +++ b/litellm-rust/crates/python-bridge/Cargo.toml @@ -19,16 +19,25 @@ huggingface = ["litellm-token-counter/huggingface"] tiktoken = ["litellm-token-counter/tiktoken"] [dependencies] +litellm-secret-value.workspace = true fancy-regex.workspace = true litellm-tracing.workspace = true litellm-spend-clickhouse.workspace = true litellm-storage-clickhouse.workspace = true litellm-host.workspace = true +async-trait.workspace = true bytes.workspace = true futures-util.workspace = true litellm-cache.workspace = true -litellm-cache-memory.workspace = true -litellm-cache-redis.workspace = true +litellm-error.workspace = true +litellm-circuit-breaker.workspace = true +litellm-redis-client.workspace = true +litellm-redis-connect.workspace = true +litellm-redis-store.workspace = true +litellm-redis-resolve.workspace = true +litellm-python-compat.workspace = true +litellm-redis-types.workspace = true +tracing.workspace = true litellm-cache-response.workspace = true serde.workspace = true litellm-auth.workspace = true @@ -47,13 +56,15 @@ litellm-secrets = { workspace = true, features = ["aws", "azure", "google", "has litellm-secrets-types.workspace = true litellm-llms-types.workspace = true litellm-router-types.workspace = true +litellm-router-core.workspace = true litellm-host-python.workspace = true +litellm-model-catalog.workspace = true litellm-token-counter = { path = "../token-counter", default-features = false } -pyo3.workspace = true +indexmap = "2.14.0" +pyo3 = { workspace = true, features = ["indexmap"] } prost.workspace = true pyo3-async-runtimes.workspace = true reqwest.workspace = true -redis = { workspace = true, features = ["tls-rustls"] } serde_json.workspace = true strum.workspace = true veil.workspace = true @@ -70,6 +81,7 @@ criterion.workspace = true futures-util.workspace = true rstest.workspace = true sqlx = { workspace = true, features = ["migrate"] } +tempfile.workspace = true tokio-tungstenite.workspace = true wiremock.workspace = true aws-sdk-secretsmanager = "1.117.0" diff --git a/litellm-rust/crates/python-bridge/src/cache/AGENTS.md b/litellm-rust/crates/python-bridge/src/cache/AGENTS.md index 41c82187373..e79562b8762 100644 --- a/litellm-rust/crates/python-bridge/src/cache/AGENTS.md +++ b/litellm-rust/crates/python-bridge/src/cache/AGENTS.md @@ -1,7 +1,7 @@ # Cache boundary -This folder owns how Rust inference reaches the selected cache: global cache selection, route admission, delegation to a Python cache and the experimental V2 native handles. Cache algorithms, storage protocols and response-cache semantics belong to their cache crates +This folder owns how Rust inference reaches the selected Python cache. Cache algorithms, storage protocols and response-cache semantics belong to their cache crates -`mod.rs` exposes the cache boundary to routes and module registration. Adapter directories remain private. `selection.rs` asks `litellm.rust_bridge.host.cache` for the live configured cache, and owns route admission and inference protocol composition for both adapters +`mod.rs` exposes the cache boundary to routes. `selection.rs` asks `litellm.rust_bridge.host.cache` for the live configured cache and owns route admission and inference protocol composition -`python/` delegates operations to the selected Python cache without discovering configuration. `native/` owns the experimental V2 native cache handles. Neither adapter depends on shared selection or the other adapter. Shared composition depends on the adapters, and routes use only the parent module's exports +`python/` delegates operations to the selected Python cache without discovering configuration. Routes use only the parent module's exports diff --git a/litellm-rust/crates/python-bridge/src/cache/mod.rs b/litellm-rust/crates/python-bridge/src/cache/mod.rs index e190b12e665..4e3847951d8 100644 --- a/litellm-rust/crates/python-bridge/src/cache/mod.rs +++ b/litellm-rust/crates/python-bridge/src/cache/mod.rs @@ -1,21 +1,31 @@ -mod native; mod python; mod selection; -pub(crate) use native::NativeCacheHandle; pub(crate) use python::{CacheCall, PythonCache}; -pub(crate) use selection::{Cached, Selection, configure, configured_native}; +pub(crate) use selection::{Cached, Selection, configure, ensure_cache_disabled}; -use litellm_cache::Error; -use pyo3::{ - exceptions::{PyNotImplementedError, PyRuntimeError, PyValueError}, - prelude::*, -}; +use pyo3::{exceptions::PyRuntimeError, prelude::*}; -fn cache_error(error: Error) -> PyErr { - match error { - Error::InvalidEntry => PyValueError::new_err(error.to_string()), - Error::UnsupportedOperation => PyNotImplementedError::new_err(error.to_string()), - _ => PyRuntimeError::new_err(error.to_string()), +#[derive(Debug, thiserror::Error)] +pub(crate) enum HandleError { + #[error("this inference route does not support configured caches")] + Unsupported, + #[error("cache operation without configured cache")] + Unconfigured, + #[error("cache arguments unavailable")] + MissingArguments, + #[error("cache reply without pending operation")] + NothingPending, +} + +impl From for PyErr { + fn from(error: HandleError) -> Self { + let message = litellm_error::Report::new(&error).to_string(); + match error { + HandleError::Unsupported + | HandleError::Unconfigured + | HandleError::MissingArguments + | HandleError::NothingPending => PyRuntimeError::new_err(message), + } } } diff --git a/litellm-rust/crates/python-bridge/src/cache/native/AGENTS.md b/litellm-rust/crates/python-bridge/src/cache/native/AGENTS.md deleted file mode 100644 index 09a34d188cd..00000000000 --- a/litellm-rust/crates/python-bridge/src/cache/native/AGENTS.md +++ /dev/null @@ -1,9 +0,0 @@ -# Native cache bindings - -This directory exposes the experimental V2 native cache handles (`NativeCacheHandle`) to Python. Cache algorithms and storage protocols remain in their cache crates - -Accept the cache object selected by the parent module. Do not read global `litellm.cache`, decide route admission, or select the Python cache adapter here - -Keep Python-facing cache classes and method signatures stable when reorganizing modules. Native internals stay private to this directory unless the shared cache boundary or Python module registration needs them - -Verify changes with the V2 cache tests using a freshly built extension. Test cache behavior, not module paths or file structure diff --git a/litellm-rust/crates/python-bridge/src/cache/native/mod.rs b/litellm-rust/crates/python-bridge/src/cache/native/mod.rs deleted file mode 100644 index da1460f90ed..00000000000 --- a/litellm-rust/crates/python-bridge/src/cache/native/mod.rs +++ /dev/null @@ -1,3 +0,0 @@ -pub(super) mod v2; - -pub(crate) use v2::NativeCacheHandle; diff --git a/litellm-rust/crates/python-bridge/src/cache/native/v2.rs b/litellm-rust/crates/python-bridge/src/cache/native/v2.rs deleted file mode 100644 index eb09b8131c3..00000000000 --- a/litellm-rust/crates/python-bridge/src/cache/native/v2.rs +++ /dev/null @@ -1,345 +0,0 @@ -use crate::cache::cache_error; -use std::{ - sync::Arc, - time::{Duration, SystemTime, UNIX_EPOCH}, -}; - -use litellm_cache::{DeleteCache, DisconnectCache, PingCache}; -use litellm_host_python::{from_py, release_gil, to_py}; -use serde_json::Value; - -use litellm_cache_memory::InMemoryCache; -use litellm_cache_redis::{RedisCache, RedisTopology}; -use litellm_cache_response::{ - CacheEntry, CacheKeyInput, ExactResponseCache, ResponseCache, ResponseCacheCodec, - ResponseCacheConfig, ResponseCacheRequest, ResponseCacheService, -}; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; - -#[pyclass( - frozen, - name = "NativeCacheHandle", - module = "litellm.rust_bridge._native" -)] -pub(crate) struct NativeCacheHandle { - service: Arc, - backend: Arc, - storage: Storage, - pid: u32, -} - -#[derive(Clone)] -enum Storage { - Memory(Arc>), - Redis(Arc>), -} - -impl NativeCacheHandle { - fn check_process(&self) -> PyResult<()> { - if self.pid != std::process::id() { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "recreate the v2 cache after fork", - )); - } - Ok(()) - } -} - -fn request(key: String, ttl: Option) -> PyResult { - let mut request: ResponseCacheRequest = ResponseCacheRequest::new(CacheKeyInput { - preset: Some(key), - ..Default::default() - }); - request.context.ttl = ttl.map(duration).transpose()?; - Ok(request) -} - -#[pymethods] -impl NativeCacheHandle { - #[staticmethod] - #[pyo3(signature = (*, ttl=600.0, capacity=200, max_entry_bytes=4194304))] - fn memory(ttl: f64, capacity: usize, max_entry_bytes: usize) -> PyResult { - let ttl = duration(ttl)?; - if capacity == 0 || max_entry_bytes == 0 { - return Err(PyValueError::new_err("cache limits must be positive")); - } - let storage = Arc::new(InMemoryCache::new(Some(capacity), Some(ttl))); - let backend = Arc::new(ResponseCache::new(storage.clone()).with_config( - ResponseCacheConfig { - namespace: "sdk".into(), - max_entry_bytes, - }, - )); - Ok(Self { - service: backend.clone(), - backend, - storage: Storage::Memory(storage), - pid: std::process::id(), - }) - } - - #[staticmethod] - #[pyo3(signature = (url, *, namespace, ttl=600.0, max_entry_bytes=4194304))] - fn redis( - py: Python<'_>, - url: &str, - namespace: String, - ttl: f64, - max_entry_bytes: usize, - ) -> PyResult { - let ttl = duration(ttl)?; - if namespace.is_empty() || max_entry_bytes == 0 { - return Err(PyValueError::new_err( - "namespace and a positive cache limit are required", - )); - } - let storage = Arc::new( - release_gil(py, || { - RedisCache::connect( - url, - &RedisTopology::Standalone, - Some(ttl), - ResponseCacheCodec, - ) - }) - .map_err(cache_error)? - .with_namespace(Some(namespace.clone())), - ); - let backend = Arc::new(ResponseCache::new(storage.clone()).with_config( - ResponseCacheConfig { - namespace, - max_entry_bytes, - }, - )); - Ok(Self { - service: backend.clone(), - backend, - storage: Storage::Redis(storage), - pid: std::process::id(), - }) - } - fn get(&self, py: Python<'_>, key: String) -> PyResult> { - self.check_process()?; - let request = request(key, None)?; - let value = - release_gil(py, || self.backend.lookup(&request, now())).map_err(cache_error)?; - to_py(py, &value) - } - - #[pyo3(signature = (key, value, *, ttl=None))] - fn set( - &self, - py: Python<'_>, - key: String, - value: &Bound<'_, PyAny>, - ttl: Option, - ) -> PyResult<()> { - self.check_process()?; - let request = request(key, ttl)?; - let value: Value = from_py(value)?; - release_gil(py, || self.backend.store(&request, value, now())).map_err(cache_error) - } - - fn async_get<'py>(&self, py: Python<'py>, key: String) -> PyResult> { - self.check_process()?; - let request = request(key, None)?; - let backend = self.backend.clone(); - crate::execution::run_async( - py, - async move { backend.async_lookup(&request, now()).await }, - cache_error, - ) - } - - #[pyo3(signature = (key, value, *, ttl=None))] - fn async_set<'py>( - &self, - py: Python<'py>, - key: String, - value: &Bound<'_, PyAny>, - ttl: Option, - ) -> PyResult> { - self.check_process()?; - let request = request(key, ttl)?; - let value: Value = from_py(value)?; - let backend = self.backend.clone(); - crate::execution::run_async( - py, - async move { backend.async_store(&request, value, now()).await }, - cache_error, - ) - } - - #[pyo3(signature = (entries, *, ttl=None))] - fn async_set_many<'py>( - &self, - py: Python<'py>, - entries: &Bound<'_, PyAny>, - ttl: Option, - ) -> PyResult> { - self.check_process()?; - let entries: Vec<(String, Value)> = from_py(entries)?; - let entries = entries - .into_iter() - .map(|(key, value)| Ok((request(key, ttl)?, value))) - .collect::>>()?; - let backend = self.backend.clone(); - crate::execution::run_async( - py, - async move { backend.async_store_batch(entries, now()).await }, - cache_error, - ) - } - - fn flush(&self, py: Python<'_>) -> PyResult> { - self.check_process()?; - let backend = self.backend.clone(); - crate::execution::run_sync(py, async move { backend.async_flush().await }, cache_error) - } - - fn async_flush<'py>(&self, py: Python<'py>) -> PyResult> { - self.check_process()?; - let backend = self.backend.clone(); - crate::execution::run_async(py, async move { backend.async_flush().await }, cache_error) - } - - fn ping<'py>(&self, py: Python<'py>) -> PyResult> { - self.check_process()?; - let storage = self.storage.clone(); - crate::execution::run_async( - py, - async move { - match storage { - Storage::Memory(_) => Ok(true), - Storage::Redis(cache) => cache.ping().await, - } - }, - cache_error, - ) - } - - fn disconnect<'py>(&self, py: Python<'py>) -> PyResult> { - self.check_process()?; - let storage = self.storage.clone(); - crate::execution::run_async( - py, - async move { - match storage { - Storage::Memory(cache) => cache.disconnect().await, - Storage::Redis(cache) => cache.disconnect().await, - } - }, - cache_error, - ) - } - - fn delete<'py>(&self, py: Python<'py>, keys: Vec) -> PyResult> { - self.check_process()?; - let storage = self.storage.clone(); - crate::execution::run_async( - py, - async move { - for key in keys { - match &storage { - Storage::Memory(cache) => cache.async_delete_cache(&key).await?, - Storage::Redis(cache) => cache.async_delete_cache(&key).await?, - } - } - Ok(()) - }, - cache_error, - ) - } -} - -fn duration(seconds: f64) -> PyResult { - Duration::try_from_secs_f64(seconds) - .ok() - .filter(|value| !value.is_zero()) - .ok_or_else(|| PyValueError::new_err("cache durations must be finite and positive")) -} - -pub(in crate::cache) fn native_handle<'py>( - configured: &Bound<'py, PyAny>, -) -> PyResult>> { - Ok(configured - .getattr_opt("cache")? - .map(|backend| backend.getattr_opt("native_handle")) - .transpose()? - .flatten() - .filter(|handle| handle.is_instance_of::())) -} - -pub(in crate::cache) fn configured( - configured: &Bound<'_, PyAny>, - kwargs: &Bound<'_, PyDict>, -) -> PyResult<( - Option>, - litellm_cache_response::CacheOptions, -)> { - let handle = native_handle(configured)?.ok_or_else(|| { - pyo3::exceptions::PyRuntimeError::new_err( - "native inference needs a v2 cache, the catalog keeps Python caches on Python", - ) - })?; - let cache = handle.extract::>()?; - cache.check_process()?; - let controls = kwargs.get_item("cache")?.filter(|value| !value.is_none()); - let controls = controls - .as_ref() - .map(|value| value.cast::()) - .transpose()?; - if let Some(controls) = controls { - for name in controls.keys() { - let name = name.extract::()?; - if !matches!( - name.as_str(), - "no-cache" | "no-store" | "ttl" | "s-maxage" | "s-max-age" | "use-cache" - ) { - return Err(PyValueError::new_err(format!( - "unsupported v2 cache control: {name}" - ))); - } - } - } - let boolean = |name: &str| -> PyResult { - controls - .map(|values| values.get_item(name)) - .transpose()? - .flatten() - .map(|value| value.extract()) - .transpose() - .map(|value| value.unwrap_or(false)) - }; - let seconds = |name: &str| -> PyResult> { - controls - .map(|values| values.get_item(name)) - .transpose()? - .flatten() - .map(|value| duration(value.extract()?)) - .transpose() - }; - Ok(( - Some(cache.service.clone()), - litellm_cache_response::CacheOptions { - policy: litellm_cache_response::CachePolicy { - caching: kwargs - .get_item("caching")? - .filter(|value| !value.is_none()) - .map(|value| value.extract()) - .transpose()?, - no_cache: boolean("no-cache")?, - no_store: boolean("no-store")?, - ttl: seconds("ttl")?, - max_age: seconds("s-max-age")?.or(seconds("s-maxage")?), - }, - scope: litellm_cache_response::CacheScope::Shared, - }, - )) -} - -fn now() -> Duration { - SystemTime::now() - .duration_since(UNIX_EPOCH) - .unwrap_or_default() -} diff --git a/litellm-rust/crates/python-bridge/src/cache/python/host.rs b/litellm-rust/crates/python-bridge/src/cache/python/host.rs index d8f8ac7c181..82b7d881df6 100644 --- a/litellm-rust/crates/python-bridge/src/cache/python/host.rs +++ b/litellm-rust/crates/python-bridge/src/cache/python/host.rs @@ -9,6 +9,7 @@ use pyo3::{ use serde_json::Value; use super::service::CacheCall; +use crate::cache::HandleError; enum Pending { Lookup(Reply, Error>>), @@ -43,16 +44,12 @@ impl PythonCache { pub fn begin(&mut self, py: Python<'_>, call: CacheCall) -> PyResult>> { let Some(cache) = self.cache.as_ref() else { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "cache operation without configured cache", - )); + return Err(HandleError::Unconfigured.into()); }; let arguments = self .arguments .as_ref() - .ok_or_else(|| { - pyo3::exceptions::PyRuntimeError::new_err("cache arguments unavailable") - })? + .ok_or(HandleError::MissingArguments)? .bind(py) .copy()?; let (method, result) = match call { @@ -97,33 +94,30 @@ impl PythonCache { py: Python<'_>, result: PyResult>, ) -> PyResult>> { - if let Err(error) = &result - && !error.is_instance_of::(py) - { - self.pending = None; - return Err(result.err().unwrap()); - } + let result = match result { + Err(error) if !error.is_instance_of::(py) => { + self.pending = None; + return Err(error); + } + result => result, + }; match self.pending.take() { Some(Pending::Lookup(reply)) => { - let value = result.map_err(|_| Error::Unavailable).and_then(|value| { + let value = result.map_err(Error::unavailable).and_then(|value| { if value.bind(py).is_none() { Ok(None) } else { from_py(value.bind(py)) .map(Some) - .map_err(|_| Error::InvalidEntry) + .map_err(Error::invalid_entry) } }); reply.send(value); } Some(Pending::Store(reply)) => { - reply.send(result.map(|_| ()).map_err(|_| Error::Unavailable)); - } - None => { - return Err(pyo3::exceptions::PyRuntimeError::new_err( - "cache reply without pending operation", - )); + reply.send(result.map(|_| ()).map_err(Error::unavailable)); } + None => return Err(HandleError::NothingPending.into()), } Ok(None) } diff --git a/litellm-rust/crates/python-bridge/src/cache/python/service.rs b/litellm-rust/crates/python-bridge/src/cache/python/service.rs index f215e5e97f5..38872b85d30 100644 --- a/litellm-rust/crates/python-bridge/src/cache/python/service.rs +++ b/litellm-rust/crates/python-bridge/src/cache/python/service.rs @@ -17,18 +17,21 @@ fn from_python(value: Value) -> Result { let output = match value.get(STREAM_EVENTS_KEY) { Some(events) => { let events: Vec = - serde_json::from_value(events.clone()).map_err(|_| Error::InvalidEntry)?; + serde_json::from_value(events.clone()).map_err(Error::invalid_entry)?; CachedOutput::Stream(events.concat()) } None => CachedOutput::Response(value), }; - serde_json::to_value(ResponseEnvelope::new("messages", output)).map_err(|_| Error::InvalidEntry) + serde_json::to_value(ResponseEnvelope::new("messages", output)).map_err(Error::invalid_entry) } fn to_python(value: Value) -> Result { let envelope: ResponseEnvelope> = - serde_json::from_value(value).map_err(|_| Error::InvalidEntry)?; - match envelope.decode("messages").ok_or(Error::InvalidEntry)? { + serde_json::from_value(value).map_err(Error::invalid_entry)?; + match envelope + .decode("messages") + .ok_or(Error::InvalidEntry(None))? + { CachedOutput::Response(response) => Ok(response), CachedOutput::Stream(text) => Ok(serde_json::json!({ STREAM_EVENTS_KEY: text.split_inclusive("\n\n").collect::>() @@ -56,7 +59,7 @@ pub(in crate::cache) fn service>( namespace: String, ) -> std::sync::Arc where - P::Error: From, + P::Error: From + std::error::Error, { std::sync::Arc::new(PythonCacheService { services, @@ -69,7 +72,7 @@ where impl> ResponseCacheService for PythonCacheService

where - P::Error: From, + P::Error: From + std::error::Error, { fn config(&self) -> &ResponseCacheConfig { &self.config @@ -84,7 +87,7 @@ where self.services .call(|reply| CacheCall::Lookup { reply }) .await - .map_err(|_| Error::Unavailable)?? + .map_err(Error::unavailable)?? .map(from_python) .transpose() }) @@ -101,7 +104,7 @@ where self.services .call(|reply| CacheCall::Store { value, reply }) .await - .map_err(|_| Error::Unavailable)? + .map_err(Error::unavailable)? }) } } diff --git a/litellm-rust/crates/python-bridge/src/cache/selection.rs b/litellm-rust/crates/python-bridge/src/cache/selection.rs index cd19533fc73..e96f079e5da 100644 --- a/litellm-rust/crates/python-bridge/src/cache/selection.rs +++ b/litellm-rust/crates/python-bridge/src/cache/selection.rs @@ -1,14 +1,10 @@ -use super::{native, python}; -use litellm_cache_response::{ - CacheOptions, CachePolicy, CacheScope, ResponseCacheService, ScopedCache, -}; +use super::{HandleError, python}; +use litellm_cache_response::{CacheOptions, CachePolicy, CacheScope, ScopedCache}; use litellm_host::{ machine::{HostServices, MachineFault}, protocol::Protocol, }; use pyo3::{prelude::*, types::PyDict}; -use std::sync::Arc; - const CACHE_HOST_MODULE: &str = "litellm.rust_bridge.host.cache"; pub(crate) struct Cached

(std::marker::PhantomData

); @@ -24,7 +20,6 @@ impl Protocol for Cached

{ enum Backend { Disabled, - Native(Arc), Python { namespace: String }, } @@ -39,11 +34,10 @@ impl Selection { services: HostServices

, ) -> (Option, CacheOptions) where - P::Error: From, + P::Error: From + std::error::Error, { let service = match self.backend { Backend::Disabled => None, - Backend::Native(service) => Some(service), Backend::Python { namespace } => Some(python::service(services, namespace)), }; ( @@ -76,37 +70,21 @@ fn selected_cache<'py>( Ok(Some(configured)) } -pub(crate) fn configured_native( +pub(crate) fn ensure_cache_disabled( py: Python<'_>, kwargs: &Bound<'_, PyDict>, call_type: &str, -) -> PyResult<( - Option>, - litellm_cache_response::CacheOptions, -)> { +) -> PyResult<()> { let Some(configured) = selected_cache(py, kwargs, call_type)? else { - return Ok(( - None, - litellm_cache_response::CacheOptions::new(litellm_cache_response::CacheScope::Shared), - )); + return Ok(()); }; - native_configuration(&configured, kwargs) -} - -fn native_configuration( - configured: &Bound<'_, PyAny>, - kwargs: &Bound<'_, PyDict>, -) -> PyResult<(Option>, CacheOptions)> { - if !configured + if configured .call_method("should_use_cache", (), Some(kwargs))? .extract::()? { - return Ok(( - None, - litellm_cache_response::CacheOptions::new(litellm_cache_response::CacheScope::Shared), - )); + return Err(HandleError::Unsupported.into()); } - native::v2::configured(configured, kwargs) + Ok(()) } pub(crate) fn configure( @@ -122,13 +100,6 @@ pub(crate) fn configure( options: CacheOptions::new(CacheScope::Shared), }); }; - if native::v2::native_handle(&cache)?.is_some() { - let (native, options) = native_configuration(&cache, arguments)?; - return Ok(Selection { - backend: native.map_or(Backend::Disabled, Backend::Native), - options, - }); - } let enabled = cache .call_method("should_use_cache", (), Some(arguments))? .extract::()?; diff --git a/litellm-rust/crates/python-bridge/src/callable.rs b/litellm-rust/crates/python-bridge/src/callable.rs index d0ed76a3e8b..605b649e1f3 100644 --- a/litellm-rust/crates/python-bridge/src/callable.rs +++ b/litellm-rust/crates/python-bridge/src/callable.rs @@ -48,14 +48,8 @@ mod tests { Err(PyErr::from_value(error.clone())) } - #[rstest::rstest] - fn only_ordinary_exceptions_are_reported_under_the_template() { - Python::initialize(); - Python::attach(|py| { - let locals = PyDict::new(py); - py.run( - pyo3::ffi::c_str!( - r#" + const FAILURES: &std::ffi::CStr = pyo3::ffi::c_str!( + r#" class CallerError(Exception): def __format__(self, specification): return 'unavailable' @@ -63,12 +57,19 @@ ordinary = CallerError('must use __format__') type_error = TypeError('signature') abort = KeyboardInterrupt('cancelled') "# - ), - Some(&locals), - Some(&locals), - ) - .unwrap(); + ); + fn failures(py: Python<'_>) -> Bound<'_, PyDict> { + let locals = PyDict::new(py); + py.run(FAILURES, Some(&locals), Some(&locals)).unwrap(); + locals + } + + #[test] + fn ordinary_exceptions_are_reported_under_the_template() { + Python::initialize(); + Python::attach(|py| { + let locals = failures(py); let original = raised(&locals, "ordinary"); let wrapped = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); assert!(wrapped.is_instance_of::(py)); @@ -78,12 +79,19 @@ abort = KeyboardInterrupt('cancelled') wrapped.value(py).str().unwrap().to_str().unwrap(), "Failed to reach the caller: unavailable" ); + }); + } - for name in ["type_error", "abort"] { - let original = raised(&locals, name); - let error = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); - assert!(error.value(py).is(&original)); - } + #[rstest::rstest] + #[case::type_error("type_error")] + #[case::keyboard_interrupt("abort")] + fn other_exceptions_pass_through_unwrapped(#[case] name: &str) { + Python::initialize(); + Python::attach(|py| { + let locals = failures(py); + let original = raised(&locals, name); + let error = wrap_failure(py, TEMPLATE, failure(&original)).unwrap_err(); + assert!(error.value(py).is(&original)); }); } diff --git a/litellm-rust/crates/python-bridge/src/coercion.rs b/litellm-rust/crates/python-bridge/src/coercion.rs index 77c8cabc22b..8cdf1e6c5bc 100644 --- a/litellm-rust/crates/python-bridge/src/coercion.rs +++ b/litellm-rust/crates/python-bridge/src/coercion.rs @@ -1,4 +1,5 @@ use litellm_core_utils::serde_compat::parse_str_bool; +use litellm_error::Report; use pyo3::{ exceptions::{PyAttributeError, PyRuntimeError, PyValueError}, prelude::*, @@ -9,15 +10,20 @@ use pyo3::{ pub(crate) enum ProjectionError { Python(PyErr), InvalidConfiguration(ProjectionDetail), - UnsupportedLiveObject(String), + UnsupportedLiveObject(ProjectionDetail), InternalSchemaFailure(ProjectionDetail), } #[derive(Debug, thiserror::Error)] pub(crate) enum ProjectionDetail { - #[error("{0}")] - Message(String), - #[error("{field}: {source}")] + #[error("{group}.{name}: expected {expected}, got {actual}")] + Expected { + group: &'static str, + name: &'static str, + expected: &'static str, + actual: String, + }, + #[error("{field}")] InvalidField { field: &'static str, #[source] @@ -30,12 +36,6 @@ pub(crate) enum ProjectionDetail { }, } -impl From for ProjectionDetail { - fn from(message: String) -> Self { - Self::Message(message) - } -} - impl From for ProjectionError { fn from(error: PyErr) -> Self { Self::Python(error) @@ -46,12 +46,12 @@ impl From for PyErr { fn from(error: ProjectionError) -> Self { match error { ProjectionError::Python(error) => error, - ProjectionError::InvalidConfiguration(detail) => { - PyValueError::new_err(detail.to_string()) + ProjectionError::InvalidConfiguration(detail) + | ProjectionError::UnsupportedLiveObject(detail) => { + PyValueError::new_err(Report::new(&detail).to_string()) } - ProjectionError::UnsupportedLiveObject(message) => PyValueError::new_err(message), ProjectionError::InternalSchemaFailure(detail) => { - PyRuntimeError::new_err(detail.to_string()) + PyRuntimeError::new_err(Report::new(&detail).to_string()) } } } @@ -124,26 +124,26 @@ impl<'py> Field<'py> { && getter.is(object.getattr("__getattribute__")?)) } - pub(crate) fn path(&self) -> String { - format!("{}.{}", self.group, self.name) - } - /// A member of this field's collection, reported under the same path. pub(crate) fn member(&self, value: Bound<'py, PyAny>) -> Self { Self::new(self.group, self.name, value) } - pub(crate) fn expected(&self, expected: &str) -> Result { - Ok(format!( - "{}: expected {expected}, got {}", - self.path(), - self.value.get_type().name()? - )) + pub(crate) fn expected( + &self, + expected: &'static str, + ) -> Result { + Ok(ProjectionDetail::Expected { + group: self.group, + name: self.name, + expected, + actual: self.value.get_type().name()?.to_string(), + }) } - pub(crate) fn invalid(&self, expected: &str) -> ProjectionError { + pub(crate) fn invalid(&self, expected: &'static str) -> ProjectionError { match self.expected(expected) { - Ok(message) => ProjectionError::InvalidConfiguration(message.into()), + Ok(detail) => ProjectionError::InvalidConfiguration(detail), Err(error) => error, } } @@ -163,24 +163,23 @@ impl<'py> Field<'py> { pub(crate) fn schema_bool(&self) -> Result { if !self.value.is_instance_of::() { return Err(ProjectionError::InternalSchemaFailure( - self.expected("a Boolean")?.into(), + self.expected("a Boolean")?, )); } Ok(self.exact_true()) } pub(crate) fn strict_string(&self) -> Result { - let value = self - .value - .cast::() - .map_err(|_| self.invalid("a string"))?; + let Ok(value) = self.value.cast::() else { + return Err(self.invalid("a string")); + }; Ok(value.to_str()?.to_owned()) } pub(crate) fn schema_string(&self) -> Result { if !self.value.is_instance_of::() { return Err(ProjectionError::InternalSchemaFailure( - self.expected("a string")?.into(), + self.expected("a string")?, )); } self.strict_string() @@ -474,12 +473,7 @@ text = Text(' False ') }); } - #[test] - fn missing_snapshot_fields_and_descriptor_attribute_errors_are_distinct() { - with_python(|py| { - let locals = PyDict::new(py); - py.run( - c" + const RAISING_ATTRIBUTES: &std::ffi::CStr = c" failure = AttributeError('descriptor failed') class Snapshot: @property @@ -491,46 +485,56 @@ class Intercepted: def __getattribute__(self, name): raise failure dynamic = Dynamic() intercepted = Intercepted() -", - Some(&locals), - Some(&locals), - ) - .unwrap(); - let snapshot = locals.get_item("snapshot").unwrap().unwrap(); - let descriptor = PyErr::from(Field::read(&snapshot, "test", "flag").err().unwrap()); +"; + + #[rstest] + #[case::descriptor("snapshot")] + #[case::dynamic("dynamic")] + #[case::intercepted("intercepted")] + fn attribute_errors_raised_by_a_snapshot_surface_as_raised(#[case] name: &str) { + with_python(|py| { + let locals = PyDict::new(py); + py.run(RAISING_ATTRIBUTES, Some(&locals), Some(&locals)) + .unwrap(); + let value = locals.get_item(name).unwrap().unwrap(); + let error = PyErr::from(Field::read(&value, "test", "flag").err().unwrap()); assert!( - descriptor + error .value(py) .is(locals.get_item("failure").unwrap().unwrap()) ); - for name in ["dynamic", "intercepted"] { - let value = locals.get_item(name).unwrap().unwrap(); - let error = PyErr::from(Field::read(&value, "test", "flag").err().unwrap()); - assert!( - error - .value(py) - .is(locals.get_item("failure").unwrap().unwrap()) - ); - } + }); + } + + #[test] + fn missing_snapshot_fields_are_runtime_errors_naming_the_field() { + with_python(|py| { + let locals = PyDict::new(py); + py.run(RAISING_ATTRIBUTES, Some(&locals), Some(&locals)) + .unwrap(); + let snapshot = locals.get_item("snapshot").unwrap().unwrap(); let missing = PyErr::from(Field::read(&snapshot, "test", "missing").err().unwrap()); assert!(missing.is_instance_of::(py)); assert!(missing.to_string().contains("test.missing")); }); } - #[test] - fn configuration_errors_name_fields_without_exposing_values() { + #[rstest] + #[case::mapping("{'secret': 'do-not-print'}")] + #[case::nested("['host.test', {'secret': 'do-not-print'}]")] + fn configuration_errors_name_fields_without_exposing_values(#[case] source: &str) { + with_python(|py| { + let field = Field::new("test", "setting", evaluate(py, source)); + let error = PyErr::from(field.falsy_optional_string().err().unwrap()); + assert!(error.is_instance_of::(py)); + assert!(error.to_string().contains("test.setting")); + assert!(!error.to_string().contains("do-not-print")); + }); + } + + #[test] + fn invalid_collections_and_flags_are_configuration_errors() { with_python(|py| { - for source in [ - "{'secret': 'do-not-print'}", - "['host.test', {'secret': 'do-not-print'}]", - ] { - let field = Field::new("test", "setting", evaluate(py, source)); - let error = PyErr::from(field.falsy_optional_string().err().unwrap()); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("test.setting")); - assert!(!error.to_string().contains("do-not-print")); - } let hosts = Field::new( "url_policy", "user_url_allowed_hosts", diff --git a/litellm-rust/crates/python-bridge/src/credentials.rs b/litellm-rust/crates/python-bridge/src/credentials.rs index 04d8e94c8fe..adc52a7e77a 100644 --- a/litellm-rust/crates/python-bridge/src/credentials.rs +++ b/litellm-rust/crates/python-bridge/src/credentials.rs @@ -2,7 +2,9 @@ //! keyword arguments and acquired on the host's own thread when the call asks for one. use crate::callable::wrap_failure; -use litellm_auth::{ResolvedCredential, SecretValue}; +use litellm_auth::ResolvedCredential; +use litellm_secret_value::SecretValue; + use pyo3::{ exceptions::PyTypeError, gc::{PyTraverseError, PyVisit}, @@ -10,10 +12,19 @@ use pyo3::{ types::{PyDict, PyString}, }; -const NOT_CALLABLE: &str = "Azure AD token provider must be callable"; const NOT_A_STRING: &str = "Azure AD token must be a string, got {}"; const FAILED: &str = "Failed to get Azure AD token: {}"; +#[derive(Debug, thiserror::Error)] +#[error("Azure AD token provider must be callable")] +struct NotCallable; + +impl From for PyErr { + fn from(error: NotCallable) -> Self { + PyTypeError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + /// The `azure_ad_token_provider` keyword argument, kept alive for the rest of the call. pub(crate) struct CallerTokenProvider { provider: Py, @@ -36,7 +47,7 @@ impl CallerTokenProvider { pub(crate) fn acquire(&self, py: Python<'_>) -> PyResult { let provider = self.provider.bind(py); if !provider.is_callable() { - return Err(PyTypeError::new_err(NOT_CALLABLE)); + return Err(NotCallable.into()); } let token = wrap_failure( py, diff --git a/litellm-rust/crates/python-bridge/src/db.rs b/litellm-rust/crates/python-bridge/src/db.rs new file mode 100644 index 00000000000..03fb30b986f --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/db.rs @@ -0,0 +1,19 @@ +use litellm_host_python::ready_future; +use pyo3::{prelude::*, types::PyNone}; + +#[pyclass(frozen)] +pub(crate) struct NativeDatabase {} + +// TODO: take typed writer/reader settings and a token provider, then open a sqlx PgPool, once the first +// repository moves to native. The Prisma DATABASE_URL carries Prisma-only params and rotating IAM tokens +#[pymethods] +impl NativeDatabase { + #[staticmethod] + fn connect(py: Python<'_>) -> PyResult> { + ready_future(py, Bound::new(py, NativeDatabase {})?.as_any()) + } + + fn close<'py>(&self, py: Python<'py>) -> PyResult> { + ready_future(py, PyNone::get(py).as_any()) + } +} diff --git a/litellm-rust/crates/python-bridge/src/diagnostics.rs b/litellm-rust/crates/python-bridge/src/diagnostics.rs index 687a090e768..5368da4f930 100644 --- a/litellm-rust/crates/python-bridge/src/diagnostics.rs +++ b/litellm-rust/crates/python-bridge/src/diagnostics.rs @@ -1,5 +1,5 @@ use litellm_host_python::{release_count, runtime_started}; -use pyo3::{exceptions::PyRuntimeError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; #[pyfunction] pub(crate) fn gil_stats(py: Python<'_>) -> PyResult> { @@ -18,8 +18,7 @@ pub(crate) fn process_state_started() -> bool { /// so the runtime can never start. Raises if it already has. Forked workers are unaffected. #[pyfunction] pub(crate) fn reserve_process_for_forking() -> PyResult<()> { - litellm_host_python::reserve_process_for_forking() - .map_err(|_| PyRuntimeError::new_err("the native runtime already started in this process")) + Ok(litellm_host_python::reserve_process_for_forking()?) } #[cfg(feature = "panic-test")] diff --git a/litellm-rust/crates/python-bridge/src/errors.rs b/litellm-rust/crates/python-bridge/src/errors.rs index 4dab0297e73..426cd861729 100644 --- a/litellm-rust/crates/python-bridge/src/errors.rs +++ b/litellm-rust/crates/python-bridge/src/errors.rs @@ -1,9 +1,12 @@ -use litellm_http::transport::Error as TransportError; +use std::error::Error as StdError; + +use litellm_error::Report; use litellm_inference::Error; use pyo3::{ exceptions::{PyRuntimeError, PyValueError}, prelude::*, }; +use reqwest::header::HeaderMap; pyo3::create_exception!( _native, @@ -16,20 +19,45 @@ pyo3::create_exception!( _native, RustUpstreamError, pyo3::exceptions::PyException, - "The provider call was already issued and failed. Args are (status, message); status is 0 when there was no HTTP response." + "The provider call was already issued and the provider answered with a failure. Args are (status, body); the headers attribute holds the provider's (name, value) response headers." ); -pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr { - match error { - Error::Transport(TransportError::Http { status, body }) => { - RustUpstreamError::new_err((status, body)) - } - other => by_fault(other.is_request(), other.to_string()), - } +const MESSAGES_REQUEST_ERROR: &str = "messages_request_error"; +/// The rejection is a parameter the model does not support, which Python raises as +/// `UnsupportedParamsError`. +const MESSAGES_UNSUPPORTED_PARAM: &str = "messages_unsupported_param"; +/// The phase label (`Connection` or `Read`) of a timed-out call, for Python's timeout text, or +/// `Body` for a streamed body read that timed out after the head arrived. +const MESSAGES_TIMEOUT: &str = "messages_timeout"; +/// No credential reached the provider, which Python raises as `AuthenticationError`. +const MESSAGES_MISSING_CREDENTIAL: &str = "messages_missing_credential"; +/// Python refuses the request only once its logger knows the resolved model and provider (the +/// provider config's request transform), so the failure logs them. +const MESSAGES_LATE_REJECTION: &str = "messages_late_rejection"; + +pub(crate) fn upstream_error( + py: Python<'_>, + status: u16, + body: &str, + headers: &HeaderMap, +) -> PyResult { + let error = RustUpstreamError::new_err((status, body.to_owned())); + let pairs: Vec<(&str, String)> = headers + .iter() + .map(|(name, value)| { + ( + name.as_str(), + String::from_utf8_lossy(value.as_bytes()).into_owned(), + ) + }) + .collect(); + error.value(py).setattr("headers", pairs)?; + Ok(error) } /// A request the caller got wrong is a `ValueError`; anything else is a `RuntimeError`. -pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr { +pub(crate) fn by_fault(is_request: bool, error: &(dyn StdError + 'static)) -> PyErr { + let message = Report::new(error).to_string(); if is_request { PyValueError::new_err(message) } else { @@ -37,17 +65,130 @@ pub(crate) fn by_fault(is_request: bool, message: String) -> PyErr { } } +/// How a route flags a rejected request for its Python route module, which raises the public +/// exception for it instead of mapping the native one. +#[derive(Clone, Copy)] +pub(crate) enum RequestMarker { + Unmarked, + Messages, +} + +/// Maps a route failure once. A core decline and a Python exception raised inside the call are +/// returned as they are, everything else is mapped and handed to `public`, the route's public +/// exception mapping. +pub(crate) fn route_failure( + py: Python<'_>, + error: Error, + marker: RequestMarker, + public: impl FnOnce(PyErr) -> PyResult, +) -> PyResult { + if let Some(decline) = error.declined() { + return Ok(RustBridgeDeclined::new_err(decline.reason)); + } + if let Some(raised) = crate::secrets::python_error(py, &error) { + return Ok(raised); + } + if let Some((status, body)) = error.upstream_response() { + let headers = error + .upstream_headers() + .iter() + .filter_map(|(name, value)| { + Some(( + reqwest::header::HeaderName::from_bytes(name.as_bytes()).ok()?, + reqwest::header::HeaderValue::from_str(value).ok()?, + )) + }) + .collect::(); + return public(upstream_error(py, status, body, &headers)?); + } + let native = match (marker, error.rejected_request()) { + (RequestMarker::Messages, Some(rejection)) => { + let native = PyValueError::new_err(Report::new(rejection).to_string()); + let value = native.value(py); + value.setattr(MESSAGES_REQUEST_ERROR, true)?; + if matches!( + rejection.downcast_ref::(), + Some( + litellm_llms::ErrorDetail::UnsupportedParameter { .. } + | litellm_llms::ErrorDetail::NonNumericThinkingBudget { .. } + ) + ) { + value.setattr(MESSAGES_UNSUPPORTED_PARAM, true)?; + } + if matches!( + rejection.downcast_ref::(), + Some(litellm_llms::ErrorDetail::NonNumericThinkingBudget { .. }) + ) { + value.setattr(MESSAGES_LATE_REJECTION, true)?; + } + native + } + (RequestMarker::Messages, None) => { + let native = by_fault(error.is_request(), &error); + let value = native.value(py); + if let Some(connect) = error.timeout() { + let phase = match (connect, error.body_read_timeout()) { + (true, _) => "Connection", + (false, true) => "Body", + (false, false) => "Read", + }; + value.setattr(MESSAGES_TIMEOUT, phase)?; + } + if error.is_missing_credential() { + value.setattr(MESSAGES_MISSING_CREDENTIAL, true)?; + } + native + } + (RequestMarker::Unmarked, _) => by_fault(error.is_request(), &error), + }; + public(native) +} + +pub(crate) fn route_error_to_pyerr(error: Error) -> PyErr { + Python::attach(|py| { + route_failure(py, error, RequestMarker::Unmarked, Ok).unwrap_or_else(std::convert::identity) + }) +} + #[cfg(test)] mod tests { + use litellm_core_utils::decline::Decline; + use reqwest::header::{HeaderValue, RETRY_AFTER}; + use super::*; + fn upstream_payload(py: Python<'_>, failure: &PyErr) -> ((u16, String), Vec<(String, String)>) { + let value = failure.value(py); + ( + value.getattr("args").unwrap().extract().unwrap(), + value.getattr("headers").unwrap().extract().unwrap(), + ) + } + #[rstest::rstest] - #[case::unsupported(Error::Unsupported("test capability"), true)] #[case::invalid_provider(Error::InvalidProvider("unknown".into()), true)] - #[case::invalid_request(Error::InvalidRequest("empty messages".into()), true)] - #[case::connection(TransportError::Connect("unreachable".into()).into(), false)] - #[case::network(TransportError::Network("timed out".into()).into(), false)] - #[case::invalid_response(Error::InvalidResponse("missing usage".into()), false)] + #[case::invalid_request( + Error::invalid_request(litellm_llms::ErrorDetail::Expected { + subject: "messages", + expected: "non-empty", + }), + true + )] + #[case::lost_connection( + litellm_http::Error::from(tokio_tungstenite::tungstenite::Error::ConnectionClosed).into(), + false + )] + #[case::invalid_response( + Error::invalid_response(litellm_llms::ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + false + )] + #[case::post_call_decline( + Error::post_call(Error::Declined(Decline::new("result rejected"))), + false + )] fn route_failures_are_terminal(#[case] error: Error, #[case] is_request: bool) { Python::initialize(); Python::attach(|py| { @@ -59,29 +200,97 @@ mod tests { } #[rstest::rstest] - fn transport_status_survives_python_mapping() { + #[case::unmarked(RequestMarker::Unmarked)] + #[case::messages(RequestMarker::Messages)] + fn a_core_decline_hands_the_call_back_to_python_before_the_public_mapping( + #[case] marker: RequestMarker, + ) { Python::initialize(); Python::attach(|py| { - let upstream = route_error_to_pyerr( - TransportError::Http { - status: 429, - body: "slow down".into(), - } - .into(), - ); - assert!(upstream.is_instance_of::(py)); + let error = Error::from(litellm_llms::Error::Declined(Decline::new( + "unrecognized request parameter", + ))); + let declined = route_failure(py, error, marker, |_| { + panic!("a decline must not reach the public mapping") + }) + .unwrap(); + assert!(declined.is_instance_of::(py)); assert_eq!( - upstream + declined .value(py) .getattr("args") .unwrap() - .extract::<(u16, String)>() + .extract::<(String,)>() .unwrap(), - (429, "slow down".into()) + ("unrecognized request parameter".into(),) ); }); } + #[rstest::rstest] + #[case::unmarked(RequestMarker::Unmarked)] + #[case::messages(RequestMarker::Messages)] + fn the_provider_answer_passes_through_with_a_headers_attribute(#[case] marker: RequestMarker) { + Python::initialize(); + Python::attach(|py| { + let error = litellm_http::Error::upstream(429, "slow down".into()).into(); + let upstream = route_failure(py, error, marker, Ok).unwrap(); + assert!(upstream.is_instance_of::(py)); + assert_eq!( + upstream_payload(py, &upstream), + ((429, "slow down".into()), Vec::new()) + ); + assert!( + upstream + .value(py) + .getattr_opt(MESSAGES_REQUEST_ERROR) + .unwrap() + .is_none() + ); + }); + } + + #[test] + fn upstream_error_keeps_the_provider_headers() { + Python::initialize(); + Python::attach(|py| { + let headers = HeaderMap::from_iter([(RETRY_AFTER, HeaderValue::from_static("17"))]); + let upstream = upstream_error(py, 503, "busy", &headers).unwrap(); + assert_eq!( + upstream_payload(py, &upstream), + ( + (503, "busy".into()), + vec![("retry-after".into(), "17".into())] + ) + ); + }); + } + + #[rstest::rstest] + #[case::ascii("busy ".repeat(80))] + #[case::unicode("é".repeat(300))] + fn long_provider_bodies_reach_python_intact( + #[case] message: String, + #[values(RequestMarker::Unmarked, RequestMarker::Messages)] marker: RequestMarker, + ) { + let envelope = serde_json::json!({ + "type": "error", + "error": {"type": "overloaded_error", "message": message}, + "request_id": "provider-request", + }); + let body = envelope.to_string(); + let error = litellm_http::Error::upstream(529, body.clone()).into(); + Python::initialize(); + Python::attach(|py| { + let upstream = route_failure(py, error, marker, Ok).unwrap(); + assert!(upstream.is_instance_of::(py)); + let ((status, returned_body), headers) = upstream_payload(py, &upstream); + assert_eq!(status, 529); + assert_eq!(returned_body, body); + assert!(headers.is_empty()); + }); + } + #[test] fn missing_api_key_stays_a_runtime_error_while_other_auth_failures_are_value_errors() { Python::initialize(); @@ -95,4 +304,83 @@ mod tests { assert!(invalid.is_instance_of::(py)); }); } + + #[test] + fn a_wrapped_failure_renders_every_layer() { + Python::initialize(); + Python::attach(|py| { + let error = Error::post_call( + litellm_http::Error::upstream(500, "hook backend down".into()).into(), + ); + let report = Report::new(&error).to_string(); + let failure = route_error_to_pyerr(error); + assert_eq!(failure.value(py).to_string(), report); + assert!(report.contains("hook backend down"), "{report}"); + }); + } + + #[rstest::rstest] + #[case::rejected_request( + Error::invalid_request(litellm_llms::ErrorDetail::UnsupportedParameter { + model: "claude".into(), + param: "top_k".into(), + value: "5".into(), + hint: String::new(), + }), + Some("claude does not support top_k=5. To drop unsupported params, set `litellm.drop_params = True`.") + )] + #[case::missing_field( + litellm_llms::Error::MissingField("max_tokens").into(), + Some("missing required field: max_tokens") + )] + #[case::unresolvable_provider(Error::InvalidProvider("openai".into()), None)] + #[case::invalid_response( + Error::invalid_response(litellm_llms::ErrorDetail::MissingField { + subject: "response", + field: "usage", + }), + None + )] + fn only_messages_request_rejections_carry_the_marker_and_python_text( + #[case] error: Error, + #[case] rejection: Option<&str>, + ) { + Python::initialize(); + Python::attach(|py| { + let unmarked = route_failure(py, error.clone(), RequestMarker::Unmarked, Ok).unwrap(); + assert!( + unmarked + .value(py) + .getattr_opt(MESSAGES_REQUEST_ERROR) + .unwrap() + .is_none() + ); + let native = route_failure(py, error, RequestMarker::Messages, Ok).unwrap(); + let marked = native + .value(py) + .getattr_opt(MESSAGES_REQUEST_ERROR) + .unwrap() + .map(|value| value.extract::().unwrap()); + assert_eq!(marked.unwrap_or(false), rejection.is_some()); + if let Some(rejection) = rejection { + assert!(native.is_instance_of::(py)); + assert_eq!(native.value(py).to_string(), rejection); + } + }); + } + + #[test] + fn a_python_secret_manager_exception_skips_the_public_mapping() { + Python::initialize(); + Python::attach(|py| { + let raised = pyo3::exceptions::PyKeyError::new_err("vault down"); + let identity = raised.value(py).clone().unbind(); + let error = Error::from(crate::secrets::read_error(py, raised)); + let surfaced = route_failure(py, error, RequestMarker::Unmarked, |_| { + panic!("the raised exception must not be remapped") + }) + .unwrap(); + assert!(surfaced.value(py).is(identity.bind(py))); + }); + } } diff --git a/litellm-rust/crates/python-bridge/src/http.rs b/litellm-rust/crates/python-bridge/src/http.rs index 57bf8aafc4e..b4e20a8f750 100644 --- a/litellm-rust/crates/python-bridge/src/http.rs +++ b/litellm-rust/crates/python-bridge/src/http.rs @@ -132,26 +132,14 @@ pub(crate) fn host_client(py: Python<'_>, variant: ClientVariant) -> PyResult PyErr { - match error { - litellm_http::Error::Read { - tls_source: TlsSource::ClientIdentity, - .. - } - | litellm_http::Error::InvalidPem { - tls_source: TlsSource::ClientIdentity, - .. - } => PyValueError::new_err( + match error.tls_source() { + Some(TlsSource::ClientIdentity) => PyValueError::new_err( "http_settings.ssl_certificate: expected a readable PEM certificate and private key", ), - litellm_http::Error::Read { - tls_source: TlsSource::CaBundle, - .. + Some(TlsSource::CaBundle) => { + PyValueError::new_err("http_settings.ssl_verify: expected a readable PEM CA bundle") } - | litellm_http::Error::InvalidPem { - tls_source: TlsSource::CaBundle, - .. - } => PyValueError::new_err("http_settings.ssl_verify: expected a readable PEM CA bundle"), - _ => PyValueError::new_err("http_settings: native HTTP client configuration is invalid"), + None => PyValueError::new_err(litellm_error::Report::new(&error).to_string()), } } @@ -274,21 +262,28 @@ settings = types.SimpleNamespace(**defaults) fn client_error_uses_tls_source_when_paths_match() { Python::initialize(); Python::attach(|py| { - let path = PathBuf::from("/shared.pem"); - let ca_error = client_error(litellm_http::Error::InvalidPem { - path: path.clone(), - message: "invalid".into(), - tls_source: TlsSource::CaBundle, - }); + let file = tempfile::NamedTempFile::new().unwrap(); + std::fs::write(file.path(), b"not a certificate").unwrap(); + let path = file.path().to_path_buf(); + let defaults = Resolution::from(&HttpSettings::default()).config; + let build = |config: HttpClientConfig| { + HttpClientPool::new(Arc::new(PublicDnsResolver)) + .client(&config, ClientVariant::Provider) + .map(drop) + .unwrap_err() + }; + let ca_error = client_error(build(HttpClientConfig { + verify: Verify::CaBundle(path.clone()), + ..defaults.clone() + })); assert_eq!( ca_error.to_string(), "ValueError: http_settings.ssl_verify: expected a readable PEM CA bundle" ); - let client_error = client_error(litellm_http::Error::InvalidPem { - path, - message: "invalid".into(), - tls_source: TlsSource::ClientIdentity, - }); + let client_error = client_error(build(HttpClientConfig { + client_certificate: Some(litellm_http::ClientIdentity::Pem(path)), + ..defaults + })); assert!(client_error.is_instance_of::(py)); assert_eq!( client_error.to_string(), @@ -297,6 +292,34 @@ settings = types.SimpleNamespace(**defaults) }); } + #[test] + fn client_error_without_a_tls_source_reports_the_build_failure() { + Python::initialize(); + Python::attach(|py| { + let error = HttpClientPool::new(Arc::new(PublicDnsResolver)) + .client( + &HttpClientConfig { + user_agent: Some("bad\nagent".into()), + ..Resolution::from(&HttpSettings::default()).config + }, + ClientVariant::Provider, + ) + .map(drop) + .unwrap_err(); + assert_eq!(error.tls_source(), None); + let report = litellm_error::Report::new(&error).to_string(); + + let failure = client_error(error); + + assert!(failure.is_instance_of::(py)); + assert_eq!(failure.value(py).str().unwrap().to_string(), report); + assert!( + report.contains("could not build the HTTP client: "), + "{report}" + ); + }); + } + #[test] fn python_settings_flow_into_the_configured_layer() { Python::initialize(); diff --git a/litellm-rust/crates/python-bridge/src/lib.rs b/litellm-rust/crates/python-bridge/src/lib.rs index 12827b76ff3..dc127e41129 100644 --- a/litellm-rust/crates/python-bridge/src/lib.rs +++ b/litellm-rust/crates/python-bridge/src/lib.rs @@ -2,14 +2,18 @@ mod cache; mod callable; mod coercion; mod credentials; +mod db; mod diagnostics; mod errors; mod execution; mod http; mod logger; mod marshal; +mod model_registry; mod preflight; mod python_settings; +mod redis; +mod router; mod routes; mod secrets; mod tokenizer; @@ -20,6 +24,8 @@ mod _native { use litellm_host_python::{ForkedAfterNativeRuntimeStarted, ProcessReservedForForking}; use pyo3::{prelude::*, types::PyModule}; + #[pymodule_export] + use crate::db::NativeDatabase; #[cfg(feature = "panic-test")] #[pymodule_export] use crate::diagnostics::_panic_for_test; @@ -30,6 +36,33 @@ mod _native { #[pymodule_export] use crate::logger::NativeDiagnosticProcessor; #[pymodule_export] + use crate::model_registry::bundled_model_cost_map; + #[pymodule_export] + use crate::redis::client::NativeRedisClient; + #[pymodule_export] + use crate::redis::client::native_redis_breaker_states; + #[pymodule_export] + use crate::redis::store::auth_cache::{NativeAuthCache, PyLocalSizes}; + #[pymodule_export] + use crate::redis::store::rate_limits::{ + NativeRateLimits, PyAtomicApplied, PyAtomicOverLimit, PyGaugeOverLimit, PyGaugesAcquired, + PyGuardedReply, PyWindowCount, + }; + #[pymodule_export] + use crate::redis::store::request::{NativeRequestRedis, NativeSend}; + #[pymodule_export] + use crate::redis::store::router_usage::NativeRouterUsage; + #[pymodule_export] + use crate::redis::store::sensitive_route_pins::NativeSensitiveRoutePins; + #[pymodule_export] + use crate::redis::store::spend_counters::{NativeSpendCounters, PySpendRead}; + #[pymodule_export] + use crate::redis::store::{ + NativePending, NativeRedisStoreError, router_cooldowns::NativeRouterCooldowns, + }; + #[pymodule_export] + use crate::router::{RouterCall, RouterCoreHandle}; + #[pymodule_export] use crate::routes::audio_transcription::{atranscription, transcription}; #[pymodule_export] use crate::routes::chat_completions::{acompletion, completion}; @@ -59,10 +92,6 @@ mod _native { fn init(module: &Bound<'_, PyModule>) -> PyResult<()> { let py = module.py(); let dict = module.dict(); - dict.set_item( - "NativeCacheHandle", - py.get_type::(), - )?; dict.set_item( "_SecretManagerRuntime", py.get_type::(), @@ -86,7 +115,6 @@ mod tests { Python::initialize(); Python::attach(|py| { let mut expected = vec![ - "NativeCacheHandle", "RustBridgeDeclined", "RustUpstreamError", "ForkedAfterNativeRuntimeStarted", @@ -107,11 +135,35 @@ mod tests { "aresponses", "ResponsesWebSocketConnection", "NativeDiagnosticProcessor", + "NativeRedisClient", + "NativeRequestRedis", + "NativeSend", + "NativeAuthCache", + "NativeRateLimits", + "WindowCount", + "AtomicApplied", + "AtomicOverLimit", + "GuardedReply", + "GaugesAcquired", + "GaugeOverLimit", + "NativePending", + "NativeRouterCooldowns", + "NativeRouterUsage", + "NativeSensitiveRoutePins", + "NativeRedisStoreError", + "NativeSpendCounters", + "SpendRead", + "LocalSizes", + "native_redis_breaker_states", "NativeClickHouseSpendConfig", "NativeClickHouseSpendStorage", + "NativeDatabase", "trace_encode_error", + "bundled_model_cost_map", "TokenCounter", "Tokenizer", + "RouterCoreHandle", + "RouterCall", "gil_stats", "process_state_started", "reserve_process_for_forking", diff --git a/litellm-rust/crates/python-bridge/src/logger/mod.rs b/litellm-rust/crates/python-bridge/src/logger/mod.rs index bf5c735360b..8f40d94f458 100644 --- a/litellm-rust/crates/python-bridge/src/logger/mod.rs +++ b/litellm-rust/crates/python-bridge/src/logger/mod.rs @@ -81,8 +81,12 @@ impl NativeDiagnosticProcessor { } } -fn processing_error(_: fancy_regex::Error) -> PyErr { - PyRuntimeError::new_err("diagnostic processing failed") +#[derive(Debug, thiserror::Error)] +#[error("diagnostic processing failed")] +struct ProcessingError(#[source] fancy_regex::Error); + +fn processing_error(error: fancy_regex::Error) -> PyErr { + PyRuntimeError::new_err(litellm_error::Report::new(&ProcessingError(error)).to_string()) } struct PythonSink { diff --git a/litellm-rust/crates/python-bridge/src/logger/tests.rs b/litellm-rust/crates/python-bridge/src/logger/tests.rs index 1fca3be720e..9a07428808a 100644 --- a/litellm-rust/crates/python-bridge/src/logger/tests.rs +++ b/litellm-rust/crates/python-bridge/src/logger/tests.rs @@ -187,6 +187,7 @@ fn native_events_reach_python_with_levels_context_reentry_and_http_deduplication locals .set_item("http_warning", wrap_pyfunction!(http_warning, py).unwrap()) .unwrap(); + locals.set_item("native", crate::native_module(py)).unwrap(); let importable = py .eval( c"__import__('importlib.util', fromlist=['util']).find_spec('dotenv') is not None", @@ -205,6 +206,7 @@ import asyncio import logging import sys sys.path.insert(0, repo_root) +sys.modules['litellm.rust_bridge._native'] = native import litellm from litellm._logging import verbose_logger, session_id_var, trace_id_var diff --git a/litellm-rust/crates/python-bridge/src/marshal.rs b/litellm-rust/crates/python-bridge/src/marshal.rs index 038fa8c9630..dec8ff0b0ce 100644 --- a/litellm-rust/crates/python-bridge/src/marshal.rs +++ b/litellm-rust/crates/python-bridge/src/marshal.rs @@ -21,10 +21,27 @@ pub(crate) struct RouteOptions { pub(crate) timeout: Option, } -fn required_object(name: &'static str, value: Value) -> PyResult> { +/// A caller argument the bridge cannot use, raised as a `ValueError`. +#[derive(Debug, thiserror::Error)] +pub(crate) enum ArgumentError { + #[error("{0} is required")] + Required(&'static str), + #[error("{0} must be a dict")] + NotADict(&'static str), + #[error("header values must be strings")] + HeaderValue, +} + +impl From for PyErr { + fn from(error: ArgumentError) -> Self { + PyValueError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + +fn required_object(name: &'static str, value: Value) -> Result, ArgumentError> { match value { Value::Object(values) => Ok(values), - _ => Err(PyValueError::new_err(format!("{name} must be a dict"))), + _ => Err(ArgumentError::NotADict(name)), } } @@ -35,7 +52,7 @@ fn optional_object( if value.is_none() { return Ok(None); } - required_object(name, from_py_argument(value)?).map(Some) + Ok(Some(required_object(name, from_py_argument(value)?)?)) } pub(crate) fn optional_timeout(timeout_seconds: Option) -> Option { @@ -57,11 +74,11 @@ pub(crate) fn python_timeout_seconds(py: Python<'_>, timeout: Py) -> PyRe pub(crate) fn required_field<'py>( fields: &Bound<'py, PyDict>, - name: &str, + name: &'static str, ) -> PyResult> { - fields + Ok(fields .get_item(name)? - .ok_or_else(|| PyValueError::new_err(format!("{name} is required"))) + .ok_or(ArgumentError::Required(name))?) } pub(crate) fn optional_field( @@ -171,21 +188,18 @@ pub(crate) fn request_input_sources<'a>( .collect()) } -pub(crate) fn marshal_headers(headers: Option) -> PyResult> { - let value = match headers { - Some(headers) => headers, - None => Value::Object(Map::new()), - }; - let Value::Object(headers) = value else { - return Err(PyValueError::new_err("headers must be a dict")); +pub(crate) fn marshal_headers( + headers: Option, +) -> Result, ArgumentError> { + let headers = match headers { + Some(headers) => required_object("headers", headers)?, + None => Map::new(), }; headers .into_iter() - .map(|(name, value)| { - value - .as_str() - .map(|value| (name, value.to_string())) - .ok_or_else(|| PyValueError::new_err("header values must be strings")) + .map(|(name, value)| match value { + Value::String(value) => Ok((name, value)), + _ => Err(ArgumentError::HeaderValue), }) .collect() } @@ -395,48 +409,52 @@ module.__getattr__ = fail float: Option, } - #[test] - fn numeric_adapters_agree_across_json_and_python_boundaries() { + #[rstest::rstest] + #[case::empty(json!({}))] + #[case::explicit_nulls(json!({"integers": null, "float": null}))] + #[case::lax_values(json!({ + "integers": [i64::MIN, i64::MAX, "9007199254740993.0", " +1_000.00 ", true, 3.0], + "float": " 1.25 ", + }))] + #[case::integer_overflow(json!({"integers": [u64::MAX]}))] + #[case::imprecise_integer_string(json!({"integers": ["1.0000000000000001"]}))] + #[case::fractional_integer(json!({"integers": [2.5]}))] + #[case::nan_string(json!({"float": "NaN"}))] + #[case::infinity_string(json!({"float": "inf"}))] + #[case::overflowing_float_string(json!({"float": "1e999"}))] + #[case::boolean_float(json!({"float": true}))] + #[case::large_integer_float(json!({"float": u64::MAX}))] + fn numeric_adapters_agree_across_json_and_python_boundaries(#[case] input: Value) { Python::initialize(); Python::attach(|py| { - for input in [ - json!({}), - json!({"integers": null, "float": null}), - json!({"integers": [i64::MIN, i64::MAX, "9007199254740993.0", " +1_000.00 ", true, 3.0], "float": " 1.25 "}), - json!({"integers": [u64::MAX]}), - json!({"integers": ["1.0000000000000001"]}), - json!({"integers": [2.5]}), - json!({"float": "NaN"}), - json!({"float": "inf"}), - json!({"float": "1e999"}), - json!({"float": true}), - json!({"float": u64::MAX}), - ] { - let expected = serde_json::from_value::(input.clone()); - let python = litellm_host_python::to_py(py, &input).unwrap(); - let actual = from_py::(python.bind(py)); - match (expected, actual) { - (Ok(expected), Ok(actual)) => { - assert_eq!(actual, expected); - let serialized = litellm_host_python::to_py(py, &actual).unwrap(); - assert_eq!( - from_py::(serialized.bind(py)).unwrap(), - serde_json::to_value(expected).unwrap() - ); - } - (Err(_), Err(_)) => {} - mismatch => panic!("boundary mismatch for {input}: {mismatch:?}"), + let expected = serde_json::from_value::(input.clone()); + let python = litellm_host_python::to_py(py, &input).unwrap(); + let actual = from_py::(python.bind(py)); + match (expected, actual) { + (Ok(expected), Ok(actual)) => { + assert_eq!(actual, expected); + let serialized = litellm_host_python::to_py(py, &actual).unwrap(); + assert_eq!( + from_py::(serialized.bind(py)).unwrap(), + serde_json::to_value(expected).unwrap() + ); } + (Err(_), Err(_)) => {} + mismatch => panic!("boundary mismatch for {input}: {mismatch:?}"), } - for source in [ - c"{'float': float('nan')}", - c"{'float': float('inf')}", - c"{'integers': [float('inf')]}", - c"{'integers': [2 ** 100]}", - ] { - let value = py.eval(source, None, None).unwrap(); - assert!(from_py::(&value).is_err()); - } + }); + } + + #[rstest::rstest] + #[case::nan(c"{'float': float('nan')}")] + #[case::infinity(c"{'float': float('inf')}")] + #[case::infinite_integer(c"{'integers': [float('inf')]}")] + #[case::integer_overflow(c"{'integers': [2 ** 100]}")] + fn non_finite_and_overflowing_python_numbers_are_rejected(#[case] source: &std::ffi::CStr) { + Python::initialize(); + Python::attach(|py| { + let value = py.eval(source, None, None).unwrap(); + assert!(from_py::(&value).is_err()); }); } diff --git a/litellm-rust/crates/python-bridge/src/model_registry.rs b/litellm-rust/crates/python-bridge/src/model_registry.rs new file mode 100644 index 00000000000..b7b7cd1fbb3 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/model_registry.rs @@ -0,0 +1,30 @@ +use pyo3::prelude::*; +use pyo3::sync::PyOnceLock; +use pyo3::types::PyBytes; + +#[pyfunction] +pub fn bundled_model_cost_map(py: Python<'_>) -> Bound<'_, PyBytes> { + static BYTES: PyOnceLock> = PyOnceLock::new(); + BYTES + .get_or_init(py, || { + PyBytes::new(py, litellm_model_catalog::BUNDLED_REGISTRY).unbind() + }) + .bind(py) + .clone() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[rstest::rstest] + fn repeated_calls_reuse_the_bundled_registry_bytes() { + Python::initialize(); + let first = Python::attach(|py| bundled_model_cost_map(py).unbind()); + Python::attach(|py| { + let second = bundled_model_cost_map(py); + assert!(first.bind(py).is(&second)); + assert_eq!(second.as_bytes(), litellm_model_catalog::BUNDLED_REGISTRY); + }); + } +} diff --git a/litellm-rust/crates/python-bridge/src/python_settings.rs b/litellm-rust/crates/python-bridge/src/python_settings.rs index eacec6859b5..0152fe8a5b9 100644 --- a/litellm-rust/crates/python-bridge/src/python_settings.rs +++ b/litellm-rust/crates/python-bridge/src/python_settings.rs @@ -13,6 +13,8 @@ pub(crate) enum PythonSettings { UrlPolicy, #[strum(serialize = "provider_defaults")] ProviderDefaults, + #[strum(serialize = "router_tiers")] + RouterTiers, #[strum(serialize = "secret_manager")] SecretManager, #[strum(serialize = "secret_manager_binding")] diff --git a/litellm-rust/crates/python-bridge/src/redis/client/credentials.rs b/litellm-rust/crates/python-bridge/src/redis/client/credentials.rs new file mode 100644 index 00000000000..21a37947c30 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/client/credentials.rs @@ -0,0 +1,47 @@ +use std::sync::Arc; + +use litellm_host_python::{PythonContext, attach_blocking}; +use litellm_redis_connect::{CredentialSource, CredentialStage, Error, Expiring, SourceFuture}; +use litellm_secret_value::SecretValue; +use pyo3::prelude::*; + +const MODULE: &str = "litellm.rust_bridge.host.redis_credentials"; + +/// A redis-py `CredentialProvider` the SDK caller passed in `cache_kwargs`, asked through the host +/// module the way redis-py's async connection asks it. +pub(crate) struct PythonCredentialSource { + provider: Arc>, + context: PythonContext, +} + +impl PythonCredentialSource { + pub(crate) fn new(provider: Py, context: PythonContext) -> Self { + Self { + provider: Arc::new(provider), + context, + } + } +} + +impl CredentialSource for PythonCredentialSource { + fn fetch(&self) -> SourceFuture<'_> { + let provider = Arc::clone(&self.provider); + let context = self.context.clone(); + Box::pin(async move { + let (username, password) = attach_blocking(context, move |py| { + py.import(MODULE)? + .getattr("credentials")? + .call1((provider.bind(py),))? + .extract::<(Option, String)>() + }) + .await + .and_then(|credentials| credentials) + .map_err(Error::credential(CredentialStage::Provider))?; + Ok(Expiring { + username, + password: SecretValue::new(password), + expires_at: None, + }) + }) + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/client/mod.rs b/litellm-rust/crates/python-bridge/src/redis/client/mod.rs new file mode 100644 index 00000000000..5b0e0ec5d94 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/client/mod.rs @@ -0,0 +1,404 @@ +mod credentials; + +use std::{ + collections::BTreeMap, + sync::{Arc, LazyLock, Mutex, PoisonError}, + time::Duration, +}; + +use litellm_circuit_breaker::Config as BreakerConfig; +use litellm_core_utils::settings::ProcessEnvironment; +use litellm_host_python::PythonContext; +use litellm_python_compat::number::{self, SecondsError}; +use litellm_redis_client::{RedisClient, RedisEvents, WeakRedisClient}; +use litellm_redis_connect::{CredentialSource, HostServices, SharedEnvironment}; +use litellm_redis_resolve::{Kwargs, PythonException}; +use litellm_redis_types::{ProviderHandle, Resolution}; +use pyo3::{ + exceptions::{PyOverflowError, PyTypeError, PyValueError}, + prelude::*, + sync::MutexExt, + types::{PyBool, PyDict, PyFloat, PyInt, PyList, PyMapping, PyString, PyTuple}, +}; + +use self::credentials::PythonCredentialSource; +use crate::redis::events::{ServiceLoggingEvents, state_label}; + +pyo3::import_exception!(redis.exceptions, RedisClusterException); + +/// The kwargs `_merge_redis_kwargs` returns for a `RedisCache`, which the resolver reads the way +/// redis-py's constructors do. A key whose value is a Python object, such as `retry`, reaches the +/// resolver by name only, so it can name the setting it cannot honor. A `credential_provider` object +/// is also kept, so the connector can ask it for credentials, and reaches the resolver as a handle. +pub(crate) struct RedisSpec { + kwargs: Kwargs, + credential_provider: Option>, +} + +impl<'a, 'py> FromPyObject<'a, 'py> for RedisSpec { + type Error = PyErr; + + fn extract(kwargs: Borrowed<'a, 'py, PyAny>) -> PyResult { + let py = kwargs.py(); + let source = PyDict::new(py); + let mapping = kwargs.cast::()?; + source.update(&mapping)?; + let plain = PyDict::new(py); + let mut host_objects = Vec::new(); + for (name, value) in source.iter() { + match value.is_none() || is_plain(&value) { + true => plain.set_item(&name, value)?, + false => host_objects.push(name.extract::()?), + } + } + let credential_provider = source + .get_item("credential_provider")? + .filter(|provider| !provider.is_none() && !is_plain(provider)) + .map(Bound::unbind); + let fields: BTreeMap = + litellm_host_python::from_py_argument(plain.as_any())?; + let objects = host_objects + .into_iter() + .map(|name| (name, serde_json::Value::Bool(true))); + Ok(Self { + kwargs: litellm_redis_resolve::kwargs(fields.into_iter().chain(objects)), + credential_provider, + }) + } +} + +/// `litellm.redis.store_settings.RedisBreakerSettings`, which `RedisCache` fills from `litellm.constants`. +#[derive(FromPyObject)] +pub(crate) struct RedisBreaker { + #[pyo3(attribute)] + failure_threshold: u32, + #[pyo3(attribute)] + recovery_timeout: f64, + #[pyo3(attribute)] + timeout_min_duration: f64, +} + +impl RedisBreaker { + fn config(&self) -> PyResult { + let seconds = |seconds: f64| { + Duration::try_from_secs_f64(seconds) + .map_err(|error| PyValueError::new_err(error.to_string())) + }; + Ok(BreakerConfig { + failure_threshold: self.failure_threshold, + recovery_timeout: seconds(self.recovery_timeout)?, + timeout_min_duration: seconds(self.timeout_min_duration)?, + }) + } +} + +/// `litellm.redis.store_settings.RedisStoreSettings`, which a `RedisCache` captures when it is built. +#[derive(FromPyObject)] +pub(crate) struct StoreSettings { + #[pyo3(attribute)] + redis_kwargs: RedisSpec, + #[pyo3(attribute)] + namespace: Option, + #[pyo3(attribute)] + default_ttl: f64, + #[pyo3(attribute)] + breaker: Option, +} + +fn is_plain(value: &Bound<'_, PyAny>) -> bool { + value.is_instance_of::() + || value.is_instance_of::() + || value.is_instance_of::() + || value.is_instance_of::() + || value.is_instance_of::() + || value.is_instance_of::() + || value.is_instance_of::() +} + +/// A client the process built, held weakly so a discarded `Router` drops its client and provider. +struct Shared { + resolved: Resolution, + client: WeakRedisClient, +} + +static CLIENTS: LazyLock>> = LazyLock::new(|| Mutex::new(Vec::new())); + +fn live_clients(py: Python<'_>) -> std::sync::MutexGuard<'static, Vec> { + let mut clients = CLIENTS + .lock_py_attached(py) + .unwrap_or_else(PoisonError::into_inner); + clients.retain(|shared| shared.client.upgrade().is_some()); + clients +} + +/// The one client for a resolved Redis config and credential provider, so every family in this process shares its pool +/// and its breaker. A config redis-py refuses building its client raises the same class here. One it only fails on +/// once a command connects, where `RedisCache` logs every failed call, leaves the family on memory with an error. +fn shared_client( + py: Python<'_>, + spec: RedisSpec, + namespace: Option, + breaker: Option, +) -> PyResult> { + let environment: SharedEnvironment = Arc::new(ProcessEnvironment); + let RedisSpec { + kwargs, + credential_provider, + } = spec; + // The registry entry's client keeps the provider alive, so no live entry shares its address + let handle = credential_provider + .as_ref() + .map(|provider| ProviderHandle::new(provider.as_ptr() as usize as u64)); + let logger = crate::logger::capture(py); + let environ = py.import("os")?.getattr("environ")?; + let resolved = match logger.scope(|| resolve(&kwargs, handle, &environ))? { + Ok(resolution) => { + logger.scope(|| { + for ignored in &resolution.ignored { + litellm_tracing::warn!( + "Redis setting {} is not supported by the Rust client and is ignored: {}", + ignored.setting, + ignored.consequence + ); + } + }); + Resolution { + namespace, + ..resolution + } + } + Err(error) => return unresolved(py, &error), + }; + let runtime = litellm_host_python::runtime_handle()?; + let events: Arc = Arc::new(ServiceLoggingEvents::new(py, runtime.clone())?); + let host = HostServices { + events: Some(events), + http: crate::http::host_client(py, litellm_http::ClientVariant::Provider)?, + environment, + credential_provider: credential_provider + .as_ref() + .map(|provider| -> PyResult> { + Ok(Arc::new(PythonCredentialSource::new( + provider.clone_ref(py), + PythonContext::capture(py)?, + ))) + }) + .transpose()?, + }; + let mut clients = live_clients(py); + if let Some(client) = clients + .iter() + .filter(|shared| shared.resolved == resolved) + .find_map(|shared| shared.client.upgrade()) + { + return Ok(Some(client)); + } + let connected = { + let _runtime = runtime.enter(); + litellm_redis_connect::connect(resolved.clone(), host, breaker) + }; + let client = match connected { + Ok(client) => client, + Err(error) => { + drop(clients); + return Ok(unconnected(py, &error)); + } + }; + clients.push(Shared { + resolved, + client: client.downgrade(), + }); + Ok(Some(client)) +} + +/// The Rust client for one `RedisCache`, built once and handed to every native store, so a store +/// never sees the redis-py kwargs or the namespace. +#[pyclass(frozen, name = "NativeRedisClient")] +pub(crate) struct NativeRedisClient { + pub(crate) client: RedisClient, + /// `RedisCache.default_ttl`, the TTL a store's write takes when its caller gives none. + pub(crate) default_ttl: Duration, +} + +#[pymethods] +impl NativeRedisClient { + /// `None` when redis-py would fail only once a command connects, which leaves every store given it on memory. + #[staticmethod] + fn connect(py: Python<'_>, settings: StoreSettings) -> PyResult> { + let StoreSettings { + redis_kwargs, + namespace, + default_ttl, + breaker, + } = settings; + let default_ttl = number::seconds(default_ttl).map_err(seconds_error)?; + let breaker = breaker.as_ref().map(RedisBreaker::config).transpose()?; + Ok( + shared_client(py, redis_kwargs, namespace, breaker)?.map(|client| Self { + client, + default_ttl, + }), + ) + } + + #[getter] + fn default_ttl_seconds(&self) -> f64 { + self.default_ttl.as_secs_f64() + } +} + +/// The breaker state of every native Redis client this process built, so the Prometheus state +/// gauge counts them next to the Python `RedisCache` breakers. +#[pyfunction] +pub(crate) fn native_redis_breaker_states(py: Python<'_>) -> Vec<&'static str> { + live_clients(py) + .iter() + .filter_map(|shared| shared.client.upgrade()) + .filter_map(|client| client.breaker_state()) + .map(|state| state_label(&state)) + .collect() +} + +/// What redis-py makes of the kwargs, with `REDIS_USERNAME` read from `environ` the way Python's Azure AD auth +/// reads `os.environ`. +fn resolve( + kwargs: &Kwargs, + provider: Option, + environ: &Bound<'_, PyAny>, +) -> PyResult> { + let redis_username: Option = environ + .call_method1("get", ("REDIS_USERNAME",))? + .extract()?; + Ok(litellm_redis_resolve::interpret( + kwargs, + provider, + redis_username.as_deref(), + )) +} + +fn unresolved( + py: Python<'_>, + error: &litellm_redis_resolve::Error, +) -> PyResult> { + let message = || litellm_error::Report::new(error).to_string(); + match error.raised_building_the_python_client() { + Some(PythonException::ValueError) => Err(PyValueError::new_err(message())), + Some(PythonException::TypeError) => Err(PyTypeError::new_err(message())), + Some(PythonException::RedisClusterException) => { + Err(RedisClusterException::new_err(message())) + } + None => Ok(memory_only(py, error)), + } +} + +// redis-py reads TLS files and fetches credentials only once a command connects, so none of these fails building +// its client +fn unconnected(py: Python<'_>, error: &litellm_redis_connect::Error) -> Option { + match error { + litellm_redis_connect::Error::Connect(_) + | litellm_redis_connect::Error::NoHostCredentialProvider + | litellm_redis_connect::Error::ProcessEnvironmentRequired { .. } + | litellm_redis_connect::Error::Credential { .. } + | litellm_redis_connect::Error::CredentialTimeout { .. } + | litellm_redis_connect::Error::ExpiredCredential + | litellm_redis_connect::Error::MissingCredential { .. } => memory_only(py, error), + } +} + +fn memory_only(py: Python<'_>, error: &(dyn std::error::Error + 'static)) -> Option { + let report = litellm_error::Report::new(error).to_string(); + crate::logger::capture(py).scope(|| { + litellm_tracing::error!( + "Redis is not usable from the Rust stores, their state stays in memory: {report}" + ) + }); + None +} + +fn seconds_error(error: SecondsError) -> PyErr { + match error { + SecondsError::Overflow => PyOverflowError::new_err(error.to_string()), + SecondsError::NotANumber => PyValueError::new_err(error.to_string()), + } +} + +#[cfg(test)] +mod tests { + use std::ffi::CStr; + + use litellm_redis_types::Auth; + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn kwargs(fields: serde_json::Value) -> Kwargs { + let fields: BTreeMap = serde_json::from_value(fields).unwrap(); + litellm_redis_resolve::kwargs(fields) + } + + #[rstest] + #[case::set(c"environ = {'REDIS_USERNAME': 'azure-user'}", Some("azure-user"))] + #[case::empty_is_unset(c"environ = {'REDIS_USERNAME': ''}", None)] + fn azure_auth_takes_its_username_from_the_environ_like_python( + #[case] source: &CStr, + #[case] expected: Option<&str>, + ) { + Python::initialize(); + Python::attach(|py| { + let locals = PyDict::new(py); + py.run(source, Some(&locals), Some(&locals)).unwrap(); + let environ = locals.get_item("environ").unwrap().unwrap(); + let kwargs = kwargs(json!({"host": "redis", "azure_redis_ad_token": "true"})); + + let resolved = resolve(&kwargs, None, &environ).unwrap().unwrap(); + + assert!( + matches!(&resolved.auth, Auth::AzureAd(azure) if azure.username.as_deref() == expected), + "{:?}", + resolved.auth + ); + }); + } + + #[rstest] + #[case::no_target(json!({"port": 6379}), Some("ValueError"))] + #[case::unenforceable(json!({"host": "redis", "ssl": true, "ssl_validate_ocsp": true}), Some("TypeError"))] + #[case::fails_only_once_a_command_connects( + json!({"host": "redis", "ssl": true, "ssl_cert_reqs": "None"}), + None + )] + fn a_config_raises_where_redis_py_raises_building_its_client( + #[case] fields: serde_json::Value, + #[case] raised: Option<&str>, + ) { + Python::initialize(); + Python::attach(|py| { + let error = litellm_redis_resolve::interpret(&kwargs(fields), None, None).unwrap_err(); + + let outcome = unresolved(py, &error) + .map(|client| client.is_some()) + .map_err(|error| error.get_type(py).name().unwrap().to_string()); + + assert_eq!( + outcome, + raised.map_or(Ok(false), |class| Err(class.to_owned())) + ); + }); + } + + #[rstest] + #[case::overflow(f64::INFINITY, "OverflowError")] + #[case::not_a_number(f64::NAN, "ValueError")] + fn a_default_ttl_python_cannot_make_an_int_raises_like_int( + #[case] seconds: f64, + #[case] raised: &str, + ) { + Python::initialize(); + Python::attach(|py| { + let error = number::seconds(seconds).map_err(seconds_error).unwrap_err(); + + assert_eq!(error.get_type(py).name().unwrap().to_string(), raised); + }); + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/events.rs b/litellm-rust/crates/python-bridge/src/redis/events.rs new file mode 100644 index 00000000000..04b0c9dbf32 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/events.rs @@ -0,0 +1,391 @@ +use std::{ + sync::Arc, + time::{Duration, SystemTime, UNIX_EPOCH}, +}; + +use litellm_circuit_breaker::{State, Transition}; +use litellm_host_python::{PythonContext, attach_blocking}; +use litellm_redis_client::{ + CallEvent, CallTag, FailureKind, Operation, Outcome, RedisEvents, current_tag, tagged, +}; +use pyo3::{prelude::*, types::PyModule}; +use tokio::runtime::Handle; + +/// The asyncio loop a Python caller ran on. `RedisCache` scheduled its service events there with +/// `asyncio.create_task`, so the native events go there too instead of a new loop per event. +#[derive(Clone)] +pub(crate) struct EventLoop(Arc>); + +impl EventLoop { + pub(crate) fn running(py: Python<'_>) -> Option { + pyo3_async_runtimes::get_running_loop(py) + .ok() + .map(|event_loop| Self(Arc::new(event_loop.unbind()))) + } +} + +/// The Python caller of one store operation. Its context carries `with_service_target` and the +/// ambient OTel span, and its `call_type` is the label `RedisCache` reports, so dashboards keep +/// their series. +pub(crate) struct RedisCall { + pub(crate) context: PythonContext, + pub(crate) event_loop: Option, + pub(crate) call_type: &'static str, +} + +impl RedisCall { + pub(crate) fn tag(self) -> CallTag { + Arc::new(self) + } + + pub(crate) async fn scope(self, operation: F) -> F::Output { + tagged(self.tag(), operation).await + } +} + +/// Reports every native Redis call and breaker transition to `ServiceLogging`. +pub(crate) struct ServiceLoggingEvents { + record_call: Arc>, + record_breaker: Arc>, + context: PythonContext, + /// The loop the client was built on, for calls whose caller had none running. + event_loop: Option, + runtime: Handle, +} + +impl ServiceLoggingEvents { + pub(crate) fn new(py: Python<'_>, runtime: Handle) -> PyResult { + const MODULE: &str = "litellm.rust_bridge.host.redis_events"; + Self::from_module(py, &py.import(MODULE)?, runtime) + } + + fn from_module( + py: Python<'_>, + module: &Bound<'_, PyModule>, + runtime: Handle, + ) -> PyResult { + Ok(Self { + record_call: Arc::new(module.getattr("record_redis_call")?.unbind()), + record_breaker: Arc::new(module.getattr("record_redis_breaker")?.unbind()), + context: PythonContext::capture(py)?, + event_loop: EventLoop::running(py), + runtime, + }) + } + + fn report(&self, context: PythonContext, record: Arc>, args: ReportArgs) { + self.runtime.spawn(async move { + let reported = attach_blocking(context, move |py| match args { + ReportArgs::Call(call) => record + .bind(py) + .call1(( + call.event_loop + .as_ref() + .map(|event_loop| event_loop.0.bind(py)), + call.call_type, + call.start_time, + call.end_time, + call.failure, + )) + .map(drop), + ReportArgs::Breaker(state) => record.bind(py).call1((state,)).map(drop), + }) + .await + .and_then(|reported| reported); + if let Err(error) = reported { + tracing::warn!(error = %error, "reporting a Redis event to ServiceLogging failed"); + } + }); + } +} + +/// One call as `record_redis_call` takes it, its times in `time.time()` seconds. +struct CallReport { + event_loop: Option, + call_type: &'static str, + start_time: f64, + end_time: f64, + failure: Option<&'static str>, +} + +enum ReportArgs { + Call(CallReport), + Breaker(&'static str), +} + +/// The wall-clock start and end of a call that just finished after `duration`. +fn wall_clock_span(duration: Duration) -> (f64, f64) { + let end = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default(); + ( + end.saturating_sub(duration).as_secs_f64(), + end.as_secs_f64(), + ) +} + +fn operation_call_type(operation: Operation) -> &'static str { + match operation { + Operation::Command(name) => name, + Operation::Script => "eval", + Operation::Pipeline { .. } => "pipeline", + } +} + +fn failure_label(outcome: Outcome) -> Option<&'static str> { + match outcome { + Outcome::Succeeded => None, + Outcome::Failed(FailureKind::Timeout) => Some("timeout"), + Outcome::Failed(FailureKind::Transport) => Some("transport"), + Outcome::Failed(FailureKind::Server) => Some("server"), + Outcome::Failed(FailureKind::Decode) => Some("decode"), + Outcome::Failed(FailureKind::BreakerOpen) => Some("breaker_open"), + } +} + +pub(crate) fn state_label(state: &State) -> &'static str { + match state { + State::Closed { .. } => "closed", + State::Open { .. } => "open", + State::HalfOpen => "half_open", + } +} + +impl RedisEvents for ServiceLoggingEvents { + fn call(&self, event: &CallEvent) { + let scoped = current_tag().and_then(|tag| tag.downcast::().ok()); + let (context, event_loop, call_type) = match scoped { + Some(call) => ( + call.context.clone(), + call.event_loop.clone().or_else(|| self.event_loop.clone()), + call.call_type, + ), + None => ( + self.context.clone(), + self.event_loop.clone(), + operation_call_type(event.operation), + ), + }; + let (start_time, end_time) = wall_clock_span(event.duration); + self.report( + context, + self.record_call.clone(), + ReportArgs::Call(CallReport { + event_loop, + call_type, + start_time, + end_time, + failure: failure_label(event.outcome), + }), + ); + } + + fn breaker(&self, transition: &Transition) { + self.report( + self.context.clone(), + self.record_breaker.clone(), + ReportArgs::Breaker(state_label(&transition.to)), + ); + } +} + +#[cfg(test)] +mod tests { + use std::{sync::Arc, time::Duration}; + + use litellm_host_python::PythonContext; + use litellm_redis_client::{CallEvent, FailureKind, Operation, Outcome, RedisEvents}; + use pyo3::{ + prelude::*, + types::{PyCFunction, PyModule, PyString}, + }; + use rstest::rstest; + use tokio::runtime::Handle; + + use super::{EventLoop, RedisCall, ServiceLoggingEvents}; + + const RECORDER: &str = " +import contextvars, time +target = contextvars.ContextVar('target', default=None) +seen = [] +def record_redis_call(event_loop, call_type, start_time, end_time, failure): + just_now = abs(time.time() - end_time) < 60 + seen.append((event_loop, call_type, round(end_time - start_time, 3), just_now, failure, target.get())) +def record_redis_breaker(state): + seen.append(('breaker', state)) +"; + + fn stand_in_loop(py: Python<'_>, name: &str) -> EventLoop { + EventLoop(Arc::new(PyString::new(py, name).into_any().unbind())) + } + + fn recorder(name: &str, client_loop: Option<&str>) -> (ServiceLoggingEvents, Py) { + Python::initialize(); + Python::attach(|py| { + let name = std::ffi::CString::new(format!("redis_events_recorder_{name}")).unwrap(); + let module = + PyModule::from_code(py, &std::ffi::CString::new(RECORDER).unwrap(), &name, &name) + .unwrap(); + let events = ServiceLoggingEvents::from_module(py, &module, Handle::current()).unwrap(); + ( + ServiceLoggingEvents { + event_loop: client_loop.map(|client_loop| stand_in_loop(py, client_loop)), + ..events + }, + module.unbind(), + ) + }) + } + + async fn seen(module: &Py, count: usize) -> String { + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + loop { + let seen: (usize, String) = Python::attach(|py| { + let seen = module.bind(py).getattr("seen").unwrap(); + (seen.len().unwrap(), seen.repr().unwrap().to_string()) + }); + if seen.0 >= count || tokio::time::Instant::now() > deadline { + return seen.1; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + } + + fn scoped_call(namespace: &Py, caller_loop: Option<&str>) -> RedisCall { + Python::attach(|py| { + namespace + .bind(py) + .getattr("target") + .unwrap() + .call_method1("set", ("spend_counter",)) + .unwrap(); + RedisCall { + context: PythonContext::capture(py).unwrap(), + event_loop: caller_loop.map(|caller_loop| stand_in_loop(py, caller_loop)), + call_type: "async_increment", + } + }) + } + + #[rstest] + #[case::on_the_callers_loop(Some("caller loop"), "caller loop")] + #[case::on_the_clients_loop_when_the_caller_had_none(None, "client loop")] + #[tokio::test(flavor = "multi_thread")] + async fn a_scoped_call_reports_the_store_labels_and_its_times_in_the_callers_context( + #[case] caller_loop: Option<&str>, + #[case] expected_loop: &str, + ) { + let (events, namespace) = recorder( + expected_loop.replace(' ', "_").as_str(), + Some("client loop"), + ); + let call = scoped_call(&namespace, caller_loop); + let event = CallEvent { + operation: Operation::Command("INCRBYFLOAT"), + duration: Duration::from_millis(250), + outcome: Outcome::Failed(FailureKind::Timeout), + }; + + call.scope(async { events.call(&event) }).await; + + assert_eq!( + seen(&namespace, 1).await, + format!( + "[('{expected_loop}', 'async_increment', 0.25, True, 'timeout', 'spend_counter')]" + ) + ); + } + + #[rstest] + #[case::opened(litellm_circuit_breaker::State::Open { since: std::time::Instant::now() }, "open")] + #[case::probing(litellm_circuit_breaker::State::HalfOpen, "half_open")] + #[tokio::test(flavor = "multi_thread")] + async fn breaker_transitions_report_python_state_names( + #[case] to: litellm_circuit_breaker::State, + #[case] expected: &str, + ) { + let (events, namespace) = recorder(expected, None); + + events.breaker(&litellm_circuit_breaker::Transition { + from: litellm_circuit_breaker::State::HalfOpen, + to, + generation: 1, + }); + + assert_eq!( + seen(&namespace, 1).await, + format!("[('breaker', '{expected}')]") + ); + } + + #[rstest] + #[tokio::test(flavor = "multi_thread")] + async fn a_transition_on_a_thread_without_a_runtime_is_still_reported() { + let (events, namespace) = recorder("plain_thread", None); + + std::thread::spawn(move || { + events.breaker(&litellm_circuit_breaker::Transition { + from: litellm_circuit_breaker::State::HalfOpen, + to: litellm_circuit_breaker::State::HalfOpen, + generation: 1, + }); + }) + .join() + .unwrap(); + + assert_eq!(seen(&namespace, 1).await, "[('breaker', 'half_open')]"); + } + + #[rstest] + #[tokio::test(flavor = "multi_thread")] + async fn an_unscoped_call_falls_back_to_the_command_name_and_the_clients_loop() { + let (events, namespace) = recorder("unscoped", None); + + events.call(&CallEvent { + operation: Operation::Pipeline { ops: 3 }, + duration: Duration::from_millis(5), + outcome: Outcome::Succeeded, + }); + + assert_eq!( + seen(&namespace, 1).await, + "[(None, 'pipeline', 0.005, True, None, None)]" + ); + } + + #[rstest] + fn running_is_the_loop_the_caller_runs_on_and_none_outside_one() { + Python::initialize(); + Python::attach(|py| { + let module = PyModule::from_code( + py, + c"import asyncio\nasync def probe_inside(probe):\n return probe() is asyncio.get_running_loop()\n", + c"event_loop_probe", + c"event_loop_probe", + ) + .unwrap(); + let probe = PyCFunction::new_closure(py, None, None, |_, _| { + Python::attach(|py| -> PyResult>> { + Ok(EventLoop::running(py).map(|event_loop| event_loop.0.clone_ref(py))) + }) + }) + .unwrap(); + let coroutine = module + .getattr("probe_inside") + .unwrap() + .call1((probe,)) + .unwrap(); + + let inside: bool = py + .import("asyncio") + .unwrap() + .call_method1("run", (coroutine,)) + .unwrap() + .extract() + .unwrap(); + + assert!(inside); + assert!(EventLoop::running(py).is_none()); + }); + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/mod.rs b/litellm-rust/crates/python-bridge/src/redis/mod.rs new file mode 100644 index 00000000000..0a54822bcd7 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/mod.rs @@ -0,0 +1,3 @@ +pub(crate) mod client; +mod events; +pub(crate) mod store; diff --git a/litellm-rust/crates/python-bridge/src/redis/store/auth_cache.rs b/litellm-rust/crates/python-bridge/src/redis/store/auth_cache.rs new file mode 100644 index 00000000000..8d7788b136e --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/auth_cache.rs @@ -0,0 +1,265 @@ +use std::sync::Arc; + +use litellm_python_compat::Value; +use litellm_redis_store::auth_cache::{AuthCache, Settings}; +use pyo3::prelude::*; + +use super::{ + NativePending, PyNumber, kind_error, pending, redis_async, redis_sync, remote, + request::{NativeRequestRedis, NativeSend}, + ttl, + value::PyValue, +}; +use crate::redis::client::NativeRedisClient; + +fn values(values: Vec>) -> Vec> { + values.into_iter().map(|value| value.map(PyValue)).collect() +} + +/// `UserApiKeyCache` with `enable_redis_auth_cache` on. The sync methods back `DualCache`'s sync +/// API and block with the GIL released, like redis-py's sync client. +#[pyclass(frozen, name = "NativeAuthCache")] +pub(crate) struct NativeAuthCache(Arc); + +#[pymethods] +impl NativeAuthCache { + #[new] + #[pyo3(signature = (client, *, memory_entries, memory_ttl_seconds = None))] + fn new( + client: Option>, + memory_entries: usize, + memory_ttl_seconds: Option, + ) -> PyResult { + Ok(Self(Arc::new(AuthCache::new( + client.map(|client| remote(&client)), + Settings { + memory_entries, + memory_ttl: ttl(memory_ttl_seconds)?, + }, + )))) + } + + #[getter] + fn has_redis(&self) -> bool { + self.0.has_redis() + } + + /// `request` answers a key its pre-call reads already found absent without another `GET`. + #[pyo3(signature = (key, request = None))] + fn get<'py>( + &self, + py: Python<'py>, + key: String, + request: Option<&NativeRequestRedis>, + ) -> PyResult> { + let store = self.0.clone(); + let request = request.map(|request| request.0.clone()); + redis_async(py, "async_get_cache", kind_error, async move { + store + .get(request.as_ref(), &key) + .await + .map(|value| value.map(PyValue)) + }) + } + + fn get_sync(&self, py: Python<'_>, key: String) -> PyResult> { + let store = self.0.clone(); + redis_sync(py, "get_cache", kind_error, async move { + store.get(None, &key).await.map(|value| value.map(PyValue)) + }) + } + + /// `update_cache_ttl` and `update_in_memory_max_size`: both partitions keep what they hold. + #[pyo3(signature = (*, memory_entries, memory_ttl_seconds = None))] + fn reconfigure(&self, memory_entries: usize, memory_ttl_seconds: Option) -> PyResult<()> { + self.0.reconfigure(Settings { + memory_entries, + memory_ttl: ttl(memory_ttl_seconds)?, + }); + Ok(()) + } + + #[pyo3(signature = (key, value, ttl_seconds = None))] + fn set<'py>( + &self, + py: Python<'py>, + key: String, + value: PyValue, + ttl_seconds: Option, + ) -> PyResult> { + let store = self.0.clone(); + let ttl = ttl(ttl_seconds)?; + redis_async(py, "async_set_cache", kind_error, async move { + store.set(&key, value.0, ttl).await + }) + } + + #[pyo3(signature = (key, value, ttl_seconds = None))] + fn set_sync( + &self, + py: Python<'_>, + key: String, + value: PyValue, + ttl_seconds: Option, + ) -> PyResult<()> { + let store = self.0.clone(); + let ttl = ttl(ttl_seconds)?; + redis_sync(py, "set_cache", kind_error, async move { + store.set_str(&key, value.0, ttl).await + }) + } + + /// Memory now and the `SET` on the batch `send` picks. + #[pyo3(signature = (key, value, ttl_seconds, send))] + fn declare_set( + &self, + key: String, + value: PyValue, + ttl_seconds: Option, + send: &NativeSend, + ) -> PyResult { + let written = self + .0 + .declare_set(send.send(), &key, value.0, ttl(ttl_seconds)?) + .map_err(kind_error)?; + Ok(pending( + "async_set_cache", + kind_error, + written, + std::convert::identity, + )) + } + + fn declare_delete(&self, key: String, send: &NativeSend) -> PyResult { + let deleted = self + .0 + .declare_delete(send.send(), &key) + .map_err(kind_error)?; + Ok(pending( + "async_delete_cache", + kind_error, + deleted, + std::convert::identity, + )) + } + + #[pyo3(signature = (entries, ttl_seconds = None))] + fn set_many<'py>( + &self, + py: Python<'py>, + entries: Vec<(String, PyValue)>, + ttl_seconds: Option, + ) -> PyResult> { + let store = self.0.clone(); + let entries = entries + .into_iter() + .map(|(key, value)| (key, value.0)) + .collect(); + let ttl = ttl(ttl_seconds)?; + redis_async(py, "async_set_cache_pipeline", kind_error, async move { + store.set_many(entries, ttl).await + }) + } + + /// On a request batch it returns before anything is sent, and `Now` is a pipeline of its own. + fn write_back<'py>( + &self, + py: Python<'py>, + entries: Vec<(String, PyValue, Option)>, + send: &NativeSend, + ) -> PyResult> { + let entries = entries + .into_iter() + .map(|(key, value, seconds)| Ok((key, value.0, ttl(seconds)?))) + .collect::>()?; + let written = self + .0 + .write_back(send.send(), entries) + .map_err(kind_error)?; + redis_async( + py, + "async_set_cache_pipeline_with_ttls", + kind_error, + written, + ) + } + + fn delete<'py>(&self, py: Python<'py>, keys: Vec) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_delete_cache", kind_error, async move { + store.delete(&keys).await + }) + } + + fn delete_sync(&self, py: Python<'_>, keys: Vec) -> PyResult<()> { + let store = self.0.clone(); + redis_sync(py, "delete_cache", kind_error, async move { + store.delete(&keys).await + }) + } + + /// One `MGET` of keys the caller found missing in memory, on the batch `send` picks; `None` + /// without Redis. + fn get_many(&self, keys: Vec, send: &NativeSend) -> Option { + let read = self.0.get_many(send.send(), &keys)?; + Some(pending("async_batch_get_cache", kind_error, read, values)) + } + + fn increment_if_present<'py>( + &self, + py: Python<'py>, + key: String, + amount: PyNumber, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_increment_if_present", kind_error, async move { + store.increment_if_present(&key, amount.0).await + }) + } + + fn get_local(&self, key: &str) -> PyResult> { + Ok(self.0.get_local(key).map_err(kind_error)?.map(PyValue)) + } + + #[pyo3(signature = (key, value, ttl_seconds = None))] + fn set_local(&self, key: &str, value: PyValue, ttl_seconds: Option) -> PyResult<()> { + self.0 + .set_local(key, value.0, ttl(ttl_seconds)?) + .map_err(kind_error) + } + + fn delete_local(&self, key: &str) -> PyResult<()> { + self.0.delete_local(key).map_err(kind_error) + } + + fn flush_local(&self) -> PyResult<()> { + self.0.flush_local().map_err(kind_error) + } + + fn len_local(&self) -> PyResult { + let (objects, key_objects) = self.0.len_local().map_err(kind_error)?; + Ok(PyLocalSizes { + objects, + key_objects, + }) + } +} + +/// Entries the management-object and the user-key partitions hold in memory. +#[pyclass(frozen, eq, get_all, name = "LocalSizes")] +#[derive(PartialEq)] +pub(crate) struct PyLocalSizes { + objects: usize, + key_objects: usize, +} + +#[pymethods] +impl PyLocalSizes { + #[new] + fn new(objects: usize, key_objects: usize) -> Self { + Self { + objects, + key_objects, + } + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/mod.rs b/litellm-rust/crates/python-bridge/src/redis/store/mod.rs new file mode 100644 index 00000000000..d9c3aabb814 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/mod.rs @@ -0,0 +1,271 @@ +pub(crate) mod auth_cache; +mod policy; +pub(crate) mod rate_limits; +pub(crate) mod request; +pub(crate) mod router_cooldowns; +pub(crate) mod router_usage; +pub(crate) mod sensitive_route_pins; +pub(crate) mod spend_counters; +mod value; + +use value::PyValue; + +use std::{ + pin::Pin, + sync::{Mutex, PoisonError}, + time::Duration, +}; + +use litellm_host_python::PythonContext; +use litellm_python_compat::number::{SecondsError, seconds}; +use litellm_redis_store::{Error, FromValue, IntoValue, Number, Pending, Remote}; +use pyo3::{ + IntoPyObjectExt, + exceptions::{PyOverflowError, PyRuntimeError, PyTypeError, PyValueError}, + prelude::*, + types::{PyBool, PyFloat, PyInt}, +}; + +use crate::execution::{run_async_value, run_sync_value}; +use crate::redis::{ + client::NativeRedisClient, + events::{EventLoop, RedisCall}, +}; + +/// The labels `RedisCache` reports for this operation, in the caller's context so its +/// `service_target` reaches the service event, dispatched on the loop the caller runs on. +fn call(py: Python<'_>, call_type: &'static str) -> PyResult { + Ok(RedisCall { + context: PythonContext::capture(py)?, + event_loop: EventLoop::running(py), + call_type, + }) +} + +/// Runs one Redis operation as a Python awaitable, reported to `ServiceLogging` as `call_type`. +fn redis_async<'py, T, E: 'static>( + py: Python<'py>, + call_type: &'static str, + error: fn(E) -> PyErr, + operation: impl Future> + Send + 'static, +) -> PyResult> +where + T: for<'a> IntoPyObject<'a> + Send + 'static, +{ + let scope = call(py, call_type)?; + run_async_value( + py, + async move { scope.scope(operation).await.map_err(error) }, + ) +} + +/// Runs one Redis operation with the GIL released, for the router's sync paths. +fn redis_sync( + py: Python<'_>, + call_type: &'static str, + error: fn(E) -> PyErr, + operation: impl Future> + Send + 'static, +) -> PyResult +where + T: Send + 'static, +{ + let scope = call(py, call_type)?; + run_sync_value( + py, + async move { scope.scope(operation).await.map_err(error) }, + ) +} + +/// A TTL in seconds, raising what `int(seconds)` raises for infinity and NaN. +fn duration(value: f64) -> PyResult { + seconds(value).map_err(|error| match error { + SecondsError::Overflow => PyOverflowError::new_err(error.to_string()), + SecondsError::NotANumber => PyValueError::new_err(error.to_string()), + }) +} + +fn ttl(value: Option) -> PyResult> { + value.map(duration).transpose() +} + +fn remote(client: &NativeRedisClient) -> Remote { + Remote { + client: client.client.clone(), + default_ttl: client.default_ttl, + } +} + +type Resolve = Pin>> + Send>>; + +/// A Redis call declared on a request's batch. `resolve` flushes that batch if it has not gone +/// out and answers the call, once. +#[pyclass(frozen, name = "NativePending")] +pub(crate) struct NativePending { + resolve: Mutex>, + call_type: &'static str, +} + +impl NativePending { + pub(crate) fn new( + call_type: &'static str, + error: fn(E) -> PyErr, + reply: impl Future> + Send + 'static, + ) -> Self + where + T: for<'a> IntoPyObject<'a> + Send + 'static, + { + let resolve = async move { + let reply = reply.await.map_err(error)?; + Python::attach(|py| reply.into_py_any(py)) + }; + Self { + resolve: Mutex::new(Some(Box::pin(resolve))), + call_type, + } + } +} + +#[pymethods] +impl NativePending { + fn resolve<'py>(&self, py: Python<'py>) -> PyResult> { + let resolve = self + .resolve + .lock() + .unwrap_or_else(PoisonError::into_inner) + .take() + .ok_or_else(|| PyRuntimeError::new_err("this Redis call was already resolved"))?; + redis_async(py, self.call_type, std::convert::identity, resolve) + } +} + +pyo3::create_exception!( + _native, + NativeRedisStoreError, + pyo3::exceptions::PyException, + "A native Redis store operation failed. Args are (kind, message), kind spelled as litellm-redis-store's ErrorKind." +); + +/// What every native store raises, its kind telling a caller which failures it may drop. +fn kind_error(error: Error) -> PyErr { + let kind: &'static str = error.kind().into(); + NativeRedisStoreError::new_err((kind, litellm_error::Report::new(&error).to_string())) +} + +/// A store op declared on the batch the caller's `NativeSend` picked, answered as `reply` shapes it. +fn pending( + call_type: &'static str, + error: fn(Error) -> PyErr, + declared: Pending, + reply: fn(T) -> R, +) -> NativePending +where + T: Send + 'static, + R: for<'a> IntoPyObject<'a> + Send + 'static, +{ + NativePending::new(call_type, error, async move { declared.await.map(reply) }) +} + +/// A Python `int` or `float` across the bridge, keeping which one it is. +#[derive(Clone, Copy, PartialEq)] +pub(crate) struct PyNumber(Number); + +impl<'a, 'py> FromPyObject<'a, 'py> for PyNumber { + type Error = PyErr; + + fn extract(value: Borrowed<'a, 'py, PyAny>) -> PyResult { + if value.is_instance_of::() && !value.is_instance_of::() { + return Ok(Self(Number::Int(value.extract()?))); + } + if value.is_instance_of::() { + return Ok(Self(Number::Float(value.extract()?))); + } + Err(PyTypeError::new_err("expected an int or a float")) + } +} + +impl<'py> IntoPyObject<'py> for PyNumber { + type Target = PyAny; + type Output = Bound<'py, PyAny>; + type Error = PyErr; + + fn into_pyobject(self, py: Python<'py>) -> PyResult { + Ok(match self.0 { + Number::Int(value) => value.into_pyobject(py)?.into_any(), + Number::Float(value) => value.into_pyobject(py)?.into_any(), + }) + } +} + +/// A store enum read from its wire spelling, so the bridge names no variant. +pub(crate) struct Spelled(pub(crate) T); + +impl<'a, 'py, T: std::str::FromStr> FromPyObject<'a, 'py> for Spelled { + type Error = PyErr; + + fn extract(value: Borrowed<'a, 'py, PyAny>) -> PyResult { + let spelling: String = value.extract()?; + spelling + .parse() + .map(Self) + .map_err(|_| PyValueError::new_err(format!("unknown spelling {spelling:?}"))) + } +} + +/// A store type whose Python side is plain data, converted through its `FromValue` and `IntoValue` +/// so the bridge never names a field. A shape the store refuses is `ValueError`. +pub(crate) struct PyData(pub(crate) T); + +impl<'a, 'py, T: FromValue> FromPyObject<'a, 'py> for PyData { + type Error = PyErr; + + fn extract(value: Borrowed<'a, 'py, PyAny>) -> PyResult { + let PyValue(value) = value.extract()?; + T::from_value(&value) + .map(Self) + .map_err(|error| PyValueError::new_err(error.to_string())) + } +} + +impl<'py, T: IntoValue> IntoPyObject<'py> for PyData { + type Target = PyAny; + type Output = Bound<'py, PyAny>; + type Error = PyErr; + + fn into_pyobject(self, py: Python<'py>) -> PyResult { + PyValue(self.0.into_value()).into_pyobject(py) + } +} + +fn unwrap_all(values: Vec>) -> Vec { + values.into_iter().map(|value| value.0).collect() +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use pyo3::prelude::*; + use rstest::rstest; + + use super::ttl; + + #[rstest] + #[case::infinity(f64::INFINITY, "OverflowError")] + #[case::nan(f64::NAN, "ValueError")] + fn a_ttl_int_refuses_raises_what_int_raises(#[case] seconds: f64, #[case] raised: &str) { + Python::initialize(); + Python::attach(|py| { + let error = ttl(Some(seconds)).unwrap_err(); + + assert_eq!(error.get_type(py).name().unwrap().to_string(), raised); + }); + } + + #[rstest] + #[case::fraction(Some(1.5), Some(Duration::from_millis(1500)))] + #[case::negative_clamps_to_zero(Some(-1.0), Some(Duration::ZERO))] + #[case::absent(None, None)] + fn a_finite_ttl_converts(#[case] seconds: Option, #[case] expected: Option) { + assert_eq!(ttl(seconds).unwrap(), expected); + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/policy.rs b/litellm-rust/crates/python-bridge/src/redis/store/policy.rs new file mode 100644 index 00000000000..ffdf7b97a2e --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/policy.rs @@ -0,0 +1,151 @@ +use std::time::Duration; + +use litellm_redis_store::TierPolicy; +use pyo3::prelude::*; + +use crate::{ + coercion::{Field, FieldSpec, ProjectionError}, + python_settings::{PythonSettings, Snapshot}, +}; + +const COOLDOWN_REFRESH: FieldSpec = + FieldSpec::new("cooldown_redis_read_interval", seconds); +const USAGE_REFRESH: FieldSpec = FieldSpec::new("usage_redis_read_interval", seconds); +const TRACKED_READS: FieldSpec = FieldSpec::new("max_redis_batch_cache_size", count); +const MEMORY_TTL: FieldSpec> = + FieldSpec::new("default_in_memory_ttl", optional_seconds); + +/// The tier values this Python process runs with, from its environment and `litellm` globals. +#[derive(Debug)] +pub(crate) struct RouterTierPolicies { + pub(crate) cooldowns: TierPolicy, + pub(crate) usage: TierPolicy, +} + +pub(crate) fn router_tier_policies(py: Python<'_>) -> PyResult { + Ok(project(&PythonSettings::RouterTiers.read(py)?)?) +} + +fn project(snapshot: &Snapshot<'_>) -> Result { + let tracked_reads = snapshot.read(&TRACKED_READS)?; + let default_in_memory_ttl = snapshot.read(&MEMORY_TTL)?; + Ok(RouterTierPolicies { + cooldowns: TierPolicy { + refresh_after: snapshot.read(&COOLDOWN_REFRESH)?, + tracked_reads, + default_in_memory_ttl, + }, + usage: TierPolicy { + refresh_after: snapshot.read(&USAGE_REFRESH)?, + tracked_reads, + default_in_memory_ttl, + }, + }) +} + +fn seconds(field: &Field<'_>) -> Result { + match field.value().extract::() { + Ok(seconds) if seconds.is_finite() && seconds >= 0.0 => { + Ok(Duration::from_secs_f64(seconds)) + } + _ => Err(field.invalid("a non-negative number of seconds")), + } +} + +fn optional_seconds(field: &Field<'_>) -> Result, ProjectionError> { + match field.value().is_none() { + true => Ok(None), + false => seconds(field).map(Some), + } +} + +fn count(field: &Field<'_>) -> Result { + field + .value() + .extract::() + .map_err(|_| field.invalid("a non-negative integer")) +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use pyo3::{prelude::*, types::PyDict}; + + use super::project; + use crate::python_settings::PythonSettings; + + fn projected(py: Python<'_>, fields: &str) -> Result { + let locals = PyDict::new(py); + let source = std::ffi::CString::new(format!( + "import types\nsettings = types.SimpleNamespace({fields})" + )) + .unwrap(); + py.run(&source, Some(&locals), Some(&locals)).unwrap(); + let snapshot = + PythonSettings::RouterTiers.snapshot(locals.get_item("settings").unwrap().unwrap()); + project(&snapshot).map_err(|error| PyErr::from(error).to_string()) + } + + #[test] + fn the_host_values_are_each_familys_refresh_and_tracked_reads() { + Python::initialize(); + Python::attach(|py| { + let policies = projected( + py, + "cooldown_redis_read_interval=2.5, usage_redis_read_interval=30, max_redis_batch_cache_size=42, default_in_memory_ttl=None", + ) + .unwrap(); + assert_eq!( + policies.cooldowns.refresh_after, + Duration::from_millis(2500) + ); + assert_eq!(policies.usage.refresh_after, Duration::from_secs(30)); + assert_eq!(policies.cooldowns.tracked_reads, 42); + assert_eq!(policies.usage.tracked_reads, 42); + assert_eq!(policies.cooldowns.default_in_memory_ttl, None); + }); + } + + #[test] + fn a_set_default_in_memory_ttl_reaches_both_familys_backfill() { + Python::initialize(); + Python::attach(|py| { + let policies = projected( + py, + "cooldown_redis_read_interval=1, usage_redis_read_interval=10, max_redis_batch_cache_size=1000, default_in_memory_ttl=7.5", + ) + .unwrap(); + assert_eq!( + policies.cooldowns.default_in_memory_ttl, + Some(Duration::from_millis(7500)) + ); + assert_eq!( + policies.usage.default_in_memory_ttl, + Some(Duration::from_millis(7500)) + ); + }); + } + + #[test] + fn a_value_python_would_not_run_with_is_a_configuration_error() { + Python::initialize(); + Python::attach(|py| { + let negative = projected( + py, + "cooldown_redis_read_interval=-1, usage_redis_read_interval=10, max_redis_batch_cache_size=1000, default_in_memory_ttl=None", + ) + .unwrap_err(); + let text = projected( + py, + "cooldown_redis_read_interval=1, usage_redis_read_interval=10, max_redis_batch_cache_size='many', default_in_memory_ttl=None", + ) + .unwrap_err(); + assert!( + negative.contains("cooldown_redis_read_interval"), + "{negative}" + ); + assert!(text.contains("max_redis_batch_cache_size"), "{text}"); + }); + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/rate_limits.rs b/litellm-rust/crates/python-bridge/src/redis/store/rate_limits.rs new file mode 100644 index 00000000000..bf6ddfbeb52 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/rate_limits.rs @@ -0,0 +1,591 @@ +use std::{sync::Arc, time::Duration}; + +use indexmap::IndexMap; +use litellm_redis_store::{ + Pending, + rate_limits::{ + AcquireReply, Applied, AtomicCounter, AtomicReply, Counter, Entry, Gauge, GuardedIncrement, + GuardedReply, Increment, Kind, Local, RateLimits, Scope, WindowCount, + }, +}; +use pyo3::{exceptions::PyValueError, prelude::*}; + +use super::{ + NativePending, PyNumber, Spelled, kind_error, redis_async, remote, + request::{NativeRequestRedis, NativeSend}, + ttl, +}; +use crate::redis::client::NativeRedisClient; + +/// The v3 rate limiter's keyspace. Memory reads and writes are synchronous like `InMemoryCache`; +/// every Redis call is a future. +#[pyclass(frozen, name = "NativeRateLimits")] +pub(crate) struct NativeRateLimits(Arc); + +/// An `int` or `float` counter, a `str` window start, or a `{slot_id: acquired_at}` registry. +#[derive(FromPyObject)] +enum PyLocal { + Number(PyNumber), + Text(String), + Slots(IndexMap), +} + +impl From for Local { + fn from(local: PyLocal) -> Self { + match local { + PyLocal::Number(number) => Self::Number(number.0), + PyLocal::Text(text) => Self::Text(text), + PyLocal::Slots(slots) => Self::Slots(slots.into_iter().collect()), + } + } +} + +/// What the memory tier holds, back as the Python value it was written as. +struct LocalValue(Local); + +impl<'py> IntoPyObject<'py> for LocalValue { + type Target = PyAny; + type Output = Bound<'py, PyAny>; + type Error = PyErr; + + fn into_pyobject(self, py: Python<'py>) -> PyResult { + Ok(match self.0 { + Local::Number(number) => PyNumber(number).into_pyobject(py)?, + Local::Text(text) => text.into_pyobject(py)?.into_any(), + Local::Slots(slots) => slots + .into_iter() + .collect::>() + .into_pyobject(py)? + .into_any(), + }) + } +} + +/// `litellm.redis.stores.rate_limits.RateLimitScope`. +#[derive(FromPyObject)] +struct PyScope { + #[pyo3(attribute)] + descriptor_key: String, + #[pyo3(attribute)] + descriptor_value: String, +} + +impl From for Scope { + fn from(scope: PyScope) -> Self { + Self { + descriptor: scope.descriptor_key, + value: scope.descriptor_value, + } + } +} + +/// `RateLimitCounter`. +#[derive(FromPyObject)] +struct PyCounter { + #[pyo3(attribute)] + scope: PyScope, + #[pyo3(attribute)] + kind: Spelled, +} + +impl From for Counter { + fn from(counter: PyCounter) -> Self { + Self { + scope: counter.scope.into(), + kind: counter.kind.0, + } + } +} + +/// `RateLimitWindow`, `RateLimitCounter` or `RateLimitGauge`, told apart by the key suffix each names. +#[derive(FromPyObject)] +struct PyEntry { + #[pyo3(attribute)] + suffix: String, + #[pyo3(attribute)] + scope: PyScope, +} + +impl TryFrom for Entry { + type Error = PyErr; + + fn try_from(entry: PyEntry) -> PyResult { + Self::new(&entry.suffix, entry.scope.into()) + .map_err(|_| PyValueError::new_err(format!("unknown key suffix {:?}", entry.suffix))) + } +} + +/// `CounterIncrement`. +#[derive(FromPyObject)] +struct PyIncrement { + #[pyo3(attribute)] + counter: PyCounter, + #[pyo3(attribute)] + amount: PyNumber, + #[pyo3(attribute)] + ttl: Option, +} + +impl From for Increment { + fn from(increment: PyIncrement) -> Self { + Self { + counter: increment.counter.into(), + amount: increment.amount.0, + ttl: increment.ttl.map(Duration::from_secs), + } + } +} + +/// `AtomicCounter`. +#[derive(FromPyObject)] +struct PyAtomicCounter { + #[pyo3(attribute)] + counter: PyCounter, + #[pyo3(attribute)] + limit: i64, + #[pyo3(attribute)] + increment: i64, + #[pyo3(attribute)] + ttl: i64, + #[pyo3(attribute)] + window_size: i64, +} + +impl From for AtomicCounter { + fn from(counter: PyAtomicCounter) -> Self { + Self { + counter: counter.counter.into(), + limit: counter.limit, + increment: counter.increment, + ttl_seconds: counter.ttl, + window_size_seconds: counter.window_size, + } + } +} + +fn all>(records: Vec

) -> Vec { + records.into_iter().map(T::from).collect() +} + +#[pyclass(frozen, eq, get_all, name = "WindowCount")] +#[derive(PartialEq)] +pub(crate) struct PyWindowCount { + window_start: String, + count: i64, +} + +#[pymethods] +impl PyWindowCount { + #[new] + fn new(window_start: String, count: i64) -> Self { + Self { + window_start, + count, + } + } +} + +/// Per counter, the count after the increment and the start of the window it landed in. +#[pyclass(frozen, eq, get_all, name = "AtomicApplied")] +#[derive(PartialEq)] +pub(crate) struct PyAtomicApplied { + counters: Vec<(i64, i64)>, +} + +#[pymethods] +impl PyAtomicApplied { + #[new] + fn new(counters: Vec<(i64, i64)>) -> Self { + Self { counters } + } +} + +#[pyclass(frozen, eq, get_all, name = "AtomicOverLimit")] +#[derive(PartialEq)] +pub(crate) struct PyAtomicOverLimit { + index: usize, + current: i64, + limit: i64, +} + +#[pymethods] +impl PyAtomicOverLimit { + #[new] + fn new(index: usize, current: i64, limit: i64) -> Self { + Self { + index, + current, + limit, + } + } +} + +#[derive(IntoPyObject)] +enum PyAtomicReply { + Applied(PyAtomicApplied), + OverLimit(PyAtomicOverLimit), +} + +#[pyclass(frozen, eq, get_all, name = "GuardedReply")] +#[derive(PartialEq)] +pub(crate) struct PyGuardedReply { + applied: bool, + count: i64, +} + +#[pymethods] +impl PyGuardedReply { + #[new] + fn new(applied: bool, count: i64) -> Self { + Self { applied, count } + } +} + +#[pyclass(frozen, eq, get_all, name = "GaugesAcquired")] +#[derive(PartialEq)] +pub(crate) struct PyGaugesAcquired { + in_flight: Vec, +} + +#[pymethods] +impl PyGaugesAcquired { + #[new] + fn new(in_flight: Vec) -> Self { + Self { in_flight } + } +} + +#[pyclass(frozen, eq, get_all, name = "GaugeOverLimit")] +#[derive(PartialEq)] +pub(crate) struct PyGaugeOverLimit { + index: usize, + in_flight: i64, + limit: i64, +} + +#[pymethods] +impl PyGaugeOverLimit { + #[new] + fn new(index: usize, in_flight: i64, limit: i64) -> Self { + Self { + index, + in_flight, + limit, + } + } +} + +#[derive(IntoPyObject)] +enum PyAcquireReply { + Acquired(PyGaugesAcquired), + OverLimit(PyGaugeOverLimit), +} + +fn window_counts(counts: Vec) -> Vec { + counts + .into_iter() + .map( + |WindowCount { + window_start, + count, + }| PyWindowCount { + window_start, + count, + }, + ) + .collect() +} + +fn atomic(reply: AtomicReply) -> PyAtomicReply { + match reply { + AtomicReply::Applied(applied) => PyAtomicReply::Applied(PyAtomicApplied { + counters: applied + .into_iter() + .map( + |Applied { + count, + window_start, + }| (count, window_start), + ) + .collect(), + }), + AtomicReply::OverLimit { + index, + current, + limit, + } => PyAtomicReply::OverLimit(PyAtomicOverLimit { + index, + current, + limit, + }), + } +} + +fn guarded(GuardedReply { applied, count }: GuardedReply) -> PyGuardedReply { + PyGuardedReply { applied, count } +} + +fn acquire(reply: AcquireReply) -> PyAcquireReply { + match reply { + AcquireReply::Acquired(in_flight) => { + PyAcquireReply::Acquired(PyGaugesAcquired { in_flight }) + } + AcquireReply::OverLimit { + index, + in_flight, + limit, + } => PyAcquireReply::OverLimit(PyGaugeOverLimit { + index, + in_flight, + limit, + }), + } +} + +/// One script call declared on the batch the caller's `NativeSend` picked; `None` without Redis. +fn script( + call_type: &'static str, + declared: Option>, + reply: fn(T) -> R, +) -> Option +where + T: Send + 'static, + R: for<'a> IntoPyObject<'a> + Send + 'static, +{ + Some(super::pending(call_type, kind_error, declared?, reply)) +} + +#[pymethods] +impl NativeRateLimits { + #[new] + #[pyo3(signature = (client, backfill_ttl_seconds = None))] + fn new( + client: Option>, + backfill_ttl_seconds: Option, + ) -> PyResult { + Ok(Self(Arc::new(RateLimits::new( + client.map(|client| remote(&client)), + ttl(backfill_ttl_seconds)?, + )))) + } + + #[getter] + fn has_redis(&self) -> bool { + self.0.has_redis() + } + + fn slot_groups(&self, scopes: Vec, force: bool) -> Vec> { + self.0.slot_groups(&all(scopes), force) + } + + fn get_local(&self, entry: PyEntry) -> PyResult> { + Ok(self + .0 + .get_local(&entry.try_into()?) + .map_err(kind_error)? + .map(LocalValue)) + } + + #[pyo3(signature = (entry, value, ttl_seconds))] + fn set_local(&self, entry: PyEntry, value: PyLocal, ttl_seconds: Option) -> PyResult<()> { + self.0 + .set_local(&entry.try_into()?, value.into(), ttl(ttl_seconds)?) + .map_err(kind_error) + } + + /// `request` answers a key its pre-call reads already found absent without another `GET`. + #[pyo3(signature = (entry, *, backfill_ttl_seconds = None, request = None))] + fn get<'py>( + &self, + py: Python<'py>, + entry: PyEntry, + backfill_ttl_seconds: Option, + request: Option<&NativeRequestRedis>, + ) -> PyResult> { + let store = self.0.clone(); + let entry = Entry::try_from(entry)?; + let backfill_ttl = ttl(backfill_ttl_seconds)?; + let request = request.map(|request| request.0.clone()); + redis_async(py, "async_get_cache", kind_error, async move { + store + .get(request.as_ref(), &entry, backfill_ttl) + .await + .map(|local| local.map(LocalValue)) + }) + } + + fn sliding_window( + &self, + counters: Vec, + now: i64, + window_size: i64, + send: &NativeSend, + ) -> Option { + script( + "batch_rate_limiter_script", + self.0 + .sliding_window(send.send(), &all(counters), now, window_size), + window_counts, + ) + } + + fn read_counters(&self, counters: Vec, send: &NativeSend) -> Option { + script( + "batch_counter_read_script", + self.0.read_counters(send.send(), &all(counters)), + std::convert::identity, + ) + } + + fn check_and_increment( + &self, + counters: Vec, + send: &NativeSend, + ) -> Option { + script( + "check_and_increment_by_n_script", + self.0.check_and_increment(send.send(), &all(counters)), + atomic, + ) + } + + fn guarded_increment( + &self, + counter: PyCounter, + expected_window_start: String, + amount: PyNumber, + ttl_seconds: Option, + send: &NativeSend, + ) -> Option { + let increment = GuardedIncrement { + counter: counter.into(), + expected_window_start, + amount: amount.0, + ttl: ttl_seconds.map(Duration::from_secs), + }; + script( + "window_guarded_token_increment_script", + self.0.guarded_increment(send.send(), &increment), + guarded, + ) + } + + fn parallel_acquire( + &self, + gauges: Vec<(PyScope, i64)>, + slot_ttl_seconds: u64, + slot_id: &str, + send: &NativeSend, + ) -> Option { + let gauges: Vec = gauges + .into_iter() + .map(|(scope, limit)| Gauge { + scope: scope.into(), + limit, + }) + .collect(); + script( + "parallel_acquire_script", + self.0.parallel_acquire( + send.send(), + &gauges, + Duration::from_secs(slot_ttl_seconds), + slot_id, + ), + acquire, + ) + } + + fn parallel_release( + &self, + gauges: Vec, + slot_id: &str, + send: &NativeSend, + ) -> Option { + script( + "parallel_release_script", + self.0.parallel_release(send.send(), &all(gauges), slot_id), + std::convert::identity, + ) + } + + fn parallel_count( + &self, + gauges: Vec, + slot_ttl_seconds: u64, + send: &NativeSend, + ) -> Option { + script( + "parallel_count_script", + self.0.parallel_count( + send.send(), + &all(gauges), + Duration::from_secs(slot_ttl_seconds), + ), + std::convert::identity, + ) + } + + fn token_increment( + &self, + increments: Vec, + send: &NativeSend, + ) -> Option { + script( + "token_increment_script", + self.0.token_increment(send.send(), &all(increments)), + std::convert::identity, + ) + } + + /// `False` without Redis. + fn token_increment_post_call( + &self, + request: &NativeRequestRedis, + increments: Vec, + ) -> bool { + self.0 + .token_increment_post_call(&request.0, all(increments)) + } + + /// `False` without Redis. + fn parallel_release_post_call( + &self, + request: &NativeRequestRedis, + gauges: Vec, + slot_id: &str, + ) -> bool { + self.0 + .parallel_release_post_call(&request.0, &all(gauges), slot_id) + } + + fn increment<'py>( + &self, + py: Python<'py>, + increment: PyIncrement, + ) -> PyResult> { + let store = self.0.clone(); + let increment = Increment::from(increment); + redis_async(py, "async_increment", kind_error, async move { + Ok(store.increment(&increment).await.map(PyNumber)) + }) + } + + fn increment_many<'py>( + &self, + py: Python<'py>, + increments: Vec, + ) -> PyResult> { + let store = self.0.clone(); + let increments: Vec = all(increments); + redis_async(py, "async_increment_pipeline", kind_error, async move { + store.increment_many(&increments).await; + Ok(()) + }) + } + + fn increment_many_post_call(&self, request: &NativeRequestRedis, increments: Vec) { + self.0 + .increment_many_post_call(&request.0, &all(increments)); + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/request.rs b/litellm-rust/crates/python-bridge/src/redis/store/request.rs new file mode 100644 index 00000000000..3436b26a16c --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/request.rs @@ -0,0 +1,60 @@ +use litellm_redis_store::{RequestRedis, Send}; +use pyo3::prelude::*; + +use super::{call, duration}; +use crate::execution::run_async_value; + +/// One request's native Redis batches. Python builds one per request scope and passes it to each +/// request-aware store call. +#[pyclass(frozen, name = "NativeRequestRedis")] +pub(crate) struct NativeRequestRedis(pub(crate) RequestRedis); + +#[pymethods] +impl NativeRequestRedis { + #[new] + fn new(py: Python<'_>, post_call_deadline_seconds: f64) -> PyResult { + let runtime = litellm_host_python::runtime_handle()?; + let deadline = duration(post_call_deadline_seconds)?; + let deadline_tag = call(py, "post_call_redis_batch")?.tag(); + Ok(Self(crate::logger::capture(py).scope(|| { + RequestRedis::with_deadline_tag(runtime, deadline, Some(deadline_tag)) + }))) + } + + fn flush_pre<'py>(&self, py: Python<'py>) -> PyResult> { + let request = self.0.clone(); + let scope = call(py, "request_redis_batch")?; + run_async_value(py, async move { + scope.scope(request.flush_pre()).await; + Ok(()) + }) + } + + fn flush_post<'py>(&self, py: Python<'py>) -> PyResult> { + let request = self.0.clone(); + let scope = call(py, "post_call_redis_batch")?; + run_async_value(py, async move { + scope.scope(request.flush_post()).await; + Ok(()) + }) + } +} + +/// The Python face of `Send`: which batch a store op rides. Each variant carries exactly what it +/// needs, so there is no phase string and no phase without a request. +#[pyclass(frozen, name = "NativeSend")] +pub(crate) enum NativeSend { + Now(), + PreCall(Py), + PostCall(Py), +} + +impl NativeSend { + pub(crate) fn send(&self) -> Send<'_> { + match self { + Self::Now() => Send::Now, + Self::PreCall(request) => Send::PreCall(&request.get().0), + Self::PostCall(request) => Send::PostCall(&request.get().0), + } + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/router_cooldowns.rs b/litellm-rust/crates/python-bridge/src/redis/store/router_cooldowns.rs new file mode 100644 index 00000000000..f9f2364da19 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/router_cooldowns.rs @@ -0,0 +1,183 @@ +use std::sync::Arc; + +use litellm_redis_store::{ + Send, + router_cooldowns::{Cooldown, RouterCooldowns}, +}; +use pyo3::{ + prelude::*, + types::{PyDict, PyTuple}, +}; + +use super::{ + NativePending, PyData, PyNumber, duration, kind_error, pending, policy::router_tier_policies, + redis_async, redis_sync, remote, request::NativeSend, +}; +use crate::redis::client::NativeRedisClient; + +/// One router's cooldown store. The router's sync completion path calls `put`, `active_sync`, +/// `minimum` and `increment_allowed_fails`, so those block with the GIL released like redis-py; the +/// `async_` twins return awaitables for async callers. +#[pyclass(frozen, name = "NativeRouterCooldowns")] +pub(crate) struct NativeRouterCooldowns(Arc); + +/// `[(model_id, cooldown), ...]` the way `RedisCache` hands the router its batch read. +struct PyActive(Vec<(String, Cooldown)>); + +impl<'py> IntoPyObject<'py> for PyActive { + type Target = PyAny; + type Output = Bound<'py, PyAny>; + type Error = PyErr; + + fn into_pyobject(self, py: Python<'py>) -> PyResult { + active_to_py(py, self.0)? + .into_pyobject(py) + .map(Bound::into_any) + } +} + +fn active_to_py( + py: Python<'_>, + active: Vec<(String, Cooldown)>, +) -> PyResult>> { + active + .into_iter() + .map(|(model_id, cooldown)| { + PyTuple::new( + py, + [ + model_id.into_pyobject(py)?.into_any(), + PyData(cooldown).into_pyobject(py)?, + ], + ) + }) + .collect() +} + +#[pymethods] +impl NativeRouterCooldowns { + #[new] + fn new(py: Python<'_>, client: Option>) -> PyResult { + let remote = client.map(|client| remote(&client)); + let policy = router_tier_policies(py)?.cooldowns; + Ok(Self(Arc::new(RouterCooldowns::new(remote, policy)))) + } + + /// Adds the Redis tier and keeps memory; `false` when one was already attached. + fn attach(&self, client: PyRef<'_, NativeRedisClient>) -> bool { + self.0.attach(remote(&client)) + } + + #[getter] + fn has_redis(&self) -> bool { + self.0.has_redis() + } + + fn put( + &self, + py: Python<'_>, + model_id: String, + cooldown: PyData, + ttl_seconds: f64, + ) -> PyResult<()> { + let store = self.0.clone(); + let ttl = duration(ttl_seconds)?; + redis_sync(py, "set_cache", kind_error, async move { + store.put_str(&model_id, &cooldown.0, ttl).await + }) + } + + fn increment_allowed_fails( + &self, + py: Python<'_>, + model_id: String, + suffix: Option, + ttl_seconds: f64, + ) -> PyResult { + let store = self.0.clone(); + let ttl = duration(ttl_seconds)?; + redis_sync(py, "increment_cache", kind_error, async move { + store + .increment_allowed_fails_int(&model_id, suffix.as_deref(), ttl) + .await + }) + } + + fn async_put<'py>( + &self, + py: Python<'py>, + model_id: String, + cooldown: PyData, + ttl_seconds: f64, + ) -> PyResult> { + let store = self.0.clone(); + let ttl = duration(ttl_seconds)?; + redis_async(py, "async_set_cache", kind_error, async move { + store.put(&model_id, &cooldown.0, ttl).await + }) + } + + fn async_increment_allowed_fails<'py>( + &self, + py: Python<'py>, + model_id: String, + suffix: Option, + ttl_seconds: f64, + ) -> PyResult> { + let store = self.0.clone(); + let ttl = duration(ttl_seconds)?; + redis_async(py, "async_increment", kind_error, async move { + store + .increment_allowed_fails(&model_id, suffix.as_deref(), ttl) + .await + }) + } + + /// The deployments still cooling down, read on the batch `send` picks. + fn active(&self, model_ids: Vec, send: &NativeSend) -> PyResult { + let active = self.0.active(send.send(), &model_ids).map_err(kind_error)?; + Ok(pending( + "async_batch_get_cache", + kind_error, + active, + PyActive, + )) + } + + /// `active` on its own with the GIL released, for the router's sync completion path. + fn active_sync<'py>( + &self, + py: Python<'py>, + model_ids: Vec, + ) -> PyResult>> { + let active = self.0.active(Send::Now, &model_ids).map_err(kind_error)?; + let active = redis_sync(py, "batch_get_cache", kind_error, active)?; + active_to_py(py, active) + } + + fn minimum(&self, py: Python<'_>, model_ids: Vec) -> PyResult> { + let store = self.0.clone(); + redis_sync(py, "batch_get_cache", kind_error, async move { + store + .minimum(&model_ids) + .await + .map(|minimum| minimum.map(PyNumber)) + }) + } + + fn delete_local(&self, model_id: &str) -> PyResult<()> { + self.0.delete_local(model_id).map_err(kind_error) + } + + fn flush_local_counters(&self) -> PyResult<()> { + self.0.flush_local_counters().map_err(kind_error) + } + + fn snapshot<'py>(&self, py: Python<'py>) -> PyResult> { + let snapshot = PyDict::new(py); + for (key, cooldown) in self.0.snapshot().map_err(kind_error)? { + snapshot.set_item(key, PyData(cooldown))?; + } + Ok(snapshot) + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/router_usage.rs b/litellm-rust/crates/python-bridge/src/redis/store/router_usage.rs new file mode 100644 index 00000000000..bad4085cfe6 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/router_usage.rs @@ -0,0 +1,184 @@ +use std::{sync::Arc, time::Duration}; + +use litellm_redis_store::router_usage::{Counter, Kind, RouterUsage}; +use pyo3::{prelude::*, types::PyDict}; + +use super::{ + PyNumber, Spelled, kind_error, policy::router_tier_policies, redis_async, redis_sync, remote, +}; +use crate::redis::client::NativeRedisClient; + +/// `litellm.redis.stores.router_usage.UsageCounter`. +#[derive(FromPyObject)] +struct PyCounter { + #[pyo3(attribute)] + deployment_id: String, + #[pyo3(attribute)] + model: Option, + #[pyo3(attribute)] + minute: String, + #[pyo3(attribute)] + kind: Spelled, +} + +impl From for Counter { + fn from(counter: PyCounter) -> Self { + Self { + deployment_id: counter.deployment_id, + model: counter.model, + minute: counter.minute, + kind: counter.kind.0, + } + } +} + +fn counters(counters: Vec) -> Vec { + counters.into_iter().map(Counter::from).collect() +} + +/// One router's global TPM/RPM counters. The `_sync` methods block with the GIL released, for the +/// Rust router's sync calls, which have no event loop to await on. +#[pyclass(frozen, name = "NativeRouterUsage")] +pub(crate) struct NativeRouterUsage(Arc); + +#[pymethods] +impl NativeRouterUsage { + #[new] + fn new(py: Python<'_>, client: Option>) -> PyResult { + let remote = client.map(|client| remote(&client)); + let policy = router_tier_policies(py)?.usage; + Ok(Self(Arc::new(RouterUsage::new(remote, policy)))) + } + + /// Adds the Redis tier and keeps memory; `false` when one was already attached. + fn attach(&self, client: PyRef<'_, NativeRedisClient>) -> bool { + self.0.attach(remote(&client)) + } + + #[getter] + fn has_redis(&self) -> bool { + self.0.has_redis() + } + + fn increment<'py>( + &self, + py: Python<'py>, + counter: PyCounter, + value: PyNumber, + ttl_seconds: u64, + ) -> PyResult> { + let store = self.0.clone(); + let counter = Counter::from(counter); + redis_async(py, "async_increment", kind_error, async move { + Ok(store + .increment(&counter, value.0, Duration::from_secs(ttl_seconds)) + .await + .map(PyNumber)) + }) + } + + /// `_increment_deployment`'s `async_increment_pipeline`, always on its own round trip. + fn increment_deployment<'py>( + &self, + py: Python<'py>, + deployment_id: String, + model: Option, + minute: String, + tokens: PyNumber, + requests: i64, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_increment_pipeline", kind_error, async move { + store + .increment_deployment( + &deployment_id, + model.as_deref(), + &minute, + tokens.0, + requests, + ) + .await + }) + } + + fn get_many<'py>( + &self, + py: Python<'py>, + counters: Vec, + ) -> PyResult> { + let store = self.0.clone(); + let counters = self::counters(counters); + redis_async(py, "async_batch_get_cache", kind_error, async move { + let read = store.get_many(&counters).await?; + Ok(read + .into_iter() + .map(|count| count.map(PyNumber)) + .collect::>()) + }) + } + + fn increment_sync( + &self, + py: Python<'_>, + counter: PyCounter, + value: PyNumber, + ttl_seconds: u64, + ) -> PyResult> { + let store = self.0.clone(); + let counter = Counter::from(counter); + redis_sync(py, "increment_cache", kind_error, async move { + Ok(store + .increment(&counter, value.0, Duration::from_secs(ttl_seconds)) + .await + .map(PyNumber)) + }) + } + + fn increment_deployment_sync( + &self, + py: Python<'_>, + deployment_id: String, + model: Option, + minute: String, + tokens: PyNumber, + requests: i64, + ) -> PyResult<()> { + let store = self.0.clone(); + redis_sync(py, "increment_pipeline", kind_error, async move { + store + .increment_deployment( + &deployment_id, + model.as_deref(), + &minute, + tokens.0, + requests, + ) + .await + }) + } + + fn get_many_sync( + &self, + py: Python<'_>, + counters: Vec, + ) -> PyResult>> { + let store = self.0.clone(); + let counters = self::counters(counters); + redis_sync(py, "batch_get_cache", kind_error, async move { + let read = store.get_many(&counters).await?; + Ok(read.into_iter().map(|count| count.map(PyNumber)).collect()) + }) + } + + fn flush_local_counters(&self) -> PyResult<()> { + self.0.flush_local_counters().map_err(kind_error) + } + + fn snapshot<'py>(&self, py: Python<'py>) -> PyResult> { + let snapshot = PyDict::new(py); + for (key, count) in self.0.snapshot().map_err(kind_error)? { + snapshot.set_item(key, PyNumber(count))?; + } + Ok(snapshot) + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/sensitive_route_pins.rs b/litellm-rust/crates/python-bridge/src/redis/store/sensitive_route_pins.rs new file mode 100644 index 00000000000..8b1b82945fb --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/sensitive_route_pins.rs @@ -0,0 +1,49 @@ +use std::sync::Arc; + +use litellm_redis_store::sensitive_route_pins::SensitiveRoutePins; +use pyo3::prelude::*; + +use super::{duration, kind_error, redis_async, remote}; +use crate::redis::client::NativeRedisClient; + +#[pyclass(frozen, name = "NativeSensitiveRoutePins")] +pub(crate) struct NativeSensitiveRoutePins(Arc); + +#[pymethods] +impl NativeSensitiveRoutePins { + #[new] + fn new(client: Option>) -> Self { + Self(Arc::new(SensitiveRoutePins::new( + client.map(|client| remote(&client)), + ))) + } + + fn get<'py>( + &self, + py: Python<'py>, + tenant: String, + session_id: String, + default_ttl_seconds: f64, + ) -> PyResult> { + let store = self.0.clone(); + let default_ttl = duration(default_ttl_seconds)?; + redis_async(py, "async_get_cache", kind_error, async move { + store.get(&tenant, &session_id, default_ttl).await + }) + } + + fn set<'py>( + &self, + py: Python<'py>, + tenant: String, + session_id: String, + model: String, + ttl_seconds: f64, + ) -> PyResult> { + let store = self.0.clone(); + let ttl = duration(ttl_seconds)?; + redis_async(py, "async_set_cache", kind_error, async move { + store.set(&tenant, &session_id, &model, ttl).await + }) + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/spend_counters.rs b/litellm-rust/crates/python-bridge/src/redis/store/spend_counters.rs new file mode 100644 index 00000000000..58b77222f5e --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/spend_counters.rs @@ -0,0 +1,256 @@ +use std::sync::Arc; + +use litellm_redis_store::{ + Number, + spend_counters::{LostSeed, Reading, SpendCounter, SpendCounters}, +}; +use pyo3::{prelude::*, types::PyDict}; + +use super::{ + NativePending, PyData, PyNumber, kind_error, pending, redis_async, remote, + request::{NativeRequestRedis, NativeSend}, + ttl, unwrap_all, +}; +use crate::redis::client::NativeRedisClient; + +/// A counter's value and whether Redis answered it, a clean miss included. +#[pyclass(frozen, eq, get_all, name = "SpendRead")] +#[derive(PartialEq)] +pub(crate) struct PySpendRead { + value: Option, + from_redis: bool, +} + +#[pymethods] +impl PySpendRead { + #[new] + fn new(value: Option, from_redis: bool) -> Self { + Self { value, from_redis } + } +} + +fn increments(increments: Vec<(PyData, PyNumber)>) -> Vec<(SpendCounter, Number)> { + increments + .into_iter() + .map(|(counter, amount)| (counter.0, amount.0)) + .collect() +} + +fn numbers(values: Option>>) -> Option>> { + values.map(|values| { + values + .into_iter() + .map(|value| value.map(PyNumber)) + .collect() + }) +} + +/// The proxy's spend counters, Redis first with this worker's memory copy behind it. +#[pyclass(frozen, name = "NativeSpendCounters")] +pub(crate) struct NativeSpendCounters(Arc); + +#[pymethods] +impl NativeSpendCounters { + #[new] + fn new(client: Option>) -> Self { + Self(Arc::new(SpendCounters::new( + client.map(|client| remote(&client)), + ))) + } + + /// Adds the Redis tier and keeps memory; `false` when one was already attached. + fn attach(&self, client: PyRef<'_, NativeRedisClient>) -> bool { + self.0.attach(remote(&client)) + } + + #[getter] + fn has_redis(&self) -> bool { + self.0.has_redis() + } + + #[getter] + fn redis_ttl_seconds(&self) -> Option { + self.0.redis_ttl().map(|ttl| ttl.as_secs_f64()) + } + + /// `("redis" | "memory", value)`: where the answer came from, a Redis miss included. + fn read<'py>( + &self, + py: Python<'py>, + counter: PyData, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_get_cache", kind_error, async move { + let reading = store.read(&counter.0).await?; + Ok(PySpendRead { + value: reading.value().map(PyNumber), + from_redis: matches!(reading, Reading::Redis(_)), + }) + }) + } + + /// One `MGET` on the batch `send` picks; `None` without Redis. It resolves to `None` when the + /// read failed, so each caller reads on its own. + fn read_many( + &self, + counters: Vec>, + send: &NativeSend, + ) -> Option { + let read = self.0.read_many(send.send(), &unwrap_all(counters))?; + Some(pending("async_batch_get_cache", kind_error, read, numbers)) + } + + fn increment<'py>( + &self, + py: Python<'py>, + counter: PyData, + amount: PyNumber, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_increment", kind_error, async move { + store.increment(&counter.0, amount.0).await.map(PyNumber) + }) + } + + fn increment_many<'py>( + &self, + py: Python<'py>, + increments: Vec<(PyData, PyNumber)>, + ) -> PyResult> { + let store = self.0.clone(); + let increments = self::increments(increments); + redis_async(py, "async_increment_pipeline", kind_error, async move { + let totals = store.increment_many(&increments).await?; + Ok(totals.into_iter().map(PyNumber).collect::>()) + }) + } + + /// Declares the increments on the request's post-call batch; `False` when there is no Redis. + fn increment_many_post_call( + &self, + request: &NativeRequestRedis, + increments: Vec<(PyData, PyNumber)>, + ) -> bool { + self.0 + .increment_many_post_call(&request.0, &self::increments(increments)) + } + + fn set_max<'py>( + &self, + py: Python<'py>, + counter: PyData, + spend: PyNumber, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_set_max", kind_error, async move { + store + .set_max(&counter.0, spend.0) + .await + .map(|stored| stored.map(PyNumber)) + }) + } + + #[pyo3(signature = (counter, spend, *, increment_if_lost))] + fn seed<'py>( + &self, + py: Python<'py>, + counter: PyData, + spend: PyNumber, + increment_if_lost: bool, + ) -> PyResult> { + let store = self.0.clone(); + let lost = if increment_if_lost { + LostSeed::Increment + } else { + LostSeed::Answer + }; + redis_async(py, "async_set_cache", kind_error, async move { + store.seed(&counter.0, spend.0, lost).await.map(PyNumber) + }) + } + + #[pyo3(signature = (counter, spend, ttl_seconds = None))] + fn set<'py>( + &self, + py: Python<'py>, + counter: PyData, + spend: PyNumber, + ttl_seconds: Option, + ) -> PyResult> { + let store = self.0.clone(); + let ttl = ttl(ttl_seconds)?; + redis_async(py, "async_set_cache", kind_error, async move { + store.set(&counter.0, spend.0, ttl).await + }) + } + + fn delete<'py>( + &self, + py: Python<'py>, + counters: Vec>, + ) -> PyResult> { + let store = self.0.clone(); + let counters = unwrap_all(counters); + redis_async(py, "async_delete_cache", kind_error, async move { + store.delete(&counters).await + }) + } + + fn refresh_ttl<'py>( + &self, + py: Python<'py>, + counter: PyData, + ) -> PyResult> { + let store = self.0.clone(); + redis_async(py, "async_refresh_ttl", kind_error, async move { + store.refresh_ttl(&counter.0).await + }) + } + + fn get_local(&self, counter: PyData) -> PyResult> { + Ok(self + .0 + .get_local(&counter.0) + .map_err(kind_error)? + .map(PyNumber)) + } + + #[pyo3(signature = (counter, spend, ttl_seconds = None))] + fn set_local( + &self, + counter: PyData, + spend: PyNumber, + ttl_seconds: Option, + ) -> PyResult<()> { + self.0 + .set_local(&counter.0, spend.0, ttl(ttl_seconds)?) + .map_err(kind_error) + } + + fn delete_local(&self, counter: PyData) -> PyResult<()> { + self.0.delete_local(&counter.0).map_err(kind_error) + } + + fn increment_local( + &self, + counter: PyData, + amount: PyNumber, + ) -> PyResult { + self.0 + .increment_local(&counter.0, amount.0) + .map(PyNumber) + .map_err(kind_error) + } + + fn flush_local(&self) -> PyResult<()> { + self.0.flush_local().map_err(kind_error) + } + + fn snapshot<'py>(&self, py: Python<'py>) -> PyResult> { + let snapshot = PyDict::new(py); + for (key, spend) in self.0.snapshot().map_err(kind_error)? { + snapshot.set_item(key, PyNumber(spend))?; + } + Ok(snapshot) + } +} diff --git a/litellm-rust/crates/python-bridge/src/redis/store/value.rs b/litellm-rust/crates/python-bridge/src/redis/store/value.rs new file mode 100644 index 00000000000..c86c9e9bb20 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/redis/store/value.rs @@ -0,0 +1,125 @@ +use litellm_python_compat::{BigInt, MAX_DEPTH, Value}; +use pyo3::{ + exceptions::{PyTypeError, PyValueError}, + prelude::*, + types::{ + PyBool, PyBytes, PyComplex, PyDict, PyFloat, PyFrozenSet, PyInt, PyList, PySet, PyString, + PyTuple, + }, +}; + +/// A plain Python value (what `json.loads` or `ast.literal_eval` returns) across the bridge. +/// Anything else, a live object included, is refused with `TypeError`. +pub(crate) struct PyValue(pub(crate) Value); + +impl<'a, 'py> FromPyObject<'a, 'py> for PyValue { + type Error = PyErr; + + fn extract(value: Borrowed<'a, 'py, PyAny>) -> PyResult { + to_value(&value, 0).map(Self) + } +} + +impl<'py> IntoPyObject<'py> for PyValue { + type Target = PyAny; + type Output = Bound<'py, PyAny>; + type Error = PyErr; + + fn into_pyobject(self, py: Python<'py>) -> PyResult { + to_python(py, self.0) + } +} + +fn to_value(value: &Bound<'_, PyAny>, depth: usize) -> PyResult { + if depth > MAX_DEPTH { + return Err(PyValueError::new_err("value is nested too deeply")); + } + let nested = |items: Bound<'_, PyAny>| -> PyResult> { + items + .try_iter()? + .map(|item| to_value(&item?, depth + 1)) + .collect() + }; + if value.is_none() { + return Ok(Value::None); + } + if let Ok(flag) = value.cast::() { + return Ok(Value::Bool(flag.is_true())); + } + if value.is_instance_of::() { + return Ok(Value::Int(match value.extract::() { + Ok(small) => small.into(), + Err(_) => value + .str()? + .to_str()? + .parse::() + .map_err(|error| PyValueError::new_err(error.to_string()))?, + })); + } + if let Ok(number) = value.cast::() { + return Ok(Value::Float(number.value())); + } + if let Ok(text) = value.cast::() { + return Ok(Value::Str(text.to_str()?.to_owned())); + } + if let Ok(bytes) = value.cast::() { + return Ok(Value::Bytes(bytes.as_bytes().to_vec())); + } + if let Ok(complex) = value.cast::() { + return Ok(Value::Complex { + re: complex.real(), + im: complex.imag(), + }); + } + if let Ok(entries) = value.cast::() { + return entries + .iter() + .map(|(key, item)| Ok((to_value(&key, depth + 1)?, to_value(&item, depth + 1)?))) + .collect::>() + .map(Value::Dict); + } + if value.is_instance_of::() { + return nested(value.clone()).map(Value::List); + } + if value.is_instance_of::() { + return nested(value.clone()).map(Value::Tuple); + } + if value.is_instance_of::() || value.is_instance_of::() { + return nested(value.clone()).map(Value::Set); + } + Err(PyTypeError::new_err(format!( + "{} is not a plain data value", + value.get_type().name()? + ))) +} + +fn to_python(py: Python<'_>, value: Value) -> PyResult> { + let all = |items: Vec| -> PyResult>> { + items.into_iter().map(|item| to_python(py, item)).collect() + }; + Ok(match value { + Value::None => py.None().into_bound(py), + Value::Bool(flag) => PyBool::new(py, flag).to_owned().into_any(), + Value::Int(number) => match i64::try_from(&number) { + Ok(small) => small.into_pyobject(py)?.into_any(), + Err(_) => py + .get_type::() + .call1((number.to_string(),))? + .into_any(), + }, + Value::Float(number) => number.into_pyobject(py)?.into_any(), + Value::Complex { re, im } => PyComplex::from_doubles(py, re, im).into_any(), + Value::Str(text) => text.into_pyobject(py)?.into_any(), + Value::Bytes(bytes) => PyBytes::new(py, &bytes).into_any(), + Value::Tuple(items) => PyTuple::new(py, all(items)?)?.into_any(), + Value::List(items) => PyList::new(py, all(items)?)?.into_any(), + Value::Set(items) => PySet::new(py, all(items)?)?.into_any(), + Value::Dict(entries) => { + let dict = PyDict::new(py); + for (key, item) in entries { + dict.set_item(to_python(py, key)?, to_python(py, item)?)?; + } + dict.into_any() + } + }) +} diff --git a/litellm-rust/crates/python-bridge/src/router/config.rs b/litellm-rust/crates/python-bridge/src/router/config.rs new file mode 100644 index 00000000000..de383a0ffa1 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/router/config.rs @@ -0,0 +1,412 @@ +//! The documents `RouterCoreHandle` reads: the router's static and dynamic config, a call's +//! request and an `apply` change set. A config key router-core does not read declines the +//! handle, so Python keeps the router on its own core. + +use std::collections::BTreeMap; + +use litellm_router_core::select::PassThrough; +use litellm_router_core::{ + ChangeSet, ConfigError, DeploymentSpec, OutcomeCallbacks, RouterConfig, RouterDynamicConfig, + RouterRequest, + config::{Fallbacks, RetryPolicy}, + host::UncountedRetries, + mock::MockFailure, + registry::IndexOverrides, +}; +use serde::{Deserialize, de::DeserializeOwned}; +use serde_json::{Map, Value}; + +/// The `RouterConfig` keys router-core reads. +const STATIC_KEYS: &[&str] = &[ + "model_list", + "routing_strategy", + "routing_groups", + "routing_group_members", + "plugins", + "cooldown_time", + "disable_cooldowns", + "allowed_fails_policy", + "retry_policy", + "model_group_retry_policy", + "retry_after", + "timeout", + "request_timeout", + "drop_params", + "enable_pre_call_checks", + "routing_plugins", + "callback_filters", + "enable_health_check_routing", + "health_check_staleness_threshold", + "background_health_check_model_groups", + "enable_weighted_failover", + "fallback_on_unsupported_params", + "provider_default_deployment_ids", +]; + +/// The `RouterDynamicConfig` keys router-core reads. +const DYNAMIC_KEYS: &[&str] = &[ + "version", + "model_alias_map", + "model_group_alias", + "fallbacks", + "default_fallbacks", + "context_window_fallbacks", + "content_policy_fallbacks", + "num_retries", + "max_fallbacks", + "allowed_fails", + "allowed_fails_is_default", + "fallback_keys", + "default_deployment", + "default_deployment_raises", +]; + +/// Router arguments that do not change routing, accepted in either config and dropped. +const IGNORED_KEYS: &[&str] = &[ + "set_verbose", + "debug_level", + "alerting_config", + "client_ttl", +]; + +/// Fallback lists Python treats as unset when empty (`context_window_fallbacks or None`). +const EMPTY_MEANS_UNSET: &[&str] = &["context_window_fallbacks", "content_policy_fallbacks"]; + +/// Which document a failure is about. +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::Display)] +pub(crate) enum Document { + #[strum(serialize = "router config")] + Static, + #[strum(serialize = "dynamic router config")] + Dynamic, + #[strum(serialize = "router request")] + Request, + #[strum(serialize = "change set")] + Changes, +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum Error { + #[error("the {document} must be a JSON object")] + NotObject { document: Document }, + #[error("router-core does not read `{key}` in the {document}")] + UnknownKey { document: Document, key: String }, + #[error("the {document} does not have the shape router-core reads")] + Shape { + document: Document, + #[source] + source: serde_json::Error, + }, + #[error("the request names no model")] + MissingModel, + #[error("router-core rejected the router config")] + Core(#[from] ConfigError), +} + +impl Error { + /// Whether router-core cannot serve this config, so Python keeps its own core. + pub(crate) fn declined(&self) -> bool { + match self { + Self::UnknownKey { .. } | Self::Core(_) => true, + Self::Shape { document, .. } => { + matches!(document, Document::Static | Document::Dynamic) + } + Self::NotObject { .. } | Self::MissingModel => false, + } + } +} + +fn object(document: Document, value: Value) -> Result, Error> { + match value { + Value::Object(object) => Ok(object), + _ => Err(Error::NotObject { document }), + } +} + +fn shaped(document: Document, object: Map) -> Result { + serde_json::from_value(Value::Object(object)) + .map_err(|source| Error::Shape { document, source }) +} + +/// Drop unset and ignored keys, then reject any key outside `known`. +fn routing_keys( + document: Document, + value: Value, + known: &[&str], +) -> Result, Error> { + let mut object = object(document, value)?; + object.retain(|key, value| { + let empty_unset = EMPTY_MEANS_UNSET.contains(&key.as_str()) + && value.as_array().is_some_and(Vec::is_empty); + !(value.is_null() || empty_unset || IGNORED_KEYS.contains(&key.as_str())) + }); + if let Some(key) = object.keys().find(|key| !known.contains(&key.as_str())) { + return Err(Error::UnknownKey { + document, + key: key.clone(), + }); + } + Ok(object) +} + +pub(crate) fn parse_router_config(value: Value) -> Result { + let object = routing_keys(Document::Static, value, STATIC_KEYS)?; + shaped(Document::Static, object) +} + +pub(crate) fn parse_dynamic_config(value: Value) -> Result { + let object = routing_keys(Document::Dynamic, value, DYNAMIC_KEYS)?; + shaped(Document::Dynamic, object) +} + +/// A call's request as Python spells it. +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct RequestSpec { + model: Option, + team_id: Option, + listing_team_id: Option, + proxy_admin: Option, + allowed_models: Option>, + admin_model: Option, + specific_deployment: Option, + num_retries: Option, + uncounted_retries: Option, + disable_fallbacks: Option, + router_weights: Option, + pinned_deployment_id: Option, + routing_strategy: Option, + filter_tags: Option, + /// `listed` or `healthy`: which pass-through pick the request makes. + pass_through: Option, + target_order: Option, + excluded_ids: Option>, + retry_skipped_ids: Option>, + pre_routes: Option, + has_image: Option, + endpoint: Option, + web_search: Option, + route_handler: Option, + decisions_media: Option, + batch_create: Option, + embedding_batch: Option, + wildcard_request: Option, + unrouted: Option, + passthrough_on_no_deployment: Option, + skip_weighted_failover: Option, + same_group_fallbacks_only: Option, + model_group_retry_policy: Option>, + retry_policy_raises: Option>, + #[serde(default, deserialize_with = "present")] + fallbacks: Option>, + #[serde(default, deserialize_with = "present")] + context_window_fallbacks: Option>, + #[serde(default, deserialize_with = "present")] + content_policy_fallbacks: Option>, + max_fallbacks: Option, + mock_failure: Option, + cooldown_time: Option, + health_check_probe: Option, + outcome_callbacks: Option, + body: Option, +} + +/// A key the request passed, even as `None`: `Some(None)`. +fn present<'de, D, T>(deserializer: D) -> Result>, D::Error> +where + D: serde::Deserializer<'de>, + T: Deserialize<'de>, +{ + Option::::deserialize(deserializer).map(Some) +} + +/// Read a request; `model` replaces the request's own when given. +pub(crate) fn parse_request(value: Value, model: Option<&str>) -> Result { + let spec: RequestSpec = shaped(Document::Request, object(Document::Request, value)?)?; + let model = model + .map(str::to_owned) + .or(spec.model) + .ok_or(Error::MissingModel)?; + Ok(RouterRequest { + model, + body: spec.body.unwrap_or_default(), + team_id: spec.team_id, + listing_team_id: spec.listing_team_id, + proxy_admin: spec.proxy_admin.unwrap_or(false), + allowed_models: spec.allowed_models.unwrap_or_default(), + admin_model: spec.admin_model, + specific_deployment: spec.specific_deployment.unwrap_or(false), + num_retries: spec.num_retries, + uncounted_retries: spec.uncounted_retries, + disable_fallbacks: spec.disable_fallbacks.unwrap_or(false), + router_weights: spec.router_weights, + pinned_deployment_id: spec.pinned_deployment_id, + routing_strategy: spec.routing_strategy, + filter_tags: spec.filter_tags.unwrap_or(false), + pass_through: match spec.pass_through.as_deref() { + None => None, + Some("healthy") => Some(PassThrough::Healthy), + Some(_) => Some(PassThrough::Listed), + }, + target_order: spec.target_order, + excluded_ids: spec.excluded_ids.unwrap_or_default(), + retry_skipped_ids: spec.retry_skipped_ids.unwrap_or_default(), + pre_routes: spec.pre_routes.unwrap_or(false), + has_image: spec.has_image.unwrap_or(false), + endpoint: spec.endpoint, + web_search: spec.web_search.unwrap_or(false), + route_handler: spec.route_handler, + decisions_media: spec.decisions_media.unwrap_or(false), + batch_create: spec.batch_create.unwrap_or(false), + embedding_batch: spec.embedding_batch.unwrap_or(false), + wildcard_request: spec.wildcard_request, + unrouted: spec.unrouted.unwrap_or(false), + passthrough_on_no_deployment: spec.passthrough_on_no_deployment.unwrap_or(false), + skip_weighted_failover: spec.skip_weighted_failover.unwrap_or(false), + same_group_fallbacks_only: spec.same_group_fallbacks_only.unwrap_or(false), + model_group_retry_policy: spec.model_group_retry_policy, + retry_policy_raises: spec.retry_policy_raises.unwrap_or_default(), + fallbacks: spec.fallbacks, + context_window_fallbacks: spec.context_window_fallbacks, + content_policy_fallbacks: spec.content_policy_fallbacks, + max_fallbacks: spec.max_fallbacks, + mock_failure: spec.mock_failure, + cooldown_time: spec.cooldown_time, + health_check_probe: spec.health_check_probe.unwrap_or(false), + outcome_callbacks: spec.outcome_callbacks.unwrap_or_default(), + payload: None, + }) +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct ChangesSpec { + /// Every deployment, replacing the current list before the upserts and deletes. + #[serde(default)] + replace: Option>, + /// Deployments added at the end, beside any that share their id. + #[serde(default)] + append: Vec, + #[serde(default)] + upsert: Vec, + #[serde(default)] + delete: Vec, + /// A new dynamic document (the same keys `RouterCoreHandle(static, dynamic)` takes). + #[serde(default)] + settings: Option, + /// The registry indexes the host wrote (`IndexOverrides`); `{}` drops them. + #[serde(default)] + index: Option, + /// The deployment at the id map's position for this id, removed with every index shifted. + #[serde(default)] + remove: Option, + /// A write to the indexes the host hands out (`HostIndexes`), field by field. + #[serde(default)] + host_index: Option, +} + +pub(crate) fn parse_changes(value: Value) -> Result { + let spec: ChangesSpec = shaped(Document::Changes, object(Document::Changes, value)?)?; + Ok(ChangeSet { + replace: spec.replace, + appends: spec.append, + upserts: spec.upsert, + deletes: spec.delete, + settings: spec.settings.map(parse_dynamic_config).transpose()?, + index: spec.index, + remove: spec.remove, + host_index: spec.host_index, + }) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + #[rstest] + #[case::unknown_key(json!({"routing_strategy_args": {"ttl": 1}}), "routing_strategy_args")] + #[case::dynamic_key_in_static(json!({"num_retries": 2}), "num_retries")] + fn an_unread_static_key_declines_naming_the_key(#[case] json: Value, #[case] key: &str) { + let error = parse_router_config(json).unwrap_err(); + assert!(error.declined()); + assert!(matches!(&error, Error::UnknownKey { key: named, .. } if named == key)); + } + + #[rstest] + #[case::null_unknown(json!({"routing_strategy_args": null}))] + #[case::ignored(json!({"set_verbose": true, "debug_level": "INFO"}))] + #[case::pre_call_checks_off(json!({"enable_pre_call_checks": false}))] + #[case::pre_call_checks_on(json!({"enable_pre_call_checks": true}))] + fn unset_and_ignored_keys_are_accepted(#[case] json: Value) { + parse_router_config(json).unwrap(); + } + + #[rstest] + #[case::not_object(json!([]), false)] + #[case::bad_shape(json!({"retry_after": "soon"}), true)] + fn a_config_that_is_not_an_object_is_a_value_error_and_a_wrong_shape_declines( + #[case] json: Value, + #[case] declined: bool, + ) { + assert_eq!(parse_router_config(json).unwrap_err().declined(), declined); + } + + #[rstest] + fn python_router_defaults_read_as_unset() { + let dynamic = parse_dynamic_config(json!({ + "num_retries": null, "fallbacks": [], "context_window_fallbacks": [], + "content_policy_fallbacks": [], "model_group_alias": {"gpt": "gpt-4"}, + })) + .unwrap(); + assert_eq!(dynamic.num_retries, None); + assert!(dynamic.context_window_fallbacks.is_none()); + assert!(dynamic.content_policy_fallbacks.is_none()); + assert_eq!(dynamic.model_alias_map["gpt"], "gpt-4"); + } + + #[rstest] + #[case::own_model(json!({"model": "gpt", "num_retries": 1}), None, "gpt")] + #[case::override_model(json!({"model": "gpt"}), Some("other"), "other")] + fn a_request_reads_its_model( + #[case] json: Value, + #[case] model: Option<&str>, + #[case] expected: &str, + ) { + assert_eq!(parse_request(json, model).unwrap().model, expected); + } + + #[rstest] + #[case::negative("negative", UncountedRetries::Negative)] + #[case::not_positive("not_positive", UncountedRetries::NotPositive)] + #[case::raises("raises", UncountedRetries::Raises)] + fn a_request_reads_a_num_retries_that_is_no_count( + #[case] spelled: &str, + #[case] expected: UncountedRetries, + ) { + let request = + parse_request(json!({"model": "gpt", "uncounted_retries": spelled}), None).unwrap(); + assert_eq!(request.uncounted_retries, Some(expected)); + assert_eq!(request.num_retries, None); + } + + #[rstest] + #[case::unknown_field(json!({"model": "gpt", "temperature": 1}))] + #[case::no_model(json!({"team_id": "t"}))] + fn a_bad_request_is_a_value_error(#[case] json: Value) { + assert!(!parse_request(json, None).unwrap_err().declined()); + } + + #[rstest] + fn a_change_set_reads_upserts_and_deletes() { + let changes = parse_changes(json!({ + "upsert": [{"model_name": "m", "litellm_params": {"model": "openai/x"}}], + "delete": ["old"], + })) + .unwrap(); + assert_eq!(changes.upserts.len(), 1); + assert_eq!(changes.deletes, ["old"]); + } +} diff --git a/litellm-rust/crates/python-bridge/src/router/host.rs b/litellm-rust/crates/python-bridge/src/router/host.rs new file mode 100644 index 00000000000..0ded98f27c5 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/router/host.rs @@ -0,0 +1,1663 @@ +//! The host behind `RouterCoreHandle`. Rust never awaits Python: each host operation the call +//! loop needs becomes a [`HostCall`] parked in the call's [`Exchange`], and the call future stays +//! pending until Python runs it and hands the [`HostCallResponse`] back through [`Driver::resume`]. +//! +//! The driver polls the call future with a no-op waker while the call's exchange is bound to the +//! current thread, so a host future finds its exchange without the router passing it through. +//! +//! The routing cache is a host operation too: [`StepCache`] parks every [`RouterCache`] op as a +//! `cache` step, and Python runs it against the router's own `DualCache` (memory plus Redis), so +//! Python and Rust read and write the same cooldowns, fail counters and usage. + +use std::{ + cell::RefCell, + future::{Future, poll_fn}, + pin::Pin, + sync::{Arc, Mutex, MutexGuard, PoisonError}, + task::{Context, Poll, Waker}, + time::Duration, +}; + +use async_trait::async_trait; +use litellm_router_core::{ + Attempt, AttemptError, Blocked, BuildAsk, BuildPath, Built, CallbackFiltered, Deployment, + Error, Host, Output, Payload, PreRoute, PreRouted, Provided, Randomness, RouterCache, + RouterEvent, RouterRequest, RouterResponse, RoutingContext, SecretRef, SelectStage, + ThreadRandomness, build::BuiltDeployment, host::ModelInfoLookup, model_info::MergedModelInfo, + precall::PreCallFacts, +}; +use pyo3::{PyResult, Python, types::PyAnyMethods}; +use serde_json::{Map, Value}; + +/// The `kind` of a step dict. +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::Display, strum::IntoStaticStr)] +pub(crate) enum StepKind { + #[strum(serialize = "attempt")] + Attempt, + #[strum(serialize = "sleep")] + Sleep, + #[strum(serialize = "allow_fallback")] + AllowFallback, + #[strum(serialize = "refuse_fallbacks")] + RefuseFallbacks, + #[strum(serialize = "select")] + Select, + #[strum(serialize = "pre_route")] + PreRoute, + #[strum(serialize = "cache")] + Cache, + #[strum(serialize = "secrets")] + Secrets, + #[strum(serialize = "validate")] + Validate, + #[strum(serialize = "provider")] + Provider, + #[strum(serialize = "added")] + Added, + #[strum(serialize = "model_info")] + ModelInfo, + #[strum(serialize = "done")] + Done, + #[strum(serialize = "failed")] + Failed, +} + +/// Which selection stage Python is asked to run. +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum Stage { + /// Pre-call checks and sub-routers: return the deployments that remain. + Plugins, + /// A strategy router-core does not implement: return the candidates with the pick first. + /// When candidates share an id, `records` holds each candidate's deployment, in order, and the + /// host answers with the pick's position, so a pick splits over every entry as Python's does. + Strategy { + strategy: String, + records: Vec, + }, + /// Tag routing: return the deployments that remain. + Tags, + /// `enable_pre_call_checks`: return the facts the filter decides by. + PreCall, + /// `_deployment_declares_no_vision`: return the candidates that declare no vision support. + TextOnly, + /// `_deployment_serves_route`: return the candidates that cannot serve `handler`'s call. + RouteIncapable { handler: String }, + /// `_decisions_deployment_refuses_media`: return the candidates that refuse the call's media. + MediaRefusing, + /// `deployments_admitting_embedding_batch`: return the candidates the embedding pick keeps. + EmbeddingBatch, + /// The override strategy's pre-call check on a deployment named by id. + OverrideCheck(String), + /// The routing plugins: run them when `run`, then return the candidate models they left. + RoutingPlugins { run: bool }, + /// The host's `CustomLogger` deployment filters: return the ids that remain, or a failure. + /// They read the ids this request already failed on, as Python's request kwargs carry them. + CallbackFilters { + excluded_ids: Vec, + skipped_ids: Vec, + target_order: Option, + }, + /// `f"{provider}/{model}"` for a name no wildcard pattern matches as spelled: return it as + /// the one entry of `models`. + ProviderRequest, +} + +/// The `op` of a `cache` step: one per [`RouterCache`] method. +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::Display, strum::IntoStaticStr)] +pub(crate) enum CacheOp { + /// `async_get_cache(key)`: answered with the stored JSON value, or `None`. + #[strum(serialize = "get")] + Get, + /// `async_set_cache(key, value, ttl)`: answered with `None`. + #[strum(serialize = "set")] + Set, + /// `async_increment(key, value, ttl)`: answered with the new number. The TTL applies when + /// the key is created. + #[strum(serialize = "increment")] + Increment, + /// `async_increment_usage(keys, value, ttl)`: `value` is one amount per key, added as + /// `_increment_deployment_usage` adds them. Answered with `None`. + #[strum(serialize = "increment_usage")] + IncrementUsage, + /// `async_batch_get_cache(keys)`: answered with one JSON value or `None` per key, in order. + #[strum(serialize = "batch_get")] + BatchGet, + /// `async_delete_cache(key)`: answered with `None`. + #[strum(serialize = "delete")] + Delete, +} + +/// Python's outcome of a `cache` step. +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum CacheOutcome { + /// `get`: the stored value (`None` when missing). + Value(Option), + /// `batch_get`: one entry per key. + Values(Vec>), + /// `increment`: the counter after the increment. + Number(f64), + /// `set`, `delete` and `increment_usage`. + Stored, +} + +impl CacheOutcome { + /// The op this outcome answers (`Stored` answers both `set` and `delete`). + fn answers(&self, op: CacheOp) -> bool { + matches!( + (self, op), + (Self::Value(_), CacheOp::Get) + | (Self::Values(_), CacheOp::BatchGet) + | (Self::Number(_), CacheOp::Increment) + | ( + Self::Stored, + CacheOp::Set | CacheOp::Delete | CacheOp::IncrementUsage + ) + ) + } +} + +/// A fallback chain's dict targets, which can carry an api key. +#[derive(Clone, PartialEq, veil::Redact)] +pub(crate) struct DictTargets(#[redact(with = "[REDACTED]")] pub(crate) Vec>); + +/// What a call is blocked on, for Python to run. +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum HostCall { + Attempt { + deployment_id: String, + /// The picked deployment as configured: `model_name`, `litellm_params`, `model_info` + /// (with its `id`). The host builds the request from it; it never looks the deployment up + /// in a Python router. + deployment: Value, + /// The name the attempt resolved: the requested name or the fallback target. + model: String, + model_group: String, + /// The group a pre-routing hook selected for `model`, when it differs. + routed: Option, + /// The `order` level of a same-group order fallback. + target_order: Option, + /// Every deployment `model` resolved to for this caller, the picked one included. + candidate_ids: Vec, + retry: u32, + fallback_depth: u32, + fallback_level: u32, + timeout: Option, + /// Request params the fallback walk's dict targets merged in. + params: Option>, + /// The walk level whose request metadata the attempt carries. + scope: u32, + max_retries: u32, + model_group_size: usize, + }, + Sleep { + seconds: f64, + }, + AllowFallback { + target: String, + }, + RefuseFallbacks { + /// The walk level whose chain the targets are. + level: u32, + targets: DictTargets, + }, + Select { + stage: Stage, + model_group: String, + /// The name the request resolved from, which strategies key their state by. + name: String, + /// Whose selector a strategy reads: `default`, `group` (with `group`) or `override`. + scope: &'static str, + group: Option, + /// The deployments the name resolved to, before health filters (the tag stage reads them). + pool: Vec, + candidates: Vec, + }, + /// Pre-routing before a pick resolves `model`: answered with the model group to resolve + /// instead (a string), a custom strategy's deployment document (a mapping), or `None`. + PreRoute { + model: String, + fallback_depth: u32, + retry: u32, + }, + /// One [`RouterCache`] op against the host's cache. `key` is set for every op but + /// `batch_get`, which sets `keys`; `value` is set for `set` (the JSON to store) and + /// `increment` (the amount); `ttl` is in seconds. + Cache { + op: CacheOp, + key: Option, + keys: Option>, + value: Option, + ttl: Option, + /// The op reads or writes this worker's in-memory tier only (`local_only=True`). + local: bool, + }, + /// The deployment build's `os.environ/` references: answered with one value per ref. + Secrets { + index: usize, + refs: Vec, + }, + /// The deployment build's `validate`, `provider` and `added` steps for one deployment. + Build { + kind: StepKind, + path: BuildPath, + index: usize, + entry: Value, + declared_id: Option, + duplicate_id: bool, + /// The provider the `provider` step resolved (`added` only). + provider: Option, + }, + /// One lookup of a model-info query ([`ModelInfoLookup`], as its `lookup` field and + /// arguments): answered as the lookup documents. + ModelInfo { + lookup: Value, + }, +} + +impl HostCall { + pub(crate) fn kind(&self) -> StepKind { + match self { + Self::Attempt { .. } => StepKind::Attempt, + Self::Sleep { .. } => StepKind::Sleep, + Self::AllowFallback { .. } => StepKind::AllowFallback, + Self::RefuseFallbacks { .. } => StepKind::RefuseFallbacks, + Self::Select { .. } => StepKind::Select, + Self::PreRoute { .. } => StepKind::PreRoute, + Self::Cache { .. } => StepKind::Cache, + Self::Secrets { .. } => StepKind::Secrets, + Self::Build { kind, .. } => *kind, + Self::ModelInfo { .. } => StepKind::ModelInfo, + } + } +} + +/// Python's response to a [`HostCall`]. +#[derive(Debug)] +pub(crate) enum HostCallResponse { + /// The attempt's total tokens, or its failure. + Attempt(Result), + /// A successful attempt whose response the provider blocked, or a stream whose outcome the + /// host reports when it ends (`deferred`), with its tokens. + Served { + total_tokens: u64, + blocked: Option, + deferred: bool, + tracks_usage: Option, + }, + Slept, + AllowFallback(bool), + /// Whether the chain a `refuse_fallbacks` step named is refused. + RefuseFallbacks(bool), + /// The candidate ids that remain, pick first. + Select(Vec), + /// A strategy's pick among candidates sharing an id: positions in the candidates, pick first. + Positions(Vec), + /// The model group to resolve instead, a custom strategy's deployment, or `None`. + PreRoute(Option), + /// The facts a `pre_call` select step asked for. + PreCall(PreCallFacts), + /// The candidate models a `routing_plugins` select step reported. + Models(Option>), + /// The ids a `callback_filters` select step left with the one a filter prefers, or the error + /// a filter raised. + Filtered(Result<(Vec, Option), AttemptError>), + /// What a pick's select or pre-route step, or a fallback check, raised on the host instead of + /// answering: the walk takes it as the try's failure. + Raised(AttemptError), + Cache(CacheOutcome), + /// One value per ref of a `secrets` step. + Secrets(Vec), + Validated(Built), + Provided(Provided), + Added(Built), + /// A `model_info` lookup's response, as the host spelled it. + ModelInfo(Value), +} + +impl HostCallResponse { + fn kind(&self) -> StepKind { + match self { + Self::Attempt(_) | Self::Served { .. } => StepKind::Attempt, + // Answers whichever step raised it (`check`). + Self::Raised(_) => StepKind::Select, + Self::Slept => StepKind::Sleep, + Self::AllowFallback(_) => StepKind::AllowFallback, + Self::RefuseFallbacks(_) => StepKind::RefuseFallbacks, + Self::Select(_) + | Self::Positions(_) + | Self::PreCall(_) + | Self::Models(_) + | Self::Filtered(_) => StepKind::Select, + Self::PreRoute(_) => StepKind::PreRoute, + Self::Cache(_) => StepKind::Cache, + Self::Secrets(_) => StepKind::Secrets, + Self::Validated(_) => StepKind::Validate, + Self::Provided(_) => StepKind::Provider, + Self::Added(_) => StepKind::Added, + Self::ModelInfo(_) => StepKind::ModelInfo, + } + } + + fn check(&self, call: &HostCall) -> Result<(), DriveError> { + if matches!(self, Self::Raised(_)) + && matches!( + call.kind(), + StepKind::Select | StepKind::PreRoute | StepKind::AllowFallback + ) + { + return Ok(()); + } + if self.kind() != call.kind() { + return Err(DriveError::WrongAnswer { + expected: call.kind(), + got: self.kind(), + }); + } + if let ( + Self::Select(ids) | Self::Filtered(Ok((ids, _))), + HostCall::Select { candidates, .. }, + ) = (self, call) + && let Some(id) = ids.iter().find(|id| !candidates.contains(id)) + { + return Err(DriveError::UnknownCandidate { id: id.clone() }); + } + if let (Self::Positions(positions), HostCall::Select { candidates, .. }) = (self, call) + && let Some(position) = positions.iter().find(|p| **p >= candidates.len()) + { + return Err(DriveError::UnknownCandidate { + id: format!("#{position}"), + }); + } + if let (Self::Secrets(values), HostCall::Secrets { refs, .. }) = (self, call) + && values.len() != refs.len() + { + return Err(DriveError::SecretsLength { + expected: refs.len(), + got: values.len(), + }); + } + if let (Self::Cache(outcome), HostCall::Cache { op, keys, .. }) = (self, call) { + if !outcome.answers(*op) { + return Err(DriveError::WrongCacheOutcome { op: *op }); + } + if let (CacheOutcome::Values(values), Some(keys)) = (outcome, keys) + && values.len() != keys.len() + { + return Err(DriveError::BatchLength { + expected: keys.len(), + got: values.len(), + }); + } + } + Ok(()) + } +} + +/// How a driven future ended well. +#[derive(Debug)] +pub(crate) enum Finish { + /// `start`: the call's response. + Call(RouterResponse), + /// `select`: the deployment picked and the model group it serves. + Selected { + deployment_id: String, + model_group: String, + /// The deployment as served: a priority group's member carries its priority as `order`. + deployment: Value, + }, + /// `healthy`: the deployments a pick would choose among, as served. + Healthy { + /// A deployment named by its id, which the listing returns alone rather than as a list. + single: bool, + deployment_ids: Vec, + deployments: Vec, + }, + /// `report_failure`: the cooldown applied, if the deployment was cooled down. + Reported { cooldown: Option }, + /// `report_success` / `count_success`: the success counter's key, if one was counted. + Counted { key: Option }, + /// `build`: the deployments it installed, in order. + Built(Vec), + /// `router_model_info`: the merged model info (`None` when the lookup answered `None`). + RouterModelInfo(Option), + /// `deployment_model_info`: the merged model info, if any. + DeploymentModelInfo(Option>), + /// A model-info query that ends with one JSON answer (`model_group_info`, + /// `configured_token_limits`, `model_listing_info`). + Value(Value), +} + +/// One step of a call: what it is blocked on, or how it ended. +#[derive(Debug)] +pub(crate) enum Step { + HostCall(HostCall), + Done(Finish), + Failed(Error), +} + +/// A misuse of the step protocol by the caller. +#[derive(Debug, thiserror::Error)] +pub(crate) enum DriveError { + #[error("the router call already finished")] + Finished, + #[error("the router call has not made a host call yet, so the outcome must be None")] + UnexpectedResponse, + #[error("the router call is waiting for an outcome of its `{expected}` step")] + MissingResponse { expected: StepKind }, + #[error("the router call is waiting for an outcome of its `{expected}` step, not `{got}`")] + WrongAnswer { expected: StepKind, got: StepKind }, + #[error("deployment `{id}` was not one of the select step's candidates")] + UnknownCandidate { id: String }, + #[error("the outcome does not answer a `{op}` cache step")] + WrongCacheOutcome { op: CacheOp }, + #[error("a `batch_get` cache step for {expected} keys was answered with {got} values")] + BatchLength { expected: usize, got: usize }, + #[error("a `secrets` step for {expected} references was answered with {got} values")] + SecretsLength { expected: usize, got: usize }, + #[error("the router call is pending on something other than a step")] + Stalled, +} + +impl DriveError { + /// Whether the caller passed a bad outcome, rather than driving a call that cannot go on. + pub(crate) fn is_request(&self) -> bool { + match self { + Self::UnexpectedResponse + | Self::MissingResponse { .. } + | Self::WrongAnswer { .. } + | Self::UnknownCandidate { .. } + | Self::WrongCacheOutcome { .. } + | Self::BatchLength { .. } + | Self::SecretsLength { .. } => true, + Self::Finished | Self::Stalled => false, + } + } +} + +/// The host call a router call is blocked on and the response that releases it. +#[derive(Default)] +struct Exchange { + call: Option, + response: Option, + /// Events the call reported since the last step, handed to Python with the next one. + events: Vec, +} + +type Shared = Arc>; + +fn lock(exchange: &Mutex) -> MutexGuard<'_, Exchange> { + exchange.lock().unwrap_or_else(PoisonError::into_inner) +} + +thread_local! { + static CURRENT: RefCell> = const { RefCell::new(None) }; +} + +/// Binds an exchange to this thread for one poll and restores the previous one after. +struct Bound(Option); + +impl Bound { + fn new(exchange: &Shared) -> Self { + Self(CURRENT.replace(Some(exchange.clone()))) + } +} + +impl Drop for Bound { + fn drop(&mut self) { + CURRENT.set(self.0.take()); + } +} + +/// Park the host call `call` in the bound exchange and wait for its response. +/// `response`, or the failure a step answered with instead ([`HostCallResponse::Raised`]). +fn answered(response: HostCallResponse) -> Result { + match response { + HostCallResponse::Raised(failure) => Err(failure), + response => Ok(response), + } +} + +async fn call_host(call: HostCall) -> HostCallResponse { + let mut call = Some(call); + poll_fn(move |_| { + CURRENT.with_borrow(|current| { + let Some(exchange) = current else { + return Poll::Pending; + }; + let mut exchange = lock(exchange); + if let Some(call) = call.take() { + exchange.call = Some(call); + return Poll::Pending; + } + exchange.response.take().map_or(Poll::Pending, Poll::Ready) + }) + }) + .await +} + +const CHECKED: &str = "the driver checks each response against its host call"; + +fn ttl_seconds(ttl: Option) -> Option { + ttl.map(|ttl| ttl.as_secs_f64()) +} + +/// An increment amount as Python's counters store it: an int when it is whole. +fn amount_value(amount: f64) -> Value { + if amount.fract() == 0.0 && amount.abs() < i64::MAX as f64 { + Value::from(amount as i64) + } else { + Value::from(amount) + } +} + +/// The host's cache: every op is a `cache` step Python runs against the router's `DualCache`. +pub(crate) struct StepCache; + +async fn cache_step( + op: CacheOp, + key: Option<&str>, + keys: Option<&[String]>, + value: Option, + ttl: Option, +) -> CacheOutcome { + call_cache(op, key, keys, value, ttl, false).await +} + +async fn call_cache( + op: CacheOp, + key: Option<&str>, + keys: Option<&[String]>, + value: Option, + ttl: Option, + local: bool, +) -> CacheOutcome { + let call = HostCall::Cache { + op, + key: key.map(str::to_owned), + keys: keys.map(<[String]>::to_vec), + value, + ttl: ttl_seconds(ttl), + local, + }; + let HostCallResponse::Cache(outcome) = call_host(call).await else { + unreachable!("{CHECKED}"); + }; + outcome +} + +#[async_trait] +impl RouterCache for StepCache { + async fn async_get_cache(&self, key: &str) -> Result, litellm_cache::Error> { + let CacheOutcome::Value(value) = + cache_step(CacheOp::Get, Some(key), None, None, None).await + else { + unreachable!("{CHECKED}"); + }; + Ok(value) + } + + async fn async_set_cache( + &self, + key: &str, + value: Value, + ttl: Option, + ) -> Result<(), litellm_cache::Error> { + cache_step(CacheOp::Set, Some(key), None, Some(value), ttl).await; + Ok(()) + } + + async fn async_increment( + &self, + key: &str, + amount: f64, + ttl: Option, + ) -> Result { + let value = Some(amount_value(amount)); + let CacheOutcome::Number(next) = + cache_step(CacheOp::Increment, Some(key), None, value, ttl).await + else { + unreachable!("{CHECKED}"); + }; + Ok(next) + } + + async fn async_batch_get_cache( + &self, + keys: &[String], + ) -> Result>, litellm_cache::Error> { + let CacheOutcome::Values(values) = + cache_step(CacheOp::BatchGet, None, Some(keys), None, None).await + else { + unreachable!("{CHECKED}"); + }; + Ok(values) + } + + async fn async_delete_cache(&self, key: &str) -> Result<(), litellm_cache::Error> { + cache_step(CacheOp::Delete, Some(key), None, None, None).await; + Ok(()) + } + + async fn async_increment_usage( + &self, + increments: &[(String, f64)], + ttl: Option, + ) -> Result<(), litellm_cache::Error> { + let keys: Vec = increments.iter().map(|(key, _)| key.clone()).collect(); + let amounts = increments + .iter() + .map(|(_, amount)| amount_value(*amount)) + .collect(); + let value = Some(Value::Array(amounts)); + cache_step(CacheOp::IncrementUsage, None, Some(&keys), value, ttl).await; + Ok(()) + } + + async fn async_get_local(&self, key: &str) -> Result, litellm_cache::Error> { + let CacheOutcome::Value(value) = + call_cache(CacheOp::Get, Some(key), None, None, None, true).await + else { + unreachable!("{CHECKED}"); + }; + Ok(value) + } + + async fn async_increment_local( + &self, + key: &str, + amount: f64, + ttl: Option, + ) -> Result { + let value = Some(amount_value(amount)); + let CacheOutcome::Number(next) = + call_cache(CacheOp::Increment, Some(key), None, value, ttl, true).await + else { + unreachable!("{CHECKED}"); + }; + Ok(next) + } +} + +/// A deployment as Python spells a `model_list` entry, with its id in `model_info`. +/// Whether two candidates share an id, which a pick answered by id can't tell apart. +fn shares_ids(candidates: &[Arc]) -> bool { + let mut seen = std::collections::HashSet::new(); + !candidates.iter().all(|d| seen.insert(d.id.as_str())) +} + +/// The candidates a host answer names, in its order. An id names every candidate that carries +/// it, so deployments sharing an id stay together through a filter, as Python's list keeps them. +fn answered_by_id(candidates: &[Arc], ids: &[String]) -> Vec> { + let mut taken = vec![false; candidates.len()]; + let mut kept = Vec::new(); + for id in ids { + for (position, deployment) in candidates.iter().enumerate() { + if !taken[position] && deployment.id == *id { + taken[position] = true; + kept.push(deployment.clone()); + } + } + } + kept +} + +pub(crate) fn deployment_record(deployment: &Deployment) -> Value { + deployment.record() +} + +/// The process's `random` module: a Python router draws its shuffle and retry jitter there, so +/// a seeded process picks the same deployments on either core. The bare module functions share +/// one generator, so these calls interleave with every other draw the process makes. +pub(crate) struct PythonRandomness; + +impl Randomness for PythonRandomness { + fn next_f64(&self) -> f64 { + Python::attach(|py| -> PyResult { + py.import("random")?.call_method0("random")?.extract() + }) + .unwrap_or_else(|_| ThreadRandomness.next_f64()) + } + + fn index_below(&self, len: usize) -> usize { + // `random.choice` is `seq[self._randbelow(len(seq))]`, and so is `randrange(len)`. + Python::attach(|py| -> PyResult { + py.import("random")? + .call_method1("randrange", (len,))? + .extract() + }) + .unwrap_or_else(|_| ThreadRandomness.index_below(len)) + } +} + +/// The [`Host`] every `RouterCoreHandle` router runs on. +pub(crate) struct StepHost; + +#[async_trait] +impl Host for StepHost { + async fn invoke(&self, attempt: &Attempt) -> Result { + let call = HostCall::Attempt { + deployment_id: attempt.deployment.id.clone(), + deployment: deployment_record(&attempt.deployment), + model: attempt.name.clone(), + model_group: attempt.model_group.clone(), + routed: attempt.routed.clone(), + target_order: attempt.target_order, + candidate_ids: attempt.candidate_ids.clone(), + retry: attempt.retry, + fallback_depth: attempt.fallback_depth, + fallback_level: attempt.fallback_level, + timeout: attempt.timeout.map(|timeout| timeout.as_secs_f64()), + params: attempt + .params + .as_ref() + .map(|params| Box::new(Value::Object(Map::clone(params)))), + scope: attempt.scope, + max_retries: attempt.max_retries, + model_group_size: attempt.model_group_size, + }; + let (result, blocked, deferred, tracks_usage) = match call_host(call).await { + HostCallResponse::Attempt(result) => (result, None, false, None), + HostCallResponse::Served { + total_tokens, + blocked, + deferred, + tracks_usage, + } => (Ok(total_tokens), blocked, deferred, tracks_usage), + _ => unreachable!("{CHECKED}"), + }; + result.map(|total_tokens| Output { + value: Payload(Arc::new(())), + total_tokens, + blocked, + deferred, + tracks_usage, + }) + } + + async fn select( + &self, + stage: &SelectStage, + ctx: &RoutingContext<'_>, + candidates: Vec>, + ) -> Result>, Error> { + let (scope, group) = ctx.scope.wire(); + let call = HostCall::Select { + stage: match stage { + SelectStage::Plugins => Stage::Plugins, + SelectStage::Strategy(strategy) => Stage::Strategy { + strategy: strategy.to_string(), + records: if shares_ids(&candidates) { + candidates.iter().map(|d| deployment_record(d)).collect() + } else { + Vec::new() + }, + }, + SelectStage::Tags => Stage::Tags, + SelectStage::TextOnly => Stage::TextOnly, + SelectStage::RouteIncapable { handler } => Stage::RouteIncapable { + handler: handler.clone(), + }, + SelectStage::MediaRefusing => Stage::MediaRefusing, + SelectStage::EmbeddingBatch => Stage::EmbeddingBatch, + SelectStage::OverrideCheck(strategy) => Stage::OverrideCheck(strategy.to_string()), + }, + pool: ctx.pool.iter().map(|d| d.id.clone()).collect(), + model_group: ctx.model_group.to_owned(), + name: ctx.name.to_owned(), + scope, + group: group.map(str::to_owned), + candidates: candidates.iter().map(|d| d.id.clone()).collect(), + }; + match answered(call_host(call).await)? { + HostCallResponse::Select(ids) => Ok(answered_by_id(&candidates, &ids)), + HostCallResponse::Positions(positions) => Ok(positions + .iter() + .map(|position| candidates[*position].clone()) + .collect()), + _ => unreachable!("{CHECKED}"), + } + } + + async fn pre_route(&self, pre: &PreRoute<'_>) -> Result, Error> { + let call = HostCall::PreRoute { + model: pre.name.to_owned(), + fallback_depth: pre.fallback_depth, + retry: pre.retry, + }; + let HostCallResponse::PreRoute(routed) = answered(call_host(call).await)? else { + unreachable!("{CHECKED}"); + }; + Ok(routed) + } + + async fn routing_plugin_models( + &self, + ctx: &RoutingContext<'_>, + run: bool, + ) -> Result>, Error> { + let (scope, group) = ctx.scope.wire(); + let call = HostCall::Select { + stage: Stage::RoutingPlugins { run }, + pool: Vec::new(), + model_group: ctx.model_group.to_owned(), + name: ctx.name.to_owned(), + scope, + group: group.map(str::to_owned), + candidates: Vec::new(), + }; + let HostCallResponse::Models(models) = answered(call_host(call).await)? else { + unreachable!("{CHECKED}"); + }; + Ok(models) + } + + async fn provider_request(&self, ctx: &RoutingContext<'_>) -> Result, Error> { + let (scope, group) = ctx.scope.wire(); + let call = HostCall::Select { + stage: Stage::ProviderRequest, + pool: Vec::new(), + model_group: ctx.model_group.to_owned(), + name: ctx.name.to_owned(), + scope, + group: group.map(str::to_owned), + candidates: Vec::new(), + }; + let HostCallResponse::Models(models) = answered(call_host(call).await)? else { + unreachable!("{CHECKED}"); + }; + Ok(models.into_iter().flatten().next()) + } + + async fn pre_call_facts( + &self, + ctx: &RoutingContext<'_>, + candidates: &[Arc], + ) -> Result { + let (scope, group) = ctx.scope.wire(); + let call = HostCall::Select { + stage: Stage::PreCall, + pool: ctx.pool.iter().map(|d| d.id.clone()).collect(), + model_group: ctx.model_group.to_owned(), + name: ctx.name.to_owned(), + scope, + group: group.map(str::to_owned), + candidates: candidates.iter().map(|d| d.id.clone()).collect(), + }; + let HostCallResponse::PreCall(facts) = answered(call_host(call).await)? else { + unreachable!("{CHECKED}"); + }; + Ok(facts) + } + + async fn callback_filters( + &self, + ctx: &RoutingContext<'_>, + candidates: Vec>, + ) -> Result { + let (scope, group) = ctx.scope.wire(); + let joined = |own: &[String], walked: &[String]| [own, walked].concat(); + let call = HostCall::Select { + stage: Stage::CallbackFilters { + excluded_ids: joined(&ctx.request.excluded_ids, ctx.excluded_ids), + skipped_ids: joined(&ctx.request.retry_skipped_ids, ctx.skipped_ids), + target_order: ctx.target_order, + }, + pool: Vec::new(), + model_group: ctx.model_group.to_owned(), + name: ctx.name.to_owned(), + scope, + group: group.map(str::to_owned), + candidates: candidates.iter().map(|d| d.id.clone()).collect(), + }; + let HostCallResponse::Filtered(result) = call_host(call).await else { + unreachable!("{CHECKED}"); + }; + let (ids, preferred) = result.map_err(Error::Attempt)?; + Ok(CallbackFiltered { + kept: answered_by_id(&candidates, &ids), + preferred, + }) + } + + async fn allow_fallback( + &self, + _request: &RouterRequest, + target: &str, + ) -> Result { + let call = HostCall::AllowFallback { + target: target.to_owned(), + }; + let HostCallResponse::AllowFallback(allowed) = answered(call_host(call).await)? else { + unreachable!("{CHECKED}"); + }; + Ok(allowed) + } + + async fn refuses_fallbacks(&self, level: u32, targets: &[Map]) -> bool { + let call = HostCall::RefuseFallbacks { + level, + targets: DictTargets(targets.to_vec()), + }; + let HostCallResponse::RefuseFallbacks(refused) = call_host(call).await else { + unreachable!("{CHECKED}"); + }; + refused + } + + /// A zero sleep is asked too: Python's retry loop awaits `asyncio.sleep(0)` between sibling + /// retries, which yields to the event loop. + async fn sleep(&self, duration: Duration) { + let call = HostCall::Sleep { + seconds: duration.as_secs_f64(), + }; + let HostCallResponse::Slept = call_host(call).await else { + unreachable!("{CHECKED}"); + }; + } + + /// Kept for Python only when it acts on it: an exhausted retry loop is stamped on the + /// error Python raises, a fallback hop tells it which target a failed walk ended on, and a + /// cooldown or a pick is reported to the host's Prometheus and service loggers (a pick also + /// joins the router cache's per-deployment count, a failed walk the router's error log). + fn observe(&self, event: &RouterEvent) { + if !matches!( + event, + RouterEvent::RetriesExhausted { .. } + | RouterEvent::Retry { .. } + | RouterEvent::Fallback { .. } + | RouterEvent::FallingBack { .. } + | RouterEvent::OrderChainFailed { .. } + | RouterEvent::FallbacksFailed { .. } + | RouterEvent::LevelFailed { .. } + | RouterEvent::PickFailed { .. } + | RouterEvent::RetriesOpened { .. } + | RouterEvent::RetryBudget { .. } + | RouterEvent::MockFailed { .. } + | RouterEvent::RetryLogged { .. } + | RouterEvent::RetryPolicyRaised { .. } + | RouterEvent::RetryCountRaised { .. } + | RouterEvent::FallbackDone { .. } + | RouterEvent::Blocked { .. } + | RouterEvent::CooldownSet { .. } + | RouterEvent::Selected { .. } + | RouterEvent::PickStarted { .. } + ) { + return; + } + CURRENT.with_borrow(|current| { + if let Some(exchange) = current { + lock(exchange).events.push(event.clone()); + } + }); + } + + fn cache(&self) -> Arc { + Arc::new(StepCache) + } + + async fn secrets(&self, index: usize, refs: &[SecretRef]) -> Result, Error> { + let HostCallResponse::Secrets(values) = call_host(HostCall::Secrets { + index, + refs: refs.to_vec(), + }) + .await + else { + unreachable!("{CHECKED}"); + }; + Ok(values) + } + + async fn validate(&self, build: &BuildAsk<'_>) -> Result { + let HostCallResponse::Validated(built) = + call_host(build_host_call(StepKind::Validate, build, None)).await + else { + unreachable!("{CHECKED}"); + }; + Ok(built) + } + + async fn provider(&self, build: &BuildAsk<'_>) -> Result { + let HostCallResponse::Provided(provided) = + call_host(build_host_call(StepKind::Provider, build, None)).await + else { + unreachable!("{CHECKED}"); + }; + Ok(provided) + } + + async fn added(&self, build: &BuildAsk<'_>, provider: Option<&str>) -> Result { + let HostCallResponse::Added(built) = + call_host(build_host_call(StepKind::Added, build, provider)).await + else { + unreachable!("{CHECKED}"); + }; + Ok(built) + } + + async fn model_info(&self, lookup: &ModelInfoLookup) -> Result { + let lookup = serde_json::to_value(lookup).unwrap_or(Value::Null); + let HostCallResponse::ModelInfo(response) = call_host(HostCall::ModelInfo { lookup }).await + else { + unreachable!("{CHECKED}"); + }; + Ok(response) + } +} + +fn build_host_call(kind: StepKind, build: &BuildAsk<'_>, provider: Option<&str>) -> HostCall { + HostCall::Build { + kind, + path: build.path, + index: build.index, + entry: Value::Object(build.entry.clone()), + declared_id: build.declared_id.map(str::to_owned), + duplicate_id: build.duplicate_id, + provider: provider.map(str::to_owned), + } +} + +type CallFuture = Pin> + Send>>; + +/// Poll `future` once with `exchange` bound. +fn poll_bound(future: Pin<&mut F>, exchange: &Shared) -> Poll { + let _bound = Bound::new(exchange); + future.poll(&mut Context::from_waker(Waker::noop())) +} + +/// Drives one router call a step at a time. +pub(crate) struct Driver { + future: Option, + exchange: Shared, + pending_call: Option, +} + +impl Driver { + pub(crate) fn new(call: impl Future> + Send + 'static) -> Self { + Self { + future: Some(Box::pin(call)), + exchange: Shared::default(), + pending_call: None, + } + } + + /// The host call the router call is blocked on, if it made one. + pub(crate) fn pending_call(&self) -> Option<&HostCall> { + self.pending_call.as_ref() + } + + /// The events reported since the last call. + pub(crate) fn take_events(&self) -> Vec { + std::mem::take(&mut lock(&self.exchange).events) + } + + /// Hand back the response to the pending host call (`None` on the first call) and run the call to + /// its next step or its end. + pub(crate) fn resume( + &mut self, + response: Option, + ) -> Result { + let Some(future) = self.future.as_mut() else { + return Err(DriveError::Finished); + }; + match (&self.pending_call, response) { + (None, None) => {} + (None, Some(_)) => return Err(DriveError::UnexpectedResponse), + (Some(call), None) => { + return Err(DriveError::MissingResponse { + expected: call.kind(), + }); + } + (Some(call), Some(response)) => { + response.check(call)?; + lock(&self.exchange).response = Some(response); + } + } + self.pending_call = None; + match poll_bound(future.as_mut(), &self.exchange) { + Poll::Ready(result) => { + self.future = None; + Ok(match result { + Ok(finish) => Step::Done(finish), + Err(error) => Step::Failed(error), + }) + } + Poll::Pending => { + let Some(call) = lock(&self.exchange).call.take() else { + self.future = None; + return Err(DriveError::Stalled); + }; + self.pending_call = Some(call.clone()); + Ok(Step::HostCall(call)) + } + } + } +} + +#[cfg(test)] +mod tests { + use std::{ + collections::HashMap, + time::{SystemTime, UNIX_EPOCH}, + }; + + use litellm_router_core::{ + FailureKind, Op, Router, RouterConfig, RouterDynamicConfig, SeededRandomness, + }; + use rstest::{fixture, rstest}; + use serde_json::{Value, json}; + + use super::*; + + fn router(config: Value, dynamic: Value) -> Router { + let config: RouterConfig = serde_json::from_value(config).unwrap(); + let dynamic: RouterDynamicConfig = serde_json::from_value(dynamic).unwrap(); + Router::builder(config, dynamic, Arc::new(StepHost)) + .randomness(Arc::new(SeededRandomness::new(7))) + .build() + .unwrap() + } + + fn deployment(group: &str, id: &str) -> Value { + json!({"model_name": group, "litellm_params": {"model": format!("openai/{id}")}, "model_info": {"id": id}}) + } + + fn start(router: &Router, model: &str) -> Driver { + let router = router.clone(); + let request = RouterRequest { + model: model.to_owned(), + ..RouterRequest::default() + }; + Driver::new(async move { + router + .call(Op::from("acompletion"), request) + .await + .map(Finish::Call) + }) + } + + fn rate_limited() -> HostCallResponse { + HostCallResponse::Attempt(Err(AttemptError::new(FailureKind::RateLimit, Some(429)))) + } + + /// A dict-backed stand-in for Python's `DualCache` that answers `cache` steps and keeps + /// every one it answered. + #[derive(Default)] + struct Store { + entries: HashMap, + calls: Vec, + } + + impl Store { + fn respond(&mut self, call: &HostCall) -> HostCallResponse { + let HostCall::Cache { + op, + key, + keys, + value, + .. + } = call + else { + panic!("not a cache step: {call:?}"); + }; + self.calls.push(call.clone()); + let key = key.clone().unwrap_or_default(); + HostCallResponse::Cache(match op { + CacheOp::Get => CacheOutcome::Value(self.entries.get(&key).cloned()), + CacheOp::Set => { + self.entries.insert(key, value.clone().unwrap()); + CacheOutcome::Stored + } + CacheOp::Increment => { + let current = self.entries.get(&key).and_then(Value::as_f64); + let next = + current.unwrap_or(0.0) + value.as_ref().and_then(Value::as_f64).unwrap(); + self.entries.insert(key, amount_value(next)); + CacheOutcome::Number(next) + } + CacheOp::IncrementUsage => { + let amounts = value.as_ref().and_then(Value::as_array).unwrap(); + for (key, amount) in keys.iter().flatten().zip(amounts) { + let current = self.entries.get(key).and_then(Value::as_f64); + let next = current.unwrap_or(0.0) + amount.as_f64().unwrap(); + self.entries.insert(key.clone(), amount_value(next)); + } + CacheOutcome::Stored + } + CacheOp::BatchGet => CacheOutcome::Values( + keys.iter() + .flatten() + .map(|key| self.entries.get(key).cloned()) + .collect(), + ), + CacheOp::Delete => { + self.entries.remove(&key); + CacheOutcome::Stored + } + }) + } + + /// Resume with `response`, answering every cache step on the way, up to the next other + /// step. + fn resume(&mut self, driver: &mut Driver, response: Option) -> Step { + let mut step = driver.resume(response).unwrap(); + while let Step::HostCall(call @ HostCall::Cache { .. }) = &step { + let response = self.respond(call); + step = driver.resume(Some(response)).unwrap(); + } + step + } + } + + #[fixture] + fn pair() -> Router { + router( + json!({"model_list": [deployment("primary", "a1"), deployment("primary", "a2")]}), + json!({}), + ) + } + + #[rstest] + fn a_first_attempt_success_is_done(pair: Router) { + let mut store = Store::default(); + let mut driver = start(&pair, "primary"); + let Step::HostCall(HostCall::Attempt { + deployment_id, + model_group, + retry: 0, + fallback_depth: 0, + .. + }) = store.resume(&mut driver, None) + else { + panic!("the call should start with an attempt"); + }; + assert_eq!(model_group, "primary"); + let Step::Done(Finish::Call(response)) = + store.resume(&mut driver, Some(HostCallResponse::Attempt(Ok(12)))) + else { + panic!("a successful attempt should finish the call"); + }; + assert_eq!(response.deployment_id, deployment_id); + assert_eq!(response.output.total_tokens, 12); + assert_eq!( + (response.attempted_retries, response.attempted_fallbacks), + (0, 0) + ); + assert!(matches!(driver.resume(None), Err(DriveError::Finished))); + } + + #[rstest] + fn cooldown_reads_are_cache_steps(pair: Router) { + let mut driver = start(&pair, "primary"); + let step = driver.resume(None).unwrap(); + let Step::HostCall(HostCall::Cache { + op: CacheOp::BatchGet, + key: None, + keys: Some(keys), + value: None, + ttl: None, + local: false, + }) = step + else { + panic!("the call should read cooldowns first, got {step:?}"); + }; + assert_eq!(keys, ["deployment:a1:cooldown", "deployment:a2:cooldown"]); + } + + #[rstest] + #[case::lone_deployment_backs_off(vec![deployment("primary", "a1")], true)] + #[case::sibling_retries_at_once( + vec![deployment("primary", "a1"), deployment("primary", "a2")], + false + )] + fn a_rate_limit_retries_after_the_backoff( + #[case] model_list: Vec, + #[case] backs_off: bool, + ) { + let router = router(json!({"model_list": model_list}), json!({"num_retries": 1})); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + assert_eq!( + store.resume(&mut driver, None).unwrap_kind(), + StepKind::Attempt + ); + let step = store.resume(&mut driver, Some(rate_limited())); + let Step::HostCall(HostCall::Sleep { seconds }) = step else { + panic!("a retry should sleep first, got {step:?}"); + }; + assert_eq!(seconds > 0.0, backs_off, "slept {seconds}s"); + let step = store.resume(&mut driver, Some(HostCallResponse::Slept)); + assert!( + matches!(step, Step::HostCall(HostCall::Attempt { retry: 1, .. })), + "expected the retry, got {step:?}" + ); + let Step::Done(Finish::Call(response)) = + store.resume(&mut driver, Some(HostCallResponse::Attempt(Ok(1)))) + else { + panic!("the retry should finish the call"); + }; + assert_eq!(response.attempted_retries, 1); + } + + #[rstest] + fn a_cooldown_python_wrote_is_read_back_by_the_next_call() { + let pair_without_retries = router( + json!({"model_list": [deployment("primary", "a1"), deployment("primary", "a2")]}), + json!({"num_retries": 0}), + ); + let mut store = Store::default(); + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_secs_f64(); + store.entries.insert( + "deployment:a1:cooldown".into(), + json!({"exception_received": "slow down", "status_code": "429", "timestamp": now, "cooldown_time": 60}), + ); + for _ in 0..8 { + let mut call = start(&pair_without_retries, "primary"); + let Step::HostCall(HostCall::Attempt { deployment_id, .. }) = + store.resume(&mut call, None) + else { + panic!("the call should attempt the healthy deployment"); + }; + assert_eq!(deployment_id, "a2", "the cooled-down deployment was picked"); + } + } + + /// The host only runs the attempt; the core records its outcome (cooldowns, fail counters, + /// usage) through the host's cache. + #[rstest] + #[case::failure(rate_limited())] + #[case::success(HostCallResponse::Attempt(Ok(7)))] + fn attempt_outcomes_are_recorded_by_the_core(#[case] outcome: HostCallResponse) { + let tracked = |id: &str| { + json!({ + "model_name": "primary", + "litellm_params": {"model": format!("openai/{id}"), "tpm": 100, "cooldown_time": 7}, + "model_info": {"id": id, "allowed_fails": 3}, + }) + }; + let router = router( + json!({"model_list": [tracked("a1"), tracked("a2")]}), + json!({"num_retries": 0}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + store.resume(&mut driver, None); + let step = store.resume(&mut driver, Some(outcome)); + assert!( + matches!(step, Step::Done(_) | Step::Failed(_)), + "got {step:?}" + ); + let writes: Vec<&HostCall> = store + .calls + .iter() + .filter(|call| { + matches!(call, HostCall::Cache { op, .. } if !matches!(op, CacheOp::Get | CacheOp::BatchGet)) + }) + .collect(); + assert!(!writes.is_empty(), "the core wrote no outcome"); + } + + #[rstest] + fn a_failed_group_falls_back_once_python_allows_it() { + let router = router( + json!({"model_list": [deployment("primary", "a1"), deployment("backup", "b1")]}), + json!({"fallbacks": [{"primary": ["backup"]}]}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + assert_eq!( + store.resume(&mut driver, None).unwrap_kind(), + StepKind::Attempt + ); + let bad_request = + HostCallResponse::Attempt(Err(AttemptError::new(FailureKind::BadRequest, Some(400)))); + let step = store.resume(&mut driver, Some(bad_request)); + assert!( + matches!(&step, Step::HostCall(HostCall::AllowFallback { target }) if target == "backup"), + "expected the fallback check, got {step:?}" + ); + let step = store.resume(&mut driver, Some(HostCallResponse::AllowFallback(true))); + assert!( + matches!(&step, Step::HostCall(HostCall::Attempt { deployment_id, fallback_depth: 1, .. }) if deployment_id == "b1"), + "expected the fallback attempt, got {step:?}" + ); + let Step::Done(Finish::Call(response)) = + store.resume(&mut driver, Some(HostCallResponse::Attempt(Ok(3)))) + else { + panic!("the fallback should finish the call"); + }; + assert_eq!(response.model_group, "backup"); + assert_eq!(response.attempted_fallbacks, 1); + } + + #[rstest] + fn a_refused_fallback_fails_with_the_attempt_error() { + let router = router( + json!({"model_list": [deployment("primary", "a1"), deployment("backup", "b1")]}), + json!({"fallbacks": [{"primary": ["backup"]}]}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + store.resume(&mut driver, None); + let bad_request = + HostCallResponse::Attempt(Err(AttemptError::new(FailureKind::BadRequest, Some(400)))); + store.resume(&mut driver, Some(bad_request)); + let Step::Failed(error) = + store.resume(&mut driver, Some(HostCallResponse::AllowFallback(false))) + else { + panic!("a refused fallback should fail the call"); + }; + assert_eq!( + error.attempt().map(|a| a.kind), + Some(FailureKind::BadRequest) + ); + } + + #[rstest] + #[case::none_while_waiting(None, StepKind::Attempt)] + #[case::wrong_kind(Some(HostCallResponse::Slept), StepKind::Attempt)] + #[case::cache_outcome(Some(HostCallResponse::Cache(CacheOutcome::Stored)), StepKind::Attempt)] + fn a_mismatched_outcome_is_rejected_and_the_step_kept( + pair: Router, + #[case] response: Option, + #[case] expected: StepKind, + ) { + let mut store = Store::default(); + let mut driver = start(&pair, "primary"); + store.resume(&mut driver, None); + let error = driver.resume(response).unwrap_err(); + assert!(error.is_request(), "{error}"); + assert_eq!(driver.pending_call().map(HostCall::kind), Some(expected)); + assert!(matches!( + store.resume(&mut driver, Some(HostCallResponse::Attempt(Ok(1)))), + Step::Done(_) + )); + } + + #[rstest] + #[case::wrong_op(HostCallResponse::Cache(CacheOutcome::Stored))] + #[case::number_for_batch(HostCallResponse::Cache(CacheOutcome::Number(1.0)))] + #[case::short_batch(HostCallResponse::Cache(CacheOutcome::Values(vec![None])))] + #[case::not_a_cache_outcome(HostCallResponse::Slept)] + fn a_mismatched_cache_outcome_is_rejected_and_the_step_kept( + pair: Router, + #[case] response: HostCallResponse, + ) { + let mut store = Store::default(); + let mut driver = start(&pair, "primary"); + let step = driver.resume(None).unwrap(); + let Step::HostCall( + call @ HostCall::Cache { + op: CacheOp::BatchGet, + .. + }, + ) = &step + else { + panic!("expected the cooldown read, got {step:?}"); + }; + let error = driver.resume(Some(response)).unwrap_err(); + assert!(error.is_request(), "{error}"); + assert_eq!(driver.pending_call(), Some(call)); + let response = store.respond(call); + assert_eq!( + store.resume(&mut driver, Some(response)).unwrap_kind(), + StepKind::Attempt + ); + } + + #[rstest] + fn a_strategy_python_runs_asks_for_a_selection_and_keeps_the_pick() { + let router = router( + json!({ + "model_list": [deployment("primary", "a1"), deployment("primary", "a2")], + "routing_strategy": "least-busy", + }), + json!({}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + let step = store.resume(&mut driver, None); + let Step::HostCall(HostCall::Select { + stage: Stage::Strategy { strategy, records }, + candidates, + .. + }) = step + else { + panic!("a host strategy should ask for a selection, got {step:?}"); + }; + assert_eq!(strategy, "least-busy"); + assert_eq!(candidates, ["a1", "a2"]); + assert!(records.is_empty(), "distinct ids send no records"); + let step = store.resume( + &mut driver, + Some(HostCallResponse::Select(vec!["a2".to_owned()])), + ); + assert!( + matches!(&step, Step::HostCall(HostCall::Attempt { deployment_id, .. }) if deployment_id == "a2"), + "expected the picked deployment, got {step:?}" + ); + } + + #[rstest] + fn a_pick_among_deployments_sharing_an_id_is_answered_by_position() { + let mut second = deployment("primary", "a1"); + second["litellm_params"]["api_base"] = json!("https://second.example"); + let router = router( + json!({ + "model_list": [deployment("primary", "a1"), second], + "routing_strategy": "least-busy", + }), + json!({}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + let step = store.resume(&mut driver, None); + let Step::HostCall(HostCall::Select { + stage: Stage::Strategy { records, .. }, + .. + }) = step + else { + panic!("a host strategy should ask for a selection, got {step:?}"); + }; + assert_eq!( + records.len(), + 2, + "each candidate sharing an id sends its record" + ); + let step = store.resume(&mut driver, Some(HostCallResponse::Positions(vec![1]))); + assert!( + matches!(&step, Step::HostCall(HostCall::Attempt { deployment, .. }) + if deployment["litellm_params"]["api_base"] == "https://second.example"), + "expected the second entry, got {step:?}" + ); + } + + #[rstest] + fn a_selection_is_driven_through_cache_steps(pair: Router) { + let request = RouterRequest { + model: "primary".to_owned(), + ..RouterRequest::default() + }; + let mut store = Store::default(); + let mut driver = Driver::new(async move { + pair.select(&Op::from("acompletion"), &request) + .await + .map(|deployment| Finish::Selected { + deployment_id: deployment.id.clone(), + model_group: deployment.model_name.clone(), + deployment: deployment_record(&deployment), + }) + }); + let Step::Done(Finish::Selected { + deployment_id, + model_group, + .. + }) = store.resume(&mut driver, None) + else { + panic!("a selection should finish with the pick"); + }; + assert!(["a1", "a2"].contains(&deployment_id.as_str())); + assert_eq!(model_group, "primary"); + assert!(matches!( + store.calls.as_slice(), + [HostCall::Cache { + op: CacheOp::BatchGet, + .. + }] + )); + } + + #[rstest] + fn dict_targets_keep_their_api_key_out_of_debug() { + let target = json!({"model": "b", "api_key": "credential-value"}); + let Value::Object(target) = target else { + unreachable!("a JSON object"); + }; + let shown = format!("{:?}", DictTargets(vec![target])); + assert!(!shown.contains("credential-value")); + assert!(shown.contains("[REDACTED]")); + } + + #[rstest] + fn a_chain_with_a_dict_target_asks_python_before_any_hop() { + let router = router( + json!({"model_list": [deployment("primary", "a1"), deployment("backup", "b1")]}), + json!({"fallbacks": [{"primary": [{"model": "backup", "api_key": "k"}]}]}), + ); + let mut store = Store::default(); + let mut driver = start(&router, "primary"); + store.resume(&mut driver, None); + let bad_request = + HostCallResponse::Attempt(Err(AttemptError::new(FailureKind::BadRequest, Some(400)))); + let step = store.resume(&mut driver, Some(bad_request)); + assert!( + matches!(&step, Step::HostCall(HostCall::RefuseFallbacks { level: 0, targets }) if targets.0.len() == 1), + "expected the chain's check, got {step:?}" + ); + let Step::Failed(error) = + store.resume(&mut driver, Some(HostCallResponse::RefuseFallbacks(true))) + else { + panic!("a refused chain should fail the call"); + }; + assert_eq!( + error.attempt().map(|a| a.kind), + Some(FailureKind::BadRequest) + ); + } + + trait StepKindOf { + fn unwrap_kind(self) -> StepKind; + } + + impl StepKindOf for Step { + fn unwrap_kind(self) -> StepKind { + match self { + Step::HostCall(call) => call.kind(), + Step::Done(_) => StepKind::Done, + Step::Failed(_) => StepKind::Failed, + } + } + } +} diff --git a/litellm-rust/crates/python-bridge/src/router/mod.rs b/litellm-rust/crates/python-bridge/src/router/mod.rs new file mode 100644 index 00000000000..6b58d63a714 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/router/mod.rs @@ -0,0 +1,2087 @@ +//! `RouterCoreHandle`: router-core behind a step-driven handle. Python drives each call through +//! [`RouterCall::next`], running every attempt, sleep, fallback check, selection and cache op the +//! call loop asks for and handing the outcome back; Rust never awaits Python. `select` is driven +//! the same way, since reading cooldowns is a cache step. + +mod config; +mod host; +mod retries; +mod stream; + +use std::{ + sync::{Arc, Mutex, PoisonError}, + time::{Duration, SystemTime}, +}; + +use litellm_error::Report; +use litellm_host_python::{from_py_argument, json_loads, to_py}; +use litellm_router_core::lookup::{ListScope, Lookup, Row}; +use litellm_router_core::{ + AttemptError, Blocked, BuildPath, Built, Deployment, DeploymentSpec, Error, FailureKind, + FailureReport, Op, OutcomeCallbacks, PreRouted, Provided, RetryCause, Router, RouterEvent, + SecretAt, SuccessReport, + build::Install, + health::{UsageMetric, current_minute, usage_key}, + registry::{DeploymentFilter, ResolveKind, ResolveScope}, + strategy::routing_context, +}; +use pyo3::{ + exceptions::{PyRuntimeError, PyValueError}, + prelude::*, + types::{PyBool, PyDict, PyList, PyString, PyTuple}, +}; +use serde_json::Value; + +use self::host::{ + CacheOp, CacheOutcome, DriveError, Driver, Finish, HostCall, HostCallResponse, + PythonRandomness, Stage, Step, StepHost, StepKind, +}; +use crate::errors::RustBridgeDeclined; + +/// The `error_kind` of a `failed` step. +#[derive(Clone, Copy, Debug, strum::IntoStaticStr)] +enum ErrorKind { + #[strum(serialize = "config")] + Config, + #[strum(serialize = "unknown_model")] + UnknownModel, + #[strum(serialize = "no_healthy_deployments")] + NoHealthyDeployments, + #[strum(serialize = "attempt")] + Attempt, + #[strum(serialize = "cache")] + Cache, + #[strum(serialize = "retired")] + Retired, + #[strum(serialize = "paused")] + Paused, + #[strum(serialize = "not_pass_through")] + NotPassThrough, + #[strum(serialize = "no_pass_through_deployments")] + NoPassThroughDeployments, + #[strum(serialize = "only_markers")] + OnlyMarkers, + #[strum(serialize = "endpoint_not_supported")] + EndpointNotSupported, + #[strum(serialize = "ptu_reserved")] + PtuReserved, + #[strum(serialize = "context_window_exceeded")] + ContextWindowExceeded, + #[strum(serialize = "unsupported_params")] + UnsupportedParams, + #[strum(serialize = "routing_plugins_left_nothing")] + RoutingPluginsLeftNothing, + #[strum(serialize = "routing_plugins_need_async")] + RoutingPluginsNeedAsync, + #[strum(serialize = "multiple_teams")] + MultipleTeams, + #[strum(serialize = "reserved")] + Reserved, + #[strum(serialize = "invalid_router_weights")] + InvalidRouterWeights, + #[strum(serialize = "non_numeric_weight")] + NonNumericWeight, + #[strum(serialize = "missing_routing_group")] + MissingRoutingGroup, + #[strum(serialize = "default_deployment_raises")] + DefaultDeploymentRaises, + #[strum(serialize = "index_out_of_range")] + IndexOutOfRange, + #[strum(serialize = "model_info")] + ModelInfo, +} + +impl From<&Error> for ErrorKind { + fn from(error: &Error) -> Self { + match error { + Error::Config(_) => Self::Config, + Error::UnknownModel { .. } => Self::UnknownModel, + Error::NoHealthyDeployments { .. } => Self::NoHealthyDeployments, + Error::Attempt(_) => Self::Attempt, + Error::Cache(_) => Self::Cache, + Error::Retired => Self::Retired, + Error::Paused { .. } => Self::Paused, + Error::NotPassThrough { .. } => Self::NotPassThrough, + Error::NoPassThroughDeployments { .. } => Self::NoPassThroughDeployments, + Error::OnlyMarkers { .. } => Self::OnlyMarkers, + Error::EndpointNotSupported { .. } => Self::EndpointNotSupported, + Error::PtuReserved { .. } => Self::PtuReserved, + Error::ContextWindowExceeded { .. } => Self::ContextWindowExceeded, + Error::UnsupportedParams { .. } => Self::UnsupportedParams, + Error::RoutingPluginsLeftNothing { .. } => Self::RoutingPluginsLeftNothing, + Error::RoutingPluginsNeedAsync => Self::RoutingPluginsNeedAsync, + Error::MultipleTeams { .. } => Self::MultipleTeams, + Error::Reserved { .. } => Self::Reserved, + Error::InvalidRouterWeights => Self::InvalidRouterWeights, + Error::NonNumericWeight { .. } => Self::NonNumericWeight, + Error::MissingRoutingGroup { .. } => Self::MissingRoutingGroup, + Error::DefaultDeploymentRaises { .. } => Self::DefaultDeploymentRaises, + Error::IndexOutOfRange { .. } => Self::IndexOutOfRange, + Error::ModelInfo { .. } => Self::ModelInfo, + } + } +} + +fn config_error_to_pyerr(error: config::Error) -> PyErr { + let message = Report::new(&error).to_string(); + if error.declined() { + RustBridgeDeclined::new_err(message) + } else { + PyValueError::new_err(message) + } +} + +fn drive_error_to_pyerr(error: DriveError) -> PyErr { + let message = Report::new(&error).to_string(); + if error.is_request() { + PyValueError::new_err(message) + } else { + PyRuntimeError::new_err(message) + } +} + +/// A document Python passes as a mapping or as its JSON text. +fn document(value: &Bound<'_, PyAny>) -> PyResult { + match value.cast::() { + Ok(text) => { + let parsed = json_loads(value.py(), text.to_str()?.as_bytes())?; + from_py_argument(parsed.bind(value.py())) + } + Err(_) => from_py_argument(value), + } +} + +/// The `OutcomeCallbacks` a reported outcome came through; none is a report no callback makes. +fn outcome_callbacks(value: Option<&Bound<'_, PyAny>>) -> PyResult { + let Some(value) = value else { + return Ok(OutcomeCallbacks::default()); + }; + serde_json::from_value(document(value)?) + .map_err(|error| PyValueError::new_err(error.to_string())) +} + +/// A router-core router whose host operations Python runs. +#[pyclass(frozen)] +pub(crate) struct RouterCoreHandle { + router: Router, +} + +#[pymethods] +impl RouterCoreHandle { + #[new] + fn new(config: &Bound<'_, PyAny>, dynamic: &Bound<'_, PyAny>) -> PyResult { + let (config, dynamic) = (document(config)?, document(dynamic)?); + let build = || { + let config = config::parse_router_config(config)?; + let dynamic = config::parse_dynamic_config(dynamic)?; + Ok(Router::builder(config, dynamic, Arc::new(StepHost)) + .randomness(Arc::new(PythonRandomness)) + .build()?) + }; + build() + .map(|router| Self { router }) + .map_err(config_error_to_pyerr) + } + + /// Start a call; nothing runs until the first `next(None)`. + fn start(&self, op: &str, request: &Bound<'_, PyAny>) -> PyResult { + let op = Op::new(op); + let request = + config::parse_request(document(request)?, None).map_err(config_error_to_pyerr)?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + router.call(op, request).await.map(Finish::Call) + })) + } + + /// Pick the deployment `model` routes to now, without calling it. Driven like `start`: its + /// steps are the cache reads and selections the pick needs, and it ends with `done` + /// (`deployment_id`, `model_group`) or `failed`. + fn select(&self, model: &str, request: &Bound<'_, PyAny>) -> PyResult { + let request = config::parse_request(document(request)?, Some(model)) + .map_err(config_error_to_pyerr)?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + let deployment = router.select(&Op::from("acompletion"), &request).await?; + Ok(Finish::Selected { + deployment_id: deployment.id.clone(), + model_group: deployment.model_name.clone(), + deployment: host::deployment_record(&deployment), + }) + })) + } + + /// List the deployments a pick for `model` would choose among, driven like `select`. + fn healthy(&self, model: &str, request: &Bound<'_, PyAny>) -> PyResult { + let request = config::parse_request(document(request)?, Some(model)) + .map_err(config_error_to_pyerr)?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + let candidates = router.healthy(&Op::from("acompletion"), &request).await?; + // A deployment named by its id is listed alone, as a dict (`_core_async_get_healthy_deployments`). + let single = candidates.kind.is_single() && candidates.healthy.len() == 1; + let deployments = candidates.healthy; + Ok(Finish::Healthy { + single, + deployment_ids: deployments.iter().map(|d| d.id.clone()).collect(), + deployments: deployments + .iter() + .map(|d| host::deployment_record(d)) + .collect(), + }) + })) + } + + /// Apply `{"upsert": [deployments], "delete": [ids], "settings": {dynamic document}}` and + /// return the new registry version. + fn apply(&self, changes: &Bound<'_, PyAny>) -> PyResult { + let changes = config::parse_changes(document(changes)?).map_err(config_error_to_pyerr)?; + self.router + .apply(changes) + .map_err(|error| config_error_to_pyerr(error.into())) + } + + /// Build `entries` (model_list entries) through the host, driven like `start`: its steps + /// are `secrets`, `validate`, `provider` and `added`, and it ends with `done` (`built`: one + /// `{"index", "id", "provider"}` per deployment installed). `path` is `create` + /// (`set_model_list`) or `add` (`add_deployment`); `install` is `replace`, `append` or + /// `upsert`. + #[pyo3(signature = (entries, path, resolve_secrets, install))] + fn build( + &self, + entries: &Bound<'_, PyAny>, + path: &str, + resolve_secrets: bool, + install: &str, + ) -> PyResult { + let entries: Vec> = + serde_json::from_value(document(entries)?) + .map_err(|error| PyValueError::new_err(format!("bad model_list: {error}")))?; + let path: BuildPath = path + .parse() + .map_err(|_| PyValueError::new_err(format!("`{path}` is not a build path")))?; + let install: Install = install + .parse() + .map_err(|_| PyValueError::new_err(format!("`{install}` is not a build install")))?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + router + .build(path, entries, resolve_secrets, install) + .await + .map(Finish::Built) + })) + } + + /// `get_router_model_info` for the deployment `query` describes (a + /// `RouterModelInfoQuery` document), driven like `start`: its steps are `model_info` + /// lookups, and it ends with `done` (`info`: the merged map or None, `user_keys`: the keys + /// whose values are the deployment's own). + fn router_model_info(&self, query: &Bound<'_, PyAny>) -> PyResult { + let query: litellm_router_core::model_info::RouterModelInfoQuery = + serde_json::from_value(document(query)?) + .map_err(|error| PyValueError::new_err(format!("bad model-info query: {error}")))?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + router + .router_model_info(&query) + .await + .map(Finish::RouterModelInfo) + })) + } + + /// `get_deployment_model_info` for deployment `model_id` serving `model_name`, driven like + /// `router_model_info`; it ends with `done` (`info`: the merged map or None). + fn deployment_model_info( + &self, + model_id: &str, + model_name: &Bound<'_, PyAny>, + ) -> PyResult { + let model_id = model_id.to_owned(); + let model_name = from_py_argument::(model_name)?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + router + .deployment_model_info(&model_id, &model_name) + .await + .map(Finish::DeploymentModelInfo) + })) + } + + /// `_set_model_group_info` for the deployments `query` names (a `GroupInfoQuery` document), + /// driven like `router_model_info`; it ends with `done` (`value`: None when no deployment + /// matched, else `{"assignments": {field: value}}`, what the merge set on the group the host + /// built from its first deployment). + fn model_group_info(&self, query: &Bound<'_, PyAny>) -> PyResult { + let query: litellm_router_core::group_info::GroupInfoQuery = + serde_json::from_value(document(query)?) + .map_err(|error| PyValueError::new_err(format!("bad group-info query: {error}")))?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + let info = router.model_group_info(&query).await?; + Ok(Finish::Value(info.map_or( + Value::Null, + |info| serde_json::json!({"assignments": info.assignments}), + ))) + })) + } + + /// `get_configured_token_limits` for the deployment with `id` and its `configured` model_info, + /// driven like `router_model_info`; it ends with `done` (`value`: `[max_input, max_output]`). + fn configured_token_limits( + &self, + id: &Bound<'_, PyAny>, + configured: &Bound<'_, PyAny>, + ) -> PyResult { + let id = from_py_argument::(id)?; + let Value::Object(configured) = document(configured)? else { + return Err(PyValueError::new_err( + "the configured model_info must be a dict", + )); + }; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + let (input, output) = router.configured_token_limits(id, &configured).await?; + Ok(Finish::Value(serde_json::json!([input, output]))) + })) + } + + /// `get_model_listing_info` over the `deployments` a listed name names, driven like + /// `router_model_info`; it ends with `done` (`value`: `{"cost_map_keys", "max_input_tokens", + /// "max_output_tokens"}`). + fn model_listing_info(&self, deployments: &Bound<'_, PyAny>) -> PyResult { + let deployments: Vec = from_py_argument(deployments)?; + let router = self.router.clone(); + Ok(RouterCall::new(async move { + let listing = router.model_listing_info(&deployments).await?; + Ok(Finish::Value(serde_json::json!({ + "cost_map_keys": listing.cost_map_keys, + "max_input_tokens": listing.max_input_tokens, + "max_output_tokens": listing.max_output_tokens, + }))) + })) + } + + /// `get_configured_mode` / `get_configured_display_name`: `value` when it is a string that is + /// not blank. + #[staticmethod] + fn configured_text(value: &Bound<'_, PyAny>) -> PyResult> { + let value = from_py_argument::(value).ok(); + Ok(litellm_router_core::group_info::configured_text( + value.as_ref(), + )) + } + + /// `get_configured_service_tiers`: each of `model_infos`' `service_tiers`, unvalidated. + #[staticmethod] + fn configured_service_tiers( + py: Python<'_>, + model_infos: &Bound<'_, PyAny>, + ) -> PyResult>> { + let model_infos: Vec> = from_py_argument(model_infos)?; + litellm_router_core::group_info::configured_service_tiers(&model_infos) + .iter() + .map(|tiers| to_py(py, tiers)) + .collect() + } + + /// The routing-cache key deployment `id` serving litellm `model` counts `metric` (`tpm`, + /// `rpm`, `itpm` or `otpm`) under in `minute` (`HH-MM`). + fn usage_key(&self, id: &str, model: &str, metric: &str, minute: &str) -> PyResult { + let metric: UsageMetric = metric + .parse() + .map_err(|_| PyValueError::new_err(format!("`{metric}` is not a usage metric")))?; + Ok(usage_key(id, model, metric, minute)) + } + + /// Keep the providers `other`'s build resolved for the deployments this router serves too. + fn adopt_providers(&self, other: &Self) { + self.router.adopt_providers(&other.router); + } + + fn registry_version(&self) -> u64 { + self.router.registry().version() + } + + /// The version the deployments before the newest appends last changed at: a list read at + /// or after it is current once it reads `deployments_from(len(list))`. + fn registry_rewritten(&self) -> u64 { + self.router.registry().rewritten() + } + + /// Record a failure the router did not see itself (a health check, an interceptor, a + /// logging callback), driven like `start`: its steps are cache ops, and it ends with `done` + /// (`cooled`, `cooldown_time`). `counts_request` also counts the failed request against the deployment's + /// per-minute requests, as an async call's failure callback does. `callbacks` is the logged + /// request's `OutcomeCallbacks`. + #[pyo3(signature = (id, failure, deployment_cooldown=false, count_only=false, requested_group=None, cooldown_time=None, uncounted=false, counts_request=false, usage_model=None, callbacks=None))] + #[allow(clippy::too_many_arguments)] + fn report_failure( + &self, + id: &str, + failure: &Bound<'_, PyDict>, + deployment_cooldown: bool, + count_only: bool, + requested_group: Option, + cooldown_time: Option, + uncounted: bool, + counts_request: bool, + usage_model: Option, + callbacks: Option<&Bound<'_, PyAny>>, + ) -> PyResult { + let failure = match attempt_outcome(failure)? { + Err(failure) => failure, + Ok(_) => { + return Err(PyValueError::new_err( + "a reported failure needs `ok: False`", + )); + } + }; + let router = self.router.clone(); + let id = id.to_owned(); + let report = FailureReport { + deployment_cooldown, + cooldown_time, + count_only, + requested_group, + uncounted, + counts_request, + usage_model, + callbacks: outcome_callbacks(callbacks)?, + }; + Ok(RouterCall::new(async move { + router + .report_failure(&id, &failure, report) + .await + .map(|cooldown| Finish::Reported { cooldown }) + })) + } + + /// Record a success a logging callback saw, driven like `report_failure`; it ends with + /// `done` (`key`: the tokens counter's key when usage was added, else None). + /// `usage_model` is the model its usage is keyed by, and `tracks_usage` the host's answer to + /// whether it counts any (see `Output::tracks_usage`). `callbacks`: see `report_failure`. + #[pyo3(signature = (id, total_tokens, requests, usage_model=None, tracks_usage=None, callbacks=None))] + fn report_success( + &self, + id: &str, + total_tokens: f64, + requests: f64, + usage_model: Option, + tracks_usage: Option, + callbacks: Option<&Bound<'_, PyAny>>, + ) -> PyResult { + let router = self.router.clone(); + let id = id.to_owned(); + let report = SuccessReport { + tracks_usage, + usage_model, + callbacks: outcome_callbacks(callbacks)?, + }; + Ok(RouterCall::new(async move { + router + .report_success(&id, total_tokens, requests, report) + .await + .map(|key| Finish::Counted { key }) + })) + } + + /// Count a success against `id`, configured or not; ends with `done` (`key`). `callbacks`: + /// see `report_failure`. + #[pyo3(signature = (id, callbacks=None))] + fn count_success( + &self, + id: &str, + callbacks: Option<&Bound<'_, PyAny>>, + ) -> PyResult { + let router = self.router.clone(); + let id = id.to_owned(); + let callbacks = outcome_callbacks(callbacks)?; + Ok(RouterCall::new(async move { + router + .count_success(&id, callbacks) + .await + .map(|key| Finish::Counted { key }) + })) + } + + /// The usage keys of this minute for every deployment of `model_group`: + /// `[(id, tpm_key, rpm_key)]`, or None when the group has no deployment. + fn usage_keys(&self, model_group: &str) -> Option> { + let minute = current_minute(SystemTime::now()); + let group = self.router.registry().group(model_group)?; + Some( + group + .iter() + .map(|deployment| { + let model = &deployment.params.model; + ( + deployment.id.clone(), + usage_key(&deployment.id, model, UsageMetric::Tpm, &minute), + usage_key(&deployment.id, model, UsageMetric::Rpm, &minute), + ) + }) + .collect(), + ) + } + + /// Every configured deployment, as `model_list` entries with their ids, in config order. + fn deployments<'py>(&self, py: Python<'py>) -> PyResult> { + let registry = self.router.registry(); + let records = registry + .deployments(&DeploymentFilter::default()) + .iter() + .map(|deployment| to_py(py, &host::deployment_record(deployment))) + .collect::>>()?; + PyList::new(py, records) + } + + /// The deployments from position `start` on, as `deployments()` lists them. + fn deployments_from<'py>(&self, py: Python<'py>, start: usize) -> PyResult> { + let registry = self.router.registry(); + let records = registry + .deployments_from(start) + .iter() + .map(|deployment| to_py(py, &host::deployment_record(deployment))) + .collect::>>()?; + PyList::new(py, records) + } + + /// The deployment with this id, or None. + fn deployment(&self, py: Python<'_>, id: &str) -> PyResult>> { + self.router + .registry() + .deployment(id) + .map(|deployment| to_py(py, &host::deployment_record(&deployment))) + .transpose() + } + + // -- registry lookups (router-core `lookup.rs`): rows the host reads its stored entries at -- + + /// `_core_get_all_deployments`, as rows. + #[pyo3(signature = (model_name, team_id=None, rename=None))] + fn lookup_all<'py>( + &self, + py: Python<'py>, + model_name: &str, + team_id: Option<&str>, + rename: Option<&str>, + ) -> PyResult> { + let registry = self.router.registry(); + rows_to_py(py, &Lookup::new(®istry).all(model_name, team_id, rename)) + } + + /// `_team_deployments_across_teams`: the rows, and how many teams own them. + fn lookup_across_teams<'py>( + &self, + py: Python<'py>, + model: &str, + ) -> PyResult<(Bound<'py, PyAny>, usize)> { + let registry = self.router.registry(); + let (rows, teams) = Lookup::new(®istry).across_teams(model); + Ok((rows_to_py(py, &rows)?, teams)) + } + + /// `_core_config_deployments`, as rows. + fn lookup_config<'py>(&self, py: Python<'py>) -> PyResult> { + let registry = self.router.registry(); + rows_to_py(py, &Lookup::new(®istry).config()) + } + + /// `_core_get_model_ids`: the ids, or the row the read raises at. + #[pyo3(signature = (model_name=None, exclude_team_models=false))] + fn lookup_model_ids<'py>( + &self, + py: Python<'py>, + model_name: Option<&str>, + exclude_team_models: bool, + ) -> PyResult<(Vec, Option>)> { + let registry = self.router.registry(); + match Lookup::new(®istry).model_ids(model_name, exclude_team_models) { + Ok(ids) => Ok((ids, None)), + Err(raw) => Ok((Vec::new(), Some(row_to_py(py, &raw)?))), + } + } + + /// `_core_get_model_list`, as rows. + #[pyo3(signature = (model_name, team_id, scope))] + fn lookup_model_list<'py>( + &self, + py: Python<'py>, + model_name: Option<&str>, + team_id: Option<&str>, + scope: &Bound<'_, PyAny>, + ) -> PyResult> { + let scope = list_scope(scope)?; + let registry = self.router.registry(); + rows_to_py( + py, + &Lookup::new(®istry).model_list(model_name, team_id, &scope), + ) + } + + /// `_core_get_model_names`, or the row the read raises at. + fn lookup_model_names<'py>( + &self, + py: Python<'py>, + team_id: Option<&str>, + scope: &Bound<'_, PyAny>, + ) -> PyResult<(Vec, Option>)> { + let scope = list_scope(scope)?; + let registry = self.router.registry(); + match Lookup::new(®istry).model_names(team_id, &scope) { + Ok(names) => Ok((names, None)), + Err(raw) => Ok((Vec::new(), Some(row_to_py(py, &raw)?))), + } + } + + /// `_core_get_model_access_groups`: `(group, names)` pairs in the order the groups appear, + /// or the row the read raises at. + fn lookup_access_groups<'py>( + &self, + py: Python<'py>, + model_name: Option<&str>, + access_group: Option<&str>, + team_id: Option<&str>, + scope: &Bound<'_, PyAny>, + ) -> PyResult<(AccessGroups, Option>)> { + let scope = list_scope(scope)?; + let registry = self.router.registry(); + match Lookup::new(®istry).access_groups(model_name, access_group, team_id, &scope) { + Ok(groups) => Ok((groups, None)), + Err(raw) => Ok((Vec::new(), Some(row_to_py(py, &raw)?))), + } + } + + /// Whether the whole list lists any row, and the name of its first row whose litellm model + /// is `model_id` (or ends in `/model_id` or `:model_id`); or the row the read raises at. + fn lookup_name_serving<'py>( + &self, + py: Python<'py>, + model_id: &str, + scope: &Bound<'_, PyAny>, + ) -> PyResult<(bool, Option, Option>)> { + let scope = list_scope(scope)?; + let registry = self.router.registry(); + match Lookup::new(®istry).name_serving(model_id, &scope) { + Ok((listed, name)) => Ok((listed, name, None)), + Err(raw) => Ok((false, None, Some(row_to_py(py, &raw)?))), + } + } + + /// Whether `model_names` (the group names) holds `name`. + fn has_model_name(&self, name: &str) -> bool { + self.router + .registry() + .host_indexes() + .model_names + .contains(&Value::String(name.to_owned())) + } + + /// `_core_team_model_has_alternatives`, or the row the read raises at. + fn lookup_team_model_has_alternatives<'py>( + &self, + py: Python<'py>, + id: &str, + ) -> PyResult<(bool, Option>)> { + let registry = self.router.registry(); + match Lookup::new(®istry).team_model_has_alternatives(id) { + Ok(found) => Ok((found, None)), + Err(raw) => Ok((false, Some(row_to_py(py, &raw)?))), + } + } + + /// The position of the group's first deployment the team may use, or the row the read + /// raises at. + fn lookup_group_usable_by_team<'py>( + &self, + py: Python<'py>, + model_group: &str, + team_id: Option<&str>, + ) -> PyResult<(Option, Option>)> { + let registry = self.router.registry(); + match Lookup::new(®istry).group_deployment_usable_by_team(model_group, team_id) { + Ok(position) => Ok((position, None)), + Err(raw) => Ok((None, Some(row_to_py(py, &raw)?))), + } + } + + /// The position of the team public name's first deployment, or the row the read raises at. + fn lookup_team_public_name<'py>( + &self, + py: Python<'py>, + model_id: &str, + team_id: &str, + ) -> PyResult<(Option, Option>)> { + let registry = self.router.registry(); + match Lookup::new(®istry).team_public_name_deployment(model_id, team_id) { + Ok(position) => Ok((position, None)), + Err(raw) => Ok((None, Some(row_to_py(py, &raw)?))), + } + } + + /// `_routable_deployments`, as rows. + fn lookup_routable<'py>( + &self, + py: Python<'py>, + model_group: &str, + team_id: Option<&str>, + ) -> PyResult> { + let registry = self.router.registry(); + rows_to_py(py, &Lookup::new(®istry).routable(model_group, team_id)) + } + + /// `_core_wildcard_route_deployments`: the team router's routes, else the global router's. + fn lookup_wildcard_route<'py>( + &self, + py: Python<'py>, + model: &str, + team_id: Option<&str>, + requests: Vec, + ) -> PyResult> { + let registry = self.router.registry(); + rows_to_py( + py, + &Lookup::new(®istry).wildcard_route(model, team_id, &requests), + ) + } + + /// `_core_get_candidate_model_ids_for_route` for the resolved name: the members' ids when it + /// names a callable routing group (`members`), else for a name that is no model group the + /// team's or the wildcard route's, else the group's; or the row the read raises at. + fn lookup_candidate_ids<'py>( + &self, + py: Python<'py>, + resolved: &str, + team_id: Option<&str>, + members: Option>, + requests: Vec, + ) -> PyResult<(Vec, Option>)> { + let registry = self.router.registry(); + let lookup = Lookup::new(®istry); + match lookup.candidate_ids(resolved, team_id, members.as_deref(), &requests) { + Ok(ids) => Ok((ids, None)), + Err(raw) => Ok((Vec::new(), Some(row_to_py(py, &raw)?))), + } + } + + /// Whether the id map holds `id` (`_has_model_id`). + fn has_model_id(&self, id: &str) -> bool { + self.router.registry().host_indexes().has_id(id) + } + + /// What the id map holds for `id`, as written (a position, or what a caller wrote), or None. + fn id_position<'py>(&self, py: Python<'py>, id: &str) -> PyResult>> { + let registry = self.router.registry(); + registry + .host_indexes() + .id_entry(id) + .map(|held| to_py(py, held).map(|value| value.into_bound(py))) + .transpose() + } + + /// The id map's entries, in order: `(id, what it holds)`. + fn id_positions<'py>(&self, py: Python<'py>) -> PyResult> { + let registry = self.router.registry(); + let pairs: Vec = registry + .host_indexes() + .id_map + .iter() + .map(|(id, held)| Value::Array(vec![Value::String(id.clone()), held.clone()])) + .collect(); + to_py(py, &Value::Array(pairs)).map(|value| value.into_bound(py)) + } + + /// What the name index holds for `name`, as written, or None when it does not index it. + fn named_positions<'py>( + &self, + py: Python<'py>, + name: &str, + ) -> PyResult>> { + let registry = self.router.registry(); + registry + .host_indexes() + .named(name) + .map(|held| to_py(py, held).map(|value| value.into_bound(py))) + .transpose() + } + + /// What the team index holds for `(team, public)`, as written, or None. + fn team_positions<'py>( + &self, + py: Python<'py>, + team: &str, + public: &str, + ) -> PyResult>> { + let registry = self.router.registry(); + registry + .host_indexes() + .team_named(team, public) + .map(|held| to_py(py, held).map(|value| value.into_bound(py))) + .transpose() + } + + /// `team_public_model_names_of`: the public names the team index holds for `team`. + fn team_public_names_of(&self, team: &str) -> Vec { + self.router + .registry() + .host_indexes() + .team_public_names_of(team) + } + + /// The positions of the deployments whose `litellm_params.model` is `model`. + fn litellm_model_positions(&self, model: &str) -> Vec { + self.router + .registry() + .deployments(&DeploymentFilter::default()) + .iter() + .enumerate() + .filter(|(_, deployment)| { + deployment.raw_params.get("model").and_then(Value::as_str) == Some(model) + }) + .map(|(position, _)| position) + .collect() + } + + /// `PatternMatchRouter.route` over the wildcard router (or `team`'s): the first pattern's + /// deployments, each with the model the wildcards matched, or None. + #[pyo3(signature = (team, request, filtered=None))] + fn pattern_route<'py>( + &self, + py: Python<'py>, + team: Option<&str>, + request: &str, + filtered: Option>, + ) -> PyResult>> { + let registry = self.router.registry(); + let indexes = registry.host_indexes(); + let router = match team { + None => Some(&indexes.patterns), + Some(team) => indexes + .team_patterns + .iter() + .find(|(held, _)| held == team) + .map(|(_, router)| router), + }; + router + .and_then(|router| router.route(request, filtered.as_deref())) + .map(|routed| to_py(py, &Value::Array(routed)).map(|value| value.into_bound(py))) + .transpose() + } + + /// `get_models_by_pattern`: the models the first matching (request, pattern) pair routes to. + fn models_by_pattern(&self, team: Option<&str>, requests: Vec) -> Option> { + let registry = self.router.registry(); + let indexes = registry.host_indexes(); + let router = match team { + None => Some(&indexes.patterns), + Some(team) => indexes + .team_patterns + .iter() + .find(|(held, _)| held == team) + .map(|(_, router)| router), + }; + router.and_then(|router| router.models_by_pattern(&requests)) + } + + /// The indexes the host hands out (`HostIndexes`): id map, name and team indexes, group and + /// deployment names, team public names, provider default ids and the pattern routers. + fn host_indexes<'py>(&self, py: Python<'py>) -> PyResult> { + let indexes = serde_json::to_value(self.router.registry().host_indexes()) + .map_err(|error| PyValueError::new_err(error.to_string()))?; + to_py(py, &indexes).map(|value| value.into_bound(py)) + } + + /// `provider_default_deployment_ids` as the indexes hold it now: the ids of the wildcard + /// deployments the cooldown check skips. + fn provider_default_ids(&self) -> Vec { + self.router + .registry() + .host_indexes() + .provider_default_ids + .clone() + } + + /// The provider the host resolved when it built the deployment with this id (its + /// `provider` build step), or None when there is no such deployment or it never went + /// through a host build. + fn provider(&self, id: &str) -> Option { + self.router + .registry() + .deployment(id) + .and_then(|deployment| deployment.resolved_provider.clone()) + } + + /// What `name` resolves to: `{"model_group", "kind", "ids"}`, or None for an unknown name. + #[pyo3(signature = (name, specific_deployment=false))] + fn resolve<'py>( + &self, + py: Python<'py>, + name: &str, + specific_deployment: bool, + ) -> PyResult>> { + let registry = self.router.registry(); + let Some(resolution) = registry.resolve( + name, + ResolveScope { + specific_deployment, + }, + ) else { + return Ok(None); + }; + let dict = PyDict::new(py); + dict.set_item("model_group", resolution.group)?; + dict.set_item("kind", format!("{:?}", resolution.kind))?; + dict.set_item( + "ids", + PyList::new(py, resolution.deployments.iter().map(|d| d.id.clone()))?, + )?; + Ok(Some(dict)) + } + + /// The deployments a pick for `model` would resolve to for `request`, before any team, + /// cooldown or strategy filter (`_core_get_routable_deployments`); empty when it refuses. + fn routable<'py>( + &self, + py: Python<'py>, + model: &str, + request: &Bound<'py, PyAny>, + ) -> PyResult> { + let request = config::parse_request(document(request)?, Some(model)) + .map_err(config_error_to_pyerr)?; + let records = litellm_router_core::resolve::routable( + &self.router.registry(), + &self.router.settings().dynamic, + &request, + ) + .iter() + .map(|d| to_py(py, &host::deployment_record(d))) + .collect::>>()?; + PyList::new(py, records) + } + + /// `_get_routing_context` for a pick of `model`: `(strategy, scope, group)`, spelled as a + /// strategy select step spells them. A name that resolves to nothing reads its routing group + /// membership as is. + #[pyo3(signature = (model, routing_strategy=None))] + fn routing_context( + &self, + model: &str, + routing_strategy: Option<&str>, + ) -> (String, &'static str, Option) { + let registry = self.router.registry(); + let settings = self.router.settings(); + let (kind, group) = registry + .resolve(model, ResolveScope::default()) + .map_or((ResolveKind::Group, model.to_owned()), |resolution| { + (resolution.kind, resolution.group) + }); + let (strategy, scope) = routing_context( + ®istry, + &settings.routing_strategy, + routing_strategy, + &kind, + &group, + ); + let (scope, group) = scope.wire(); + (strategy.to_string(), scope, group.map(str::to_owned)) + } + + fn retire(&self) { + self.router.retire(); + } + + /// What may still take over a stream from `origin` that fails before content, and the + /// retry budget for a failure of `kind` (see `stream::policy`). + #[pyo3(signature = (request, origin, kind=None))] + fn stream_policy<'py>( + &self, + py: Python<'py>, + request: &Bound<'py, PyAny>, + origin: &Bound<'py, PyAny>, + kind: Option<&str>, + ) -> PyResult> { + stream::policy(py, &self.router, request, origin, kind) + } + + /// Retry a stream from `origin` that failed before content within its group, driven like + /// `start`: `done` when a retry returned a response, `failed` with the newest failure. + fn retry_stream( + &self, + op: &str, + request: &Bound<'_, PyAny>, + origin: &Bound<'_, PyAny>, + failure: &Bound<'_, PyDict>, + ) -> PyResult { + stream::recover( + &self.router, + stream::Recovery::Retry, + op, + request, + origin, + failure, + ) + } + + /// The fallback walk for a stream from `origin` that failed before content, driven like + /// `start`. + fn fallback_stream( + &self, + op: &str, + request: &Bound<'_, PyAny>, + origin: &Bound<'_, PyAny>, + failure: &Bound<'_, PyDict>, + ) -> PyResult { + stream::recover( + &self.router, + stream::Recovery::Fallback, + op, + request, + origin, + failure, + ) + } + + /// `get_num_retries_from_retry_policy` for a failure shaped like an attempt outcome. + #[pyo3(signature = (failure, model_group=None))] + fn retry_policy_retries( + &self, + failure: &Bound<'_, PyDict>, + model_group: Option<&str>, + ) -> PyResult> { + retries::policy_retries(&self.router, failure, model_group) + } + + /// `_core_get_allowed_fails_from_policy` for a failure shaped like an attempt outcome. + fn allowed_fails_for(&self, failure: &Bound<'_, PyDict>) -> PyResult> { + retries::allowed_fails(&self.router, failure) + } + + /// `should_retry_this_error` as a predicate, for a failure shaped like an attempt outcome. + fn should_retry( + &self, + failure: &Bound<'_, PyDict>, + counts: (usize, usize), + fallbacks: (bool, bool, bool), + ) -> PyResult { + retries::should_retry(failure, counts, fallbacks) + } +} + +/// One router call, driven a step at a time by Python. +#[pyclass(frozen)] +pub(crate) struct RouterCall { + driver: Mutex, +} + +impl RouterCall { + fn new( + future: impl std::future::Future> + Send + 'static, + ) -> Self { + Self { + driver: Mutex::new(Driver::new(future)), + } + } +} + +#[pymethods] +impl RouterCall { + /// Hand back the outcome of the last step (`None` on the first call) and return the next + /// step, or the call's end. + #[pyo3(signature = (outcome))] + fn next<'py>( + &self, + py: Python<'py>, + outcome: Option>, + ) -> PyResult> { + let mut driver = self.driver.lock().unwrap_or_else(PoisonError::into_inner); + let response = match driver.pending_call() { + None => match outcome { + None => None, + Some(_) => return Err(drive_error_to_pyerr(DriveError::UnexpectedResponse)), + }, + Some(call) => Some(host_call_response(call, outcome)?), + }; + let step = driver.resume(response).map_err(drive_error_to_pyerr)?; + let dict = step_dict(py, step)?; + dict.set_item("events", events_list(py, driver.take_events())?)?; + Ok(dict) + } +} + +/// `(group, names)` pairs, in the order the groups appear. +type AccessGroups = Vec<(String, Vec)>; + +/// Lookup rows as the host reads them. +fn rows_to_py<'py>(py: Python<'py>, rows: &[Row]) -> PyResult> { + let rows = Value::Array(rows.iter().map(Row::wire).collect()); + to_py(py, &rows).map(|value| value.into_bound(py)) +} + +fn row_to_py<'py>(py: Python<'py>, row: &Row) -> PyResult> { + to_py(py, &row.wire()).map(|value| value.into_bound(py)) +} + +/// `{"aliases": [{alias, target}], "groups": [{name, members}], "wildcard": [requests]}`. +fn list_scope(scope: &Bound<'_, PyAny>) -> PyResult { + serde_json::from_value(document(scope)?) + .map_err(|error| PyValueError::new_err(format!("bad lookup scope: {error}"))) +} + +/// Read Python's outcome for `call`. +fn host_call_response( + call: &HostCall, + outcome: Option>, +) -> PyResult { + let missing = || { + drive_error_to_pyerr(DriveError::MissingResponse { + expected: call.kind(), + }) + }; + // `{"raised": True, ...}`: the step's work raised, reported as an attempt's failure. + if let HostCall::Select { .. } | HostCall::PreRoute { .. } | HostCall::AllowFallback { .. } = + call + && let Some(failure) = outcome.as_ref().and_then(|o| o.cast::().ok()) + && failure.contains("raised")? + { + return match attempt_outcome(failure)? { + Err(error) => Ok(HostCallResponse::Raised(error)), + Ok(_) => Err(PyValueError::new_err("a `raised` outcome is a failure")), + }; + } + match call { + HostCall::Sleep { .. } => Ok(HostCallResponse::Slept), + HostCall::AllowFallback { .. } => Ok(HostCallResponse::AllowFallback( + outcome.ok_or_else(missing)?.extract()?, + )), + HostCall::RefuseFallbacks { .. } => Ok(HostCallResponse::RefuseFallbacks( + outcome.ok_or_else(missing)?.extract()?, + )), + HostCall::Select { + stage: Stage::PreCall, + .. + } => { + let facts = from_py_argument(&outcome.ok_or_else(missing)?)?; + serde_json::from_value(facts) + .map(HostCallResponse::PreCall) + .map_err(|error| PyValueError::new_err(format!("bad pre-call facts: {error}"))) + } + HostCall::Select { + stage: Stage::RoutingPlugins { .. } | Stage::ProviderRequest, + .. + } => { + // `{"models": [...] | None}`: a bare None would read as no outcome. + let outcome = outcome.ok_or_else(missing)?; + let models = outcome.cast::()?.get_item("models")?; + Ok(HostCallResponse::Models(match models { + Some(models) if !models.is_none() => Some(models.extract()?), + _ => None, + })) + } + HostCall::Select { + stage: Stage::CallbackFilters { .. }, + .. + } => { + // The remaining ids, alone or paired with the id a filter prefers, or a failure + // mapping as an attempt reports one. + let outcome = outcome.ok_or_else(missing)?; + if let Ok(failure) = outcome.cast::() { + return match attempt_outcome(failure)? { + Err(error) => Ok(HostCallResponse::Filtered(Err(error))), + Ok(_) => Err(PyValueError::new_err( + "a `callback_filters` outcome is the remaining ids or a failure", + )), + }; + } + if let Ok(answer) = outcome.cast::() { + return Ok(HostCallResponse::Filtered(Ok(answer.extract()?))); + } + Ok(HostCallResponse::Filtered(Ok((outcome.extract()?, None)))) + } + HostCall::Select { + stage: Stage::Strategy { records, .. }, + .. + } if !records.is_empty() => Ok(HostCallResponse::Positions( + outcome.ok_or_else(missing)?.extract()?, + )), + HostCall::Select { .. } => Ok(HostCallResponse::Select( + outcome.ok_or_else(missing)?.extract()?, + )), + HostCall::PreRoute { .. } => pre_route_outcome(outcome).map(HostCallResponse::PreRoute), + HostCall::Attempt { .. } => { + let outcome = outcome.ok_or_else(missing)?; + let outcome = outcome.cast::()?; + let blocked = match outcome.get_item("blocked")? { + Some(value) if !value.is_none() => { + let name: String = value.extract()?; + Some(name.parse::().map_err(|_| { + PyValueError::new_err(format!( + "`{name}` is not a way a response is blocked" + )) + })?) + } + Some(_) | None => None, + }; + let deferred = outcome + .get_item("deferred")? + .is_some_and(|flag| flag.is_truthy().unwrap_or(false)); + let tracks_usage = outcome + .get_item("tracks_usage")? + .map(|flag| flag.extract::>()) + .transpose()? + .flatten(); + attempt_outcome(outcome).map(|result| match result { + Ok(total_tokens) if blocked.is_some() || deferred || tracks_usage.is_some() => { + HostCallResponse::Served { + total_tokens, + blocked, + deferred, + tracks_usage, + } + } + result => HostCallResponse::Attempt(result), + }) + } + HostCall::Cache { op, .. } => cache_outcome(*op, outcome).map(HostCallResponse::Cache), + HostCall::Secrets { .. } => Ok(HostCallResponse::Secrets(from_py_argument( + &outcome.ok_or_else(missing)?, + )?)), + // `None` is a response here (no row, no entry), so a missing outcome reads as one. + HostCall::ModelInfo { .. } => Ok(HostCallResponse::ModelInfo(match outcome { + Some(outcome) if !outcome.is_none() => from_py_argument(&outcome)?, + _ => Value::Null, + })), + HostCall::Build { kind, .. } => { + let outcome = outcome.ok_or_else(missing)?; + let outcome = outcome.cast::()?; + let invalid = outcome + .get_item("invalid")? + .is_some_and(|flag| flag.is_truthy().unwrap_or(false)); + match kind { + StepKind::Provider => Ok(HostCallResponse::Provided(if invalid { + Provided::Invalid + } else { + Provided::Provider(match outcome.get_item("provider")? { + Some(provider) if !provider.is_none() => Some(provider.extract()?), + _ => None, + }) + })), + _ => { + let built = if invalid { + Built::Invalid + } else { + match outcome.get_item("entry")? { + Some(entry) if !entry.is_none() => { + Built::Entry(from_py_argument(&entry)?) + } + _ => Built::Skip, + } + }; + Ok(if *kind == StepKind::Validate { + HostCallResponse::Validated(built) + } else { + HostCallResponse::Added(built) + }) + } + } + } + } +} + +/// A pre-route response: `None`, a model group name, or a custom strategy's deployment document. +fn pre_route_outcome(outcome: Option>) -> PyResult> { + let Some(outcome) = outcome.filter(|outcome| !outcome.is_none()) else { + return Ok(None); + }; + if let Ok(model) = outcome.cast::() { + return Ok(Some(PreRouted::Model(model.to_str()?.to_owned()))); + } + // `(model, group)`: the model to resolve, served as the group a session binding stamped. + if let Ok(pair) = outcome.cast::() + && pair.len() == 2 + { + return Ok(Some(PreRouted::Bound { + model: pair.get_item(0)?.extract()?, + group: pair.get_item(1)?.extract()?, + })); + } + let spec: DeploymentSpec = serde_json::from_value(document(&outcome)?) + .map_err(|error| PyValueError::new_err(error.to_string()))?; + let deployment = Deployment::from_spec(spec) + .map_err(|error| PyValueError::new_err(Report::new(&error).to_string()))?; + Ok(Some(PreRouted::Deployment(Arc::new(deployment)))) +} + +/// `get`: the JSON value or None; `batch_get`: a list of JSON values or None, one per key; +/// `increment`: the new int or float; `set` and `delete`: None. +fn cache_outcome(op: CacheOp, outcome: Option>) -> PyResult { + let outcome = outcome.filter(|outcome| !outcome.is_none()); + let json = |value: Option| value.filter(|value| !value.is_null()); + match op { + CacheOp::Get => outcome + .map(|outcome| from_py_argument::(&outcome)) + .transpose() + .map(|value| CacheOutcome::Value(json(value))), + CacheOp::BatchGet => { + let outcome = outcome.ok_or_else(|| { + PyValueError::new_err("a `batch_get` cache outcome must be a list, not None") + })?; + if !outcome.is_instance_of::() { + return Err(PyValueError::new_err( + "a `batch_get` cache outcome must be a list with one entry per key", + )); + } + let values: Vec> = from_py_argument(&outcome)?; + Ok(CacheOutcome::Values(values.into_iter().map(json).collect())) + } + CacheOp::Increment => outcome + .filter(|outcome| !outcome.is_instance_of::()) + .and_then(|outcome| outcome.extract::().ok()) + .map(CacheOutcome::Number) + .ok_or_else(|| { + PyValueError::new_err("an `increment` cache outcome must be the new int or float") + }), + CacheOp::Set | CacheOp::Delete | CacheOp::IncrementUsage => match outcome { + None => Ok(CacheOutcome::Stored), + Some(_) => Err(PyValueError::new_err(format!( + "a `{op}` cache step has no outcome, so pass None" + ))), + }, + } +} + +/// `{"ok": True, "total_tokens": int}` (an attempt's may add `"blocked": str`, `"deferred": bool` +/// and `"tracks_usage": bool`) or `{"ok": False, "kind": str, "status": int | None, +/// "retry_after": float | None, "num_retries": int | None, "detail": str | None, +/// "outside_deployment": bool, "caller_scoped": bool, "status_absent": bool, +/// "uncounted_request": bool}`. +pub(crate) fn attempt_outcome(outcome: &Bound<'_, PyDict>) -> PyResult> { + fn item<'py, T: FromPyObjectOwned<'py>>( + outcome: &Bound<'py, PyDict>, + key: &str, + ) -> PyResult> { + match outcome.get_item(key)? { + Some(value) if !value.is_none() => Ok(Some(value.extract().map_err(Into::into)?)), + Some(_) | None => Ok(None), + } + } + let ok: bool = item(outcome, "ok")? + .ok_or_else(|| PyValueError::new_err("an attempt outcome needs `ok`"))?; + if ok { + return Ok(Ok(item(outcome, "total_tokens")?.unwrap_or(0))); + } + let kind: String = item(outcome, "kind")? + .ok_or_else(|| PyValueError::new_err("a failed attempt outcome needs `kind`"))?; + let kind: FailureKind = kind + .parse() + .map_err(|_| PyValueError::new_err(format!("`{kind}` is not a router failure kind")))?; + let retry_after = item::(outcome, "retry_after")? + .map(|seconds| { + Duration::try_from_secs_f64(seconds) + .map_err(|_| PyValueError::new_err("`retry_after` must be a non-negative number")) + }) + .transpose()?; + let mut failure = AttemptError::new(kind, item(outcome, "status")?); + failure.retry_after = retry_after; + failure.num_retries = item(outcome, "num_retries")?; + failure.failed_deployment_id = item(outcome, "failed_deployment_id")?; + failure.detail = item::(outcome, "detail")?.map(Into::into); + failure.recorded = item::(outcome, "recorded")?.unwrap_or(false); + failure.names_provider = item::(outcome, "llm_provider")?.unwrap_or(false); + failure.agentic_followup = item::(outcome, "agentic_followup")?.unwrap_or(false); + failure.outside_deployment = item::(outcome, "outside_deployment")?.unwrap_or(false); + failure.status_absent = item::(outcome, "status_absent")?.unwrap_or(false); + failure.uncounted_request = item::(outcome, "uncounted_request")?.unwrap_or(false); + failure.caller_scoped = item::(outcome, "caller_scoped")?.unwrap_or(false); + failure.classes = item::>(outcome, "classes")? + .unwrap_or_default() + .into_iter() + .map(Into::into) + .collect(); + Ok(Err(failure)) +} + +/// `[{"kind": "retries_exhausted", "model_group": str, "attempted": int, "max": int} +/// | {"kind": "fallback", "to": str, "order": int | None, "level": int}]`. +/// The `cause` of a `retry_logged` or `retry_policy_raised` event, with what names it. +fn cause_items(dict: &Bound<'_, PyDict>, cause: RetryCause) -> PyResult<()> { + match cause { + RetryCause::Attempt => dict.set_item("cause", "attempt"), + RetryCause::Level => dict.set_item("cause", "level"), + RetryCause::Other => dict.set_item("cause", "other"), + RetryCause::Pick(picked) => { + dict.set_item("cause", "pick")?; + pick_items(dict, &picked.0) + } + } +} + +/// A pick's error, as the `failed` step names it (`error_kind` and what names it), with the +/// model its message names (`failed_model`) when the error carries one. +fn pick_items(dict: &Bound<'_, PyDict>, error: &Error) -> PyResult<()> { + error_items(dict, error)?; + match error { + Error::NoHealthyDeployments { model, .. } | Error::UnknownModel { model } => { + dict.set_item("failed_model", model) + } + _ => Ok(()), + } +} + +/// `error_kind` and the fields that name the error. +fn error_items(dict: &Bound<'_, PyDict>, error: &Error) -> PyResult<()> { + dict.set_item("error", Report::new(error).to_string())?; + dict.set_item("error_kind", <&'static str>::from(ErrorKind::from(error)))?; + match error { + Error::EndpointNotSupported { model, message } => { + dict.set_item("error_model", model)?; + dict.set_item("error_detail", message) + } + Error::OnlyMarkers { model } + | Error::UnknownModel { model } + | Error::NoPassThroughDeployments { model } + | Error::PtuReserved { model } + | Error::MultipleTeams { model } + | Error::DefaultDeploymentRaises { model } + | Error::IndexOutOfRange { model } => dict.set_item("error_model", model), + Error::ContextWindowExceeded { model, detail } => { + dict.set_item("error_model", model)?; + dict.set_item("error_detail", detail) + } + Error::UnsupportedParams { model, params } => { + dict.set_item("error_model", model)?; + dict.set_item("error_params", PyList::new(dict.py(), params)?) + } + Error::RoutingPluginsLeftNothing { models } => { + dict.set_item("error_params", PyList::new(dict.py(), models)?) + } + Error::Reserved { + model, + until, + timezone, + } => { + dict.set_item("error_model", model)?; + dict.set_item("error_detail", format!("{until} {timezone}")) + } + Error::NoHealthyDeployments { + request_ids: Some(ids), + .. + } => dict.set_item("error_request_ids", PyList::new(dict.py(), ids)?), + Error::NonNumericWeight { metric } => dict.set_item("error_detail", metric), + Error::MissingRoutingGroup { group } => dict.set_item("error_model", group), + Error::ModelInfo { exception, detail } => { + dict.set_item("error_exception", *exception)?; + dict.set_item("error_detail", detail) + } + _ => Ok(()), + } +} + +fn events_list(py: Python<'_>, events: Vec) -> PyResult> { + let list = PyList::empty(py); + for event in events { + let dict = PyDict::new(py); + match event { + RouterEvent::RetriesExhausted { + model_group, + attempted, + max, + level, + picked, + } => { + dict.set_item("kind", "retries_exhausted")?; + dict.set_item("model_group", model_group)?; + dict.set_item("attempted", attempted)?; + dict.set_item("max", max)?; + dict.set_item("level", level)?; + dict.set_item("picked", picked)?; + } + RouterEvent::Retry { retry, .. } => { + dict.set_item("kind", "retry")?; + dict.set_item("retry", retry)?; + } + RouterEvent::FallingBack { level, model } => { + dict.set_item("kind", "falling_back")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + } + RouterEvent::OrderChainFailed { level, last_hop } => { + dict.set_item("kind", "order_chain_failed")?; + dict.set_item("level", level)?; + dict.set_item("last_hop", last_hop)?; + } + RouterEvent::FallbacksFailed { + level, + chain, + last_hop, + } => { + dict.set_item("kind", "fallbacks_failed")?; + dict.set_item("level", level)?; + dict.set_item("chain", chain)?; + dict.set_item("last_hop", last_hop)?; + } + RouterEvent::Fallback { + from, + to, + order, + depth, + level, + scope, + parent, + max_fallbacks, + unroutable, + } => { + dict.set_item("kind", "fallback")?; + dict.set_item("from", from)?; + dict.set_item("to", to)?; + dict.set_item("order", order)?; + dict.set_item("depth", depth)?; + dict.set_item("level", level)?; + dict.set_item("scope", scope)?; + dict.set_item("parent", parent)?; + dict.set_item("max_fallbacks", max_fallbacks)?; + if let Some(model) = unroutable { + dict.set_item("unroutable", to_py(py, &model)?)?; + } + } + RouterEvent::Blocked { + level, + model, + blocked, + } => { + dict.set_item("kind", "blocked")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + dict.set_item("blocked", <&'static str>::from(blocked))?; + } + RouterEvent::FallbackDone { + from, + to, + level, + scope, + parent, + succeeded, + } => { + dict.set_item("kind", "fallback_done")?; + dict.set_item("from", from)?; + dict.set_item("to", to)?; + dict.set_item("level", level)?; + dict.set_item("scope", scope)?; + dict.set_item("parent", parent)?; + dict.set_item("succeeded", succeeded)?; + } + RouterEvent::RetryLogged { + scope, + level, + model, + attempted_retries, + cause, + } => { + dict.set_item("kind", "retry_logged")?; + dict.set_item("scope", scope)?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + dict.set_item("attempted_retries", attempted_retries)?; + cause_items(&dict, cause)?; + } + RouterEvent::RetryPolicyRaised { + level, + model, + cause, + } => { + dict.set_item("kind", "retry_policy_raised")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + cause_items(&dict, cause)?; + } + RouterEvent::RetryCountRaised { + level, + model, + cause, + } => { + dict.set_item("kind", "retry_count_raised")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + cause_items(&dict, cause)?; + } + RouterEvent::RetriesOpened { + scope, + max_retries, + model_group_size, + } => { + dict.set_item("kind", "retries_opened")?; + dict.set_item("scope", scope)?; + dict.set_item("max_retries", max_retries)?; + dict.set_item("model_group_size", model_group_size)?; + } + RouterEvent::RetryBudget { scope, max_retries } => { + dict.set_item("kind", "retry_budget")?; + dict.set_item("scope", scope)?; + dict.set_item("max_retries", max_retries)?; + } + RouterEvent::PickFailed { + level, + model, + error, + } => { + dict.set_item("kind", "pick_failed")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + pick_items(&dict, &error.0)?; + } + RouterEvent::LevelFailed { + level, + model, + error, + handling, + } => { + dict.set_item("kind", "level_failed")?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + dict.set_item("handling", handling)?; + pick_items(&dict, &error.0)?; + } + RouterEvent::MockFailed { + mock, + level, + model, + num_retries, + } => { + dict.set_item("kind", "mock_failed")?; + dict.set_item("mock", <&'static str>::from(mock))?; + dict.set_item("level", level)?; + dict.set_item("model", model)?; + dict.set_item("num_retries", num_retries)?; + } + RouterEvent::CooldownSet { + deployment_id, + cooldown, + } => { + dict.set_item("kind", "cooldown_set")?; + dict.set_item("deployment_id", deployment_id)?; + dict.set_item("cooldown_time", cooldown.as_secs_f64())?; + } + RouterEvent::PickStarted { model } => { + dict.set_item("kind", "pick_started")?; + dict.set_item("model", model)?; + } + RouterEvent::Selected { + deployment_id, + pick, + .. + } => { + dict.set_item("kind", "selected")?; + dict.set_item("deployment_id", deployment_id)?; + dict.set_item("pick_seconds", pick.as_secs_f64())?; + } + _ => continue, + } + list.append(dict)?; + } + Ok(list) +} + +fn step_dict(py: Python<'_>, step: Step) -> PyResult> { + let dict = PyDict::new(py); + let kind = match &step { + Step::HostCall(call) => call.kind(), + Step::Done(_) => StepKind::Done, + Step::Failed(_) => StepKind::Failed, + }; + dict.set_item("kind", <&'static str>::from(kind))?; + match step { + Step::HostCall(call) => host_call_items(py, &dict, call)?, + Step::Done(finish) => done_items(&dict, finish)?, + Step::Failed(error) => { + let attempt = error.attempt(); + error_items(&dict, &error)?; + dict.set_item("attempt_failed", attempt.is_some())?; + dict.set_item("status", attempt.and_then(|failure| failure.status))?; + dict.set_item( + "failure_kind", + attempt.map(|failure| <&'static str>::from(failure.kind)), + )?; + } + } + Ok(dict) +} + +fn host_call_items(py: Python<'_>, dict: &Bound<'_, PyDict>, call: HostCall) -> PyResult<()> { + match call { + HostCall::Attempt { + deployment_id, + deployment, + model, + model_group, + routed, + target_order, + candidate_ids, + retry, + fallback_depth, + fallback_level, + timeout, + params, + scope, + max_retries, + model_group_size, + } => { + dict.set_item("routed", routed)?; + dict.set_item("scope", scope)?; + dict.set_item("max_retries", max_retries)?; + dict.set_item("model_group_size", model_group_size)?; + dict.set_item("deployment_id", deployment_id)?; + dict.set_item("deployment", to_py(py, &deployment)?)?; + dict.set_item("model", model)?; + dict.set_item("model_group", model_group)?; + dict.set_item("target_order", target_order)?; + dict.set_item("candidate_ids", PyList::new(py, candidate_ids)?)?; + dict.set_item("retry", retry)?; + dict.set_item("fallback_depth", fallback_depth)?; + dict.set_item("fallback_level", fallback_level)?; + dict.set_item("timeout", timeout)?; + dict.set_item("params", params.map(|p| to_py(py, &p)).transpose()?) + } + HostCall::Sleep { seconds } => dict.set_item("seconds", seconds), + HostCall::AllowFallback { target } => dict.set_item("target", target), + HostCall::RefuseFallbacks { level, targets } => { + dict.set_item("level", level)?; + let targets = targets.0.iter().cloned().map(Value::Object).collect(); + dict.set_item("targets", to_py(py, &Value::Array(targets))?) + } + HostCall::Select { + stage, + model_group, + name, + scope, + group, + pool, + candidates, + } => { + let (stage, strategy) = match stage { + Stage::Plugins => ("plugins", None), + Stage::Strategy { strategy, records } => { + if !records.is_empty() { + dict.set_item("records", to_py(py, &Value::Array(records))?)?; + } + ("strategy", Some(strategy)) + } + Stage::Tags => ("tags", None), + Stage::PreCall => ("pre_call", None), + Stage::TextOnly => ("text_only", None), + Stage::RouteIncapable { handler } => { + dict.set_item("handler", handler)?; + ("route_incapable", None) + } + Stage::MediaRefusing => ("media_refusing", None), + Stage::EmbeddingBatch => ("embedding_batch", None), + Stage::OverrideCheck(strategy) => ("override_check", Some(strategy)), + Stage::ProviderRequest => ("provider_request", None), + Stage::CallbackFilters { + excluded_ids, + skipped_ids, + target_order, + } => { + dict.set_item("excluded_ids", excluded_ids)?; + dict.set_item("skipped_ids", skipped_ids)?; + dict.set_item("target_order", target_order)?; + ("callback_filters", None) + } + Stage::RoutingPlugins { run } => { + dict.set_item("run", run)?; + ("routing_plugins", None) + } + }; + dict.set_item("pool", PyList::new(py, pool)?)?; + dict.set_item("stage", stage)?; + dict.set_item("strategy", strategy)?; + dict.set_item("model_group", model_group)?; + dict.set_item("model", name)?; + dict.set_item("scope", scope)?; + dict.set_item("group", group)?; + dict.set_item("candidates", PyList::new(py, candidates)?) + } + HostCall::PreRoute { + model, + fallback_depth, + retry, + } => { + dict.set_item("model", model)?; + dict.set_item("fallback_depth", fallback_depth)?; + dict.set_item("retry", retry) + } + HostCall::Cache { + op, + key, + keys, + value, + ttl, + local, + } => { + dict.set_item("op", <&'static str>::from(op))?; + dict.set_item("key", key)?; + dict.set_item("keys", keys.map(|keys| PyList::new(py, keys)).transpose()?)?; + dict.set_item("value", value.map(|value| to_py(py, &value)).transpose()?)?; + dict.set_item("ttl", ttl)?; + dict.set_item("local", local) + } + HostCall::Secrets { index, refs } => { + dict.set_item("index", index)?; + let refs = refs + .into_iter() + .map(|secret| { + let item = PyDict::new(py); + item.set_item("name", secret.name)?; + match secret.at { + SecretAt::Param(key) => { + item.set_item("key", key)?; + item.set_item("env_only", false)?; + } + SecretAt::DataSource { source, key } => { + item.set_item("source", source)?; + item.set_item("key", key)?; + item.set_item("env_only", true)?; + } + } + Ok(item) + }) + .collect::>>()?; + dict.set_item("refs", PyList::new(py, refs)?) + } + HostCall::Build { + kind: _, + path, + index, + entry, + declared_id, + duplicate_id, + provider, + } => { + dict.set_item("path", <&'static str>::from(path))?; + dict.set_item("index", index)?; + dict.set_item("entry", to_py(py, &entry)?)?; + dict.set_item("declared_id", declared_id)?; + dict.set_item("duplicate_id", duplicate_id)?; + dict.set_item("provider", provider) + } + HostCall::ModelInfo { lookup } => { + if let Value::Object(lookup) = lookup { + for (key, value) in lookup { + dict.set_item(key, to_py(py, &value)?)?; + } + } + Ok(()) + } + } +} + +fn done_items(dict: &Bound<'_, PyDict>, finish: Finish) -> PyResult<()> { + match finish { + Finish::Call(response) => { + dict.set_item("deployment_id", response.deployment_id)?; + dict.set_item("model_group", response.model_group)?; + dict.set_item("attempted_retries", response.attempted_retries)?; + dict.set_item("max_retries", response.max_retries)?; + dict.set_item("attempted_fallbacks", response.attempted_fallbacks)?; + dict.set_item("total_tokens", response.output.total_tokens) + } + Finish::Selected { + deployment_id, + model_group, + deployment, + } => { + dict.set_item("deployment_id", deployment_id)?; + dict.set_item("model_group", model_group)?; + dict.set_item("deployment", to_py(dict.py(), &deployment)?) + } + Finish::Reported { cooldown } => { + dict.set_item("cooled", cooldown.is_some())?; + dict.set_item("cooldown_time", cooldown.map(|c| c.as_secs_f64())) + } + Finish::Built(built) => { + let py = dict.py(); + let records = built + .into_iter() + .map(|deployment| { + let item = PyDict::new(py); + item.set_item("index", deployment.index)?; + item.set_item( + "id", + deployment.spec.model_info.get("id").and_then(Value::as_str), + )?; + item.set_item("provider", deployment.spec.provider)?; + Ok(item) + }) + .collect::>>()?; + dict.set_item("built", PyList::new(py, records)?) + } + Finish::Counted { key } => dict.set_item("key", key), + Finish::RouterModelInfo(merged) => { + let py = dict.py(); + match merged { + Some(merged) => { + dict.set_item("info", to_py(py, &Value::Object(merged.info))?)?; + dict.set_item("user_keys", merged.user_keys) + } + None => dict.set_item("info", py.None()), + } + } + Finish::DeploymentModelInfo(info) => { + let py = dict.py(); + match info { + Some(info) => dict.set_item("info", to_py(py, &Value::Object(info))?), + None => dict.set_item("info", py.None()), + } + } + Finish::Value(value) => dict.set_item("value", to_py(dict.py(), &value)?), + Finish::Healthy { + single, + deployment_ids, + deployments, + } => { + dict.set_item("single", single)?; + dict.set_item("deployment_ids", deployment_ids)?; + let records = deployments + .iter() + .map(|d| to_py(dict.py(), d)) + .collect::>>()?; + dict.set_item("deployments", PyList::new(dict.py(), records)?) + } + } +} + +#[cfg(test)] +mod tests { + use pyo3::{ffi::c_str, prelude::*, types::PyDict}; + use rstest::rstest; + + const CONFIG: &str = r#"{"model_list": [ + {"model_name": "primary", "litellm_params": {"model": "openai/a"}, "model_info": {"id": "a1"}} + ]}"#; + + const PAIR: &str = r#"{"model_list": [ + {"model_name": "primary", "litellm_params": {"model": "openai/a"}, "model_info": {"id": "a1"}}, + {"model_name": "primary", "litellm_params": {"model": "openai/b"}, "model_info": {"id": "a2"}} + ]}"#; + + #[rstest] + fn python_drives_calls_and_selections_through_the_handle() { + Python::initialize(); + Python::attach(|py| { + let locals = PyDict::new(py); + locals.set_item("native", crate::native_module(py)).unwrap(); + locals.set_item("config", CONFIG).unwrap(); + locals.set_item("pair", PAIR).unwrap(); + py.run( + c_str!( + r#" +import time +store = {} +asked = [] + +def answer_cache(step): + asked.append(step) + op, key = step["op"], step["key"] + if op == "get": + return store.get(key) + if op == "batch_get": + return [store.get(k) for k in step["keys"]] + if op == "set": + store[key] = step["value"] + return None + if op == "increment": + store[key] = store.get(key, 0) + step["value"] + return store[key] + if op == "delete": + store.pop(key, None) + return None + raise AssertionError(op) + +def drive(call, outcome): + step = call.next(outcome) + while step["kind"] == "cache": + step = call.next(answer_cache(step)) + return step + +handle = native.RouterCoreHandle(config, {"num_retries": 1}) +picked = drive(handle.select("primary", {}), None) +assert (picked["kind"], picked["deployment_id"], picked["model_group"]) == ("done", "a1", "primary"), picked +assert [step["op"] for step in asked] == ["batch_get"] +assert list(asked[0]) == ["kind", "op", "key", "keys", "value", "ttl", "local", "events"], list(asked[0]) +call = handle.start("acompletion", '{"model": "primary"}') +version = handle.registry_version() +assert handle.apply({"delete": ["gone"]}) == version + 1 +steps = [drive(call, None)] +steps.append(drive(call, {"ok": False, "kind": "RateLimitError", "status": 429})) +steps.append(drive(call, None)) +steps.append(drive(call, {"ok": True, "total_tokens": 5})) +kinds = [step["kind"] for step in steps] + +cooled = native.RouterCoreHandle(pair, {"num_retries": 0}) +call = cooled.start("acompletion", {"model": "primary"}) +attempt = drive(call, None) +tried = attempt["deployment_id"] +assert attempt["deployment"]["model_info"]["id"] == tried, attempt +assert attempt["deployment"]["litellm_params"]["model"].startswith("openai/"), attempt +assert [d["model_info"]["id"] for d in cooled.deployments()] == ["a1", "a2"] +assert cooled.deployment("a2")["model_name"] == "primary" +assert cooled.resolve("primary")["ids"] == ["a1", "a2"] +assert cooled.resolve("nope") is None +assert drive(call, {"ok": False, "kind": "RateLimitError", "status": 429})["kind"] == "failed" +# The core records the failure itself: the 429 cools the deployment down in the host's store. +assert f"deployment:{tried}:cooldown" in store, store +other = drive(cooled.select("primary", {}), None) +assert other["kind"] == "done" and other["deployment_id"] != tried, other + +call = handle.select("primary", {}) +step = call.next(None) +assert step["op"] == "batch_get" +for bad in ([], [None, None], "nope", None): + try: + call.next(bad) + except ValueError: + pass + else: + raise AssertionError(f"{bad!r} should not answer a batch_get of one key") +assert drive(call, [None])["kind"] == "done" +"# + ), + Some(&locals), + Some(&locals), + ) + .unwrap(); + let kinds: Vec = locals + .get_item("kinds") + .unwrap() + .unwrap() + .extract() + .unwrap(); + assert_eq!(kinds, ["attempt", "sleep", "attempt", "done"]); + }); + } + + #[rstest] + #[case::unknown_key(r#"{"routing_strategy_args": {}}"#, true)] + #[case::bad_json("{", false)] + #[case::not_an_object("[]", false)] + fn construction_failures_raise_the_class_python_reads( + #[case] config: &str, + #[case] declined: bool, + ) { + Python::initialize(); + Python::attach(|py| { + let error = crate::native_module(py) + .getattr("RouterCoreHandle") + .and_then(|handle| handle.call1((config, "{}"))) + .unwrap_err(); + assert_eq!( + error.is_instance_of::(py), + declined, + "{error}" + ); + assert_eq!( + error.is_instance_of::(py), + !declined, + "{error}" + ); + }); + } +} diff --git a/litellm-rust/crates/python-bridge/src/router/retries.rs b/litellm-rust/crates/python-bridge/src/router/retries.rs new file mode 100644 index 00000000000..949aae96b34 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/router/retries.rs @@ -0,0 +1,55 @@ +//! The retry decisions `RouterCoreHandle` exposes as members: `get_num_retries_from_retry_policy` +//! and `should_retry_this_error`, decided by router-core for a failure Python describes the way it +//! describes a failed attempt. + +use litellm_router_core::{ + AttemptError, Router, + retry::{RetryContext, resolve_retry_policy, should_retry as core_should_retry}, +}; +use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; + +fn failure(outcome: &Bound<'_, PyDict>) -> PyResult { + match super::attempt_outcome(outcome)? { + Err(failure) => Ok(failure), + Ok(_) => Err(PyValueError::new_err("a failure needs `ok: False`")), + } +} + +pub(crate) fn policy_retries( + router: &Router, + outcome: &Bound<'_, PyDict>, + model_group: Option<&str>, +) -> PyResult> { + let failure = failure(outcome)?; + let settings = router.settings(); + let policy = match model_group { + Some(group) => resolve_retry_policy( + settings.retry_policy.as_ref(), + group, + Some(&settings.model_group_retry_policy), + ), + None => settings.retry_policy.as_ref(), + }; + Ok(policy.and_then(|policy| policy.retries_for(&failure))) +} + +pub(crate) fn allowed_fails(router: &Router, outcome: &Bound<'_, PyDict>) -> PyResult> { + Ok(router.settings().allowed_fails_for(&failure(outcome)?)) +} + +pub(crate) fn should_retry( + outcome: &Bound<'_, PyDict>, + (healthy, all): (usize, usize), + (context_window, content_policy, regular): (bool, bool, bool), +) -> PyResult { + Ok(core_should_retry( + &failure(outcome)?, + RetryContext { + healthy, + all, + has_context_window_fallbacks: context_window, + has_content_policy_fallbacks: content_policy, + has_regular_fallbacks: regular, + }, + )) +} diff --git a/litellm-rust/crates/python-bridge/src/router/stream.rs b/litellm-rust/crates/python-bridge/src/router/stream.rs new file mode 100644 index 00000000000..f85d73e0778 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/router/stream.rs @@ -0,0 +1,94 @@ +//! The stream-recovery entry points of `RouterCoreHandle`: what may still take over a stream that +//! failed before content, the same-group stream retry, and the fallback walk re-entered from a +//! stream. The host's stream wrappers call them; the calls are driven like `start`. + +use litellm_router_core::{ + FailureKind, Op, Router, + call::{StreamOrigin, StreamPolicy}, +}; +use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; + +use super::{RouterCall, attempt_outcome, config, config_error_to_pyerr, document, host::Finish}; + +/// A stream recovery a host wrapper asks for. +#[derive(Clone, Copy)] +pub(super) enum Recovery { + /// `_astream_retry_same_group`. + Retry, + /// The fallback walk re-entered from the stream wrapper. + Fallback, +} + +fn origin(value: &Bound<'_, PyAny>) -> PyResult { + serde_json::from_value(document(value)?) + .map_err(|error| PyValueError::new_err(format!("a stream origin: {error}"))) +} + +fn kind(kind: Option<&str>) -> PyResult> { + kind.map(|kind| { + kind.parse() + .map_err(|_| PyValueError::new_err(format!("`{kind}` is not a router failure kind"))) + }) + .transpose() +} + +/// `{"can_retry", "can_fall_back", "refusal_fallback", "retry_budget": int | None, +/// "policy_applies": bool}`. +pub(super) fn policy<'py>( + py: Python<'py>, + router: &Router, + request: &Bound<'py, PyAny>, + stream_origin: &Bound<'py, PyAny>, + failure_kind: Option<&str>, +) -> PyResult> { + let request = config::parse_request(document(request)?, None).map_err(config_error_to_pyerr)?; + let StreamPolicy { + can_retry, + can_fall_back, + refusal_fallback, + retry_budget, + } = router.stream_policy(&request, &origin(stream_origin)?, kind(failure_kind)?); + let dict = PyDict::new(py); + dict.set_item("can_retry", can_retry)?; + dict.set_item("can_fall_back", can_fall_back)?; + dict.set_item("refusal_fallback", refusal_fallback)?; + dict.set_item("retry_budget", retry_budget.map(|(budget, _)| budget))?; + dict.set_item( + "policy_applies", + retry_budget.is_some_and(|(_, applies)| applies), + )?; + Ok(dict) +} + +/// A recovery call for a stream from `stream_origin` that failed with `failure`. +pub(super) fn recover( + router: &Router, + recovery: Recovery, + op: &str, + request: &Bound<'_, PyAny>, + stream_origin: &Bound<'_, PyAny>, + failure: &Bound<'_, PyDict>, +) -> PyResult { + let op = Op::new(op); + let request = config::parse_request(document(request)?, None).map_err(config_error_to_pyerr)?; + let stream_origin = origin(stream_origin)?; + let Err(failure) = attempt_outcome(failure)? else { + return Err(PyValueError::new_err("a stream failure needs `ok: False`")); + }; + let router = router.clone(); + Ok(RouterCall::new(async move { + let response = match recovery { + Recovery::Retry => { + router + .retry_stream(op, request, stream_origin, failure) + .await + } + Recovery::Fallback => { + router + .fallback_stream(op, request, stream_origin, failure) + .await + } + }; + response.map(Finish::Call) + })) +} diff --git a/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs b/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs index e62673d4d4f..21a57123921 100644 --- a/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs +++ b/litellm-rust/crates/python-bridge/src/routes/audio_transcription.rs @@ -13,7 +13,7 @@ use crate::{ }; async fn execute( - http: Result, + http: litellm_http::Client, secrets: std::sync::Arc, audio: Value, optional_params: Map, @@ -27,7 +27,7 @@ async fn execute( extra_headers, timeout, } = options; - AudioTranscriptionRoute::new(http?, crate::http::resources().auth.clone(), secrets) + AudioTranscriptionRoute::new(http, crate::http::resources().auth.clone(), secrets) .execute(AudioTranscriptionRequest { model: &model, audio, @@ -48,7 +48,8 @@ pub(crate) fn transcription(py: Python<'_>, call: NativeCall<'_>) -> PyResult( litellm_host_python::from_py_argument(&required_field(&arguments, "audio")?)?; let options = value_route_options(&arguments)?; let optional_params = optional_object_field(&arguments, "optional_params")?.unwrap_or_default(); - let http = crate::http::provider_client(py, &call.kwargs, true)?; + let http = + crate::http::provider_client(py, &call.kwargs, true)?.map_err(crate::http::client_error)?; let secrets = crate::secrets::source(py)?; run_async( py, diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs index 5ad9f8df8b8..4db9b37cbb8 100644 --- a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs @@ -1,3 +1,4 @@ +use litellm_core_utils::decline::Decline; use std::convert::Infallible; use crate::routes::inference::InferenceHost; @@ -44,9 +45,9 @@ impl PythonBinding for ChatCompletionsPythonHost { .get("stream") .is_some_and(|value| value == &serde_json::Value::Bool(true)) { - return Err(InvokeError::Native(Error::Unsupported( + return Err(InvokeError::Native(Error::Declined(Decline::new( "native Python chat_completions streaming", - ))); + )))); } Ok(call) } @@ -79,7 +80,7 @@ impl PythonBinding for ChatCompletionsPythonHost { self.0.error(py, error) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs index d78d141f0f2..1bcdc18a765 100644 --- a/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs @@ -4,7 +4,7 @@ use std::sync::Arc; use host::ChatCompletionsPythonHost; use litellm_auth::AuthServices; -use litellm_cache_response::{CachePolicy, ScopedCache}; +use litellm_cache_response::CachePolicy; use litellm_callbacks_legacy_python::LoggingOperation; use litellm_host::{call::HostedMachine, protocol::Protocol}; use litellm_inference_chat::{ChatCompletionsRoute, route::ChatCompletions}; @@ -57,10 +57,6 @@ impl InferenceRoute for ChatCompletionsRoute { Self::new(http, auth, secrets) } - fn with_cache(self, cache: ScopedCache) -> Self { - self.with_cache(cache) - } - fn machine( self, call: ::Request, diff --git a/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs b/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs index f8df7a9de23..80d7bd5360e 100644 --- a/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs +++ b/litellm-rust/crates/python-bridge/src/routes/clickhouse_spend.rs @@ -8,24 +8,13 @@ use pyo3::{ }; fn map_error(error: Error) -> PyErr { - use litellm_storage_clickhouse::Error as StorageError; - match error { - Error::InsertTooLarge | Error::Storage(StorageError::InsertTooLarge) => { - PyOverflowError::new_err(error.to_string()) - } - Error::InvalidRow - | Error::InvalidLimit(_) - | Error::InvalidSchema - | Error::Storage( - StorageError::InvalidRow - | StorageError::InvalidLimit(_) - | StorageError::InvalidTable - | StorageError::InvalidSchema - | StorageError::EmptySql - | StorageError::InvalidParameters - | StorageError::InvalidQuery, - ) => PyValueError::new_err(error.to_string()), - Error::Migration(_) | Error::Storage(_) => PyRuntimeError::new_err(error.to_string()), + let message = litellm_error::Report::new(&error).to_string(); + if error.is_insert_too_large() { + PyOverflowError::new_err(message) + } else if error.is_request() { + PyValueError::new_err(message) + } else { + PyRuntimeError::new_err(message) } } @@ -99,26 +88,22 @@ mod tests { use super::*; use rstest::rstest; + use litellm_storage_clickhouse::Error as StorageError; + #[rstest] - #[case::row(Error::InvalidRow, "ValueError")] - #[case::insert_limit(Error::InvalidLimit("CLICKHOUSE_TRACE_MAX_INSERT_BYTES"), "ValueError")] - #[case::insert_timeout( - Error::Storage(litellm_storage_clickhouse::Error::InvalidLimit( - "CLICKHOUSE_INSERT_TIMEOUT_SECONDS" - )), + #[case::timestamp(Error::NonIntegerTimestamp { field: "start_time" }, "ValueError")] + #[case::insert_limit( + StorageError::InvalidLimit("CLICKHOUSE_TRACE_MAX_INSERT_BYTES").into(), "ValueError" )] - #[case::insert_budget(Error::InsertTooLarge, "OverflowError")] - #[case::schema( - Error::Storage(litellm_storage_clickhouse::Error::SchemaFailed(503)), - "RuntimeError" - )] + #[case::insert_budget(StorageError::InsertTooLarge.into(), "OverflowError")] + #[case::schema(StorageError::SchemaFailed(503).into(), "RuntimeError")] #[case::migration( Error::Migration(sqlx::migrate::MigrateError::VersionMismatch(6)), "RuntimeError" )] #[case::storage( - Error::Storage(litellm_storage_clickhouse::Error::InvalidUrl), + Config::new("spend".into(), "ftp://localhost", 1).err().unwrap(), "RuntimeError" )] fn spend_failures_preserve_public_exception_types( @@ -127,7 +112,7 @@ mod tests { ) { Python::initialize(); Python::attach(|py| { - let message = error.to_string(); + let message = litellm_error::Report::new(&error).to_string(); let exception = map_error(error); assert_eq!(exception.get_type(py).name().unwrap(), exception_name); assert_eq!( @@ -136,4 +121,15 @@ mod tests { ); }); } + + #[test] + fn spend_failures_render_their_cause() { + let error = Config::new("spend".into(), "not a url", 1).err().unwrap(); + let cause = std::error::Error::source(&error).unwrap().to_string(); + Python::initialize(); + Python::attach(|py| { + let message = map_error(error).value(py).str().unwrap().to_string(); + assert!(message.ends_with(&cause), "{message}"); + }); + } } diff --git a/litellm-rust/crates/python-bridge/src/routes/embeddings.rs b/litellm-rust/crates/python-bridge/src/routes/embeddings.rs index 0f73eefedd5..0935f42f85a 100644 --- a/litellm-rust/crates/python-bridge/src/routes/embeddings.rs +++ b/litellm-rust/crates/python-bridge/src/routes/embeddings.rs @@ -1,14 +1,23 @@ +use litellm_error::Report; use pyo3::prelude::*; use super::NativeCall; use crate::errors::RustBridgeDeclined; +#[derive(Debug, thiserror::Error)] +#[error("native embeddings route is not implemented")] +struct NotImplemented; + +impl From for PyErr { + fn from(error: NotImplemented) -> Self { + RustBridgeDeclined::new_err(Report::new(&error).to_string()) + } +} + #[pyfunction] pub(crate) fn embedding(call: NativeCall<'_>) -> PyResult> { drop(call); - Err(RustBridgeDeclined::new_err( - "native embeddings route is not implemented", - )) + Err(NotImplemented.into()) } #[pyfunction] diff --git a/litellm-rust/crates/python-bridge/src/routes/inference.rs b/litellm-rust/crates/python-bridge/src/routes/inference.rs index c9eb60e47ed..1245c126c4e 100644 --- a/litellm-rust/crates/python-bridge/src/routes/inference.rs +++ b/litellm-rust/crates/python-bridge/src/routes/inference.rs @@ -1,16 +1,14 @@ use std::sync::Arc; use litellm_auth::AuthServices; -use litellm_cache_response::{CachePolicy, CacheScope, ScopedCache}; +use litellm_cache_response::CachePolicy; use litellm_callbacks_legacy_python::LoggingOperation; use litellm_core_utils::get_llm_provider_logic::get_custom_llm_provider; use litellm_host::{call::HostedMachine, protocol::Protocol}; use litellm_host_python::{PythonBinding, PythonHostCalls, from_py}; -use litellm_http::transport::Error as TransportError; use litellm_inference::Error; use litellm_secrets::source::SecretSource; use pyo3::{ - exceptions::PyValueError, gc::{PyTraverseError, PyVisit}, prelude::*, types::PyDict, @@ -20,8 +18,10 @@ use serde_json::{Map, Value}; use super::{NativeCall, RequestView}; use crate::{ - errors::{RustUpstreamError, route_error_to_pyerr}, - marshal::{RouteOptions, optional_timeout, public_response, python_timeout_seconds}, + errors::{RequestMarker, route_failure}, + marshal::{ + ArgumentError, RouteOptions, optional_timeout, public_response, python_timeout_seconds, + }, }; pub(super) struct InferenceHost { @@ -44,7 +44,7 @@ impl InferenceHost { &mut self, py: Python<'_>, arguments: &Bound<'_, PyDict>, - input: &str, + input: &'static str, ) -> PyResult { self.view.resolve(py, arguments)?; let argument = |name: &str| self.argument(py, arguments, name); @@ -57,7 +57,7 @@ impl InferenceHost { .map(|value| python_timeout_seconds(py, value.unbind())) .transpose()? .flatten(); - let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?; + let model = string("model")?.ok_or(ArgumentError::Required("model"))?; let custom_llm_provider = string("custom_llm_provider")?; let provider = get_custom_llm_provider(&model, custom_llm_provider.as_deref()) .map_or("", |resolved| resolved.custom_llm_provider); @@ -82,10 +82,7 @@ impl InferenceHost { .transpose()?, timeout: optional_timeout(timeout), }, - input: from_py( - &argument(input)? - .ok_or_else(|| PyValueError::new_err(format!("{input} is required")))?, - )?, + input: from_py(&argument(input)?.ok_or(ArgumentError::Required(input))?)?, params, }) } @@ -120,29 +117,20 @@ impl InferenceHost { } pub fn error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source.source_error()) - { - return Ok(original); - } - let native = match error { - Error::Transport(TransportError::Http { status, body }) => { - let error = RustUpstreamError::new_err((status, body)); - error - .value(py) - .setattr("headers", Vec::<(String, String)>::new())?; - error - } - other => route_error_to_pyerr(other), - }; - let mapped = py - .import(self.module)? - .getattr("map_failure")? - .call1((native.value(py), self.view.request(py)?))?; - Ok(PyErr::from_value(mapped)) + route_failure(py, error, RequestMarker::Unmarked, |native| { + let mapped = py + .import(self.module)? + .getattr("map_failure")? + .call1((native.value(py), self.view.request(py)?))?; + Ok(PyErr::from_value(mapped)) + }) } } +pub(super) fn host_call_error(_: &PyErr) -> Error { + Error::invalid_request(litellm_llms::ErrorDetail::HostCall) +} + pub(super) trait InferenceRoute: Sized + 'static { type Protocol: Protocol; const OPERATION: LoggingOperation; @@ -154,7 +142,6 @@ pub(super) trait InferenceRoute: Sized + 'static { auth: Arc, secrets: Arc, ) -> Self; - fn with_cache(self, cache: ScopedCache) -> Self; fn machine( self, call: ::Request, @@ -188,12 +175,8 @@ where crate::http::resources().auth.clone(), crate::secrets::source(py)?, ); - let (cache, cache_options) = crate::cache::configured_native(py, arguments, call_type)?; - let route = match cache { - Some(cache) => route.with_cache(ScopedCache::new(cache, CacheScope::Shared)), - None => route, - }; - Ok(route.machine(request, cache_options.policy)) + crate::cache::ensure_cache_disabled(py, arguments, call_type)?; + Ok(route.machine(request, CachePolicy::default())) }, host, hooks, diff --git a/litellm-rust/crates/python-bridge/src/routes/messages/host.rs b/litellm-rust/crates/python-bridge/src/routes/messages/host.rs index 5e445fcaa7d..af614f17360 100644 --- a/litellm-rust/crates/python-bridge/src/routes/messages/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/messages/host.rs @@ -3,7 +3,6 @@ use litellm_host_python::{PythonHostCalls, PythonOwned}; use bytes::Bytes; use litellm_host_python::{InvokeError, PythonBinding, from_py, to_py}; -use litellm_http::transport::Error as TransportError; use litellm_inference_messages::{ Error, MessagesCall, MessagesSettings, MessagesShaping, litellm_params, messages_body, route::{Messages, MessagesStreamHead}, @@ -12,7 +11,7 @@ use litellm_llms::base_llm::messages::context::MessagesModelCapabilities; use litellm_llms_types::headers::ProviderSpecificHeaders; use litellm_router_types::LitellmParams; use pyo3::{ - exceptions::{PyException, PyValueError}, + exceptions::PyException, gc::{PyTraverseError, PyVisit}, prelude::*, types::{PyBytes, PyDict}, @@ -20,24 +19,24 @@ use pyo3::{ use serde_json::{Map, Value}; use crate::{ - errors::{RustUpstreamError, route_error_to_pyerr}, - marshal::{optional_timeout, project_optional_fields, public_response, python_timeout_seconds}, + errors::{RequestMarker, route_failure}, + marshal::{ + ArgumentError, optional_timeout, project_optional_fields, public_response, + python_timeout_seconds, + }, python_settings::missing_module, routes::RequestView, }; const ROUTE_HOST_MODULE: &str = "litellm.rust_bridge.host.routes.messages"; -const REQUEST_ERROR_MARKER: &str = "messages_request_error"; fn merge_headers( forwarded: Option>, extra_headers: Option>, ) -> Option> { - let merged: Map = forwarded - .into_iter() - .flatten() - .chain(extra_headers.into_iter().flatten()) - .collect(); + let merged = litellm_inference_messages::merge_messages_header_sources( + forwarded.into_iter().chain(extra_headers), + ); (!merged.is_empty()).then_some(merged) } @@ -68,29 +67,6 @@ fn project_litellm_params<'py>( Ok(litellm_params(fields.into_iter().chain(folded).collect())) } -fn native_error(py: Python<'_>, error: Error) -> PyResult { - match error { - Error::Transport(TransportError::Http { status, body }) => { - let error = RustUpstreamError::new_err((status, body)); - error - .value(py) - .setattr("headers", Vec::<(String, String)>::new())?; - Ok(error) - } - Error::InvalidRequest(message) => { - let error = PyValueError::new_err(message.to_string()); - error.value(py).setattr(REQUEST_ERROR_MARKER, true)?; - Ok(error) - } - Error::MissingField(field) => { - let error = PyValueError::new_err(format!("missing required field: {field}")); - error.value(py).setattr(REQUEST_ERROR_MARKER, true)?; - Ok(error) - } - other => Ok(route_error_to_pyerr(other)), - } -} - /// The Python side of the Messages route: projects the prepared arguments and builds the /// public response, chunks and exceptions. pub(super) struct MessagesPythonHost { @@ -115,9 +91,8 @@ impl MessagesPythonHost { let string = |name: &str| -> PyResult> { argument(name)?.map(|value| value.extract()).transpose() }; - let model = string("model")?.ok_or_else(|| PyValueError::new_err("model is required"))?; - let messages = - argument("messages")?.ok_or_else(|| PyValueError::new_err("messages is required"))?; + let model = string("model")?.ok_or(ArgumentError::Required("model"))?; + let messages = argument("messages")?.ok_or(ArgumentError::Required("messages"))?; let fields = self.view.body_params(py, arguments, ROUTE_HOST_MODULE)?; let body = [ ("model".to_string(), Value::String(model.clone())), @@ -130,10 +105,19 @@ impl MessagesPythonHost { .map(|value| python_timeout_seconds(py, value.unbind())) .transpose()? .flatten(); - let custom_llm_provider = string("custom_llm_provider")?; - let shaping = self.shaping(py, &model, custom_llm_provider.as_deref(), arguments)?; let api_key = string("api_key")?; let api_base = string("api_base")?; + // Python's handler resolves a bare model name (`claude-sonnet-4-5`) through + // `get_llm_provider`'s model lists, which the native prefix lookup does not carry. + let custom_llm_provider = match string("custom_llm_provider")? { + Some(provider) => Some(provider), + None => py + .import(ROUTE_HOST_MODULE)? + .getattr("inferred_provider")? + .call1((&model, api_base.as_deref()))? + .extract()?, + }; + let shaping = self.shaping(py, &model, custom_llm_provider.as_deref(), arguments)?; let extra_headers = self.merged_headers(py, arguments)?; let provider_specific_header = self.provider_specific_header(py, arguments)?; let litellm_params = project_litellm_params(argument, |name| module_global(py, name))?; @@ -276,16 +260,13 @@ impl PythonBinding for MessagesPythonHost { } fn map_error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source.source_error()) - { - return Ok(original); - } - Ok(self.map_failure(py, native_error(py, error)?)) + route_failure(py, error, RequestMarker::Messages, |native| { + Ok(self.map_failure(py, native)) + }) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } @@ -459,33 +440,9 @@ mod tests { projected, litellm_params(serde_json::from_value(fields).unwrap()).unwrap() ), - (Err(error), Err(())) => assert!(matches!(error, Error::InvalidRequest(_))), + (Err(error), Err(())) => assert!(error.is_request(), "{error:?}"), (projected, expected) => panic!("got {projected:?}, expected {expected:?}"), } }); } - - #[rstest] - #[case::rejected_request(Error::InvalidRequest("does not support top_k=5".into()), true)] - #[case::missing_field(Error::MissingField("max_tokens"), true)] - #[case::unresolvable_provider(Error::InvalidProvider("openai".into()), false)] - #[case::upstream_failure( - Error::Transport(TransportError::Http { status: 400, body: "bad".into() }), - false, - )] - fn only_request_rejections_carry_the_request_error_marker( - #[case] error: Error, - #[case] marked: bool, - ) { - Python::initialize(); - Python::attach(|py| { - let native = native_error(py, error).unwrap(); - let marker = native - .value(py) - .getattr_opt(REQUEST_ERROR_MARKER) - .unwrap() - .map(|value| value.extract::().unwrap()); - assert_eq!(marker.unwrap_or(false), marked); - }); - } } diff --git a/litellm-rust/crates/python-bridge/src/routes/messages/mod.rs b/litellm-rust/crates/python-bridge/src/routes/messages/mod.rs index c788760b268..a9ae01bd7ee 100644 --- a/litellm-rust/crates/python-bridge/src/routes/messages/mod.rs +++ b/litellm-rust/crates/python-bridge/src/routes/messages/mod.rs @@ -50,9 +50,18 @@ fn run_messages(py: Python<'_>, call: NativeCall<'_>, asynchronous: bool) -> PyR ) } +/// Python's sync Messages handler raises for every native provider before it sends anything +/// or opens a logging object (`BaseLLMHTTPHandler.anthropic_messages_handler`), and Rust copies +/// that until both sides serve sync calls. +const SYNC_NOT_IMPLEMENTED: &str = "anthropic_messages_handler is not implemented for sync calls"; + #[pyfunction] -pub(crate) fn messages(py: Python<'_>, call: NativeCall<'_>) -> PyResult> { - run_messages(py, call, false) +#[pyo3(signature = (call))] +pub(crate) fn messages(call: NativeCall<'_>) -> PyResult> { + drop(call); + Err(pyo3::exceptions::PyValueError::new_err( + SYNC_NOT_IMPLEMENTED, + )) } #[pyfunction] diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs index f85d7d4c7cb..71eb743138d 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/document.rs @@ -9,6 +9,29 @@ use pyo3::{ types::{PyBytes, PyString, PyType}, }; +/// A `type='file'` document the bridge cannot read, with the text Python's OCR path raises. +#[derive(Debug, thiserror::Error)] +enum DocumentError { + #[error( + "document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes" + )] + MissingFile, + #[error( + "OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object." + )] + BareString, + #[error( + "Unsupported file input type: {0}. Expected pathlib.Path, bytes, or a file-like object." + )] + UnsupportedType(String), +} + +impl From for PyErr { + fn from(error: DocumentError) -> Self { + PyValueError::new_err(litellm_error::Report::new(&error).to_string()) + } +} + /// A `type='file'` document as projected: paths and bytes are typed inputs already; a /// file-like object is a reader the projection consumes once every other field is read. pub(super) enum FileDocumentInput { @@ -45,25 +68,18 @@ impl FromPyObject<'_, '_> for FileDocumentInput { Err(error) if error.is_instance_of::(py) => None, Err(error) => return Err(error), }; - let missing = || { - PyValueError::new_err( - "document with type='file' must include a 'file' field containing a pathlib.Path, file-like object, or bytes", - ) - }; - let file = document.get_item("file").map_err(|error| { - if error.is_instance_of::(py) { - missing() - } else { - error + let file = match document.get_item("file") { + Ok(file) => file, + Err(error) if error.is_instance_of::(py) => { + return Err(DocumentError::MissingFile.into()); } - })?; + Err(error) => return Err(error), + }; if file.is_none() { - return Err(missing()); + return Err(DocumentError::MissingFile.into()); } if file.is_instance_of::() { - return Err(PyValueError::new_err( - "OCR file input does not accept bare str values. Pass bytes, a pathlib.Path, or a file-like object.", - )); + return Err(DocumentError::BareString.into()); } static PATH_LIKE: PyOnceLock> = PyOnceLock::new(); if file.is_instance(PATH_LIKE.import(py, "os", "PathLike")?)? { @@ -81,10 +97,7 @@ impl FromPyObject<'_, '_> for FileDocumentInput { } match PythonFileReader::from_file_like(&file)? { Some(reader) => Ok(Self::Deferred { reader, mime_type }), - None => Err(PyValueError::new_err(format!( - "Unsupported file input type: {}. Expected pathlib.Path, bytes, or a file-like object.", - file.get_type(), - ))), + None => Err(DocumentError::UnsupportedType(file.get_type().to_string()).into()), } } } @@ -92,6 +105,7 @@ impl FromPyObject<'_, '_> for FileDocumentInput { #[cfg(test)] mod tests { use pyo3::{exceptions::PyTypeError, types::PyDict}; + use rstest::rstest; use super::*; @@ -108,40 +122,47 @@ mod tests { } } - #[test] - fn extraction_validates_required_file_and_optional_mime_type() { + enum Rejection { + Value(&'static str), + Type, + } + + #[rstest] + #[case::missing_file(c"{}", Rejection::Value("must include a 'file' field"))] + #[case::none_file(c"{'file': None}", Rejection::Value("must include a 'file' field"))] + #[case::none_mime_type(c"{'file': b'abc', 'mime_type': None}", Rejection::Type)] + #[case::numeric_mime_type(c"{'file': b'abc', 'mime_type': 7}", Rejection::Type)] + #[case::bare_str(c"{'file': 'scan.pdf'}", Rejection::Value("bare str"))] + #[case::unsupported_type( + c"{'file': object()}", + Rejection::Value("Unsupported file input type") + )] + fn extraction_rejects_invalid_documents( + #[case] expression: &std::ffi::CStr, + #[case] rejection: Rejection, + ) { Python::initialize(); Python::attach(|py| { - for expression in [c"{}", c"{'file': None}"] { - let document = py.eval(expression, None, None).unwrap(); - let error = document.extract::().err().unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("must include a 'file' field")); - } - for expression in [ - c"{'file': b'abc', 'mime_type': None}", - c"{'file': b'abc', 'mime_type': 7}", - ] { - let document = py.eval(expression, None, None).unwrap(); - let error = document.extract::().err().unwrap(); - assert!(error.is_instance_of::(py)); - } let error = py - .eval(c"{'file': 'scan.pdf'}", None, None) + .eval(expression, None, None) .unwrap() .extract::() .err() .unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("bare str")); - let error = py - .eval(c"{'file': object()}", None, None) - .unwrap() - .extract::() - .err() - .unwrap(); - assert!(error.is_instance_of::(py)); - assert!(error.to_string().contains("Unsupported file input type")); + match rejection { + Rejection::Value(fragment) => { + assert!(error.is_instance_of::(py)); + assert!(error.to_string().contains(fragment), "{error}"); + } + Rejection::Type => assert!(error.is_instance_of::(py)), + } + }); + } + + #[test] + fn extraction_keeps_bytes_and_their_mime_type() { + Python::initialize(); + Python::attach(|py| { let document = py .eval(c"{'file': b'abc', 'mime_type': 'image/png'}", None, None) .unwrap(); diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs index 2068b6a6e4b..31736c7b654 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/errors.rs @@ -1,128 +1,203 @@ +use std::io::ErrorKind; + +use litellm_error::Report; use litellm_llms::base_llm::ocr::error::Error; use pyo3::{ exceptions::{PyFileNotFoundError, PyOSError}, prelude::*, }; -use crate::errors::{RustUpstreamError, by_fault}; +use crate::errors::{by_fault, upstream_error}; + +const REQUEST_FORMAT_ERROR: &str = "ocr_request_format_error"; pub(super) fn to_pyerr(error: Error) -> PyErr { - let status = error.http_status_code(); - let mapped = Python::attach(|py| -> PyResult { - Ok(match error { - Error::Provider { - status, - body, - headers, - } => upstream_error(py, status, body, headers)?, - Error::Transport(litellm_http::transport::Error::Http { status, body }) => { - upstream_error(py, status, body, Vec::new())? - } - Error::RequestFormat => { - let error = by_fault(true, Error::RequestFormat.to_string()); - error - .value(py) - .setattr("ocr_request_format_error", true) - .ok(); - error - } - Error::FileRead { path, source } if source.kind() == std::io::ErrorKind::NotFound => { - PyFileNotFoundError::new_err(format!("File not found: {}", path.display())) - } - Error::FileRead { source, .. } => PyOSError::new_err(source.to_string()), - other => by_fault(is_request(&other), other.to_string()), - }) - }) - .unwrap_or_else(|error| error); - attach_status(mapped, status) + Python::attach(|py| native(py, error).unwrap_or_else(std::convert::identity)) } -fn is_request(error: &Error) -> bool { - error.is_request() - || matches!( - error, - Error::Auth(_) - | Error::InvalidProvider(_) - | Error::InvalidRequest(_) - | Error::MissingField(_) - | Error::MissingDocumentUrl - ) -} - -fn upstream_error( +/// Maps an OCR route failure once. A Python exception raised inside the call is returned as it +/// is, everything else is mapped and handed to `public`, the route's public exception mapping. +pub(super) fn route_failure( py: Python<'_>, - status: u16, - body: String, - headers: Vec<(String, String)>, + error: Error, + public: impl FnOnce(PyErr) -> PyResult, ) -> PyResult { - let error = RustUpstreamError::new_err((status, body)); - error.value(py).setattr("headers", headers)?; - Ok(error) + match crate::secrets::python_error(py, &error) { + Some(raised) => Ok(raised), + None => public(native(py, error)?), + } } -fn attach_status(error: PyErr, status: Option) -> PyErr { - if let Some(status) = status { - Python::attach(|py| { - let value = error.value(py); - value.setattr("status_code", status).ok(); - value.setattr("message", value.to_string()).ok(); - }); +fn native(py: Python<'_>, error: Error) -> PyResult { + let mapped = match (&error, error.upstream_response()) { + (_, Some((status, body, headers))) => upstream_error(py, status, body, headers)?, + (Error::RequestFormat, None) => { + let mapped = by_fault(true, &error); + mapped.value(py).setattr(REQUEST_FORMAT_ERROR, true)?; + mapped + } + (Error::FileRead { path, cause }, None) if cause.kind() == ErrorKind::NotFound => { + PyFileNotFoundError::new_err(format!("File not found: {}", path.display())) + } + (Error::FileRead { cause, .. }, None) => { + PyOSError::new_err(Report::new(cause.as_ref()).to_string()) + } + (_, None) => by_fault(error.is_request(), &error), + }; + if let Some(status) = error.http_status_code() { + let value = mapped.value(py); + value.setattr("status_code", status)?; + value.setattr("message", value.str()?)?; } - error + Ok(mapped) } #[cfg(test)] mod tests { - use pyo3::exceptions::PyValueError; + use pyo3::exceptions::{PyRuntimeError, PyValueError}; + use rstest::rstest; use super::*; + use crate::errors::RustUpstreamError; - #[test] - fn preserves_python_validation_and_provider_details() { + fn status_code(py: Python<'_>, mapped: &PyErr) -> Option { + mapped + .value(py) + .getattr_opt("status_code") + .unwrap() + .map(|status| status.extract().unwrap()) + } + + fn file_read(kind: ErrorKind) -> Error { + Error::FileRead { + path: "/missing/scan.pdf".into(), + cause: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")), + } + } + + #[rstest] + #[case::provider( + Error::Provider { + status: 429, + body: r#"{"message":"rate limited"}"#.into(), + headers: Box::new(reqwest::header::HeaderMap::from_iter([( + reqwest::header::RETRY_AFTER, + reqwest::header::HeaderValue::from_static("17"), + )])), + }, + vec![("retry-after".into(), "17".into())] + )] + #[case::transport(litellm_http::Error::upstream(429, r#"{"message":"rate limited"}"#.into()).into(), vec![])] + fn a_provider_answer_keeps_its_status_body_and_headers( + #[case] error: Error, + #[case] headers: Vec<(String, String)>, + ) { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(Error::MissingDocumentUrl); - assert!(mapped.is_instance_of::(py)); - assert_eq!(mapped.value(py).to_string(), "Document URL is required"); - assert_eq!( - mapped - .value(py) - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); - let mapped = to_pyerr(Error::Provider { - status: 429, - body: r#"{"message":"rate limited"}"#.to_string(), - headers: vec![("Retry-After".to_string(), "17".to_string())], - }); + let mapped = to_pyerr(error); assert!(mapped.is_instance_of::(py)); - let headers: Vec<(String, String)> = mapped - .value(py) - .getattr("headers") - .and_then(|headers| headers.extract()) - .expect("OCR failures retain provider headers"); - assert_eq!(headers, vec![("Retry-After".to_string(), "17".to_string())]); - let args: (u16, String) = mapped - .value(py) - .getattr("args") - .and_then(|args| args.extract()) - .expect("OCR failures retain status and unprefixed provider message"); - assert_eq!(args, (429, r#"{"message":"rate limited"}"#.to_string())); + let value = mapped.value(py); + let args: (u16, String) = value.getattr("args").unwrap().extract().unwrap(); + assert_eq!(args, (429, r#"{"message":"rate limited"}"#.into())); + let kept: Vec<(String, String)> = value.getattr("headers").unwrap().extract().unwrap(); + assert_eq!(kept, headers); + assert_eq!(status_code(py, &mapped), Some(429)); + }); + } - let mapped = to_pyerr(Error::InvalidRequest("invalid format".into())); + #[rstest] + #[case::not_found(Some(404), 404, "OCR document download failed")] + #[case::rate_limited(Some(429), 429, "OCR document download failed")] + #[case::timeout(None, 408, "OCR document download timed out")] + #[tokio::test] + async fn document_download_failures_keep_the_upstream_exception( + #[case] response_status: Option, + #[case] expected_status: u16, + #[case] expected_body: &str, + ) { + use std::time::Duration; + + use litellm_http::{Client, media::MediaFetcher}; + use litellm_llms::base_llm::ocr::{ + document::inline_remote_document, transformation::OcrConnection, + }; + use litellm_llms_types::formats::ocr::OcrDocument; + use tokio::{ + io::{AsyncReadExt, AsyncWriteExt}, + net::TcpListener, + }; + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let server = tokio::spawn(async move { + let (mut socket, _) = listener.accept().await.unwrap(); + let mut request = [0_u8; 1]; + socket.read_exact(&mut request).await.unwrap(); + if let Some(status) = response_status { + socket.write_all(format!("HTTP/1.1 {status} Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n").as_bytes()).await.unwrap(); + } else { + std::future::pending::<()>().await; + } + }); + let connection = OcrConnection { + timeout: Duration::from_millis(100), + ..OcrConnection::default() + }; + let error = inline_remote_document( + &MediaFetcher::for_test(Client::for_test()), + OcrDocument::DocumentUrl { + document_url: format!("http://{address}/document.pdf"), + extra_fields: Default::default(), + }, + &connection, + ) + .await + .unwrap_err(); + server.abort(); + if response_status.is_none() { + assert!(litellm_error::find::(&error).is_some()); + } + Python::initialize(); + Python::attach(|py| { + let mapped = to_pyerr(error); + assert!(mapped.is_instance_of::(py), "{mapped}"); + let args: (u16, String) = mapped.value(py).getattr("args").unwrap().extract().unwrap(); + assert_eq!(args, (expected_status, expected_body.into())); + assert_eq!(status_code(py, &mapped), Some(expected_status)); + }); + } + + #[rstest] + #[case::missing_document_url(Error::MissingDocumentUrl)] + #[case::invalid_features(Error::Features)] + #[case::invalid_provider(Error::InvalidProvider("unknown".into()))] + fn request_faults_are_value_errors_with_status_400(#[case] error: Error) { + Python::initialize(); + Python::attach(|py| { + let report = Report::new(&error).to_string(); + let mapped = to_pyerr(error); assert!(mapped.is_instance_of::(py)); - assert_eq!( - mapped - .value(py) - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); + assert_eq!(mapped.value(py).to_string(), report); + assert_eq!(status_code(py, &mapped), Some(400)); + }); + } + + #[rstest] + #[case::missing_api_key(litellm_auth::Error::MissingApiKey { + provider: "Mistral", + environment_variable: "MISTRAL_API_KEY", + } + .into())] + #[case::missing_credential(Error::MissingCredential { + credential: "Azure AI credentials", + guidance: "set AZURE_AI_API_KEY", + })] + fn a_missing_credential_is_a_statusless_runtime_error(#[case] error: Error) { + Python::initialize(); + Python::attach(|py| { + let mapped = to_pyerr(error); + assert!(mapped.is_instance_of::(py)); + assert_eq!(status_code(py, &mapped), None); }); } @@ -135,19 +210,12 @@ mod tests { assert!(mapped.is_instance_of::(py)); assert!( value - .getattr("ocr_request_format_error") + .getattr(REQUEST_FORMAT_ERROR) .unwrap() .extract::() .unwrap() ); - assert_eq!( - value - .getattr("status_code") - .unwrap() - .extract::() - .unwrap(), - 400 - ); + assert_eq!(status_code(py, &mapped), Some(400)); assert_eq!( value .getattr("message") @@ -159,18 +227,11 @@ mod tests { }); } - fn file_read(kind: std::io::ErrorKind) -> Error { - Error::FileRead { - path: "/missing/scan.pdf".into(), - source: std::sync::Arc::new(std::io::Error::new(kind, "disk said no")), - } - } - #[test] fn missing_files_map_to_file_not_found_naming_the_path() { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(file_read(std::io::ErrorKind::NotFound)); + let mapped = to_pyerr(file_read(ErrorKind::NotFound)); assert!(mapped.is_instance_of::(py)); assert_eq!( mapped.value(py).to_string(), @@ -183,28 +244,32 @@ mod tests { fn other_file_read_failures_map_to_os_error_with_the_io_message() { Python::initialize(); Python::attach(|py| { - let mapped = to_pyerr(file_read(std::io::ErrorKind::PermissionDenied)); + let mapped = to_pyerr(file_read(ErrorKind::PermissionDenied)); assert!(mapped.is_instance_of::(py)); assert!(!mapped.is_instance_of::(py)); assert_eq!(mapped.value(py).to_string(), "disk said no"); }); } - #[rstest::rstest] + #[rstest] #[case::oversized(Error::TooLarge { limit: 7 })] - #[case::malformed_field(Error::ResponseField { path: "pages[0].index".into() })] - fn response_failures_are_statusless_runtime_errors(#[case] error: Error) { + #[case::malformed_field(Error::response_field( + "pages[0].index", + "-1".parse::().unwrap_err() + ))] + fn response_failures_are_statusless_runtime_errors( + #[case] error: Error, + #[values("status_code", "ocr_request_format_error", "headers")] attribute: &str, + ) { Python::initialize(); Python::attach(|py| { - let message = error.to_string(); + let report = Report::new(&error).to_string(); let mapped = to_pyerr(error); let value = mapped.value(py); - assert!(mapped.is_instance_of::(py)); + assert!(mapped.is_instance_of::(py)); assert!(!mapped.is_instance_of::(py)); - assert_eq!(value.to_string(), message); - for attribute in ["status_code", "ocr_request_format_error", "headers"] { - assert!(!value.hasattr(attribute).unwrap(), "{attribute}"); - } + assert_eq!(value.to_string(), report); + assert!(!value.hasattr(attribute).unwrap()); }); } } diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs index fc1b70392e2..c21ce6d76b6 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/host.rs @@ -12,8 +12,8 @@ use pyo3::{ }; use super::{ - errors::to_pyerr as ocr_error_to_pyerr, - project::{OcrHostHandles, SETTINGS_ERROR_MARKER, project_request}, + errors::route_failure, + project::{OcrHostHandles, ROUTE_HOST_MODULE, SETTINGS_ERROR_MARKER, project_request}, }; use crate::{marshal::public_response, routes::RequestView}; @@ -77,7 +77,7 @@ impl OcrPythonHost { _ => "", }; let mapped = py - .import("litellm.rust_bridge.host.routes.ocr") + .import(ROUTE_HOST_MODULE) .and_then(|module| module.getattr("map_failure")) .and_then(|map| map.call1((error.value(py), self.view.request(py)?, provider))) .and_then(|mapped| mapped.extract::>().map_err(PyErr::from)); @@ -115,7 +115,7 @@ impl PythonBinding for OcrPythonHost { py: Python<'_>, response: LiteLLMOcrResponse, ) -> PyResult> { - public_response(py, "litellm.rust_bridge.host.routes.ocr", &response) + public_response(py, ROUTE_HOST_MODULE, &response) } fn encode_stream_head( @@ -135,16 +135,11 @@ impl PythonBinding for OcrPythonHost { } fn map_error(&self, py: Python<'_>, error: Error) -> PyResult { - if let Error::Secret(source) = &error - && let Some(original) = crate::secrets::python_error(py, source) - { - return Ok(original); - } - Ok(self.map_failure(py, ocr_error_to_pyerr(error))) + route_failure(py, error, |native| Ok(self.map_failure(py, native))) } - fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string()) + fn host_error(_: &PyErr) -> Error { + Error::HostCall } } diff --git a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs index fe2f83bff46..73a2922804a 100644 --- a/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs +++ b/litellm-rust/crates/python-bridge/src/routes/ocr/project.rs @@ -1,22 +1,26 @@ -use litellm_auth::SecretValue; use litellm_auth::VertexParams; use litellm_host_python::{from_py, present}; use litellm_inference_ocr::{ types::{LiteLLMOcrRequest, OcrDocumentInput}, wire::{OcrWireRequest, consumed_optional_params, decode_document, decode_request_input}, }; -use litellm_llms::base_llm::ocr::error::Error; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use litellm_llms::base_llm::ocr::{error::Error, transformation::DROP_PARAMS_PARAM}; +use litellm_secret_value::SecretValue; +use pyo3::{prelude::*, types::PyDict}; use serde_json::{Map, Value}; use super::{document::FileDocumentInput, errors::to_pyerr as ocr_error_to_pyerr}; use crate::{ coercion::FieldSpec, credentials::{self, CallerTokenProvider}, - marshal::{project_optional_fields, python_timeout_seconds, request_input_sources}, + marshal::{ + ArgumentError, project_optional_fields, python_timeout_seconds, request_input_sources, + }, python_settings::{PythonSettings, Snapshot}, }; +pub(super) const ROUTE_HOST_MODULE: &str = "litellm.rust_bridge.host.routes.ocr"; + /// What the host keeps after projection: the caller's token callable that answers the /// token operation, and the provider name the failure mapping reports. pub(super) struct OcrHostHandles { @@ -30,9 +34,9 @@ struct OcrArguments<'a, 'py> { } impl<'py> OcrArguments<'_, 'py> { - fn lookup(&self, name: &str) -> PyResult> { - litellm_host_python::lookup(self.kwargs, self.bound, name)? - .ok_or_else(|| PyValueError::new_err(format!("missing argument: {name}"))) + fn lookup(&self, name: &'static str) -> PyResult> { + Ok(litellm_host_python::lookup(self.kwargs, self.bound, name)? + .ok_or(ArgumentError::Required(name))?) } fn model(&self) -> PyResult { @@ -92,6 +96,7 @@ impl ProjectedDocument { { ocr_error_to_pyerr(Error::RequestField { path: "document.type".into(), + cause: None, }) } else { error @@ -136,6 +141,15 @@ pub(super) fn project_request( Some(defaults) => fold_module_globals(bound.py(), optional_params, &globals, &defaults)?, None => optional_params, }; + let optional_params = if names.contains(&DROP_PARAMS_PARAM) { + fold_drop_params( + bound.py(), + optional_params, + present(kwargs, bound, DROP_PARAMS_PARAM)?, + )? + } else { + optional_params + }; let input_sources = request_input_sources( kwargs, names @@ -213,13 +227,32 @@ fn fold_module_globals( .collect()) } +fn fold_drop_params( + py: Python<'_>, + optional_params: Map, + requested: Option>, +) -> PyResult> { + let drops: bool = py + .import(ROUTE_HOST_MODULE)? + .getattr("drops_unsupported_params")? + .call1((requested,))? + .extract()?; + Ok(optional_params + .into_iter() + .filter(|(name, _)| name != DROP_PARAMS_PARAM) + .chain(drops.then(|| (DROP_PARAMS_PARAM.to_string(), Value::Bool(true)))) + .collect()) +} + /// A misconfigured litellm setting is the operator's error, not the request's or the /// provider's, so the host raises it as is instead of mapping it onto a provider failure. pub(super) const SETTINGS_ERROR_MARKER: &str = "native_settings_error"; fn settings_error(py: Python<'_>, error: PyErr) -> PyErr { - error.value(py).setattr(SETTINGS_ERROR_MARKER, true).ok(); - error + match error.value(py).setattr(SETTINGS_ERROR_MARKER, true) { + Ok(()) => error, + Err(failure) => failure, + } } #[cfg(test)] diff --git a/litellm-rust/crates/python-bridge/src/routes/responses/host.rs b/litellm-rust/crates/python-bridge/src/routes/responses/host.rs index 9c430ba7868..b797e334c6a 100644 --- a/litellm-rust/crates/python-bridge/src/routes/responses/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/responses/host.rs @@ -1,3 +1,4 @@ +use litellm_core_utils::decline::Decline; use std::convert::Infallible; use crate::routes::inference::InferenceHost; @@ -118,9 +119,9 @@ impl PythonBinding for ResponsesPythonHost { .get("stream") .is_some_and(|value| value == &serde_json::Value::Bool(true)) { - return Err(InvokeError::Native(Error::Unsupported( + return Err(InvokeError::Native(Error::Declined(Decline::new( "native Python responses streaming", - ))); + )))); } Ok(call) } @@ -141,8 +142,10 @@ impl PythonBinding for ResponsesPythonHost { head: ::StreamHead, ) -> PyResult> { let _ = head; - Err(pyo3::exceptions::PyRuntimeError::new_err( - "native Python Responses streaming is not supported", + Err(crate::errors::route_error_to_pyerr( + Error::invalid_response(litellm_llms::ErrorDetail::Unhandled( + "native Python responses streaming", + )), )) } @@ -152,8 +155,10 @@ impl PythonBinding for ResponsesPythonHost { chunk: ::Chunk, ) -> PyResult> { let _ = chunk; - Err(pyo3::exceptions::PyRuntimeError::new_err( - "native Python Responses streaming is not supported", + Err(crate::errors::route_error_to_pyerr( + Error::invalid_response(litellm_llms::ErrorDetail::Unhandled( + "native Python responses streaming", + )), )) } @@ -161,7 +166,7 @@ impl PythonBinding for ResponsesPythonHost { self.host.error(py, error) } fn host_error(error: &PyErr) -> Error { - Error::InvalidRequest(error.to_string().into()) + crate::routes::inference::host_call_error(error) } } diff --git a/litellm-rust/crates/python-bridge/src/routes/responses/mod.rs b/litellm-rust/crates/python-bridge/src/routes/responses/mod.rs index 9c045d65c13..cc4d85b4d26 100644 --- a/litellm-rust/crates/python-bridge/src/routes/responses/mod.rs +++ b/litellm-rust/crates/python-bridge/src/routes/responses/mod.rs @@ -5,7 +5,7 @@ use std::sync::Arc; use host::ResponsesPythonHost; use litellm_auth::AuthServices; -use litellm_cache_response::{CachePolicy, ScopedCache}; +use litellm_cache_response::CachePolicy; use litellm_callbacks_legacy_python::LoggingOperation; use litellm_host::{call::HostedMachine, protocol::Protocol}; use litellm_inference_responses::{ResponsesRoute, route::Responses}; @@ -49,10 +49,6 @@ impl InferenceRoute for ResponsesRoute { Self::new(http, auth, secrets) } - fn with_cache(self, cache: ScopedCache) -> Self { - self.with_cache(cache) - } - fn machine( self, call: ::Request, diff --git a/litellm-rust/crates/python-bridge/src/routes/responses/websocket.rs b/litellm-rust/crates/python-bridge/src/routes/responses/websocket.rs index 615a838a597..c89185dc21f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/responses/websocket.rs +++ b/litellm-rust/crates/python-bridge/src/routes/responses/websocket.rs @@ -1,7 +1,9 @@ -use litellm_inference_responses::websocket::ResponsesWebSocketConnection as RustResponsesWebSocketConnection; use pyo3::prelude::*; use serde_json::Value; +use litellm_http::ClientVariant; +use litellm_inference_responses::websocket::ResponsesWebSocketConnection as RustResponsesWebSocketConnection; + use crate::{ errors::route_error_to_pyerr, marshal::{marshal_headers, optional_timeout}, @@ -25,10 +27,12 @@ impl ResponsesWebSocketConnection { ) -> PyResult> { let headers = marshal_headers(headers)?; let timeout = optional_timeout(timeout_seconds); + let client = crate::http::host_client(py, ClientVariant::WebSocket)?; crate::execution::run_async_value(py, async move { - let inner = RustResponsesWebSocketConnection::connect_url(&url, &headers, timeout) - .await - .map_err(route_error_to_pyerr)?; + let inner = + RustResponsesWebSocketConnection::connect_url(&client, &url, &headers, timeout) + .await + .map_err(route_error_to_pyerr)?; Ok(ResponsesWebSocketConnection { inner }) }) } diff --git a/litellm-rust/crates/python-bridge/src/routes/token_counter.rs b/litellm-rust/crates/python-bridge/src/routes/token_counter.rs index 2c26311231f..4b10eed087f 100644 --- a/litellm-rust/crates/python-bridge/src/routes/token_counter.rs +++ b/litellm-rust/crates/python-bridge/src/routes/token_counter.rs @@ -2,6 +2,7 @@ use crate::execution::run_async; use std::sync::Arc; use std::{num::NonZero, thread::available_parallelism}; +use litellm_error::{Report, Source}; use litellm_host_python::enter_native; use litellm_token_counter::{ CountableRequest, Error, InputTokenCount, TokenCounter as CoreTokenCounter, @@ -50,10 +51,10 @@ impl TokenCounter { let _slot = encode_slots .acquire_owned() .await - .map_err(|error| Error::Task(error.to_string()))?; + .map_err(|error| Error::Task(Source::new(error)))?; tokio::task::spawn_blocking(move || count_body(&counter, &body)) .await - .map_err(|error| Error::Task(error.to_string()))? + .map_err(|error| Error::Task(Source::new(error)))? }, token_count_error_to_pyerr, ) @@ -70,19 +71,12 @@ fn count_body(counter: &CoreTokenCounter, body: &[u8]) -> Result PyErr { - let message = error.to_string(); - match error { - Error::Load(_) - | Error::Ranks(_) - | Error::UnicodeClasses - | Error::UnsupportedTokenizer(_) => PyValueError::new_err(message), - Error::RequestParse(_) - | Error::MissingInput - | Error::FloatText - | Error::ContentBlock - | Error::ArrayItems - | Error::JsonSerialization(_) - | Error::JsonUtf8(_) => RustBridgeDeclined::new_err(message), - Error::Encode(_) | Error::Decode(_) | Error::Task(_) => PyRuntimeError::new_err(message), + let message = Report::new(&error).to_string(); + if error.is_unsupported() { + RustBridgeDeclined::new_err(message) + } else if error.is_load() { + PyValueError::new_err(message) + } else { + PyRuntimeError::new_err(message) } } diff --git a/litellm-rust/crates/python-bridge/src/secrets/callback.rs b/litellm-rust/crates/python-bridge/src/secrets/callback.rs index 71a06bfdc39..af6485a56e1 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/callback.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/callback.rs @@ -2,9 +2,11 @@ use std::{future::Future, pin::Pin, sync::Arc}; use litellm_core_utils::settings::Lookup; use litellm_host_python::{PythonContext, attach_blocking}; +use litellm_secret_value::SecretValue; use litellm_secrets::{ - Error, ExternalSecretManager, KeyManagementSettings, KeyManagementSystem, Secret, SecretValue, + Error, ExternalSecretManager, KeyManagementSettings, KeyManagementSystem, Secret, }; + use pyo3::{ exceptions::PyException, prelude::*, @@ -192,7 +194,7 @@ handler.get_secret_from_manager = get_secret_from_manager KeyManagementSettings::default(), )), Arc::new(move |_: &str| fallback.map(str::to_owned)), - OidcResolver::new(litellm_http::Client::plain_for_test()), + OidcResolver::new(litellm_http::Client::for_test()), ) .with_failure_policy(FailurePolicy::EnvironmentFallback); (resolver, locals, handler) diff --git a/litellm-rust/crates/python-bridge/src/secrets/config.rs b/litellm-rust/crates/python-bridge/src/secrets/config.rs index d1a1eafd468..c1203dcbd89 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/config.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/config.rs @@ -1,7 +1,9 @@ use std::sync::Arc; +use litellm_secret_value::SecretValue; use litellm_secrets::{SecretManager, SecretManagerState}; -use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem, SecretValue}; +use litellm_secrets_types::{AccessMode, KeyManagementSettings, KeyManagementSystem}; + use pyo3::prelude::*; use serde_json::Value; @@ -128,9 +130,7 @@ pub(crate) fn read(py: Python<'_>) -> PyResult { .system .is_some_and(|system| system != backend.system()) { - return Err(pyo3::exceptions::PyValueError::new_err( - "native secret manager system does not match configuration", - )); + return Err(super::ManagerError::SystemMismatch.into()); } Ok(SecretManagerSnapshot { client: SecretManagerClient::Native(Box::new(backend)), diff --git a/litellm-rust/crates/python-bridge/src/secrets/error.rs b/litellm-rust/crates/python-bridge/src/secrets/error.rs index 5ccfd0a1554..ca51ee79921 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/error.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/error.rs @@ -1,7 +1,66 @@ use std::fmt; +use litellm_error::Report; use litellm_secrets::Error; -use pyo3::{exceptions::PyBaseException, prelude::*}; +use pyo3::{ + exceptions::{PyBaseException, PyRuntimeError, PyValueError}, + prelude::*, +}; + +/// A secret manager call the bridge rejects, with the text Python's own managers raise. +#[derive(Debug, thiserror::Error)] +pub(crate) enum ManagerError { + #[error("native secret manager must be recreated after fork")] + Forked, + #[error("unknown secret manager system")] + UnknownSystem(#[source] serde_json::Error), + #[error("invalid secret manager settings")] + Settings(#[source] litellm_error::Source), + #[error("native secret manager system does not match configuration")] + SystemMismatch, + #[error("secret manager does not support provider reads")] + ProviderReads, + #[error("invalid timeout")] + Timeout(#[source] std::time::TryFromFloatSecsError), + #[error("No secret found in CyberArk Secret Manager for {0}")] + CyberarkSecretMissing(String), + #[error("Current secret {0} not found")] + CurrentMissing(String), + #[error("Failed to verify new secret {0}")] + ReplacementMissing(String), + #[error("New secret value mismatch")] + ReplacementMismatch, + #[error("native secret manager mutation is unavailable")] + MutationUnsupported, + #[error(transparent)] + Secret(Error), +} + +impl ManagerError { + pub(crate) fn settings(error: impl std::error::Error + Send + Sync + 'static) -> Self { + Self::Settings(litellm_error::Source::new(error)) + } +} + +impl From for PyErr { + fn from(error: ManagerError) -> Self { + let message = Report::new(&error).to_string(); + match error { + ManagerError::Forked => PyRuntimeError::new_err(message), + ManagerError::UnknownSystem(_) + | ManagerError::Settings(_) + | ManagerError::SystemMismatch + | ManagerError::ProviderReads + | ManagerError::Timeout(_) + | ManagerError::CyberarkSecretMissing(_) + | ManagerError::CurrentMissing(_) + | ManagerError::ReplacementMissing(_) + | ManagerError::ReplacementMismatch + | ManagerError::MutationUnsupported + | ManagerError::Secret(_) => PyValueError::new_err(message), + } + } +} #[derive(thiserror::Error)] #[error("Python secret manager failed")] @@ -14,18 +73,23 @@ impl fmt::Debug for PythonSecretError { } pub(super) fn external_error(py: Python<'_>, error: PyErr) -> Error { - Error::ExternalManager(Box::new(PythonSecretError(error.into_value(py)))) + Error::ExternalManager(litellm_error::Source::new(PythonSecretError( + error.into_value(py), + ))) } -pub(super) fn read_error(py: Python<'_>, error: PyErr) -> Error { - Error::ExternalRead(Box::new(PythonSecretError(error.into_value(py)))) +pub(crate) fn read_error(py: Python<'_>, error: PyErr) -> Error { + Error::ExternalRead(litellm_error::Source::new(PythonSecretError( + error.into_value(py), + ))) } -pub(crate) fn python_error(py: Python<'_>, error: &Error) -> Option { - let (Error::ExternalManager(source) | Error::ExternalRead(source)) = error else { - return None; - }; - source - .downcast_ref::() +/// The exception a Python secret manager raised anywhere in the chain, so a route surfaces it +/// unchanged instead of wrapping it. +pub(crate) fn python_error( + py: Python<'_>, + error: &(dyn std::error::Error + 'static), +) -> Option { + litellm_error::find::(error) .map(|error| PyErr::from_value(error.0.clone_ref(py).into_bound(py).into_any())) } diff --git a/litellm-rust/crates/python-bridge/src/secrets/mod.rs b/litellm-rust/crates/python-bridge/src/secrets/mod.rs index cddd47d08db..57042e6a32e 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/mod.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/mod.rs @@ -11,7 +11,9 @@ mod vault; use std::sync::Arc; -pub(crate) use error::python_error; +#[cfg(test)] +pub(crate) use error::read_error; +pub(crate) use error::{ManagerError, python_error}; use litellm_secrets::source::{EnvironmentSecrets, SecretSource}; use pyo3::prelude::*; use python::PythonSecrets; diff --git a/litellm-rust/crates/python-bridge/src/secrets/mutation.rs b/litellm-rust/crates/python-bridge/src/secrets/mutation.rs index eaab06a5b12..b04060651ac 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/mutation.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/mutation.rs @@ -1,7 +1,8 @@ use super::operations::{PythonMutationError, PythonMutationResponse}; +use litellm_error::Report; use litellm_host_python::{json_loads, to_py}; use litellm_secrets::cyberark; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; pub(super) fn mutation_value( result: Result, @@ -23,18 +24,7 @@ pub(super) fn mutation_value( &serde_json::json!({"status": "error", "message": message}), ) } - Err(PythonMutationError::CurrentMissing(name)) => Err(PyValueError::new_err(format!( - "Current secret {name} not found" - ))), - Err(PythonMutationError::ReplacementMissing(name)) => Err(PyValueError::new_err(format!( - "Failed to verify new secret {name}" - ))), - Err(PythonMutationError::ReplacementMismatch) => { - Err(PyValueError::new_err("New secret value mismatch")) - } - Err(PythonMutationError::Unsupported) => Err(PyValueError::new_err( - "native secret manager mutation is unavailable", - )), + Err(PythonMutationError::Manager(error)) => Err(error.into()), }) } @@ -43,18 +33,35 @@ fn cyberark_failure( name: &str, failure: cyberark::WriteFailure, ) -> PyResult { - let message = match failure.source { + let message = if failure.source.is_unsafe_secret_name() { + format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) + } else { + cyberark_message( + py, + failure.source, + failure.authentication, + failure.request_url.as_ref(), + )? + }; + Ok(if failure.authentication { + format!("Could not authenticate to CyberArk Conjur: {message}") + } else { + message + }) +} + +fn cyberark_message( + py: Python<'_>, + source: cyberark::Error, + authentication: bool, + request_url: Option<&reqwest::Url>, +) -> PyResult { + Ok(match source { cyberark::Error::Status(status) | cyberark::Error::AuthStatus(status) => { - let url = failure - .request_url - .as_ref() - .map_or("", reqwest::Url::as_str); + let url = request_url.map_or("", reqwest::Url::as_str); http_message(py, "POST", url, status)? } - cyberark::Error::Operation(litellm_secrets_types::Error::UnsafeSecretName) => { - format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) - } - cyberark::Error::Http(source) if failure.authentication => match os_error_code(&source) { + cyberark::Error::Http(source) if authentication => match os_error_code(&source) { Some(code) => { let reason = py.import("os")?.getattr("strerror")?.call1((code,))?; py.import("builtins")? @@ -63,17 +70,12 @@ fn cyberark_failure( .str()? .extract()? } - None => cyberark::Error::Http(source).to_string(), + None => Report::new(&cyberark::Error::Http(source)).to_string(), }, cyberark::Error::Http(source) if source.is_connect() => { "All connection attempts failed".to_owned() } - source => source.to_string(), - }; - Ok(if failure.authentication { - format!("Could not authenticate to CyberArk Conjur: {message}") - } else { - message + source => Report::new(&source).to_string(), }) } diff --git a/litellm-rust/crates/python-bridge/src/secrets/operations.rs b/litellm-rust/crates/python-bridge/src/secrets/operations.rs index 8e338741aff..027da1e4718 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/operations.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/operations.rs @@ -4,6 +4,8 @@ use litellm_secrets::cyberark::AuthenticationRetry; use litellm_secrets::{Error, SecretManager}; use litellm_secrets_types::{PythonSecretRead, SecretOperationContext}; +use super::ManagerError; + pub(super) struct PythonReadRequest { pub secret_name: String, pub primary_secret_name: Option, @@ -62,21 +64,24 @@ pub(super) async fn read_python_provider( #[derive(Debug)] pub(super) enum PythonMutationError { - Unsupported, + Manager(ManagerError), Vault(Box), CyberarkWrite { name: String, failure: Box, }, - CurrentMissing(String), - ReplacementMissing(String), - ReplacementMismatch, +} + +impl From for PythonMutationError { + fn from(error: ManagerError) -> Self { + Self::Manager(error) + } } pub(super) async fn write_python_provider( manager: &SecretManager, name: &str, - value: &litellm_secrets::SecretValue, + value: &litellm_secret_value::SecretValue, ) -> Result { match manager { SecretManager::Cyberark(client) => { @@ -89,7 +94,7 @@ pub(super) async fn write_python_provider( })?; Ok(write_success(name)) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } @@ -115,7 +120,7 @@ pub(super) async fn delete_python_provider( "message": "CyberArk Conjur does not support direct secret deletion. Use policy updates to remove variables.", })) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } @@ -123,7 +128,7 @@ pub(super) async fn rotate_python_provider( manager: &SecretManager, current_name: &str, new_name: &str, - value: &litellm_secrets::SecretValue, + value: &litellm_secret_value::SecretValue, ) -> Result { match manager { SecretManager::Cyberark(client) => { @@ -138,7 +143,7 @@ pub(super) async fn rotate_python_provider( .flatten() .is_none() { - return Err(PythonMutationError::CurrentMissing(current_name.to_owned())); + return Err(ManagerError::CurrentMissing(current_name.to_owned()).into()); } client .write_with_retry( @@ -157,16 +162,16 @@ pub(super) async fn rotate_python_provider( .await .ok() .flatten() - .ok_or_else(|| PythonMutationError::ReplacementMissing(new_name.to_owned()))?; + .ok_or_else(|| ManagerError::ReplacementMissing(new_name.to_owned()))?; if actual != *value { - return Err(PythonMutationError::ReplacementMismatch); + return Err(ManagerError::ReplacementMismatch.into()); } if current_name != new_name { client.invalidate_cached_secret(current_name).await; } Ok(write_success(new_name)) } - _ => Err(PythonMutationError::Unsupported), + _ => Err(ManagerError::MutationUnsupported.into()), } } fn write_success(name: &str) -> serde_json::Value { @@ -181,7 +186,7 @@ pub(super) enum PythonMutationResponse { pub(super) async fn write_python_provider_with_context( manager: &SecretManager, name: &str, - value: &litellm_secrets::SecretValue, + value: &litellm_secret_value::SecretValue, context: &litellm_secrets_types::SecretWriteContext, ) -> Result { if let (SecretManager::HashicorpVault(client), SecretOperationContext::Hashicorp(operation)) = @@ -230,7 +235,7 @@ pub(super) async fn rotate_python_provider_with_context( manager: &SecretManager, current_name: &str, new_name: &str, - value: &litellm_secrets::SecretValue, + value: &litellm_secret_value::SecretValue, context: &SecretOperationContext, ) -> Result { if let (SecretManager::HashicorpVault(client), SecretOperationContext::Hashicorp(context)) = diff --git a/litellm-rust/crates/python-bridge/src/secrets/provider.rs b/litellm-rust/crates/python-bridge/src/secrets/provider.rs index 5f4cd193fcf..55343fb63bb 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/provider.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/provider.rs @@ -1,12 +1,16 @@ use std::time::Duration; use super::operations::PythonReadRequest; -use litellm_secrets::{KeyManagementSystem, SecretValue}; +use litellm_secret_value::SecretValue; +use litellm_secrets::KeyManagementSystem; + use litellm_secrets_types::{ AwsOperationContext, CyberarkOperationContext, GoogleOperationContext, HashicorpOperationContext, SecretOperationContext, }; -use pyo3::{exceptions::PyValueError, prelude::*, types::PyDict}; +use pyo3::{prelude::*, types::PyDict}; + +use super::ManagerError; pub(super) fn read_request( system: KeyManagementSystem, @@ -39,9 +43,7 @@ pub(super) fn read_request( SecretOperationContext::Google(GoogleOperationContext::default()) } _ => { - return Err(PyValueError::new_err( - "secret manager does not support provider reads", - )); + return Err(ManagerError::ProviderReads.into()); } }; Ok(PythonReadRequest { @@ -89,9 +91,7 @@ fn aws_context( fn read_timeout(value: Option<&Bound<'_, PyAny>>) -> PyResult> { read_timeout_seconds(value)? - .map(|value| { - Duration::try_from_secs_f64(value).map_err(|_| PyValueError::new_err("invalid timeout")) - }) + .map(|value| Ok(Duration::try_from_secs_f64(value).map_err(ManagerError::Timeout)?)) .transpose() } diff --git a/litellm-rust/crates/python-bridge/src/secrets/python.rs b/litellm-rust/crates/python-bridge/src/secrets/python.rs index 2f040bff773..c5d9e3565b3 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/python.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/python.rs @@ -2,7 +2,9 @@ use std::sync::Arc; use futures_util::future::BoxFuture; use litellm_host_python::{PythonContext, attach_blocking}; -use litellm_secrets::{Error, SecretValue, source::SecretSource}; +use litellm_secret_value::SecretValue; +use litellm_secrets::{Error, source::SecretSource}; + use pyo3::prelude::*; use super::error::external_error; diff --git a/litellm-rust/crates/python-bridge/src/secrets/resolved.rs b/litellm-rust/crates/python-bridge/src/secrets/resolved.rs index 5a606ab1039..752b533886e 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/resolved.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/resolved.rs @@ -4,10 +4,9 @@ use futures_util::future::BoxFuture; use litellm_core_utils::settings::ProcessEnvironment; use litellm_host_python::PythonContext; use litellm_http::Client; +use litellm_secret_value::SecretValue; use litellm_secrets::source::SecretSource; -use litellm_secrets::{ - Error, FailurePolicy, OidcResolver, SecretManagerState, SecretResolver, SecretValue, -}; +use litellm_secrets::{Error, FailurePolicy, OidcResolver, SecretManagerState, SecretResolver}; use super::config::SecretManagerSnapshot; @@ -84,7 +83,7 @@ mod tests { } async fn resolve(state: Arc, name: &'static str) -> Option { - ResolvedSecrets::from_state(state, litellm_http::Client::plain_for_test()) + ResolvedSecrets::from_state(state, litellm_http::Client::for_test()) .resolve(&[name]) .await .unwrap() @@ -182,7 +181,7 @@ mod tests { .await; let source = ResolvedSecrets::from_state( state(&server, KeyManagementSettings::default()), - litellm_http::Client::plain_for_test(), + litellm_http::Client::for_test(), ); let snapshot = source.resolve(&[declared]).await.unwrap(); assert_eq!(snapshot.get(undeclared), None); @@ -248,7 +247,7 @@ mod tests { async fn oidc_failures_are_not_converted_to_missing_secrets() { let result = ResolvedSecrets::from_state( Arc::new(SecretManagerState::default()), - litellm_http::Client::plain_for_test(), + litellm_http::Client::for_test(), ) .resolve(&["oidc/"]) .await; diff --git a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs index 72ba91ded16..0d52b0aa629 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/runtime.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/runtime.rs @@ -7,12 +7,9 @@ use litellm_secrets::{ read_secret_from_python_manager, }; use litellm_secrets_types::PythonSecretRead; -use pyo3::{ - exceptions::{PyAttributeError, PyRuntimeError, PyValueError}, - prelude::*, - types::PyDict, -}; +use pyo3::{exceptions::PyAttributeError, prelude::*, types::PyDict}; +use super::ManagerError; use crate::execution::{run_async_value, run_sync_value}; const SECRET_MANAGER_MODULE: &str = "litellm.rust_bridge.host.secret_manager"; @@ -35,9 +32,7 @@ pub(crate) struct NativeSecretManager { impl NativeSecretManager { pub(super) fn backend(&self) -> PyResult { if self.pid != std::process::id() { - return Err(PyRuntimeError::new_err( - "native secret manager must be recreated after fork", - )); + return Err(ManagerError::Forked.into()); } Ok(self.backend.clone()) } @@ -60,7 +55,7 @@ impl NativeSecretManager { enterprise_enabled, ) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) })?; Ok(Self { backend, @@ -82,7 +77,7 @@ impl NativeSecretManager { enterprise_enabled: bool, ) -> PyResult { let system = serde_json::from_value(serde_json::Value::String(system.to_owned())) - .map_err(|_| PyValueError::new_err("unknown secret manager system"))?; + .map_err(ManagerError::UnknownSystem)?; let settings = parse_settings(settings)?; Self::build( py, @@ -146,7 +141,7 @@ impl NativeSecretManager { .collect::>>()?; let settings = from_py::>(&config.getattr("settings")?) - .map_err(|_| PyValueError::new_err("invalid secret manager settings"))?; + .map_err(ManagerError::settings)?; let attributes = config .getattr("settings_attributes")? .extract::>()?; @@ -161,11 +156,11 @@ impl NativeSecretManager { system: serde_json::from_value(serde_json::Value::String( config.getattr("system")?.extract()?, )) - .map_err(|_| PyValueError::new_err("unknown secret manager system"))?, + .map_err(ManagerError::UnknownSystem)?, settings: serde_json::from_value(serde_json::Value::Object( settings.into_iter().chain(setting_overrides).collect(), )) - .map_err(|_| PyValueError::new_err("invalid secret manager settings"))?, + .map_err(ManagerError::settings)?, environment: captured .into_iter() .filter(|(key, _)| !environment_attributes.contains_key(key)) @@ -190,12 +185,8 @@ impl NativeSecretManager { } #[getter] - fn system(&self) -> String { - serde_json::to_value(self.configuration.system) - .expect("serializable system") - .as_str() - .expect("string system") - .to_owned() + fn system(&self) -> &'static str { + self.configuration.system.into() } #[pyo3(signature = (name, settings=None))] @@ -238,7 +229,7 @@ impl NativeSecretManager { run_sync_value(py, async move { super::operations::read_python_provider(&backend, &request, &ProcessEnvironment) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) .and_then(|value| python_secret_value(value, &request.secret_name)) }) } @@ -264,7 +255,7 @@ impl NativeSecretManager { run_async_value(py, async move { super::operations::read_python_provider(&backend, &request, &ProcessEnvironment) .await - .map_err(|error| PyValueError::new_err(error.to_string())) + .map_err(|error| ManagerError::Secret(error).into()) .and_then(|value| python_secret_value(value, &request.secret_name)) }) } @@ -316,7 +307,7 @@ impl NativeSecretManager { super::operations::write_python_provider_with_context( &backend, &secret_name, - &litellm_secrets::SecretValue::new(secret_value), + &litellm_secret_value::SecretValue::new(secret_value), &context, ) .await, @@ -374,7 +365,7 @@ impl NativeSecretManager { &backend, ¤t_secret_name, &new_secret_name, - &litellm_secrets::SecretValue::new(new_secret_value), + &litellm_secret_value::SecretValue::new(new_secret_value), &context, ) .await, @@ -438,8 +429,8 @@ fn cached( fn parse_settings(value: Option<&Bound<'_, PyAny>>) -> PyResult { value .map(|value| { - serde_json::from_value(from_py::(value)?) - .map_err(|_| PyValueError::new_err("invalid secret manager settings")) + Ok(serde_json::from_value(from_py::(value)?) + .map_err(ManagerError::settings)?) }) .transpose() .map(Option::unwrap_or_default) @@ -466,11 +457,10 @@ fn python_read_error( name: &str, error: litellm_secrets::Error, ) -> PyErr { - let message = match (system, error) { + match (system, error) { (KeyManagementSystem::Cyberark, litellm_secrets::Error::ManagedSecretMissing) => { - format!("No secret found in CyberArk Secret Manager for {name}") + ManagerError::CyberarkSecretMissing(name.to_owned()).into() } - (_, error) => error.to_string(), - }; - PyValueError::new_err(message) + (_, error) => ManagerError::Secret(error).into(), + } } diff --git a/litellm-rust/crates/python-bridge/src/secrets/vault.rs b/litellm-rust/crates/python-bridge/src/secrets/vault.rs index 79281662848..30e3bd2c413 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/vault.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/vault.rs @@ -2,8 +2,11 @@ mod operation; pub(super) use operation::{Failure, FailureKind, FailureStage, delete, rotate, write}; -use litellm_core_utils::exception_mapping_utils::{TimeoutPhase, timeout_message}; -use litellm_secrets::hashicorp::{Error, RawOperationError}; +use litellm_core_utils::exception_mapping_utils::{ + TimeoutPhase, timeout_message as transport_timeout_message, +}; +use litellm_error::Report; +use litellm_secrets::hashicorp::RawOperationError; use pyo3::prelude::*; use super::mutation::{error_value, http_message, json_value}; @@ -16,40 +19,39 @@ pub(super) fn failure_value( failure: Failure, context: &ErrorContext, ) -> PyResult> { - let message = match *failure.kind { - FailureKind::Native(RawOperationError::Http { - method, - url, - status, - body, - }) => match failure.stage { - FailureStage::Current(name) if status == 404 => { - format!("Current secret {name} not found") + let message = match (&*failure.kind, failure.stage) { + (FailureKind::Native(error), stage) if let Some(response) = error.http_failure() => { + match stage { + FailureStage::Current(name) if response.status == 404 => { + format!("Current secret {name} not found") + } + FailureStage::Replacement(name) if response.status == 404 => { + format!("Failed to verify new secret {name}") + } + FailureStage::Current(_) => format!( + "HTTP error occurred while checking current secret: {}", + response_text(py, &response.body)? + ), + FailureStage::Replacement(_) => format!( + "HTTP error occurred while verifying new secret: {}", + response_text(py, &response.body)? + ), + FailureStage::Mutation => { + http_message(py, &response.method, &response.url, response.status)? + } } - FailureStage::Replacement(name) if status == 404 => { - format!("Failed to verify new secret {name}") - } - FailureStage::Current(_) => format!( - "HTTP error occurred while checking current secret: {}", - response_text(py, &body)? - ), - FailureStage::Replacement(_) => format!( - "HTTP error occurred while verifying new secret: {}", - response_text(py, &body)? - ), - FailureStage::Mutation => http_message(py, &method, &url, status)?, - }, - FailureKind::ValueMismatch { expected, actual } => { - let actual = json_value(py, &actual)?; + } + (FailureKind::ValueMismatch { expected, actual }, _) => { + let actual = json_value(py, actual)?; format!( "New secret value mismatch. Expected: {}, Got: {}", expected.expose(), actual.bind(py).str()? ) } - kind => { + (kind, stage) => { let message = cause_message(py, kind, context)?; - match failure.stage { + match stage { FailureStage::Current(_) => { format!("Error checking current secret: {message}") } @@ -63,86 +65,91 @@ pub(super) fn failure_value( error_value(py, message) } -fn cause_message(py: Python<'_>, kind: FailureKind, context: &ErrorContext) -> PyResult { +fn cause_message(py: Python<'_>, kind: &FailureKind, context: &ErrorContext) -> PyResult { Ok(match kind { - FailureKind::Native(RawOperationError::Local(error)) => error.to_string(), + FailureKind::Native(error) => native_message(py, error, context)?, FailureKind::UnsafeName(name) => { format!("Invalid secret_name {}", name.into_pyobject(py)?.repr()?) } - FailureKind::Native(RawOperationError::Timeout { method, elapsed }) => { - if method == "POST" { - let kwargs = pyo3::types::PyDict::new(py); - kwargs.set_item( - "message", - timeout_message( - TimeoutPhase::Read, - context.timeout, - Some(elapsed.as_secs_f64()), - ), - )?; - kwargs.set_item("model", "default-model-name")?; - kwargs.set_item("llm_provider", "litellm-httpx-handler")?; - py.import("litellm.exceptions")? - .getattr("Timeout")? - .call((), Some(&kwargs))? - .str()? - .extract()? - } else if context.aiohttp { - "Timeout on reading data from socket".to_owned() - } else { - String::new() - } - } - FailureKind::Native(RawOperationError::Transport(source)) => { - if let Some(error) = request_error(&source) { - if error.is_timeout() { - String::new() - } else if error.is_connect() { - "All connection attempts failed".to_owned() - } else { - "HashiCorp Vault request failed".to_owned() - } - } else { - "HashiCorp Vault request failed".to_owned() - } - } FailureKind::MissingGet(value) => { - let value = json_value(py, &value)?; + let value = json_value(py, value)?; match value.bind(py).getattr("get") { Err(error) => error.value(py).str()?.extract()?, Ok(_) => "HashiCorp Vault response payload is malformed".to_owned(), } } - FailureKind::Json(body) => match json_value(py, &body) { + FailureKind::Json(body) => match json_value(py, body) { Err(error) => error.value(py).str()?.extract()?, Ok(_) => "HashiCorp Vault response payload is malformed".to_owned(), }, - FailureKind::Native(RawOperationError::Authentication { - source, - url, - certificate, - }) => { - let message = match source { - Error::LoginStatus { status } => http_message(py, "POST", &url, status)?, - error => error.to_string(), - }; - let mechanism = if certificate { "TLS cert" } else { "AppRole" }; - format!("Could not authenticate to Vault via {mechanism}: {message}") - } - FailureKind::Native(RawOperationError::Http { - method, - url, - status, - .. - }) => http_message(py, &method, &url, status)?, FailureKind::ValueMismatch { .. } => "New secret value mismatch".to_owned(), }) } -fn request_error<'a>(error: &'a (dyn std::error::Error + 'static)) -> Option<&'a reqwest::Error> { - error - .downcast_ref::() - .or_else(|| error.source().and_then(request_error)) +fn native_message( + py: Python<'_>, + error: &RawOperationError, + context: &ErrorContext, +) -> PyResult { + if let Some(error) = error.local() { + return Ok(Report::new(error).to_string()); + } + if let Some((method, elapsed)) = error.timeout() { + return timeout_message(py, method, elapsed, context); + } + if let Some(response) = error.http_failure() { + return http_message(py, &response.method, &response.url, response.status); + } + if let Some(login) = error.authentication() { + let message = match login.source.login_status() { + Some(status) => http_message(py, "POST", login.url, status)?, + None => Report::new(login.source).to_string(), + }; + let mechanism = if login.certificate { + "TLS cert" + } else { + "AppRole" + }; + return Ok(format!( + "Could not authenticate to Vault via {mechanism}: {message}" + )); + } + Ok(match error.transport() { + Some(error) if error.status() == Some(408) => String::new(), + Some(error) if error.is_connect() => "All connection attempts failed".to_owned(), + Some(_) | None => "HashiCorp Vault request failed".to_owned(), + }) +} + +fn timeout_message( + py: Python<'_>, + method: &str, + elapsed: std::time::Duration, + context: &ErrorContext, +) -> PyResult { + if method != "POST" { + return Ok(if context.aiohttp { + "Timeout on reading data from socket".to_owned() + } else { + String::new() + }); + } + let kwargs = pyo3::types::PyDict::new(py); + kwargs.set_item( + "message", + transport_timeout_message( + TimeoutPhase::Read, + context.timeout, + Some(elapsed.as_secs_f64()), + ), + )?; + kwargs.set_item("model", "default-model-name")?; + kwargs.set_item("llm_provider", "litellm-httpx-handler")?; + py.import("litellm.exceptions")? + .getattr("Timeout")? + .call((), Some(&kwargs))? + .str()? + .extract() } fn response_text(py: Python<'_>, body: &[u8]) -> PyResult { diff --git a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs index cf56beb5efe..b99dff7aeac 100644 --- a/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs +++ b/litellm-rust/crates/python-bridge/src/secrets/vault/operation.rs @@ -1,9 +1,8 @@ use std::collections::HashMap; -use litellm_secrets::{ - SecretValue, - hashicorp::{Error, HashicorpVault, RawOperationError}, -}; +use litellm_secret_value::SecretValue; +use litellm_secrets::hashicorp::{HashicorpVault, RawOperationError}; + use litellm_secrets_types::{HashicorpOperationContext, SecretWriteContext}; use serde_json::value::RawValue; @@ -53,12 +52,10 @@ impl Failure { } fn native_failure(name: &str, error: RawOperationError) -> Failure { - match error { - RawOperationError::Local(Error::InvalidSecretName(_)) => { - FailureKind::UnsafeName(name.to_owned()).into() - } - error => FailureKind::Native(error).into(), + if error.is_unsafe_secret_name() { + return FailureKind::UnsafeName(name.to_owned()).into(); } + FailureKind::Native(error).into() } pub(crate) async fn write( diff --git a/litellm-rust/crates/python-bridge/src/tokenizer.rs b/litellm-rust/crates/python-bridge/src/tokenizer.rs index df219d55eb6..2026b2fdc42 100644 --- a/litellm-rust/crates/python-bridge/src/tokenizer.rs +++ b/litellm-rust/crates/python-bridge/src/tokenizer.rs @@ -8,16 +8,18 @@ use std::sync::Arc; #[cfg(feature = "fast")] use std::sync::OnceLock; +use litellm_error::Report; use litellm_host_python::{enter_native, release_gil}; #[cfg(feature = "fast")] use litellm_token_counter::fast::{FastCounter, FastTokenizer}; use litellm_token_counter::{Error, TextCodec}; use pyo3::{exceptions::PyUnicodeEncodeError, prelude::*, types::PyString}; -#[cfg(any(feature = "tiktoken", feature = "huggingface"))] -use pyo3::exceptions::PyValueError; #[cfg(feature = "huggingface")] -use pyo3::{exceptions::PyIOError, types::PyDict}; +use pyo3::{ + exceptions::{PyIOError, PyTypeError, PyValueError}, + types::PyDict, +}; #[cfg(feature = "tiktoken")] use pyo3::{ exceptions::{PyKeyError, PyRuntimeError}, @@ -51,6 +53,70 @@ pub(crate) fn load_tiktoken(py: Python<'_>, encoding: &str) -> PyResult for PyErr { + fn from(error: TokenizerError) -> Self { + let message = Report::new(&error).to_string(); + match error { + #[cfg(not(all(feature = "tiktoken", feature = "huggingface")))] + TokenizerError::MissingFeature(_) => RustBridgeDeclined::new_err(message), + #[cfg(all(feature = "tiktoken", feature = "huggingface"))] + TokenizerError::NotTiktoken | TokenizerError::NotHuggingFace => { + PyValueError::new_err(message) + } + #[cfg(feature = "tiktoken")] + TokenizerError::NoVocabulary | TokenizerError::EncodeSpecial(_) => { + PyRuntimeError::new_err(message) + } + #[cfg(feature = "tiktoken")] + TokenizerError::DecodeBytes(_) => PyKeyError::new_err(message), + #[cfg(feature = "huggingface")] + TokenizerError::Direction { .. } | TokenizerError::EncodingJson(_) => { + PyValueError::new_err(message) + } + #[cfg(feature = "huggingface")] + TokenizerError::PretokenizedMismatch => PyTypeError::new_err(message), + #[cfg(feature = "huggingface")] + TokenizerError::Download(_) => PyIOError::new_err(message), + } + } +} + pub(crate) enum Codec { #[cfg(feature = "tiktoken")] Tiktoken(TiktokenTokenizer), @@ -100,9 +166,7 @@ impl Tokenizer { #[cfg(not(feature = "tiktoken"))] { let _ = (py, encoding); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the tiktoken feature", - )) + Err(TokenizerError::MissingFeature("tiktoken").into()) } } @@ -118,9 +182,7 @@ impl Tokenizer { #[cfg(not(feature = "huggingface"))] { let _ = (py, tokenizer_json); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the huggingface feature", - )) + Err(TokenizerError::MissingFeature("huggingface").into()) } } @@ -144,16 +206,14 @@ impl Tokenizer { .getattr("hf_hub_download")? .call((), Some(&kwargs))? .extract()?; - let json = - release_gil(py, || std::fs::read_to_string(path)).map_err(PyIOError::new_err)?; + let json = release_gil(py, || std::fs::read_to_string(path)) + .map_err(TokenizerError::Download)?; Self::from_json(py, &json) } #[cfg(not(feature = "huggingface"))] { let _ = (py, identifier, revision, token); - Err(RustBridgeDeclined::new_err( - "tokenizer backend requires the huggingface feature", - )) + Err(TokenizerError::MissingFeature("huggingface").into()) } } @@ -199,7 +259,7 @@ impl Tokenizer { let tokenizer = self.tiktoken()?; let text = self.text(text)?; release_gil(py, || tokenizer.encode_special(&text, &allowed)) - .map_err(PyRuntimeError::new_err) + .map_err(|error| TokenizerError::EncodeSpecial(error.into()).into()) } /// tiktoken's `encode_with_unstable`: `(stable_tokens, completions)`. @@ -258,8 +318,8 @@ impl Tokenizer { fn decode_bytes<'py>(&self, py: Python<'py>, ids: Vec) -> PyResult> { enter_native()?; let tokenizer = self.tiktoken()?; - let bytes = - release_gil(py, || tokenizer.decode_bytes(&ids)).map_err(PyKeyError::new_err)?; + let bytes = release_gil(py, || tokenizer.decode_bytes(&ids)) + .map_err(|error| TokenizerError::DecodeBytes(error.into()))?; Ok(PyBytes::new(py, &bytes)) } @@ -480,15 +540,16 @@ impl Tokenizer { match *self.inner { Codec::Tiktoken(ref tokenizer) => Ok(tokenizer), #[cfg(feature = "huggingface")] - Codec::HuggingFace(_) => Err(PyValueError::new_err("requires a tiktoken encoding")), + Codec::HuggingFace(_) => Err(TokenizerError::NotTiktoken.into()), } } #[cfg(feature = "tiktoken")] fn vocabulary(&self) -> PyResult<&Vocabulary> { - self.tiktoken()?.vocabulary().ok_or_else(|| { - PyRuntimeError::new_err("this encoding was built without its vocabulary") - }) + Ok(self + .tiktoken()? + .vocabulary() + .ok_or(TokenizerError::NoVocabulary)?) } #[cfg(feature = "huggingface")] @@ -496,7 +557,7 @@ impl Tokenizer { match *self.inner { Codec::HuggingFace(ref tokenizer) => Ok(tokenizer), #[cfg(feature = "tiktoken")] - Codec::Tiktoken(_) => Err(PyValueError::new_err("requires a Hugging Face tokenizer")), + Codec::Tiktoken(_) => Err(TokenizerError::NotHuggingFace.into()), } } } @@ -530,21 +591,21 @@ impl Sequence { match (self, is_pretokenized) { (Self::Text(text), false) => Ok(text.into()), (Self::Words(words), true) => Ok(words.into()), - _ => Err(pyo3::exceptions::PyTypeError::new_err( - "input must match is_pretokenized", - )), + _ => Err(TokenizerError::PretokenizedMismatch.into()), } } } #[cfg(feature = "huggingface")] -fn direction(value: &str, left: T, right: T, what: &str) -> PyResult { +fn direction(value: &str, left: T, right: T, what: &'static str) -> PyResult { match value { "left" => Ok(left), "right" => Ok(right), - other => Err(PyValueError::new_err(format!( - "invalid {what} direction {other:?}: expected 'left' or 'right'" - ))), + other => Err(TokenizerError::Direction { + what, + value: other.to_owned(), + } + .into()), } } @@ -564,7 +625,7 @@ impl HuggingFaceEncoding { fn new(json: Option<&str>) -> PyResult { let inner = match json { Some(json) => encoding_from_json(json) - .map_err(|error| PyValueError::new_err(error.to_string()))?, + .map_err(|error| TokenizerError::EncodingJson(error.into()))?, None => Encoding::default(), }; Ok(Self { inner }) @@ -586,7 +647,7 @@ impl HuggingFaceEncoding { py: Python<'py>, ) -> PyResult<(Bound<'py, pyo3::types::PyType>, (String,))> { let json = encoding_to_json(&self.inner) - .map_err(|error| PyValueError::new_err(error.to_string()))?; + .map_err(|error| TokenizerError::EncodingJson(error.into()))?; Ok((py.get_type::(), (json,))) } diff --git a/litellm-rust/crates/python-compat/benches/formats.rs b/litellm-rust/crates/python-compat/benches/formats.rs index f0b83cb1658..2f23b882e19 100644 --- a/litellm-rust/crates/python-compat/benches/formats.rs +++ b/litellm-rust/crates/python-compat/benches/formats.rs @@ -10,7 +10,7 @@ use std::{hint::black_box, time::Duration}; use criterion::{ - BatchSize, BenchmarkGroup, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main, + BenchmarkGroup, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main, measurement::WallTime, }; use litellm_python_compat::{Value, json, literal::literal_eval, pickle, repr::repr}; @@ -72,14 +72,9 @@ fn formats(c: &mut Criterion) { let mut group = c.benchmark_group("cached_completion"); text_formats(&mut group, &text, &value); binary_formats(&mut group, &value, &pickled); - // `from_json` consumes its input, so each iteration gets a freshly parsed one. group.throughput(Throughput::Bytes(dumped.len() as u64)); - group.bench_function("from_json", |bencher| { - bencher.iter_batched( - || serde_json::from_str::(&dumped).expect("dumps output parses"), - json::from_json, - BatchSize::SmallInput, - ) + group.bench_function("json_loads", |bencher| { + bencher.iter(|| json::loads(black_box(&dumped))) }); group.finish(); } diff --git a/litellm-rust/crates/python-compat/src/error.rs b/litellm-rust/crates/python-compat/src/error.rs index 43990eb7602..db09925dde9 100644 --- a/litellm-rust/crates/python-compat/src/error.rs +++ b/litellm-rust/crates/python-compat/src/error.rs @@ -6,19 +6,23 @@ use crate::MAX_DEPTH; pub enum Error { #[error("malformed Python literal at byte {0}")] InvalidLiteral(usize), + #[error("malformed JSON at byte {0}")] + InvalidJson(usize), #[error("unhashable type: '{0}'")] Unhashable(&'static str), #[error("value nests deeper than {MAX_DEPTH} levels")] TooDeep, - #[error("invalid pickle: {0}")] - InvalidPickle(String), + #[error("invalid pickle")] + InvalidPickle(#[source] serde_pickle::Error), + #[error("could not write pickle")] + PickleWrite(#[source] serde_pickle::Error), #[error("Object of type {0} is not JSON serializable")] NotJsonSerializable(&'static str), #[error("keys must be str, int, float, bool or None, not {0}")] InvalidJsonKey(&'static str), #[error("Out of range float values are not JSON compliant")] NonFiniteFloat, - #[error("integer does not fit in the JSON number range")] + #[error("integer does not fit in 64 bits")] IntegerOutOfRange, #[error("Object of type {0} cannot be pickled as plain data")] NotPicklable(&'static str), diff --git a/litellm-rust/crates/python-compat/src/json.rs b/litellm-rust/crates/python-compat/src/json.rs index d3436519ac4..37cc88fec78 100644 --- a/litellm-rust/crates/python-compat/src/json.rs +++ b/litellm-rust/crates/python-compat/src/json.rs @@ -1,14 +1,19 @@ -//! `json.dumps` with CPython's default options, and the JSON value `json.loads` returns. +//! `json.dumps` with CPython's default options, and `json.loads` with its default decoder. //! //! Defaults are `ensure_ascii=True`, `allow_nan=True`, separators `(", ", ": ")`, and no key -//! sorting. Python's `json.loads` is mapped by [`from_json`]; a `serde_json` number that does -//! not fit `i64` or `u64` arrives as a float, where Python would keep an `int`. +//! sorting on the way out, and `strict=True` with `NaN`, `Infinity` and `-Infinity` read as floats +//! on the way in. -use std::fmt::Write; +use std::{ + collections::{HashMap, hash_map::Entry}, + fmt::Write, + str::FromStr, +}; +use num_bigint::BigInt; use serde_json::{Map, Number}; -use crate::{Error, Value, repr::float_repr}; +use crate::{Error, MAX_DEPTH, Value, repr::float_repr}; /// `json.dumps(value)`. pub fn dumps(value: &Value) -> Result { @@ -44,6 +49,260 @@ pub fn from_json(value: serde_json::Value) -> Value { } } +/// `json.loads(text)`. Integers keep full precision, and a repeated key keeps its first position +/// with its last value, as `dict(pairs)` does. An escape that leaves a lone surrogate fails, since +/// a Rust `String` cannot hold one. +pub fn loads(text: &str) -> Result { + let mut decoder = Decoder { + text, + bytes: text.as_bytes(), + pos: 0, + depth: 0, + }; + decoder.skip_whitespace(); + let value = decoder.value()?; + decoder.skip_whitespace(); + if decoder.pos != decoder.bytes.len() { + return Err(decoder.error()); + } + Ok(value) +} + +/// CPython's `json.scanner` and `scanstring`, one pass over the bytes. +struct Decoder<'a> { + text: &'a str, + bytes: &'a [u8], + pos: usize, + depth: usize, +} + +impl Decoder<'_> { + fn error(&self) -> Error { + Error::InvalidJson(self.pos) + } + + fn peek(&self) -> Option { + self.bytes.get(self.pos).copied() + } + + fn eat(&mut self, token: &str) -> bool { + let found = self.bytes[self.pos..].starts_with(token.as_bytes()); + if found { + self.pos += token.len(); + } + found + } + + fn expect(&mut self, token: &str) -> Result<(), Error> { + if self.eat(token) { + Ok(()) + } else { + Err(self.error()) + } + } + + fn skip_whitespace(&mut self) { + while matches!(self.peek(), Some(b' ' | b'\t' | b'\n' | b'\r')) { + self.pos += 1; + } + } + + fn value(&mut self) -> Result { + match self.peek() { + Some(b'"') => self.string().map(Value::Str), + Some(b'[') => self.nested(Self::array), + Some(b'{') => self.nested(Self::object), + _ if self.eat("null") => Ok(Value::None), + _ if self.eat("true") => Ok(Value::Bool(true)), + _ if self.eat("false") => Ok(Value::Bool(false)), + _ if self.eat("NaN") => Ok(Value::Float(f64::NAN)), + _ if self.eat("Infinity") => Ok(Value::Float(f64::INFINITY)), + _ if self.eat("-Infinity") => Ok(Value::Float(f64::NEG_INFINITY)), + _ => self.number(), + } + } + + fn nested(&mut self, parse: fn(&mut Self) -> Result) -> Result { + self.depth += 1; + if self.depth > MAX_DEPTH { + return Err(Error::TooDeep); + } + let value = parse(self)?; + self.depth -= 1; + Ok(value) + } + + fn array(&mut self) -> Result { + self.expect("[")?; + self.skip_whitespace(); + let mut values = Vec::new(); + if self.eat("]") { + return Ok(Value::List(values)); + } + loop { + values.push(self.value()?); + self.skip_whitespace(); + if self.eat("]") { + return Ok(Value::List(values)); + } + self.expect(",")?; + self.skip_whitespace(); + } + } + + fn object(&mut self) -> Result { + self.expect("{")?; + self.skip_whitespace(); + let mut entries = Vec::new(); + let mut positions = HashMap::<_, usize>::new(); + if self.eat("}") { + return Ok(Value::Dict(entries)); + } + loop { + if self.peek() != Some(b'"') { + return Err(self.error()); + } + let key = self.string()?; + self.skip_whitespace(); + self.expect(":")?; + self.skip_whitespace(); + let value = self.value()?; + match positions.entry(key) { + Entry::Occupied(position) => entries[*position.get()].1 = value, + Entry::Vacant(position) => { + entries.push((Value::Str(position.key().clone()), value)); + position.insert(entries.len() - 1); + } + } + self.skip_whitespace(); + if self.eat("}") { + return Ok(Value::Dict(entries)); + } + self.expect(",")?; + self.skip_whitespace(); + } + } + + /// `scanstring` with `strict=True`: control characters must be escaped. + fn string(&mut self) -> Result { + self.expect("\"")?; + let mut out = String::new(); + loop { + let start = self.pos; + while self + .peek() + .is_some_and(|byte| byte != b'"' && byte != b'\\' && byte >= 0x20) + { + self.pos += 1; + } + out.push_str(&self.text[start..self.pos]); + match self.peek() { + Some(b'"') => { + self.pos += 1; + return Ok(out); + } + Some(b'\\') => { + self.pos += 1; + out.push(self.escape()?); + } + _ => return Err(self.error()), + } + } + } + + fn escape(&mut self) -> Result { + let escaped = match self.peek() { + Some(b'"') => '"', + Some(b'\\') => '\\', + Some(b'/') => '/', + Some(b'b') => '\u{8}', + Some(b'f') => '\u{c}', + Some(b'n') => '\n', + Some(b'r') => '\r', + Some(b't') => '\t', + Some(b'u') => { + self.pos += 1; + return self.unicode_escape(); + } + _ => return Err(self.error()), + }; + self.pos += 1; + Ok(escaped) + } + + /// `\uXXXX`, joining a high surrogate with the low one escaped right after it. + fn unicode_escape(&mut self) -> Result { + let unit = self.hex4()?; + let code = if (0xd800..0xdc00).contains(&unit) { + self.expect("\\u")?; + let low = self.hex4()?; + if !(0xdc00..0xe000).contains(&low) { + return Err(self.error()); + } + 0x10000 + ((unit - 0xd800) << 10) + (low - 0xdc00) + } else { + unit + }; + char::from_u32(code).ok_or_else(|| self.error()) + } + + fn hex4(&mut self) -> Result { + let digits = self + .text + .get(self.pos..self.pos + 4) + .filter(|digits| digits.bytes().all(|byte| byte.is_ascii_hexdigit())) + .ok_or_else(|| self.error())?; + self.pos += 4; + u32::from_str_radix(digits, 16).map_err(|_| self.error()) + } + + /// `NUMBER_RE`: `(-?(?:0|[1-9][0-9]*))(\.[0-9]+)?([eE][-+]?[0-9]+)?`, an `int` unless a + /// fraction or exponent matched. + fn number(&mut self) -> Result { + let start = self.pos; + self.eat("-"); + match self.peek() { + Some(b'0') => self.pos += 1, + Some(b'1'..=b'9') => self.digits(), + _ => return Err(Error::InvalidJson(start)), + } + let integer_end = self.pos; + if self.peek() == Some(b'.') && self.bytes.get(self.pos + 1).is_some_and(u8::is_ascii_digit) + { + self.pos += 1; + self.digits(); + } + if matches!(self.peek(), Some(b'e' | b'E')) { + let exponent = self.pos; + self.pos += 1; + if matches!(self.peek(), Some(b'+' | b'-')) { + self.pos += 1; + } + if self.peek().is_some_and(|byte| byte.is_ascii_digit()) { + self.digits(); + } else { + self.pos = exponent; + } + } + let text = &self.text[start..self.pos]; + if self.pos == integer_end { + BigInt::from_str(text) + .map(Value::Int) + .map_err(|_| Error::InvalidJson(start)) + } else { + f64::from_str(text) + .map(Value::Float) + .map_err(|_| Error::InvalidJson(start)) + } + } + + fn digits(&mut self) { + while self.peek().is_some_and(|byte| byte.is_ascii_digit()) { + self.pos += 1; + } + } +} + /// `json.loads(json.dumps(value))` as a `serde_json` value: tuples become arrays and dict /// keys are coerced to strings as `json.dumps` does. Non-finite floats, which Python writes /// as `NaN` and `Infinity`, have no `serde_json` form and fail with [`Error::NonFiniteFloat`]. @@ -166,3 +425,86 @@ fn write_string(out: &mut String, text: &str) { } out.push('"'); } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::{dumps, loads}; + use crate::{Error, Value, repr::repr}; + + // Expected values are the repr of CPython 3.14 json.loads(text) + #[rstest] + #[case::nan("NaN", "nan")] + #[case::infinity("Infinity", "inf")] + #[case::negative_infinity("-Infinity", "-inf")] + #[case::non_finite_in_a_list("[NaN, Infinity, -Infinity]", "[nan, inf, -inf]")] + #[case::beyond_u64("18446744073709551616", "18446744073709551616")] + #[case::exponent_overflow("1e400", "inf")] + #[case::upper_exponent("1E+2", "100.0")] + #[case::negative_zero_int("-0", "0")] + #[case::negative_zero_float("-0.0", "-0.0")] + #[case::repeated_key(r#"{"a": 1, "b": 2, "a": 3}"#, "{'a': 3, 'b': 2}")] + #[case::surrogate_pair(r#""😀""#, "'😀'")] + #[case::upper_hex_surrogate_pair(r#""😀""#, "'😀'")] + #[case::escapes( + r#"{"k": [true, false, null, 1.5, "é\/"]}"#, + "{'k': [True, False, None, 1.5, 'é/']}" + )] + #[case::delete_is_not_a_control_character("\"\x7f\"", "'\\x7f'")] + #[case::surrounding_whitespace(" \n[1]\t", "[1]")] + #[case::empty_containers("[{}, []]", "[{}, []]")] + fn loads_reads_like_python(#[case] text: &str, #[case] expected: &str) { + assert_eq!(repr(&loads(text).unwrap()), expected); + } + + // CPython 3.14 json.loads raises JSONDecodeError on each + #[rstest] + #[case::lowercase_nan("nan")] + #[case::signed_nan("-NaN")] + #[case::plus_sign("+1")] + #[case::bare_minus("-")] + #[case::leading_zero("01")] + #[case::bare_point("1.")] + #[case::bare_exponent("1e")] + #[case::trailing_data("Infinity1")] + #[case::trailing_comma("[1,]")] + #[case::missing_comma("[1 2]")] + #[case::missing_colon(r#"{"a" 1}"#)] + #[case::unquoted_key("{1: 2}")] + #[case::raw_newline_in_string("\"a\nb\"")] + #[case::unknown_escape(r#""\x41""#)] + #[case::empty("")] + #[case::byte_order_mark("\u{feff}1")] + fn loads_rejects_what_python_rejects(#[case] text: &str) { + assert!(matches!(loads(text), Err(Error::InvalidJson(_)))); + } + + #[rstest] + fn a_lone_surrogate_is_a_known_limit() { + assert!(matches!(loads(r#""\ud800""#), Err(Error::InvalidJson(_)))); + } + + #[rstest] + fn loads_reads_back_the_non_finite_floats_dumps_writes() { + let written = dumps(&Value::List(vec![ + Value::Float(f64::NAN), + Value::Float(f64::INFINITY), + Value::Float(f64::NEG_INFINITY), + ])) + .unwrap(); + + assert_eq!(repr(&loads(&written).unwrap()), "[nan, inf, -inf]"); + } + + #[rstest] + fn loads_rejects_nesting_past_max_depth() { + let text = format!( + "{}{}", + "[".repeat(crate::MAX_DEPTH + 1), + "]".repeat(crate::MAX_DEPTH + 1) + ); + + assert!(matches!(loads(&text), Err(Error::TooDeep))); + } +} diff --git a/litellm-rust/crates/python-compat/src/lib.rs b/litellm-rust/crates/python-compat/src/lib.rs index b23aff409f2..2cc0eeac397 100644 --- a/litellm-rust/crates/python-compat/src/lib.rs +++ b/litellm-rust/crates/python-compat/src/lib.rs @@ -7,9 +7,10 @@ //! |---|---| //! | [`literal`] | `ast.literal_eval(text)` | //! | [`repr`] | `repr(value)` and `str(value)` | -//! | [`json`] | `json.dumps(value)`, and `json.loads(json.dumps(value))` as a JSON value | +//! | [`json`] | `json.dumps(value)` and `json.loads(text)` | //! | [`pickle`] | `pickle.loads(data)` and `pickle.dumps(value)` for plain data | //! | [`truthy`] | `bool(value)` | +//! | [`number`] | `int(text)`, `float(text)`, and `int(seconds)` as a `Duration` | //! //! [`Value`] is the closed data model these formats share. Live Python objects //! (descriptors, `__bool__`, `__str__`, callbacks) are out of scope: those belong to the @@ -20,10 +21,12 @@ //! (string hashes are randomized per process). Sets keep their literal order. //! - `str` values are Rust `String`s, so lone surrogates cannot be represented. //! - [`pickle::loads`] decodes `tuple`, `set`, and `frozenset` as lists. +//! - [`number`] reads ASCII digits only, while Python's `int` and `float` accept any Unicode decimal digit. mod error; pub mod json; pub mod literal; +pub mod number; pub mod pickle; pub mod repr; pub mod truthy; diff --git a/litellm-rust/crates/python-compat/src/literal.rs b/litellm-rust/crates/python-compat/src/literal.rs index 12e9036572f..336c6dfcb2e 100644 --- a/litellm-rust/crates/python-compat/src/literal.rs +++ b/litellm-rust/crates/python-compat/src/literal.rs @@ -58,13 +58,6 @@ impl Term { kind: Kind::Other, } } - - fn is_number(&self) -> bool { - matches!( - self.value, - Value::Int(_) | Value::Float(_) | Value::Complex { .. } - ) - } } struct Parser<'a> { @@ -204,18 +197,25 @@ impl Parser<'_> { self.enter()?; let operand = self.unary()?; self.depth -= 1; - if operand.kind != Kind::Constant || !operand.is_number() { + if operand.kind != Kind::Constant { return Err(Error::InvalidLiteral(self.offset + at)); } - let value = if !negative { - operand.value - } else { - match operand.value { - Value::Int(value) => Value::Int(-value), - Value::Float(value) => Value::Float(-value), - Value::Complex { re, im } => Value::Complex { re: -re, im: -im }, - _ => unreachable!("checked numeric above"), - } + let value = match (operand.value, negative) { + (value @ (Value::Int(_) | Value::Float(_) | Value::Complex { .. }), false) => value, + (Value::Int(value), true) => Value::Int(-value), + (Value::Float(value), true) => Value::Float(-value), + (Value::Complex { re, im }, true) => Value::Complex { re: -re, im: -im }, + ( + Value::None + | Value::Bool(_) + | Value::Str(_) + | Value::Bytes(_) + | Value::List(_) + | Value::Tuple(_) + | Value::Dict(_) + | Value::Set(_), + _, + ) => return Err(Error::InvalidLiteral(self.offset + at)), }; Ok(Term { value, diff --git a/litellm-rust/crates/python-compat/src/number.rs b/litellm-rust/crates/python-compat/src/number.rs new file mode 100644 index 00000000000..6e2f067ae25 --- /dev/null +++ b/litellm-rust/crates/python-compat/src/number.rs @@ -0,0 +1,121 @@ +use std::{str::FromStr, time::Duration}; + +use num_bigint::BigInt; + +/// `int(text)` in base 10, or `None` where Python raises `ValueError`. +pub fn int(text: &str) -> Option { + parse(text) +} + +/// `float(text)`, or `None` where Python raises `ValueError`. +pub fn float(text: &str) -> Option { + parse(text) +} + +/// Why [`seconds`] refused a value, as `int(value)` would. Messages are CPython's. +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub enum SecondsError { + /// `OverflowError`. Finite values past `Duration::MAX` land here too, where Python builds a big int. + #[error("cannot convert float infinity to integer")] + Overflow, + /// `ValueError`. + #[error("cannot convert float NaN to integer")] + NotANumber, +} + +/// A float of seconds as a `Duration`, refusing what `int(value)` refuses. Finite negative values clamp to zero. +pub fn seconds(value: f64) -> Result { + match value { + value if value.is_nan() => Err(SecondsError::NotANumber), + value if value.is_infinite() => Err(SecondsError::Overflow), + value => Duration::try_from_secs_f64(value.max(0.0)).map_err(|_| SecondsError::Overflow), + } +} + +fn parse(text: &str) -> Option { + let text = text.trim(); + let digit_at = |index: Option| { + index + .and_then(|index| text.as_bytes().get(index)) + .is_some_and(u8::is_ascii_digit) + }; + let underscores_between_digits = text + .match_indices('_') + .all(|(index, _)| digit_at(index.checked_sub(1)) && digit_at(Some(index + 1))); + if !underscores_between_digits { + return None; + } + text.replace('_', "").parse().ok() +} + +#[cfg(test)] +mod tests { + use num_bigint::BigInt; + use rstest::rstest; + + use std::time::Duration; + + use super::{SecondsError, float, int, seconds}; + + // Expected values are CPython 3.14 int(text) + #[rstest] + #[case::trailing_newline("2\n", Some(2))] + #[case::surrounding_spaces_and_underscore(" 6_380 ", Some(6380))] + #[case::plus_sign("+5", Some(5))] + #[case::negative_zero("-0", Some(0))] + #[case::leading_zeros("007", Some(7))] + #[case::unicode_spaces("\u{a0}7\u{2003}", Some(7))] + #[case::beyond_i64("99999999999999999999", Some(99_999_999_999_999_999_999_i128))] + #[case::double_underscore("1__0", None)] + #[case::leading_underscore("_1", None)] + #[case::trailing_underscore("1_", None)] + #[case::decimal_point("5.0", None)] + #[case::information_separator("\x1c7", None)] + #[case::empty("", None)] + #[case::sign_only("+", None)] + #[case::non_ascii_digit_is_a_known_limit("\u{667}", None)] + fn int_parses_like_python(#[case] text: &str, #[case] expected: Option) { + assert_eq!(int(text), expected.map(BigInt::from)); + } + + // Expected values are CPython 3.14 float(text) + #[rstest] + #[case::underscore_and_newline(" 1_0.5\n", Some(10.5))] + #[case::exponent("1e3", Some(1000.0))] + #[case::leading_point(".5", Some(0.5))] + #[case::trailing_point("5.", Some(5.0))] + #[case::plus_sign("+1.5", Some(1.5))] + #[case::infinity_word("Infinity", Some(f64::INFINITY))] + #[case::infinity_with_space("infinity ", Some(f64::INFINITY))] + #[case::underscore_before_exponent("1_e3", None)] + #[case::underscore_after_exponent("1e_3", None)] + #[case::underscore_after_point("1._5", None)] + #[case::underscore_after_word("inf_", None)] + #[case::hexadecimal("0x10", None)] + #[case::bare_exponent("1.5e", None)] + fn float_parses_like_python(#[case] text: &str, #[case] expected: Option) { + assert_eq!(float(text), expected); + } + + #[rstest] + fn float_reads_a_signed_nan_like_python() { + assert!(float("-NaN").is_some_and(f64::is_nan)); + } + + // CPython 3.14: int(inf) and int(-inf) raise OverflowError, int(nan) raises ValueError + #[rstest] + #[case::fraction(1.5, Ok(Duration::from_millis(1500)))] + #[case::zero(0.0, Ok(Duration::ZERO))] + #[case::negative_clamps_to_zero(-3.0, Ok(Duration::ZERO))] + #[case::infinity(f64::INFINITY, Err(SecondsError::Overflow))] + #[case::negative_infinity(f64::NEG_INFINITY, Err(SecondsError::Overflow))] + #[case::past_duration_max(1e20, Err(SecondsError::Overflow))] + #[case::nan(f64::NAN, Err(SecondsError::NotANumber))] + #[case::negative_nan(-f64::NAN, Err(SecondsError::NotANumber))] + fn seconds_refuses_what_int_refuses( + #[case] value: f64, + #[case] expected: Result, + ) { + assert_eq!(seconds(value), expected); + } +} diff --git a/litellm-rust/crates/python-compat/src/pickle.rs b/litellm-rust/crates/python-compat/src/pickle.rs index bff7442dc6b..dd7ac6a0e4f 100644 --- a/litellm-rust/crates/python-compat/src/pickle.rs +++ b/litellm-rust/crates/python-compat/src/pickle.rs @@ -22,10 +22,8 @@ pub fn loads(data: &[u8]) -> Result { let mut deserializer = serde_pickle::Deserializer::new(data, DeOptions::new()); let value = Seed { depth: 0 } .deserialize(&mut deserializer) - .map_err(|error| Error::InvalidPickle(error.to_string()))?; - deserializer - .end() - .map_err(|error| Error::InvalidPickle(error.to_string()))?; + .map_err(Error::InvalidPickle)?; + deserializer.end().map_err(Error::InvalidPickle)?; Ok(value) } @@ -33,8 +31,7 @@ pub fn loads(data: &[u8]) -> Result { /// default. Sets and complex numbers are rejected rather than silently changing type. pub fn dumps(value: &Value) -> Result, Error> { check_picklable(value, 0)?; - serde_pickle::to_vec(&Pickled(value), SerOptions::new()) - .map_err(|error| Error::InvalidPickle(error.to_string())) + serde_pickle::to_vec(&Pickled(value), SerOptions::new()).map_err(Error::PickleWrite) } fn check_picklable(value: &Value, depth: usize) -> Result<(), Error> { @@ -66,7 +63,7 @@ impl Serialize for Pickled<'_> { Value::Bool(value) => serializer.serialize_bool(*value), Value::Int(value) => { let value = i64::try_from(value) - .map_err(|_| serde::ser::Error::custom("integer out of i64 range"))?; + .map_err(|_| serde::ser::Error::custom(Error::IntegerOutOfRange))?; serializer.serialize_i64(value) } Value::Float(value) => serializer.serialize_f64(*value), @@ -94,7 +91,7 @@ impl Serialize for Pickled<'_> { map.end() } value @ (Value::Set(_) | Value::Complex { .. }) => Err(serde::ser::Error::custom( - format!("{} cannot be pickled as plain data", value.type_name()), + Error::NotPicklable(value.type_name()), )), } } diff --git a/litellm-rust/crates/python-compat/tests/fixtures.rs b/litellm-rust/crates/python-compat/tests/fixtures.rs index 9c202fdc5f1..3f9c6fd7010 100644 --- a/litellm-rust/crates/python-compat/tests/fixtures.rs +++ b/litellm-rust/crates/python-compat/tests/fixtures.rs @@ -76,6 +76,11 @@ const KNOWN: &[(&str, &str, &str)] = &[ "literal_eval", "deeper than MAX_DEPTH: rejected for stack safety, where CPython still parses it", ), + ( + "nested_150", + "json.loads", + "deeper than MAX_DEPTH: rejected for stack safety, where CPython still parses it", + ), ( "nested_150", "pickle.loads", @@ -341,3 +346,19 @@ fn to_json_matches_python_json_round_trip(fixtures: &Fixtures) { } mismatches.finish(); } + +/// CPython gives back the same text from `json.dumps(json.loads(text))` for every text its +/// `json.dumps` wrote here, non-finite floats included, so `loads` must too. +#[rstest] +fn loads_reads_back_what_python_json_dumps_wrote(fixtures: &Fixtures) { + let mut mismatches = Mismatches::default(); + for row in &fixtures.rows { + let Some(text) = &row.json else { continue }; + let actual = match json::loads(text).and_then(|value| json::dumps(&value)) { + Ok(text) => text, + Err(error) => format!("error: {error}"), + }; + mismatches.check(row, "json.loads", text, &actual); + } + mismatches.finish(); +} diff --git a/litellm-rust/crates/redis-client/AGENTS.md b/litellm-rust/crates/redis-client/AGENTS.md index c73a4ed8c4d..6135919c539 100644 --- a/litellm-rust/crates/redis-client/AGENTS.md +++ b/litellm-rust/crates/redis-client/AGENTS.md @@ -1,12 +1,12 @@ - The one async Redis client for the gateway and the Rust cache crates. It owns connecting, timeouts, error classes and the key namespace - It knows nothing about domains, so key builders, TTL policy, value formats and fallback decisions stay in the crate that owns the state. Every write takes its TTL from the caller, and there is no default - Async only. Build one `RedisClient` per Redis instance at the process boundary and clone it. Never add a pool, a sync API, `spawn_blocking`, or a client built inside a route -- Set every redis-rs timeout and retry explicitly. Its defaults (500ms response, 1s connect, 6 reconnect retries) are not the Python proxy's (5s socket timeout from `REDIS_SOCKET_TIMEOUT`) +- Every connection sets TCP_NODELAY, like redis-py's `socket.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)`. Set every redis-rs timeout and retry explicitly, citing the redis-py 5.3.1 or `litellm/` source next to each constant. Its defaults (500ms response, 1s connect, 6 reconnect retries) are not the Python proxy's. `Timeouts` takes `socket_timeout` and `socket_connect_timeout`, and the connect timeout falls back to the socket timeout like redis-py. Standalone reconnects on the next command with no retry of the failed one, matching redis-py's `Retry(NoBackoff(), 0)`. redis-rs keeps a socket whose command timed out, so `Renewing` in `connection.rs` swaps in a fresh `ConnectionManager`, or a fresh cluster connection like redis-py's cluster `aclose`, after a timeout. Without it a socket that went silent times out every command until TCP gives up on it. Cluster keeps redis-py's 3 `cluster_error_retry_attempts` and waits a flat 250ms between them, which is redis-py's CLUSTERDOWN sleep. redis-py's jittered per-node reconnect backoff has no redis-rs equivalent. The retry set also differs, redis-rs retries LOADING and READONLY internally, while redis-py 5.3.1 raises them. Do not claim exact retry parity - Resolving config from `config.yaml` / `REDIS_*` env the way `litellm/_redis.py` does belongs to the consumer that reads that config, not here. `RedisConfig` takes already-resolved values ## Errors - `Transport`, `Server`, `Decode`, each wrapping the redis-rs error as `#[source]`. `ConnectError` is separate because only `connect` returns it -- Classify in `From for Error` by `ErrorKind`. LOADING, CLUSTERDOWN, MASTERDOWN, TRYAGAIN and READONLY are typed server replies, but they mean the node can't serve right now, so they are `Transport`. Codes redis-rs doesn't type (WRONGTYPE) come back as `ErrorKind::Extension` and are `Server`. Each code gets its own rstest case built from a parsed RESP error +- Classify with `Error::classify(error, deployment)`, never a bare `From`, because redis-py (5.3.1) decides by topology. `Transport` is what Python's `_redis_health_error_types` counts: LOADING everywhere, CLUSTERDOWN and MASTERDOWN only on cluster (only `ClusterParser` maps them), READONLY only under Sentinel (where `SentinelManagedConnection` turns it into `ConnectionError` and rediscovers), plus NOAUTH, WRONGPASS and the `ERR` messages redis-py maps to `ConnectionError`, matched exactly like `parse_error`. TRYAGAIN is `Server`, since the cluster client already retried it. Each code gets its own rstest case built from a parsed RESP error - A `Server` error never drops the connection - `Transport` includes timeouts, where the command may still have applied. Callers of counters and locks treat it as an unknown outcome - Never swallow an error or turn it into a miss. The caller picks the fallback @@ -17,27 +17,61 @@ - Every public call (each command, `eval`, one `run` of a pipeline) takes one admission through `guarded` / `admit` in `client.rs`. Never call the connection around it - `Transport` counts as a failure (`Timeout` when `is_timeout()`, else `Hard`). `Server` and `Decode` count as nothing, since Redis answered and the request was bad. A probe that gets one still reopens, through the dropped-probe rule in `circuit-breaker`. A pipeline counts one failure if any op came back `Transport`, else one success - An open breaker returns `Error::BreakerOpen` without touching the socket -- Python doesn't count MASTERDOWN, TRYAGAIN or READONLY as health failures. Here they are `Transport` and do count ## Namespace - Applied once, here, to every key. A key that already starts with `{namespace}:` is left alone, matching `RedisCache.check_and_fix_namespace` - The rule never changes, because Python and Rust pods share keys in the same Redis ## Commands -- One typed method per command, built with `redis::cmd`. Empty batches (MGET, DEL) return without a round trip, since Redis rejects an empty MGET +- A command is a value, one constructor in `commands.rs` returning `Command`: its alone and pipelined forms and its reply decode. `RedisClient::send`, `Pipeline::with` and `RedisBatch::add` are the only entry points, and each applies its namespace to the keys once. A new command is one constructor, never a method on an executor. `ping` and `subscribe` stay methods. Empty batches (MGET, DEL) return without a round trip, since Redis rejects an empty MGET +- A store attaches its value codec with `Command::map`, never a keyspace codec inside this crate - On cluster, redis-rs splits MGET and DEL by slot and returns MGET values in input order. Other multi-key commands are not split, so a new one needs a cross-slot test -- TTLs are `Duration` and are sent in milliseconds (`PX`, `PEXPIRE`, `PTTL`) +- Every command is the one `RedisCache` sends, argument for argument, so `MONITOR` on a mixed fleet shows one shape. A difference from main's Python is a defect even when the result is the same +- TTLs are `Duration` and are sent in whole seconds (`SET ... EX`, `EXPIRE`, `TTL`), like redis-py's `int(ttl)`. `set_nx` sends `EX` before `NX`, the order redis-py builds +- Numbers go out as `Amount`, written the way redis-py encodes them: an `int` with `str()` and a `float` with `repr()` from `python-compat` +- Increments are plain commands, never a script. `incr_by` is `increment_cache`: `INCRBY`, `TTL`, then `EXPIRE` only on a key with no TTL, each its own round trip. `incr_by_float` alone is `async_increment`: `INCRBYFLOAT` with `EXPIRE` (`Expiry::Refresh`) or `TTL` (`Expiry::KeepExisting`) in one round trip, and for `KeepExisting` a third, `EXPIRE`, only on a key with no TTL. Pipelined it is `RedisBatch`'s `_Increment`, `INCRBYFLOAT` then `EXPIRE` when an expiry is given, `KeepExisting` lowered to `Refresh`. `Plan` keeps only the cases whose alone and pipelined forms differ ## Scripts and pipelines - A `Script` declares every key it touches in KEYS. On cluster an undeclared key works or fails depending on which node owns its slot, so it can pass tests and break in production -- `eval` goes through redis-rs `invoke_async`, which loads the script on NOSCRIPT. Inside a pipeline nothing reloads, so a NOSCRIPT op reruns alone with `invoke_async` +- `eval` goes through redis-rs `invoke_async`, which loads the script on NOSCRIPT. Inside a pipeline a script is sent as `EVALSHA`, like `RedisBatch`. A `NOSCRIPT` reply means that op ran nothing, so it alone runs again after the pipeline through `invoke_async` (`EVALSHA`, `SCRIPT LOAD`, `EVALSHA`), like `RedisBatch`'s `_settle_alone`. It then lands after the ops around it, as it does in Python +- `Plan::Script` keeps a script call apart from plain commands, and the one resend takes only a `ScriptCall` whose reply was `NOSCRIPT`. Nothing can resend a pipeline or a command that already ran, so a retry can never apply an increment twice +- `RedisBatch::pipelined` is one `Pipeline` whose ops answer as typed `Declared`s. A store that sends its own pipeline uses it rather than decoding `run`'s raw `Value`s - A pipeline is never atomic. Every op gets its own `Result`, from `.ignore_errors()`, so one rejected op leaves the others alone. A transport failure fails every op of that round trip -- redis-rs rejects an async cluster pipeline whose ops span slots. `Pipeline` groups ops by the slot of their first key, runs the groups concurrently and returns results in input order -- Pipeline ops take one key each. A multi-key op would span slots inside one group -- Command builders live in `commands.rs` and are shared by `RedisClient` and `Pipeline`, so a command is written once +- redis-rs rejects an async cluster pipeline whose ops span slots. `Pipeline` groups ops by the slot of their first key and runs the slots concurrently, returning results in input order. Within a slot, runs of idempotent ops (GET, SET, DEL, EXPIRE) share a request and every script op gets its own, sent in order. redis-rs retries a cluster request whole, so a script op batched with others would be applied twice when only a neighbour failed +- Pipeline ops take one key each. A multi-key op would span slots inside one group. An op can send more than one command for that key (`INCRBYFLOAT` then `EXPIRE`), and settles from the first reply, failing if any of its replies failed +- Command constructors live in `commands.rs` and every executor sends them, so a command is written once -## Tests +## Pub/sub +- Classic `PUBLISH` / `SUBSCRIBE`, never sharded, since redis-py publishes and subscribes through one plain client and cluster broadcasts classic messages to every node. Payloads are raw bytes, and the codec belongs to the store that owns the channel +- Channel names never get the namespace. Python builds `{namespace}:{channel}` itself, so the store does too +- `publish` is a command like any other, guarded by the breaker and reported to events +- `subscribe` returns a `Stream` that opens its own RESP2 `PubSub` connection when first polled, like redis-py's `PubSub`, to the endpoint, the first reachable cluster node, or the master Sentinel names at that moment. It never touches the breaker or events, like Python's subscriber, which reads off its own connection +- After a lost connection or a failed subscribe it waits and resubscribes, 5s doubling to 60s and reset by a successful subscribe, like `AuthCacheInvalidationSubscriber._run`. Messages published while it is down are lost. The connection has no idle ping, so a socket that dies silently is only noticed by TCP +- redis-rs's `PubSub` takes no credentials provider, so a provider is asked for its current credentials once when the subscription connects, which is when redis-py's pubsub connection asks it. The connect is bounded by the connect timeout + +## Batches +- `RedisBatch` is litellm's `RedisBatch` (`litellm/caching/redis_batch.py`). It keeps each declared op as data and, on a standalone or Sentinel Redis, sends them later as one `Pipeline`, so it inherits every pipeline rule above. On a cluster it never pipelines and runs each op on its own, concurrently, the way `_settle_alone` calls the `RedisCache` method behind it. An increment there keeps a TTL the key already has, as `async_increment` does, where the pipelined one refreshes it like `_Increment` +- `mget` is one `MGET` of the keys in order with repeats read once, like `_MGet`, and answers one value per key it was given, in that order. Use it instead of one `get` per key whenever Python reads several keys in one call +- It knows nothing about requests. The host that owns a request (Axum handler, pyo3 adapter) builds one and passes it down as a value, never through a task-local +- `RedisBatch::alone` runs every op on its own like the cluster path, so a fresh alone batch is "send it now" and `Declared` is the one reply handle on every route +- Each op returns a `Declared` backed by a oneshot. `flush` sends everything declared so far, one flush at a time, and an empty flush makes no round trip. `resolve` flushes only when its op has not gone out yet, so an op declared after another caller resolved costs a second round trip. Stages that know their reads up front declare all of them and call `flush` once +- An op whose flush was cancelled resolves as `Transport`, because it may already have reached Redis +- `settled` waits for whichever flush carries the op and never sends one itself, for follow-ups that must not force a flush. `RedisClient::same_client` tells clones of one client apart from a second connection, so a host can keep one batch per client - Against real Redis from `litellm-redis-testing`, never a mocked wire or asserted command bytes +- Tests that only need the public API live in `tests/.rs`, with client builders in `tests/support`. Inline `#[cfg(test)]` modules are only for private items (`error.rs`, `namespace.rs`) +- Observe connections from the server side (`RedisDeployment::clients`, `cli_on_every_node`), never by reaching into `RedisClient`'s connection - Every command test runs on each topology through `#[values(Topology::Standalone, Topology::Cluster)]`. A test is single-topology only when the behavior exists on one (a `db` number has no cluster meaning) -- Assert values and Redis state. Outages come from the container (`restart`, `freeze` / `thaw`), not `DEBUG`, which the official image disables +- Assert values and Redis state. Outages come from the container (`restart`, `freeze` / `thaw`), not `DEBUG`, which the official image disables. A socket that never answers again comes from `TcpProxy::blackhole_open_connections`, since `thaw` revives the same socket - Use a short client timeout in outage tests so they stay fast. Never sleep to wait for a state; poll with a deadline + +## TLS, credentials and Sentinel +- `TlsOptions` follows redis-py 5.3.1. It verifies the certificate by default, skips the hostname check by default (`ssl_check_hostname=False`), and trusts the system roots plus `ca_certs` +- rustls only reads unencrypted keys, so a PKCS#8 `ENCRYPTED PRIVATE KEY` is decrypted with `key_password` before it reaches redis-rs. A legacy `Proc-Type: 4,ENCRYPTED` PEM key is refused with `KeyError::LegacyEncryption` rather than decrypted by hand-rolled OpenSSL key derivation +- redis-rs loads the system trust store for every TLS connection and fails if that fails, even with `ca_certs` set. Where the macOS keychain is blocked (agent sandboxes), run the TLS tests with `SSL_CERT_FILE=/etc/ssl/cert.pem` +- `Credentials::Static` are sent at connect, and URL userinfo wins over them, like redis-py's `from_url` +- A url's `?protocol=` is ignored and every connection speaks RESP2, which every reply decode assumes +- `Credentials::Provider` wraps a redis-rs `StreamingCredentialsProvider`. redis-rs takes the first item at connect and re-`AUTH`s live connections on every later item, on each cluster node. Providers push, redis-rs never polls. With a provider, redis-rs overwrites URL userinfo at connect, which matches Python stripping `user:pass@` from the URL +- Sentinel discovers the master on each new connection and drops it on a transport failure or a READONLY reply, to one command or inside a pipeline, like redis-py's `SentinelConnectionPool` +- redis-rs sends `ROLE` to the master a Sentinel names before using it, which redis-py never does. That probe gets the data credentials (a provider's current pair) and TLS, so the Sentinel client is built on each discovery. redis-rs gives the probe no way to skip the hostname check, and drops the client certificate when the certificate is not verified, so a TLS master whose certificate does not name the announced address, or that requires a client certificate while verification is off, fails discovery where Python connects +- Sentinel monitors get `sentinel_password`, the data username when that password is set, and the TLS mode and certificates. Python's `_sentinel_auth_kwargs` forwards every other connection kwarg too, and it sends the username even without a sentinel password +- A `Debug` impl never shows a secret. Follow `.agents/skills/rust-redaction` diff --git a/litellm-rust/crates/redis-client/Cargo.toml b/litellm-rust/crates/redis-client/Cargo.toml index eab57b36423..d3d3226abc6 100644 --- a/litellm-rust/crates/redis-client/Cargo.toml +++ b/litellm-rust/crates/redis-client/Cargo.toml @@ -6,12 +6,28 @@ license.workspace = true repository.workspace = true [dependencies] +tracing.workspace = true litellm-circuit-breaker.workspace = true +litellm-secret-value.workspace = true +litellm-python-compat.workspace = true +litellm-redis-types.workspace = true futures-util.workspace = true -redis = { workspace = true, features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] } +redis = { workspace = true, features = [ + "cluster-async", + "connection-manager", + "sentinel", + "tls-rustls-insecure", + "token-based-authentication", + "tokio-rustls-comp", +] } +pem = "4.0.0" +pkcs8 = { version = "0.10.2", features = ["3des", "encryption", "std"] } +rustls-native-certs.workspace = true +socket2 = "0.6.5" thiserror.workspace = true +tokio = { workspace = true, features = ["rt", "sync", "time"] } [dev-dependencies] litellm-redis-testing.workspace = true +rustls.workspace = true rstest.workspace = true -tokio.workspace = true diff --git a/litellm-rust/crates/redis-client/src/batch.rs b/litellm-rust/crates/redis-client/src/batch.rs new file mode 100644 index 00000000000..84fc1643424 --- /dev/null +++ b/litellm-rust/crates/redis-client/src/batch.rs @@ -0,0 +1,147 @@ +use std::{ + io, + sync::{Arc, Mutex, MutexGuard}, +}; + +use futures_util::future::join_all; +use redis::{RedisError, Value}; +use tokio::sync::oneshot::{self, error::TryRecvError}; + +use crate::{Command, Error, Pipeline, RedisClient, commands::Op}; + +/// Ops declared by many callers and sent together, like litellm's `RedisBatch`: one pipeline on a +/// standalone or Sentinel Redis, and each op on its own on a cluster or in an alone batch. +#[derive(Clone)] +pub struct RedisBatch(Arc); + +#[derive(Clone, Copy)] +enum Mode { + /// litellm's `RedisBatch`: one pipeline, or each op on its own on a cluster. + Batch, + /// Each op on its own, the way the `RedisCache` method behind it sends it. + Alone, + /// One redis-py pipeline on every topology, split by slot on a cluster. + Pipeline, +} + +struct Shared { + client: RedisClient, + mode: Mode, + pending: Mutex, + flushing: tokio::sync::Mutex<()>, +} + +#[derive(Default)] +struct Pending { + ops: Vec, + replies: Vec>>, +} + +type Decode = Box Result + Send>; + +#[must_use] +pub struct Declared { + batch: RedisBatch, + reply: oneshot::Receiver>, + decode: Decode, +} + +impl RedisBatch { + pub(crate) fn new(client: RedisClient) -> Self { + Self::build(client, Mode::Batch) + } + + /// A batch whose every op goes out on its own, the way the `RedisCache` method behind it sends + /// it, so a fresh one is "send it now": the first `resolve` sends it. + pub fn alone(client: RedisClient) -> Self { + Self::build(client, Mode::Alone) + } + + /// A batch sent as one redis-py pipeline, like `async_increment_pipeline`, on every topology. + pub fn pipelined(client: RedisClient) -> Self { + Self::build(client, Mode::Pipeline) + } + + fn build(client: RedisClient, mode: Mode) -> Self { + Self(Arc::new(Shared { + client, + mode, + pending: Mutex::new(Pending::default()), + flushing: tokio::sync::Mutex::new(()), + })) + } + + pub fn add(&self, command: Command) -> Declared { + let (op, decode) = command.prepare(self.0.client.namespace()); + let (sender, reply) = oneshot::channel(); + let mut pending = self.pending(); + pending.ops.push(op); + pending.replies.push(sender); + Declared { + batch: self.clone(), + reply, + decode, + } + } + + pub async fn flush(&self) { + let _flushing = self.0.flushing.lock().await; + let Pending { ops, replies } = std::mem::take(&mut *self.pending()); + if replies.is_empty() { + return; + } + let client = &self.0.client; + let alone = match self.0.mode { + Mode::Alone => true, + Mode::Batch => client.is_cluster(), + Mode::Pipeline => false, + }; + let results = match alone { + true => join_all(ops.into_iter().map(|op| client.run_alone(op))).await, + false => { + client + .run(&Pipeline::of(client.namespace().clone(), ops)) + .await + } + }; + for (reply, result) in replies.into_iter().zip(results) { + let _ = reply.send(result); + } + } + + fn pending(&self) -> MutexGuard<'_, Pending> { + self.0 + .pending + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + } +} + +impl Declared { + pub async fn resolve(mut self) -> Result { + match self.reply.try_recv() { + Ok(reply) => return reply.and_then(self.decode), + Err(TryRecvError::Closed) => return Err(abandoned()), + Err(TryRecvError::Empty) => {} + } + self.batch.flush().await; + self.settled().await + } + + pub async fn settled(self) -> Result { + let reply = self.reply.await.unwrap_or_else(|_| Err(abandoned())); + reply.and_then(self.decode) + } + + /// Every reply in order, flushing their batch once if nobody has yet. + pub async fn all(declared: impl IntoIterator) -> Vec> { + join_all(declared.into_iter().map(Self::resolve)).await + } +} + +fn abandoned() -> Error { + Error::Transport(RedisError::from(io::Error::new( + io::ErrorKind::Interrupted, + "the batch flush carrying this op was cancelled", + ))) +} diff --git a/litellm-rust/crates/redis-client/src/client.rs b/litellm-rust/crates/redis-client/src/client.rs index a02b0847663..3cf468e0d3f 100644 --- a/litellm-rust/crates/redis-client/src/client.rs +++ b/litellm-rust/crates/redis-client/src/client.rs @@ -1,21 +1,43 @@ use std::{ future::Future, + sync::{Arc, Mutex, PoisonError, Weak}, time::{Duration, Instant}, }; -use litellm_circuit_breaker::{Admission, Breaker, Failure}; -use redis::{Cmd, FromRedisValue, IntoConnectionInfo, ToRedisArgs, ToSingleRedisArg, Value}; +use futures_util::stream::BoxStream; +use litellm_circuit_breaker::{Admission, Breaker, Failure, State}; +use redis::{Cmd, RedisError, Value}; use crate::{ - ConnectError, Endpoint, Error, Namespace, Pipeline, RedisConfig, Script, commands, + CallEvent, Command, ConnectError, Error, FailureKind, Namespace, Operation, Outcome, Pipeline, + RedisBatch, RedisConfig, RedisEvents, + commands::{self, Op, Plan, Trips}, connection::Connection, + pubsub::Subscriber, }; +/// One Redis client. Clones share the connection, the breaker and the subscriber. #[derive(Clone)] -pub struct RedisClient { - pub(crate) connection: Connection, +pub struct RedisClient(Arc); + +/// A handle that does not keep the client alive, for a registry that must not outlive its users. +#[derive(Clone)] +pub struct WeakRedisClient(Weak); + +impl WeakRedisClient { + pub fn upgrade(&self) -> Option { + self.0.upgrade().map(RedisClient) + } +} + +struct Shared { + connection: Connection, + subscriber: Subscriber, namespace: Namespace, breaker: Option, + events: Option>, + health_check: Option, + last_used: Mutex, } #[derive(Debug, PartialEq, Eq)] @@ -27,122 +49,157 @@ pub enum Ttl { impl RedisClient { pub fn new(config: RedisConfig) -> Result { - let connection = match &config.endpoint { - Endpoint::Url(url) => { - Connection::standalone(url.as_str().into_connection_info()?, config.timeout)? - } - Endpoint::Host(host) => { - Connection::standalone(host.connection_info()?, config.timeout)? - } - Endpoint::Cluster(cluster) => Connection::cluster(cluster, config.timeout)?, - }; - Ok(Self { + let connection = Connection::new( + &config.target, + &config.credentials, + config.timeouts, + config.keepalive, + )?; + Ok(Self(Arc::new(Shared { + subscriber: Subscriber::new( + &config.target, + &config.credentials, + config.timeouts, + config.keepalive, + &connection, + )?, connection, namespace: config.namespace, breaker: config.breaker, - }) + events: config.events, + health_check: config.health_check, + last_used: Mutex::new(Instant::now()), + }))) + } + + pub fn downgrade(&self) -> WeakRedisClient { + WeakRedisClient(Arc::downgrade(&self.0)) + } + + /// The breaker's current state, or none when the breaker is disabled. + pub fn breaker_state(&self) -> Option { + self.0.breaker.as_ref().map(Breaker::state) } pub async fn ping(&self) -> Result<(), Error> { - self.query(&commands::ping()).await - } - - pub async fn get(&self, key: &str) -> Result, Error> { - self.query(&commands::get(self.key(key))).await - } - - pub async fn mget(&self, keys: &[&str]) -> Result>, Error> { - if keys.is_empty() { - return Ok(Vec::new()); - } - self.query(&commands::mget(self.keys(keys))).await - } - - pub async fn set( - &self, - key: &str, - value: impl ToSingleRedisArg, - ttl: Duration, - ) -> Result<(), Error> { - self.query(&commands::set(self.key(key), value, ttl)).await - } - - pub async fn set_nx( - &self, - key: &str, - value: impl ToSingleRedisArg, - ttl: Duration, - ) -> Result { - let reply: Option = self - .query(&commands::set_nx(self.key(key), value, ttl)) - .await?; - Ok(reply.is_some()) - } - - pub async fn del(&self, keys: &[&str]) -> Result { - if keys.is_empty() { - return Ok(0); - } - self.query(&commands::del(self.keys(keys))).await - } - - pub async fn expire(&self, key: &str, ttl: Duration) -> Result { - self.query(&commands::expire(self.key(key), ttl)).await - } - - pub async fn ttl(&self, key: &str) -> Result { - let reply: i64 = self.query(&commands::ttl(self.key(key))).await?; - Ok(match reply { - -2 => Ttl::Missing, - -1 => Ttl::Persistent, - millis => Ttl::Expires(Duration::from_millis(millis.unsigned_abs())), - }) - } - - pub async fn incr_by(&self, key: &str, delta: i64) -> Result { - self.query(&commands::incr_by(self.key(key), delta)).await - } - - pub async fn incr_by_float(&self, key: &str, delta: f64) -> Result { - self.query(&commands::incr_by_float(self.key(key), delta)) - .await - } - - pub async fn rpush(&self, key: &str, values: impl ToRedisArgs) -> Result { - self.query(&commands::rpush(self.key(key), values)).await - } - - pub async fn eval( - &self, - script: &Script, - keys: &[&str], - args: impl ToRedisArgs, - ) -> Result { - let keys = self.keys(keys); - let args = args.to_redis_args(); - self.guarded(async { - Ok(script - .invocation(&keys, &args) - .invoke_async(&mut self.connection.clone()) - .await?) + self.guarded(Operation::Command("PING"), async { + commands::ping() + .query_async(&mut self.0.connection.clone()) + .await }) .await } + /// Sends one command on its own, the way the `RedisCache` method behind it sends it. + pub async fn send(&self, command: Command) -> Result { + let (op, decode) = command.prepare(&self.0.namespace); + self.run_alone(op).await.and_then(decode) + } + + pub(crate) async fn run_alone(&self, op: Op) -> Result { + let Op { + operation, plan, .. + } = op; + self.guarded(operation, async { + let mut connection = self.0.connection.clone(); + match plan { + Plan::Reply(value) => Ok(value), + Plan::Commands(commands) => match <[Cmd; 1]>::try_from(commands) { + Ok([command]) => command.query_async(&mut connection).await, + Err(commands) => { + let mut pipeline = redis::pipe(); + for command in commands { + pipeline.add_command(command); + } + let replies: Vec = pipeline.query_async(&mut connection).await?; + Ok(replies.into_iter().next().unwrap_or(Value::Nil)) + } + }, + Plan::Script(call) => { + call.script + .invocation(&call.keys, &call.args) + .invoke_async(&mut connection) + .await + } + Plan::KeepingTtl { + increment, + key, + ttl, + trips, + } => { + let (value, current): (Value, i64) = match trips { + Trips::Shared => { + redis::pipe() + .add_command(increment) + .add_command(commands::ttl_cmd(&key)) + .query_async(&mut connection) + .await? + } + Trips::Separate => ( + increment.query_async(&mut connection).await?, + commands::ttl_cmd(&key).query_async(&mut connection).await?, + ), + }; + if current == -1 { + redis::cmd("EXPIRE") + .arg(&key) + .arg(ttl.as_secs()) + .exec_async(&mut connection) + .await?; + } + Ok(value) + } + } + }) + .await + } + + /// Messages published on `channel`, on a connection of its own. After the connection is lost + /// it reconnects and resubscribes, and messages published meanwhile are not delivered. + pub fn subscribe(&self, channel: &str) -> BoxStream<'static, Vec> { + self.0.subscriber.clone().messages(channel.to_owned()) + } + pub fn pipeline(&self) -> Pipeline { - Pipeline::new(self.namespace.clone()) + Pipeline::new(self.0.namespace.clone()) + } + + pub(crate) fn namespace(&self) -> &Namespace { + &self.0.namespace + } + + pub fn is_cluster(&self) -> bool { + matches!(self.0.connection, Connection::Cluster(_)) + } + + pub fn same_client(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) + } + + pub fn batch(&self) -> RedisBatch { + RedisBatch::new(self.clone()) } pub async fn run(&self, pipeline: &Pipeline) -> Vec> { + let operation = Operation::Pipeline { + ops: pipeline.len(), + }; + let started = Instant::now(); let admission = match self.admit() { Ok(admission) => admission, Err(_) => { + self.observe( + operation, + started, + Outcome::Failed(FailureKind::BreakerOpen), + ); return (0..pipeline.len()) .map(|_| Err(Error::BreakerOpen)) .collect(); } }; - let results = pipeline.run(&self.connection).await; + self.check_health().await; + let results = pipeline.run(&self.0.connection).await; let transport = results.iter().find_map(|result| { result .as_ref() @@ -150,31 +207,76 @@ impl RedisClient { .filter(|error| matches!(error, Error::Transport(_))) }); settle(admission, transport); + let failure = transport.or_else(|| results.iter().find_map(|result| result.as_ref().err())); + let outcome = failure.map_or(Outcome::Succeeded, |error| { + Outcome::Failed(FailureKind::of(error)) + }); + self.observe(operation, started, outcome); results } - fn key(&self, key: &str) -> String { - self.namespace.apply(key) - } - - fn keys(&self, keys: &[&str]) -> Vec { - keys.iter().map(|key| self.key(key)).collect() - } - - async fn query(&self, command: &Cmd) -> Result { - self.guarded(async { Ok(command.query_async(&mut self.connection.clone()).await?) }) - .await - } - - async fn guarded(&self, call: impl Future>) -> Result { - let admission = self.admit()?; - let result = call.await; - settle(admission, result.as_ref().err()); + async fn guarded( + &self, + operation: Operation, + call: impl Future>, + ) -> Result { + let started = Instant::now(); + let result = match self.admit() { + Ok(admission) => { + self.check_health().await; + let result = call + .await + .map_err(|error| Error::classify(error, self.0.connection.deployment())); + settle(admission, result.as_ref().err()); + result + } + Err(error) => Err(error), + }; + self.observe(operation, started, Outcome::of(&result)); result } + fn observe(&self, operation: Operation, started: Instant, outcome: Outcome) { + *self + .0 + .last_used + .lock() + .unwrap_or_else(PoisonError::into_inner) = Instant::now(); + if let Some(events) = &self.0.events { + events.call(&CallEvent { + operation, + duration: started.elapsed(), + outcome, + }); + } + } + + /// redis-py pings a connection that sat idle longer than `health_check_interval` before + /// sending the next command on it. A ping that times out or finds the socket dropped leaves + /// that connection behind, as a command would, so the command then goes out on a fresh one, + /// where redis-py 5.3.1 disconnects and fails the command with the ping's error. The ping is + /// not a call of the caller's and never reaches the breaker or the events. + async fn check_health(&self) { + let Some(interval) = self.0.health_check else { + return; + }; + let idle = self + .0 + .last_used + .lock() + .unwrap_or_else(PoisonError::into_inner) + .elapsed(); + if idle < interval { + return; + } + if let Err(error) = self.0.connection.ping_every_node().await { + tracing::debug!(%error, "idle Redis connection failed its health check, replacing it"); + } + } + fn admit(&self) -> Result, Error> { - self.breaker + self.0 + .breaker .as_ref() .map(|breaker| { breaker @@ -198,450 +300,3 @@ fn settle(admission: Option, error: Option<&Error>) { Some(Error::Server(_) | Error::Decode(_) | Error::BreakerOpen) => drop(admission), } } - -#[cfg(test)] -mod tests { - use std::time::{Duration, Instant}; - - use litellm_redis_testing::RedisServer; - use rstest::rstest; - - use litellm_circuit_breaker::{Breaker, Config, State}; - - use super::{RedisClient, Ttl}; - use crate::{ - DEFAULT_TIMEOUT, Endpoint, Error, HostEndpoint, Namespace, RedisConfig, - test_support::{Server, Topology, client_id}, - }; - - #[rstest] - #[tokio::test] - async fn namespaces_isolate_keys_on_the_server( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let a = server.client("a", DEFAULT_TIMEOUT).await; - let b = server.client("b", DEFAULT_TIMEOUT).await; - let bare = server.client("", DEFAULT_TIMEOUT).await; - - a.set("x", "1", Duration::from_secs(60)).await.unwrap(); - - assert_eq!(a.get::("x").await.unwrap().as_deref(), Some("1")); - assert_eq!(a.get::("a:x").await.unwrap().as_deref(), Some("1")); - assert_eq!( - bare.get::("a:x").await.unwrap().as_deref(), - Some("1") - ); - assert_eq!(b.get::("x").await.unwrap(), None); - } - - #[rstest] - #[tokio::test] - async fn a_rejected_command_keeps_the_connection( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - client.rpush("list", "item").await.unwrap(); - let before = client_id(&client, "list").await; - - let rejected = client.incr_by("list", 1).await; - - assert!(matches!(rejected, Err(Error::Server(_))), "{rejected:?}"); - assert_eq!(client.rpush("list", "next").await.unwrap(), 2); - assert_eq!(client_id(&client, "list").await, before); - } - - #[rstest] - #[tokio::test] - async fn a_reply_of_another_type_is_a_decode_error_and_a_missing_key_is_none( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - client - .set("float", "15.0", Duration::from_secs(60)) - .await - .unwrap(); - - let decoded = client.get::("float").await; - - assert!(matches!(decoded, Err(Error::Decode(_))), "{decoded:?}"); - assert_eq!(client.get::("missing").await.unwrap(), None); - } - - #[rstest] - #[tokio::test] - async fn a_timed_out_command_does_not_leak_its_reply_into_the_next_one( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", Duration::from_millis(500)).await; - client - .set("first", "1", Duration::from_secs(60)) - .await - .unwrap(); - client - .set("second", "2", Duration::from_secs(60)) - .await - .unwrap(); - - server.freeze().await; - let timed_out = client.get::("first").await; - server.thaw().await; - - assert!( - matches!(timed_out, Err(Error::Transport(_))), - "{timed_out:?}" - ); - assert_eq!( - client.get::("second").await.unwrap().as_deref(), - Some("2") - ); - } - - #[rstest] - #[tokio::test] - async fn the_same_client_recovers_after_redis_restarts( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", Duration::from_millis(500)).await; - - server.restart().await; - - let deadline = Instant::now() + Duration::from_secs(30); - let recovered = loop { - match client.incr_by("after_restart", 1).await { - Ok(value) => break Some(value), - Err(_) if Instant::now() < deadline => { - tokio::time::sleep(Duration::from_millis(200)).await - } - Err(_) => break None, - } - }; - assert_eq!(recovered, Some(1)); - } - - #[rstest] - #[tokio::test] - async fn a_client_created_while_redis_is_down_works_once_redis_is_up( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - server.stop().await; - - let client = server.client("", Duration::from_millis(500)).await; - let while_down = client.ping().await; - server.resume().await; - - assert!( - matches!(while_down, Err(Error::Transport(_))), - "{while_down:?}" - ); - let deadline = Instant::now() + Duration::from_secs(30); - let recovered = loop { - match client.ping().await { - Ok(()) => break true, - Err(_) if Instant::now() < deadline => { - tokio::time::sleep(Duration::from_millis(200)).await - } - Err(_) => break false, - } - }; - assert!(recovered); - } - - #[rstest] - #[tokio::test] - async fn empty_batches_succeed_without_asking_redis( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - - assert_eq!( - client.mget::(&[]).await.unwrap(), - Vec::>::new() - ); - assert_eq!(client.del(&[]).await.unwrap(), 0); - } - - #[rstest] - #[tokio::test] - async fn mget_and_del_span_slots_in_input_order( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - client.set("a", "1", Duration::from_secs(60)).await.unwrap(); - client.set("c", "3", Duration::from_secs(60)).await.unwrap(); - - let values = client.mget::(&["c", "b", "a"]).await.unwrap(); - let deleted = client.del(&["a", "b", "c"]).await.unwrap(); - - assert_eq!( - values, - vec![Some("3".to_owned()), None, Some("1".to_owned())] - ); - assert_eq!(deleted, 2); - assert_eq!( - client.mget::(&["a", "c"]).await.unwrap(), - vec![None, None] - ); - } - - #[rstest] - #[tokio::test] - async fn set_nx_only_writes_a_missing_key( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - - let first = client - .set_nx("lock", "owner-1", Duration::from_secs(60)) - .await - .unwrap(); - let second = client - .set_nx("lock", "owner-2", Duration::from_secs(60)) - .await - .unwrap(); - - assert!(first); - assert!(!second); - assert_eq!( - client.get::("lock").await.unwrap().as_deref(), - Some("owner-1") - ); - } - - #[rstest] - #[tokio::test] - async fn ttl_reports_missing_persistent_and_remaining_time( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - client.incr_by("persistent", 1).await.unwrap(); - client - .set("expiring", "1", Duration::from_secs(60)) - .await - .unwrap(); - client.incr_by_float("expired_later", 0.5).await.unwrap(); - assert!( - client - .expire("expired_later", Duration::from_secs(30)) - .await - .unwrap() - ); - - assert_eq!(client.ttl("missing").await.unwrap(), Ttl::Missing); - assert_eq!(client.ttl("persistent").await.unwrap(), Ttl::Persistent); - assert!(matches!( - client.ttl("expiring").await.unwrap(), - Ttl::Expires(left) if left > Duration::from_secs(50) && left <= Duration::from_secs(60) - )); - assert!(matches!( - client.ttl("expired_later").await.unwrap(), - Ttl::Expires(left) if left > Duration::from_secs(20) && left <= Duration::from_secs(30) - )); - assert!( - !client - .expire("missing", Duration::from_secs(30)) - .await - .unwrap() - ); - } - - #[rstest] - #[tokio::test] - async fn incr_by_float_accumulates_fractions( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - - client.incr_by_float("spend", 0.25).await.unwrap(); - - assert_eq!(client.incr_by_float("spend", 0.5).await.unwrap(), 0.75); - } - - #[rstest] - #[tokio::test] - async fn concurrent_commands_on_one_cloned_client_all_apply( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - - let tasks: Vec<_> = (0..100) - .map(|_| { - let client = client.clone(); - tokio::spawn(async move { - for _ in 0..10 { - client.incr_by("shared", 1).await.unwrap(); - } - }) - }) - .collect(); - for task in tasks { - task.await.unwrap(); - } - - assert_eq!(client.get::("shared").await.unwrap(), Some(1000)); - } - - fn breaker(failure_threshold: u32) -> Breaker { - Breaker::new( - Config { - failure_threshold, - recovery_timeout: Duration::from_millis(200), - timeout_min_duration: Duration::ZERO, - }, - Box::new(|_| {}), - ) - } - - #[rstest] - #[tokio::test] - async fn an_open_breaker_rejects_without_waiting_on_redis_and_a_probe_closes_it( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let breaker = breaker(2); - let client = server - .guarded_client("", Duration::from_millis(300), Some(breaker.clone())) - .await; - - server.freeze().await; - let first = client.get::("k").await; - let second = client.get::("k").await; - let started = Instant::now(); - let third = client.get::("k").await; - let rejected_in = started.elapsed(); - server.thaw().await; - - assert!(matches!(first, Err(Error::Transport(_))), "{first:?}"); - assert!(matches!(second, Err(Error::Transport(_))), "{second:?}"); - assert!(matches!(third, Err(Error::BreakerOpen)), "{third:?}"); - assert!(rejected_in < Duration::from_millis(50), "{rejected_in:?}"); - - let deadline = Instant::now() + Duration::from_secs(30); - let recovered = loop { - match client.get::("k").await { - Ok(value) => break Some(value), - Err(_) if Instant::now() < deadline => { - tokio::time::sleep(Duration::from_millis(50)).await - } - Err(_) => break None, - } - }; - assert_eq!(recovered, Some(None)); - assert!( - matches!(breaker.state(), State::Closed { .. }), - "{:?}", - breaker.state() - ); - } - - #[rstest] - #[tokio::test] - async fn rejected_commands_never_open_the_breaker( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let breaker = breaker(2); - let client = server - .guarded_client("", DEFAULT_TIMEOUT, Some(breaker.clone())) - .await; - client.rpush("list", "item").await.unwrap(); - - let rejected = [ - client.incr_by("list", 1).await, - client.incr_by("list", 1).await, - client.incr_by("list", 1).await, - ]; - - assert!( - matches!(rejected[2], Err(Error::Server(_))), - "{:?}", - rejected[2] - ); - assert!( - matches!(breaker.state(), State::Closed { hard: 0, .. }), - "{:?}", - breaker.state() - ); - } - - #[rstest] - #[tokio::test] - async fn an_open_breaker_fails_every_pipeline_op_without_sending( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server - .guarded_client("", Duration::from_millis(300), Some(breaker(1))) - .await; - let pipeline = client.pipeline().incr_by("a", 1).incr_by("c", 1); - - server.freeze().await; - let failed = client.run(&pipeline).await; - let rejected = client.run(&pipeline).await; - server.thaw().await; - - assert!( - matches!(failed[0], Err(Error::Transport(_))), - "{:?}", - failed[0] - ); - assert!( - matches!(rejected[0], Err(Error::BreakerOpen)), - "{:?}", - rejected[0] - ); - assert!( - matches!(rejected[1], Err(Error::BreakerOpen)), - "{:?}", - rejected[1] - ); - } - - #[rstest] - #[tokio::test] - async fn a_host_endpoint_connects_to_its_database() { - let server = RedisServer::start().await.unwrap(); - let port = server.url().rsplit(':').next().unwrap().parse().unwrap(); - let connect = |endpoint| { - RedisClient::new(RedisConfig { - endpoint, - namespace: Namespace::default(), - timeout: DEFAULT_TIMEOUT, - breaker: None, - }) - }; - let in_db_3 = connect(Endpoint::Host(HostEndpoint { - host: "127.0.0.1".to_owned(), - port, - username: None, - password: None, - tls: false, - db: 3, - })) - .unwrap(); - let in_db_0 = connect(Endpoint::Url(server.url().to_owned())).unwrap(); - let via_url = connect(Endpoint::Url(format!("{}/3", server.url()))).unwrap(); - - in_db_3 - .set("k", "3", Duration::from_secs(60)) - .await - .unwrap(); - - assert_eq!(in_db_0.get::("k").await.unwrap(), None); - assert_eq!( - via_url.get::("k").await.unwrap().as_deref(), - Some("3") - ); - } -} diff --git a/litellm-rust/crates/redis-client/src/commands.rs b/litellm-rust/crates/redis-client/src/commands.rs index fb8a4a28c94..5f8c1eff3f7 100644 --- a/litellm-rust/crates/redis-client/src/commands.rs +++ b/litellm-rust/crates/redis-client/src/commands.rs @@ -1,62 +1,407 @@ use std::time::Duration; -use redis::{Cmd, ToRedisArgs, ToSingleRedisArg}; +use litellm_python_compat::repr::float_repr; +use redis::{ + Cmd, ErrorKind, FromRedisValue, RedisError, RedisWrite, ToRedisArgs, ToSingleRedisArg, Value, +}; + +use crate::{Error, Namespace, Operation, Script, Ttl, script::ScriptCall}; + +/// How an increment treats the key's TTL, the two ways `RedisCache` increments with one. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Expiry { + /// `async_increment(refresh_ttl=True)` and every pipelined increment: `EXPIRE` after each write. + Refresh(Duration), + /// `async_increment(refresh_ttl=False)`: read the `TTL` and `EXPIRE` only a key that has none. + /// Pipelined it is lowered to [`Expiry::Refresh`], like `RedisBatch.increment`. + KeepExisting(Duration), +} + +/// A number written the way redis-py encodes it: an `int` with `str()`, a `float` with `repr()`. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Amount { + Int(i64), + Float(f64), +} + +impl ToRedisArgs for Amount { + fn write_redis_args(&self, out: &mut W) { + match self { + Self::Int(value) => out.write_arg(value.to_string().as_bytes()), + Self::Float(value) => out.write_arg(float_repr(*value).as_bytes()), + } + } +} + +impl ToSingleRedisArg for Amount {} + +type Build = Box) -> Plan + Send>; +type Decode = Box Result + Send>; + +/// One Redis command as a value: what it sends alone and pipelined, and how its reply decodes. Every +/// executor (`RedisClient::send`, `Pipeline::with`, `RedisBatch::add`) applies its namespace to the +/// keys once. +#[must_use] +pub struct Command { + operation: Operation, + keys: Vec, + idempotent: bool, + build: Build, + decode: Decode, +} + +/// What a command sends once its keys carry the namespace. Only the cases whose alone and pipelined +/// forms differ get a variant of their own. +pub(crate) enum Plan { + /// Answered without a round trip, like an empty `MGET` or `DEL`. + Reply(Value), + /// Sent together; the reply is the first command's, failing if any failed. + Commands(Vec), + /// Alone through `invoke_async`, pipelined as `EVALSHA` resent once on `NOSCRIPT`. + Script(ScriptCall), + /// Alone the increment reads the `TTL` and sends `EXPIRE` only to a key without one, pipelined it + /// is the increment then `EXPIRE`. + KeepingTtl { + increment: Cmd, + key: String, + ttl: Duration, + trips: Trips, + }, +} + +#[derive(Clone, Copy)] +pub(crate) enum Trips { + /// `async_increment`: the increment and its `TTL` in one round trip. + Shared, + /// `increment_cache`: the increment and its `TTL` each on its own round trip. + Separate, +} + +/// A command with the namespace applied, ready for an executor to send. +pub(crate) struct Op { + pub(crate) operation: Operation, + pub(crate) key: String, + pub(crate) idempotent: bool, + pub(crate) plan: Plan, +} + +impl Plan { + /// What a pipeline sends for this op. + pub(crate) fn pipelined(&self) -> Vec { + match self { + Self::Reply(_) => Vec::new(), + Self::Commands(commands) => commands.clone(), + Self::Script(call) => vec![call.script.evalsha(&call.keys, &call.args)], + Self::KeepingTtl { + increment, + key, + ttl, + .. + } => vec![increment.clone(), expire_cmd(key, *ttl)], + } + } +} + +impl Command { + fn new( + operation: Operation, + keys: Vec, + idempotent: bool, + build: impl FnOnce(Vec) -> Plan + Send + 'static, + decode: impl FnOnce(Value) -> Result + Send + 'static, + ) -> Self { + Self { + operation, + keys, + idempotent, + build: Box::new(build), + decode: Box::new(decode), + } + } + + /// Attaches a codec over the decoded reply, such as a store's value format. + pub fn map(self, decode: fn(T) -> Result) -> Command + where + T: 'static, + { + let inner = self.decode; + Command { + operation: self.operation, + keys: self.keys, + idempotent: self.idempotent, + build: self.build, + decode: Box::new(move |value| inner(value).and_then(decode)), + } + } + + pub(crate) fn prepare(self, namespace: &Namespace) -> (Op, Decode) { + let keys: Vec = self.keys.iter().map(|key| namespace.apply(key)).collect(); + let key = keys.first().cloned().unwrap_or_default(); + let op = Op { + operation: self.operation, + key, + idempotent: self.idempotent, + plan: (self.build)(keys), + }; + (op, self.decode) + } +} + +fn decode(value: Value) -> Result { + T::from_redis_value(value).map_err(|error| Error::Decode(error.into())) +} + +fn owned(keys: &[&str]) -> Vec { + keys.iter().map(|key| (*key).to_owned()).collect() +} + +fn single(keys: Vec) -> String { + keys.into_iter().next().unwrap_or_default() +} + +fn expire_cmd(key: &str, ttl: Duration) -> Cmd { + redis::cmd("EXPIRE").arg(key).arg(ttl.as_secs()).to_owned() +} + +pub(crate) fn ttl_cmd(key: &str) -> Cmd { + redis::cmd("TTL").arg(key).to_owned() +} + +/// A reply whose shape a codec over it refuses, as `what` describes. +pub fn unexpected_reply(what: &'static str) -> Error { + Error::Decode(RedisError::from((ErrorKind::UnexpectedReturnType, what))) +} + +pub fn get(key: &str) -> Command> { + Command::new( + Operation::Command("GET"), + vec![key.to_owned()], + true, + |keys| Plan::Commands(vec![redis::cmd("GET").arg(single(keys)).to_owned()]), + decode, + ) +} + +/// One `MGET` of `keys` in order, repeated keys read once, like `RedisBatch.mget`, answering one +/// value per key of `keys` in their order. No keys is no round trip, since Redis rejects an empty +/// `MGET`. Pipelined, every key must share a hash slot on a cluster, and a batch never pipelines +/// there. +pub fn mget(keys: &[&str]) -> Command>> { + let unique = keys.iter().fold(Vec::::new(), |mut unique, key| { + if !unique.iter().any(|seen| seen == key) { + unique.push((*key).to_owned()); + } + unique + }); + let positions: Vec = keys + .iter() + .filter_map(|key| unique.iter().position(|seen| seen == key)) + .collect(); + Command::new( + Operation::Command("MGET"), + unique, + true, + |keys| match keys.is_empty() { + true => Plan::Reply(Value::Array(Vec::new())), + false => Plan::Commands(vec![redis::cmd("MGET").arg(keys).to_owned()]), + }, + move |reply| { + let values: Vec = decode(reply)?; + let in_key_order = positions + .iter() + .map(|&position| values.get(position).cloned()) + .collect::>>() + .ok_or_else(|| unexpected_reply("MGET answered fewer values than keys"))?; + decode(Value::Array(in_key_order)) + }, + ) +} + +// redis-py's `set(ex=...)` takes whole seconds, as `int(ttl)` or a `timedelta`'s `int(total_seconds())` +pub fn set(key: &str, value: impl ToSingleRedisArg, ttl: Duration) -> Command<()> { + let value = value.to_redis_args().concat(); + Command::new( + Operation::Command("SET"), + vec![key.to_owned()], + true, + move |keys| { + Plan::Commands(vec![ + redis::cmd("SET") + .arg(single(keys)) + .arg(value) + .arg("EX") + .arg(ttl.as_secs()) + .to_owned(), + ]) + }, + decode, + ) +} + +/// `SET ... EX ... NX`, `EX` before `NX` as redis-py builds it. `true` when this call wrote the key. +pub fn set_nx(key: &str, value: impl ToSingleRedisArg, ttl: Duration) -> Command { + let value = value.to_redis_args().concat(); + Command::new( + Operation::Command("SET"), + vec![key.to_owned()], + false, + move |keys| { + Plan::Commands(vec![ + redis::cmd("SET") + .arg(single(keys)) + .arg(value) + .arg("EX") + .arg(ttl.as_secs()) + .arg("NX") + .to_owned(), + ]) + }, + |value| decode::>(value).map(|reply| reply.is_some()), + ) +} + +/// One multi-key `DEL`, no round trip for no keys. Pipelined, every key must share a hash slot. +pub fn del(keys: &[&str]) -> Command { + Command::new( + Operation::Command("DEL"), + owned(keys), + true, + |keys| match keys.is_empty() { + true => Plan::Reply(Value::Int(0)), + false => Plan::Commands(vec![redis::cmd("DEL").arg(keys).to_owned()]), + }, + decode, + ) +} + +pub fn expire(key: &str, ttl: Duration) -> Command { + Command::new( + Operation::Command("EXPIRE"), + vec![key.to_owned()], + true, + move |keys| Plan::Commands(vec![expire_cmd(&single(keys), ttl)]), + decode, + ) +} + +pub fn ttl(key: &str) -> Command { + Command::new( + Operation::Command("TTL"), + vec![key.to_owned()], + true, + |keys| Plan::Commands(vec![ttl_cmd(&single(keys))]), + |value| { + Ok(match decode::(value)? { + -2 => Ttl::Missing, + -1 => Ttl::Persistent, + seconds => Ttl::Expires(Duration::from_secs(seconds.unsigned_abs())), + }) + }, + ) +} + +/// `RedisCache.increment_cache`: `INCRBY`, then `TTL`, then `EXPIRE` only when the key has no TTL, +/// each its own round trip. +pub fn incr_by(key: &str, delta: i64, ttl: Duration) -> Command { + Command::new( + Operation::Command("INCRBY"), + vec![key.to_owned()], + false, + move |keys| { + let key = single(keys); + Plan::KeepingTtl { + increment: redis::cmd("INCRBY").arg(&key).arg(delta).to_owned(), + key, + ttl, + trips: Trips::Separate, + } + }, + decode, + ) +} + +/// `INCRBYFLOAT`, with the TTL handled as `expiry` says, or left alone without one. Alone it is +/// `RedisCache.async_increment`, pipelined `RedisBatch.increment`. +pub fn incr_by_float(key: &str, amount: Amount, expiry: Option) -> Command { + Command::new( + Operation::Command("INCRBYFLOAT"), + vec![key.to_owned()], + false, + move |keys| { + let key = single(keys); + let increment = redis::cmd("INCRBYFLOAT").arg(&key).arg(amount).to_owned(); + match expiry { + None => Plan::Commands(vec![increment]), + Some(Expiry::Refresh(ttl)) => { + Plan::Commands(vec![increment, expire_cmd(&key, ttl)]) + } + Some(Expiry::KeepExisting(ttl)) => Plan::KeepingTtl { + increment, + key, + ttl, + trips: Trips::Shared, + }, + } + }, + decode, + ) +} + +/// A registered script, like redis-py's `Script`. Every key it touches must be in `keys`. +pub fn eval( + script: &Script, + keys: &[&str], + args: impl ToRedisArgs, +) -> Command { + let script = script.clone(); + let args = args.to_redis_args(); + Command::new( + Operation::Script, + owned(keys), + false, + move |keys| Plan::Script(ScriptCall { script, keys, args }), + decode, + ) +} + +/// A script sent with its source as `EVAL` every time, like redis-py's `Redis.eval`, for the scripts +/// `RedisCache` runs that way rather than through `register_script`. +pub fn eval_source( + source: &'static str, + keys: &[&str], + args: impl ToRedisArgs, +) -> Command { + let args = args.to_redis_args(); + Command::new( + Operation::Script, + owned(keys), + false, + move |keys| { + Plan::Commands(vec![ + redis::cmd("EVAL") + .arg(source) + .arg(keys.len()) + .arg(keys) + .arg(args) + .to_owned(), + ]) + }, + decode, + ) +} + +/// The channel is sent as given, without the namespace, like redis-py's `publish`. +pub fn publish(channel: &str, payload: &[u8]) -> Command<()> { + let command = redis::cmd("PUBLISH").arg(channel).arg(payload).to_owned(); + Command::new( + Operation::Command("PUBLISH"), + Vec::new(), + false, + move |_| Plan::Commands(vec![command]), + |_| Ok(()), + ) +} pub(crate) fn ping() -> Cmd { redis::cmd("PING") } - -pub(crate) fn get(key: String) -> Cmd { - redis::cmd("GET").arg(key).to_owned() -} - -pub(crate) fn mget(keys: Vec) -> Cmd { - redis::cmd("MGET").arg(keys).to_owned() -} - -pub(crate) fn set(key: String, value: impl ToSingleRedisArg, ttl: Duration) -> Cmd { - redis::cmd("SET") - .arg(key) - .arg(value) - .arg("PX") - .arg(millis(ttl)) - .to_owned() -} - -pub(crate) fn set_nx(key: String, value: impl ToSingleRedisArg, ttl: Duration) -> Cmd { - redis::cmd("SET") - .arg(key) - .arg(value) - .arg("NX") - .arg("PX") - .arg(millis(ttl)) - .to_owned() -} - -pub(crate) fn del(keys: Vec) -> Cmd { - redis::cmd("DEL").arg(keys).to_owned() -} - -pub(crate) fn expire(key: String, ttl: Duration) -> Cmd { - redis::cmd("PEXPIRE").arg(key).arg(millis(ttl)).to_owned() -} - -pub(crate) fn ttl(key: String) -> Cmd { - redis::cmd("PTTL").arg(key).to_owned() -} - -pub(crate) fn incr_by(key: String, delta: i64) -> Cmd { - redis::cmd("INCRBY").arg(key).arg(delta).to_owned() -} - -pub(crate) fn incr_by_float(key: String, delta: f64) -> Cmd { - redis::cmd("INCRBYFLOAT").arg(key).arg(delta).to_owned() -} - -pub(crate) fn rpush(key: String, values: impl ToRedisArgs) -> Cmd { - redis::cmd("RPUSH").arg(key).arg(values).to_owned() -} - -fn millis(duration: Duration) -> u64 { - u64::try_from(duration.as_millis()).unwrap_or(u64::MAX) -} diff --git a/litellm-rust/crates/redis-client/src/config.rs b/litellm-rust/crates/redis-client/src/config.rs index 64934c68891..f43ab4feb85 100644 --- a/litellm-rust/crates/redis-client/src/config.rs +++ b/litellm-rust/crates/redis-client/src/config.rs @@ -1,152 +1,458 @@ -use std::{fmt, time::Duration}; +use std::{fmt, path::PathBuf, pin::Pin, sync::Arc, time::Duration}; -use redis::{ConnectionAddr, ConnectionInfo, IntoConnectionInfo, RedisConnectionInfo, RedisResult}; +use futures_util::{Stream, StreamExt}; + +use redis::{ + BasicAuth, ClientTlsConfig, ConnectionAddr, ConnectionInfo, ErrorKind, IntoConnectionInfo, + ProtocolVersion, RedisConnectionInfo, RedisError, RedisResult, StreamingCredentialsProvider, + TlsCertificates, TlsMode, io::tcp::TcpSettings, +}; use litellm_circuit_breaker::Breaker; +use litellm_secret_value::SecretValue; -use crate::Namespace; +use pkcs8::EncryptedPrivateKeyInfo; +use crate::{ConnectError, KeyError, Namespace, RedisEvents}; +use litellm_redis_types::{Cluster, RedisUrl, Target, TlsOptions}; + +// litellm/constants.py REDIS_SOCKET_TIMEOUT, which _coerce_redis_kwargs_types sets when socket_timeout is unset pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(5); +// below the idle timeouts of common cloud load balancers (AWS NLB 350s, Azure Load Balancer 240s), which the +// OS default of two hours never probes within +const KEEPALIVE_IDLE: Duration = Duration::from_secs(60); pub struct RedisConfig { - pub endpoint: Endpoint, + pub target: Target, + pub credentials: Credentials, pub namespace: Namespace, - pub timeout: Duration, + pub timeouts: Timeouts, pub breaker: Option, + pub events: Option>, + /// Ping every node before the next command once the client sat idle this long, like + /// redis-py's `health_check_interval`, so a connection the server dropped silently is + /// replaced before a real command pays for it. + pub health_check: Option, + /// Probe a connection with TCP keepalive once it sat idle, like redis-py's `socket_keepalive`. + pub keepalive: bool, } -#[derive(Clone, PartialEq)] -pub enum Endpoint { - Url(String), - Host(HostEndpoint), - Cluster(ClusterEndpoint), +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Timeouts { + pub response: Duration, + pub connect: Duration, } -#[derive(Clone, PartialEq)] -pub struct HostEndpoint { - pub host: String, - pub port: u16, - pub username: Option, - pub password: Option, - pub tls: bool, - pub db: i64, -} - -#[derive(Clone, PartialEq)] -pub struct ClusterEndpoint { - pub nodes: Vec, - pub username: Option, - pub password: Option, - pub tls: bool, -} - -#[derive(Clone, Debug, PartialEq)] -pub struct ClusterNode { - pub host: String, - pub port: u16, -} - -impl ClusterEndpoint { - pub(crate) fn addrs(&self) -> impl Iterator + '_ { - self.nodes - .iter() - .map(|node| addr(&node.host, node.port, self.tls)) - } -} - -impl HostEndpoint { - pub(crate) fn connection_info(&self) -> RedisResult { - let settings = RedisConnectionInfo::default().set_db(self.db); - let settings = match &self.username { - Some(username) => settings.set_username(username), - None => settings, - }; - let settings = match &self.password { - Some(password) => settings.set_password(password), - None => settings, - }; - Ok(addr(&self.host, self.port, self.tls) - .into_connection_info()? - .set_redis_settings(settings)) - } -} - -fn addr(host: &str, port: u16, tls: bool) -> ConnectionAddr { - match tls { - true => ConnectionAddr::TcpTls { - host: host.to_owned(), - port, - insecure: false, - tls_params: None, - }, - false => ConnectionAddr::Tcp(host.to_owned(), port), - } -} - -impl fmt::Debug for Endpoint { - fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::Url(url) => formatter - .debug_tuple("Url") - .field(&redacted_url(url)) - .finish(), - Self::Host(host) => formatter - .debug_struct("Host") - .field("host", &host.host) - .field("port", &host.port) - .field("username", &host.username) - .field("password", &host.password.as_ref().map(|_| "")) - .field("tls", &host.tls) - .field("db", &host.db) - .finish(), - Self::Cluster(cluster) => formatter - .debug_struct("Cluster") - .field("nodes", &cluster.nodes) - .field("username", &cluster.username) - .field("password", &cluster.password.as_ref().map(|_| "")) - .field("tls", &cluster.tls) - .finish(), +impl Timeouts { + pub fn new(socket_timeout: Duration, connect_timeout: Option) -> Self { + Self { + response: socket_timeout, + connect: connect_timeout.unwrap_or(socket_timeout), } } } -fn redacted_url(url: &str) -> String { - match url - .split_once("://") - .and_then(|(scheme, rest)| Some((scheme, rest.rsplit_once('@')?.1))) - { - Some((scheme, host)) => format!("{scheme}://@{host}"), - None => url.to_owned(), +impl Default for Timeouts { + fn default() -> Self { + Self::new(DEFAULT_TIMEOUT, None) } } +#[derive(Clone, Debug)] +pub enum Credentials { + Static { + username: Option, + password: Option, + }, + Provider(CredentialsProvider), +} + +#[derive(Clone)] +pub struct CredentialsProvider(pub Arc); + +impl fmt::Debug for CredentialsProvider { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("CredentialsProvider") + .finish_non_exhaustive() + } +} + +impl StreamingCredentialsProvider for CredentialsProvider { + fn subscribe(&self) -> Pin> + Send + 'static>> { + self.0.subscribe() + } +} + +impl Credentials { + pub const NONE: Self = Self::Static { + username: None, + password: None, + }; +} + +pub(crate) fn tls_mode(tls: &TlsOptions) -> TlsMode { + match tls.verify_certificate { + true => TlsMode::Secure, + false => TlsMode::Insecure, + } +} + +pub(crate) fn tls_certificates(tls: &TlsOptions) -> Result { + let client_tls = match (&tls.certfile, &tls.keyfile) { + (Some(certfile), keyfile) => Some(ClientTlsConfig { + client_cert: read(certfile)?, + client_key: client_key( + keyfile.as_ref().unwrap_or(certfile), + tls.key_password.as_ref(), + )?, + }), + (None, _) => None, + }; + let from_file: Option> = tls.ca_certs.as_ref().map(read).transpose()?; + let from_directory: Vec> = tls + .ca_path + .as_ref() + .map(read_directory) + .transpose()? + .unwrap_or_default(); + let configured: Vec> = from_file + .into_iter() + .chain(tls.ca_data.iter().map(|pem| pem.clone().into_bytes())) + .chain(from_directory) + .collect(); + Ok(TlsCertificates { + client_tls, + root_cert: (!configured.is_empty()).then(|| with_system_roots(configured.join(&b'\n'))), + }) +} + +/// Every regular file in `path`, in name order, as redis-py hands `ssl_ca_path` to +/// `load_verify_locations(capath=...)` +fn read_directory(path: &PathBuf) -> Result>, ConnectError> { + let tls_file = |source| ConnectError::TlsFile { + path: path.display().to_string(), + source, + }; + let mut files: Vec = std::fs::read_dir(path) + .map_err(tls_file)? + .map(|entry| entry.map(|entry| entry.path())) + .collect::>() + .map_err(tls_file)?; + files.sort(); + files + .iter() + .filter(|file| file.is_file()) + .map(read) + .collect() +} + +pub(crate) fn apply_tls( + tls: &TlsOptions, + info: ConnectionInfo, +) -> Result { + let tls_info = redis::Client::build_with_tls(info, tls_certificates(tls)?)? + .get_connection_info() + .clone(); + let mut addr = match tls_info.addr().clone() { + ConnectionAddr::TcpTls { + host, + port, + tls_params, + .. + } => ConnectionAddr::TcpTls { + host, + port, + insecure: !tls.verify_certificate, + tls_params, + }, + addr => addr, + }; + addr.set_danger_accept_invalid_hostnames(!tls.check_hostname); + Ok(tls_info.set_addr(addr)) +} + +fn with_system_roots(configured: Vec) -> Vec { + let system = rustls_native_certs::load_native_certs(); + let system_pem = system + .certs + .iter() + .map(|cert| pem::encode(&pem::Pem::new("CERTIFICATE", cert.to_vec()))); + configured + .into_iter() + .chain(b"\n".iter().copied()) + .chain(system_pem.flat_map(String::into_bytes)) + .collect() +} + +/// The key as unencrypted PEM, decrypting a PKCS#8 `ENCRYPTED PRIVATE KEY` with `ssl_password` the way +/// `load_cert_chain(password=...)` does, since rustls only reads unencrypted keys. Like OpenSSL, a password next to +/// an unencrypted key is unused +fn client_key(path: &PathBuf, password: Option<&SecretValue>) -> Result, ConnectError> { + let contents = read(path)?; + let key_error = |reason| ConnectError::TlsKey { + path: path.display().to_string(), + reason, + }; + let Ok(sections) = pem::parse_many(&contents) else { + return Ok(contents); + }; + let encrypted = sections + .iter() + .find(|section| section.tag() == "ENCRYPTED PRIVATE KEY"); + match (encrypted, password) { + (Some(_), None) => Err(key_error(KeyError::MissingPassword)), + (Some(section), Some(password)) => { + let decrypted = EncryptedPrivateKeyInfo::try_from(section.contents()) + .and_then(|info| info.decrypt(password.expose())) + .map_err(|source| key_error(KeyError::Decrypt(source)))?; + Ok(pem::encode(&pem::Pem::new("PRIVATE KEY", decrypted.as_bytes())).into_bytes()) + } + (None, _) if sections.iter().any(is_legacy_encrypted) => { + Err(key_error(KeyError::LegacyEncryption)) + } + (None, _) => Ok(contents), + } +} + +fn is_legacy_encrypted(section: &pem::Pem) -> bool { + section + .headers() + .get("Proc-Type") + .is_some_and(|proc_type| proc_type.contains("ENCRYPTED")) +} + +fn read(path: &PathBuf) -> Result, ConnectError> { + std::fs::read(path).map_err(|source| ConnectError::TlsFile { + path: path.display().to_string(), + source, + }) +} + +pub(crate) fn addr(host: &str, port: u16, tls: Option<&TlsOptions>) -> ConnectionAddr { + match tls { + Some(tls) => ConnectionAddr::TcpTls { + host: host.to_owned(), + port, + insecure: !tls.verify_certificate, + tls_params: None, + }, + None => ConnectionAddr::Tcp(host.to_owned(), port), + } +} + +pub(crate) fn cluster_addrs(cluster: &Cluster) -> impl Iterator + '_ { + cluster + .nodes + .iter() + .map(|node| addr(&node.host, node.port, cluster.tls.as_ref())) +} + +pub(crate) fn tcp_settings(keepalive: bool) -> TcpSettings { + let settings = TcpSettings::default().set_nodelay(true); + match keepalive { + true => settings.set_keepalive(socket2::TcpKeepalive::new().with_time(KEEPALIVE_IDLE)), + false => settings, + } +} + +pub(crate) fn connection_info( + target: &Target, + credentials: &Credentials, + keepalive: bool, +) -> Result { + let (info, tls) = match target { + Target::Url(url) => (url_info(url, credentials)?, url.tls.as_ref()), + Target::Standalone(host) => { + let info = addr(&host.host, host.port, host.tls.as_ref()).into_connection_info()?; + let settings = info.redis_settings().clone().set_db(host.db); + (info.set_redis_settings(settings), host.tls.as_ref()) + } + Target::Unix(socket) => { + let info = ConnectionAddr::Unix(socket.path.clone()).into_connection_info()?; + let settings = info.redis_settings().clone().set_db(socket.db); + (info.set_redis_settings(settings), None) + } + Target::Cluster(_) | Target::Sentinel(_) => { + unreachable!("cluster and sentinel endpoints build their own connection info") + } + }; + let info = with_static_credentials(info, credentials).set_tcp_settings(tcp_settings(keepalive)); + match (tls, info.addr()) { + (Some(tls), ConnectionAddr::TcpTls { .. }) => apply_tls(tls, info), + _ => Ok(info), + } +} + +// parse_url reads ?db= over the path, and the url's userinfo over ?username= and ?password=. A credential +// provider supersedes both, as it does in redis-py. ?protocol=3 would switch redis-rs to RESP3, which no +// reply decode here expects, so the protocol is ignored as the resolver warns +fn url_info(url: &RedisUrl, credentials: &Credentials) -> Result { + let info = url.url.expose().into_connection_info()?; + let settings = info + .redis_settings() + .clone() + .set_protocol(ProtocolVersion::RESP2); + let settings = match url.db { + Some(db) => settings.set_db(db), + None => settings, + }; + let info = info.set_redis_settings(settings); + Ok(match credentials { + Credentials::Static { .. } => with_static_credentials( + info, + &Credentials::Static { + username: url.username.clone(), + password: url.password.clone(), + }, + ), + Credentials::Provider(_) => info, + }) +} + +fn with_static_credentials(info: ConnectionInfo, credentials: &Credentials) -> ConnectionInfo { + let settings = static_credentials(info.redis_settings().clone(), credentials); + info.set_redis_settings(settings) +} + +fn static_credentials( + settings: RedisConnectionInfo, + credentials: &Credentials, +) -> RedisConnectionInfo { + let Credentials::Static { username, password } = credentials else { + return settings; + }; + let settings = match (username, settings.username()) { + (Some(username), None) => settings.set_username(username.expose()), + _ => settings, + }; + match (password, settings.password()) { + (Some(password), None) => settings.set_password(password.expose()), + _ => settings, + } +} + +/// `settings` with the credentials a connection opened now sends. A provider is asked once for its +/// current pair, which is when redis-py asks its `credential_provider`. +pub(crate) async fn authenticated( + settings: RedisConnectionInfo, + credentials: &Credentials, +) -> RedisResult { + let Credentials::Provider(provider) = credentials else { + return Ok(static_credentials(settings, credentials)); + }; + let current = provider.subscribe().next().await.ok_or_else(|| { + RedisError::from(( + ErrorKind::AuthenticationFailed, + "the Redis credential provider ended before giving credentials", + )) + })??; + Ok(settings + .set_username(current.username()) + .set_password(current.password())) +} + #[cfg(test)] mod tests { + use litellm_redis_types::{RedisUrl, UnixSocket}; use rstest::rstest; - use super::{Endpoint, HostEndpoint}; + use super::*; #[rstest] - #[case::url_with_password(Endpoint::Url("redis://user:secret@cache:6379/0".to_owned()), "redis://@cache:6379/0")] - #[case::url_with_at_in_password(Endpoint::Url("rediss://:p@ss@cache:6380".to_owned()), "rediss://@cache:6380")] - #[case::host_password( - Endpoint::Host(HostEndpoint { - host: "cache".to_owned(), - port: 6379, - username: None, - password: Some("secret".to_owned()), - tls: false, - db: 0, - }), - "" - )] - fn debug_output_never_shows_the_password(#[case] endpoint: Endpoint, #[case] shown: &str) { - let debug = format!("{endpoint:?}"); + #[case::host(Target::Standalone(crate::Standalone { host: "redis".into(), port: 6379, tls: None, db: 0 }))] + #[case::url(Target::Url(RedisUrl { + url: SecretValue::new("redis://cache:6379".to_owned()), + db: None, + username: None, + password: None, + tls: None, + }))] + fn keepalive_probes_an_idle_socket_only_when_enabled(#[case] target: Target) { + let probed = |keepalive| { + connection_info(&target, &Credentials::NONE, keepalive) + .unwrap() + .tcp_settings() + .keepalive() + .is_some() + }; - assert!( - !debug.contains("secret") && !debug.contains("p@ss"), - "{debug}" + assert!(probed(true)); + assert!(!probed(false)); + } + + #[test] + fn a_unix_socket_replaces_host_port_and_tls_like_redis_py() { + let target = Target::Unix(UnixSocket { + path: "/tmp/redis.sock".into(), + db: 3, + }); + + let info = connection_info(&target, &Credentials::NONE, true).unwrap(); + + assert_eq!(*info.addr(), ConnectionAddr::Unix("/tmp/redis.sock".into())); + assert_eq!(info.redis_settings().db(), 3); + } + + #[rstest] + #[case::the_query_db_wins_over_the_path("redis://cache:6379/1", Some(3), None, 3, None)] + #[case::the_path_db_without_a_query_db("redis://cache:6379/1", None, None, 1, None)] + #[case::the_query_password_fills_a_missing_userinfo( + "redis://cache:6379", + None, + Some("query"), + 0, + Some("query") + )] + #[case::the_userinfo_wins_over_the_query_password( + "redis://:userinfo@cache:6379", + None, + Some("query"), + 0, + Some("userinfo") + )] + fn a_url_reads_its_query_like_parse_url( + #[case] url: &str, + #[case] db: Option, + #[case] password: Option<&str>, + #[case] expected_db: i64, + #[case] expected_password: Option<&str>, + ) { + let target = Target::Url(RedisUrl { + url: SecretValue::new(url), + tls: None, + db, + username: None, + password: password.map(SecretValue::new), + }); + let static_password = Credentials::Static { + username: None, + password: Some(SecretValue::new("static")), + }; + + let info = connection_info(&target, &static_password, true).unwrap(); + + assert_eq!(info.redis_settings().db(), expected_db); + assert_eq!( + info.redis_settings().password(), + expected_password.or(Some("static")) ); - assert!(debug.contains(shown), "{debug}"); + } + + #[rstest] + #[case::by_number("redis://cache:6379/?protocol=3")] + #[case::by_name("redis://cache:6379/?protocol=resp3")] + fn a_url_never_switches_the_protocol_from_resp2(#[case] url: &str) { + let target = Target::Url(RedisUrl { + url: SecretValue::new(url), + tls: None, + db: None, + username: None, + password: None, + }); + + let info = connection_info(&target, &Credentials::NONE, true).unwrap(); + + assert_eq!(info.redis_settings().protocol(), ProtocolVersion::RESP2); } } diff --git a/litellm-rust/crates/redis-client/src/connection.rs b/litellm-rust/crates/redis-client/src/connection.rs index aaf26f105da..c1f82490b3b 100644 --- a/litellm-rust/crates/redis-client/src/connection.rs +++ b/litellm-rust/crates/redis-client/src/connection.rs @@ -1,53 +1,161 @@ -use std::time::Duration; - -use redis::{ - Cmd, ConnectionInfo, Pipeline, RedisFuture, RedisResult, Value, - aio::{ConnectionLike, ConnectionManager, ConnectionManagerConfig}, - cluster::{ClusterClientBuilder, ClusterConfig}, - cluster_async::ClusterConnection, +use std::{ + sync::{ + Arc, Mutex, PoisonError, + atomic::{AtomicU64, Ordering}, + }, + time::Duration, }; -use crate::ClusterEndpoint; +use redis::{ + AsyncConnectionConfig, Cmd, ConnectionAddr, ConnectionInfo, ErrorKind, Pipeline, + RedisConnectionInfo, RedisError, RedisFuture, RedisResult, ServerErrorKind, TlsCertificates, + TlsMode, Value, + aio::{ConnectionLike, ConnectionManager, ConnectionManagerConfig, MultiplexedConnection}, + cluster::{ClusterClientBuilder, ClusterConfig}, + cluster_async::ClusterConnection, + cluster_routing::{MultipleNodeRoutingInfo, ResponsePolicy, RoutingInfo}, + sentinel::{SentinelClient, SentinelClientBuilder, SentinelServerType}, +}; -const RETRIES: u32 = 3; +use crate::{ + Cluster, ConnectError, Credentials, Sentinel, Target, Timeouts, + config::{self, addr}, + error::Deployment, +}; + +// redis-py 5.3.1 redis/asyncio/connection.py: without retry options a connection gets Retry(NoBackoff(), 0) +const RECONNECT_RETRIES: usize = 0; +// redis-py 5.3.1 redis/asyncio/cluster.py: cluster_error_retry_attempts=3 +const CLUSTER_RETRIES: u32 = 3; +// redis-py 5.3.1 redis/asyncio/cluster.py sleeps a flat 0.25s before retrying after CLUSTERDOWN. +// redis-rs draws the wait from [min, max), so a 1ms window pins it to 250ms +const CLUSTER_RETRY_WAIT_MS: u64 = 250; #[derive(Clone)] pub(crate) enum Connection { - Standalone(ConnectionManager), - Cluster(ClusterConnection), + Standalone(Renewing), + Cluster(Renewing), + Sentinel(SentinelConnection), } impl Connection { - pub(crate) fn standalone(info: ConnectionInfo, timeout: Duration) -> RedisResult { - let config = ConnectionManagerConfig::new() - .set_connection_timeout(Some(timeout)) - .set_response_timeout(Some(timeout)) - .set_number_of_retries(RETRIES as usize); - let manager = ConnectionManager::new_lazy_with_config(redis::Client::open(info)?, config)?; - Ok(Self::Standalone(manager)) + pub(crate) fn new( + target: &Target, + credentials: &Credentials, + timeouts: Timeouts, + keepalive: bool, + ) -> Result { + match target { + Target::Url(_) | Target::Standalone(_) | Target::Unix(_) => Self::standalone( + config::connection_info(target, credentials, keepalive)?, + credentials, + timeouts, + ), + Target::Cluster(cluster) => Self::cluster(cluster, credentials, timeouts, keepalive), + Target::Sentinel(sentinel) => Ok(Self::Sentinel(SentinelConnection::new( + sentinel, + credentials, + timeouts, + keepalive, + )?)), + } } - pub(crate) fn cluster(endpoint: &ClusterEndpoint, timeout: Duration) -> RedisResult { - let builder = ClusterClientBuilder::new(endpoint.addrs()) - .connection_timeout(timeout) - .response_timeout(timeout) - .retries(RETRIES); - let builder = match &endpoint.username { - Some(username) => builder.username(username), + pub(crate) fn deployment(&self) -> Deployment { + match self { + Self::Standalone(_) => Deployment::Standalone, + Self::Cluster(_) => Deployment::Cluster, + Self::Sentinel(_) => Deployment::Sentinel, + } + } + + fn standalone( + info: ConnectionInfo, + credentials: &Credentials, + timeouts: Timeouts, + ) -> Result { + let config = ConnectionManagerConfig::new() + .set_connection_timeout(Some(timeouts.connect)) + .set_response_timeout(Some(timeouts.response)) + .set_number_of_retries(RECONNECT_RETRIES); + let config = match credentials { + Credentials::Provider(provider) => config.set_credentials_provider(provider.clone()), + Credentials::Static { .. } => config, + }; + let client = redis::Client::open(info)?; + Ok(Self::Standalone(Renewing::new(move || { + ConnectionManager::new_lazy_with_config(client.clone(), config.clone()) + })?)) + } + + fn cluster( + endpoint: &Cluster, + credentials: &Credentials, + timeouts: Timeouts, + keepalive: bool, + ) -> Result { + let builder = ClusterClientBuilder::new(config::cluster_addrs(endpoint)) + .tcp_settings(config::tcp_settings(keepalive)) + .connection_timeout(timeouts.connect) + .response_timeout(timeouts.response) + .retries(CLUSTER_RETRIES) + .retry_wait_formula(0, 2) + .min_retry_wait(CLUSTER_RETRY_WAIT_MS) + .max_retry_wait(CLUSTER_RETRY_WAIT_MS + 1); + let builder = match &endpoint.tls { + Some(tls) => builder + .tls(config::tls_mode(tls)) + .certs(config::tls_certificates(tls)?) + .danger_accept_invalid_hostnames(!tls.check_hostname), None => builder, }; - let builder = match &endpoint.password { - Some(password) => builder.password(password), - None => builder, + let builder = match credentials { + Credentials::Static { username, password } => { + let builder = match username { + Some(username) => builder.username(username.expose()), + None => builder, + }; + match password { + Some(password) => builder.password(password.expose()), + None => builder, + } + } + Credentials::Provider(provider) => builder.set_credentials_provider(provider.clone()), }; - let config = ClusterConfig::new() - .set_connection_timeout(timeout) - .set_response_timeout(timeout); - Ok(Self::Cluster( - builder - .build()? - .get_pending_async_connection_with_config(config), - )) + // The builder's response timeout bounds the whole request, but with_config overwrites each node's + // response timeout with this one even when unset, so it has to be repeated here + let config = ClusterConfig::new().set_response_timeout(timeouts.response); + let client = builder.build()?; + Ok(Self::Cluster(Renewing::new(move || { + Ok(client.get_pending_async_connection_with_config(config.clone())) + })?)) + } +} + +impl Connection { + /// `PING` on the one multiplexed connection, or on every cluster node, so a dead socket is + /// replaced before a real command uses it. + pub(crate) async fn ping_every_node(&self) -> RedisResult<()> { + let ping = redis::cmd("PING"); + match self { + Self::Cluster(renewing) => { + let (generation, mut connection) = renewing.current(); + let reply = connection + .route_command( + ping, + RoutingInfo::MultiNode(( + MultipleNodeRoutingInfo::AllNodes, + Some(ResponsePolicy::AllSucceeded), + )), + ) + .await; + renewing.settle(generation, &reply); + reply.map(drop) + } + Self::Standalone(_) | Self::Sentinel(_) => { + ping.query_async::(&mut self.clone()).await.map(drop) + } + } } } @@ -56,6 +164,7 @@ impl ConnectionLike for Connection { match self { Self::Standalone(connection) => connection.req_packed_command(cmd), Self::Cluster(connection) => connection.req_packed_command(cmd), + Self::Sentinel(connection) => connection.req_packed_command(cmd), } } @@ -68,6 +177,7 @@ impl ConnectionLike for Connection { match self { Self::Standalone(connection) => connection.req_packed_commands(pipeline, offset, count), Self::Cluster(connection) => connection.req_packed_commands(pipeline, offset, count), + Self::Sentinel(connection) => connection.req_packed_commands(pipeline, offset, count), } } @@ -75,6 +185,441 @@ impl ConnectionLike for Connection { match self { Self::Standalone(connection) => connection.get_db(), Self::Cluster(connection) => connection.get_db(), + Self::Sentinel(connection) => connection.get_db(), } } } + +/// A connection swapped for a fresh one once a command on it times out, like redis-py disconnecting +/// a connection whose read timed out. redis-rs keeps the socket after a timeout and the cluster +/// client retries on it, so a socket that went silent would time out every command until TCP gives +/// up on it. +#[derive(Clone)] +pub(crate) struct Renewing(Arc>); + +type Open = Box RedisResult + Send + Sync>; + +struct RenewingState { + open: Open, + current: Mutex<(u64, C)>, +} + +impl Renewing { + fn new(open: impl Fn() -> RedisResult + Send + Sync + 'static) -> RedisResult { + let first = open()?; + Ok(Self(Arc::new(RenewingState { + open: Box::new(open), + current: Mutex::new((0, first)), + }))) + } + + fn current(&self) -> (u64, C) { + self.0 + .current + .lock() + .unwrap_or_else(PoisonError::into_inner) + .clone() + } + + /// Replaces the connection of `generation` when its reply timed out, unless another timeout + /// already replaced it. + fn settle(&self, generation: u64, reply: &RedisResult) { + if !reply.as_ref().is_err_and(RedisError::is_timeout) { + return; + } + let mut current = self + .0 + .current + .lock() + .unwrap_or_else(PoisonError::into_inner); + if current.0 != generation { + return; + } + match (self.0.open)() { + Ok(fresh) => *current = (generation + 1, fresh), + Err(error) => { + tracing::warn!(%error, "replacing a timed-out Redis connection failed") + } + } + } +} + +impl ConnectionLike for Renewing { + fn req_packed_command<'a>(&'a mut self, cmd: &'a Cmd) -> RedisFuture<'a, Value> { + Box::pin(async move { + let (generation, mut connection) = self.current(); + let reply = connection.req_packed_command(cmd).await; + self.settle(generation, &reply); + reply + }) + } + + fn req_packed_commands<'a>( + &'a mut self, + pipeline: &'a Pipeline, + offset: usize, + count: usize, + ) -> RedisFuture<'a, Vec> { + Box::pin(async move { + let (generation, mut connection) = self.current(); + let replies = connection + .req_packed_commands(pipeline, offset, count) + .await; + self.settle(generation, &replies); + replies + }) + } + + fn get_db(&self) -> i64 { + self.current().1.get_db() + } +} + +#[derive(Clone)] +pub(crate) struct SentinelConnection(Arc); + +struct SentinelState { + discovery: tokio::sync::Mutex<()>, + current: Mutex>, + generations: AtomicU64, + endpoint: Sentinel, + credentials: Credentials, + tls: Option<(TlsMode, TlsCertificates)>, + config: AsyncConnectionConfig, + connect_timeout: Duration, + keepalive: bool, +} + +impl SentinelConnection { + fn new( + endpoint: &Sentinel, + credentials: &Credentials, + timeouts: Timeouts, + keepalive: bool, + ) -> Result { + let tls = match &endpoint.tls { + Some(tls) => Some((config::tls_mode(tls), config::tls_certificates(tls)?)), + None => None, + }; + let state = SentinelState { + discovery: tokio::sync::Mutex::new(()), + current: Mutex::new(None), + generations: AtomicU64::new(0), + endpoint: endpoint.clone(), + credentials: credentials.clone(), + tls, + config: async_config(credentials, timeouts), + connect_timeout: timeouts.connect, + keepalive, + }; + state.sentinel_client(&RedisConnectionInfo::default())?; + Ok(Self(Arc::new(state))) + } + + fn cached(&self) -> Option<(u64, MultiplexedConnection)> { + self.0 + .current + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .clone() + } + + async fn connection(&self) -> RedisResult<(u64, MultiplexedConnection)> { + if let Some(current) = self.cached() { + return Ok(current); + } + let _discovering = self.0.discovery.lock().await; + if let Some(current) = self.cached() { + return Ok(current); + } + let connection = self + .discover() + .await? + .get_multiplexed_async_connection_with_config(&self.0.config) + .await?; + let mut current = self + .0 + .current + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let generation = self.0.generations.fetch_add(1, Ordering::Relaxed); + *current = Some((generation, connection.clone())); + Ok((generation, connection)) + } + + pub(crate) async fn master(&self) -> RedisResult { + self.discover().await + } + + async fn discover(&self) -> RedisResult { + let master = tokio::time::timeout(self.0.connect_timeout, self.ask_sentinels()) + .await + .map_err(|_| timed_out("asking the Sentinels for the master timed out"))??; + self.master_client(master.get_connection_info().addr()) + } + + /// redis-rs sends `ROLE` to the master a Sentinel names before trusting it, so that probe gets + /// the data credentials and TLS a connection opened now would use. The Sentinel client is built + /// per discovery since a provider's credentials change after `new`. + async fn ask_sentinels(&self) -> RedisResult { + let data_auth = + config::authenticated(RedisConnectionInfo::default(), &self.0.credentials).await?; + self.0.sentinel_client(&data_auth)?.async_get_client().await + } + + fn master_client(&self, master: &ConnectionAddr) -> Result { + let (host, port) = match master { + ConnectionAddr::Tcp(host, port) | ConnectionAddr::TcpTls { host, port, .. } => { + (host.as_str(), *port) + } + _ => { + return Err(RedisError::from(( + ErrorKind::InvalidClientConfig, + "Sentinel named a master that is not a TCP address", + ))); + } + }; + let endpoint = Target::Standalone(crate::Standalone { + host: host.to_owned(), + port, + tls: self.0.endpoint.tls.clone(), + db: self.0.endpoint.db, + }); + let info = config::connection_info(&endpoint, &self.0.credentials, self.0.keepalive) + .map_err(|error| match error { + ConnectError::Redis(error) => error, + ConnectError::TlsFile { path, .. } => RedisError::from(( + ErrorKind::InvalidClientConfig, + "reading a Redis TLS file", + path, + )), + ConnectError::TlsKey { path, reason } => RedisError::from(( + ErrorKind::InvalidClientConfig, + "reading the Redis TLS key", + format!("{path}: {reason}"), + )), + })?; + redis::Client::open(info) + } + + fn forget(&self, generation: u64) { + let mut current = self + .0 + .current + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if current + .as_ref() + .is_some_and(|(cached, _)| *cached == generation) + { + *current = None; + } + } +} + +impl SentinelState { + fn sentinel_client(&self, data_auth: &RedisConnectionInfo) -> RedisResult { + let endpoint = &self.endpoint; + let sentinels = endpoint + .monitors + .iter() + .map(|node| addr(&node.host, node.port, endpoint.tls.as_ref())); + let builder = SentinelClientBuilder::new( + sentinels, + &endpoint.service_name, + SentinelServerType::Master, + )?; + let (username, password) = match &endpoint.monitor_auth { + Some(auth) => (auth.username.as_ref(), auth.password.as_ref()), + None => (None, None), + }; + let builder = match username { + Some(username) => builder.set_client_to_sentinel_username(username.expose()), + None => builder, + }; + let builder = match password { + Some(password) => builder.set_client_to_sentinel_password(password.expose()), + None => builder, + }; + let builder = match data_auth.username() { + Some(username) => builder.set_client_to_redis_username(username), + None => builder, + }; + let builder = match data_auth.password() { + Some(password) => builder.set_client_to_redis_password(password), + None => builder, + }; + let builder = match &self.tls { + Some((mode, certificates)) => builder + .set_client_to_sentinel_tls_mode(*mode) + .set_client_to_sentinel_certificates(certificates.clone()) + .set_client_to_redis_tls_mode(*mode) + .set_client_to_redis_certificates(certificates.clone()), + None => builder, + }; + builder.build() + } +} + +pub(crate) fn async_config(credentials: &Credentials, timeouts: Timeouts) -> AsyncConnectionConfig { + let config = AsyncConnectionConfig::new() + .set_connection_timeout(Some(timeouts.connect)) + .set_response_timeout(Some(timeouts.response)); + match credentials { + Credentials::Provider(provider) => config.set_credentials_provider(provider.clone()), + Credentials::Static { .. } => config, + } +} + +fn timed_out(message: &'static str) -> RedisError { + RedisError::from(std::io::Error::new(std::io::ErrorKind::TimedOut, message)) +} + +fn needs_rediscovery(error: &RedisError) -> bool { + matches!( + crate::Error::classify(error.clone(), Deployment::Sentinel), + crate::Error::Transport(_) + ) +} + +fn replied_read_only(values: &[Value]) -> bool { + values.iter().any(|value| { + matches!(value, Value::ServerError(error) if error.kind() == Some(ServerErrorKind::ReadOnly)) + }) +} + +impl ConnectionLike for SentinelConnection { + fn req_packed_command<'a>(&'a mut self, cmd: &'a Cmd) -> RedisFuture<'a, Value> { + Box::pin(async move { + let (generation, mut connection) = self.connection().await?; + let reply = connection.req_packed_command(cmd).await; + let rediscover = match &reply { + Ok(value) => replied_read_only(std::slice::from_ref(value)), + Err(error) => needs_rediscovery(error), + }; + if rediscover { + self.forget(generation); + } + reply + }) + } + + fn req_packed_commands<'a>( + &'a mut self, + pipeline: &'a Pipeline, + offset: usize, + count: usize, + ) -> RedisFuture<'a, Vec> { + Box::pin(async move { + let (generation, mut connection) = self.connection().await?; + let replies = connection + .req_packed_commands(pipeline, offset, count) + .await; + let rediscover = match &replies { + Ok(values) => replied_read_only(values), + Err(error) => needs_rediscovery(error), + }; + if rediscover { + self.forget(generation); + } + replies + }) + } + + fn get_db(&self) -> i64 { + self.0.endpoint.db + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Node; + use litellm_redis_testing::{RedisSentinel, SERVICE_NAME}; + + fn endpoint(server: &RedisSentinel) -> Sentinel { + let (host, port) = server.sentinel(); + Sentinel { + monitors: vec![Node { + host: host.to_owned(), + port, + }], + service_name: SERVICE_NAME.to_owned(), + monitor_auth: None, + tls: None, + db: 0, + } + } + + #[rstest::rstest] + #[tokio::test] + async fn a_late_failure_does_not_discard_the_reconnected_sentinel_master() { + let server = RedisSentinel::start().await.unwrap(); + let mut client = SentinelConnection::new( + &endpoint(&server), + &Credentials::NONE, + Timeouts::default(), + true, + ) + .unwrap(); + let (old_generation, _) = client.connection().await.unwrap(); + let original: i64 = redis::cmd("CLIENT") + .arg("ID") + .query_async(&mut client) + .await + .unwrap(); + client.forget(old_generation); + let replacement: i64 = redis::cmd("CLIENT") + .arg("ID") + .query_async(&mut client) + .await + .unwrap(); + assert_ne!(original, replacement); + + client.forget(old_generation); + let after_late_failure: i64 = redis::cmd("CLIENT") + .arg("ID") + .query_async(&mut client) + .await + .unwrap(); + + assert_eq!(replacement, after_late_failure); + } + + #[rstest::rstest] + #[tokio::test] + async fn a_read_only_reply_to_one_command_rediscovers_the_master() { + let server = RedisSentinel::start().await.unwrap(); + let mut client = SentinelConnection::new( + &endpoint(&server), + &Credentials::NONE, + Timeouts::default(), + true, + ) + .unwrap(); + let replica = redis::Client::open(format!("redis://127.0.0.1:{}", server.replica_port())) + .unwrap() + .get_multiplexed_async_connection() + .await + .unwrap(); + let stale = client.0.generations.fetch_add(1, Ordering::Relaxed); + *client.0.current.lock().unwrap() = Some((stale, replica)); + + let on_replica = redis::cmd("SET") + .arg("k") + .arg("1") + .exec_async(&mut client) + .await; + let on_master = redis::cmd("SET") + .arg("k") + .arg("1") + .exec_async(&mut client) + .await; + + assert_eq!( + on_replica.unwrap_err().kind(), + ErrorKind::Server(ServerErrorKind::ReadOnly) + ); + on_master.unwrap(); + } +} diff --git a/litellm-rust/crates/redis-client/src/error.rs b/litellm-rust/crates/redis-client/src/error.rs index ed3b2c74004..54e55615e60 100644 --- a/litellm-rust/crates/redis-client/src/error.rs +++ b/litellm-rust/crates/redis-client/src/error.rs @@ -13,32 +13,93 @@ pub enum Error { } #[derive(Debug, thiserror::Error)] -#[error("connecting to Redis")] -pub struct ConnectError(#[from] RedisError); +pub enum ConnectError { + #[error("connecting to Redis")] + Redis(#[from] RedisError), + #[error("reading the Redis TLS file {path}")] + TlsFile { + path: String, + #[source] + source: std::io::Error, + }, + #[error("reading the Redis TLS key {path}")] + TlsKey { + path: String, + #[source] + reason: KeyError, + }, +} -impl From for Error { - fn from(error: RedisError) -> Self { - match error.kind() { - ErrorKind::Server( - ServerErrorKind::BusyLoading - | ServerErrorKind::ClusterDown - | ServerErrorKind::MasterDown - | ServerErrorKind::TryAgain - | ServerErrorKind::ReadOnly, - ) => Self::Transport(error), - ErrorKind::Server(_) | ErrorKind::Extension => Self::Server(error), - ErrorKind::Parse | ErrorKind::UnexpectedReturnType => Self::Decode(error), +#[derive(Debug, thiserror::Error)] +pub enum KeyError { + #[error("the key is encrypted, so ssl_password is required")] + MissingPassword, + #[error("ssl_password does not decrypt the key, or the key uses an unsupported cipher")] + Decrypt(#[source] pkcs8::Error), + #[error( + "the key uses legacy OpenSSL PEM encryption, which Rust cannot read. Convert it to PKCS#8 with `openssl pkcs8 -topk8`" + )] + LegacyEncryption, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Deployment { + Standalone, + Cluster, + Sentinel, +} + +// redis-py 5.3.1 redis/_parsers/base.py EXCEPTION_CLASSES["ERR"] entries that subclass ConnectionError +const CONNECTION_ERR_MESSAGES: &[&str] = &[ + "max number of clients reached", + "invalid password", + "Client sent AUTH, but no password is set", + "AUTH called without any password configured for the default user. Are you sure your configuration is correct?", +]; + +impl Error { + pub(crate) fn classify(error: RedisError, deployment: Deployment) -> Self { + match (error.kind(), deployment) { + (ErrorKind::Parse | ErrorKind::UnexpectedReturnType, _) => Self::Decode(error), + (ErrorKind::Server(kind), _) if is_unavailable(kind, deployment) => { + Self::Transport(error) + } + (ErrorKind::Server(ServerErrorKind::ResponseError), _) + if error + .detail() + .is_some_and(|detail| CONNECTION_ERR_MESSAGES.contains(&detail)) => + { + Self::Transport(error) + } + (ErrorKind::Extension, _) if matches!(error.code(), Some("NOAUTH" | "WRONGPASS")) => { + Self::Transport(error) + } + (ErrorKind::Server(_) | ErrorKind::Extension | ErrorKind::InvalidClientConfig, _) => { + Self::Server(error) + } _ => Self::Transport(error), } } } +fn is_unavailable(kind: ServerErrorKind, deployment: Deployment) -> bool { + matches!( + (kind, deployment), + (ServerErrorKind::BusyLoading, _) + | ( + ServerErrorKind::ClusterDown | ServerErrorKind::MasterDown, + Deployment::Cluster + ) + | (ServerErrorKind::ReadOnly, Deployment::Sentinel) + ) +} + #[cfg(test)] mod tests { use redis::{RedisError, Value}; use rstest::rstest; - use super::Error; + use super::{Deployment, Error}; fn reply(resp: &str) -> RedisError { match redis::parse_redis_value(resp.as_bytes()).unwrap() { @@ -47,23 +108,101 @@ mod tests { } } + fn is_transport(resp: &str, deployment: Deployment) -> bool { + match Error::classify(reply(resp), deployment) { + Error::Transport(_) => true, + Error::Server(_) => false, + other => panic!("{resp:?} classified as {other:?}"), + } + } + #[rstest] #[case::wrong_type("-WRONGTYPE Operation against a key holding the wrong kind of value\r\n")] #[case::no_script("-NOSCRIPT No matching script\r\n")] #[case::cross_slot("-CROSSSLOT Keys in request don't hash to the same slot\r\n")] #[case::generic("-ERR value is not an integer or out of range\r\n")] #[case::no_perm("-NOPERM this user has no permissions to access one of the keys\r\n")] - fn rejections_are_server_errors(#[case] resp: &str) { - assert!(matches!(Error::from(reply(resp)), Error::Server(_))); + #[case::try_again("-TRYAGAIN Multiple keys request during rehashing of slot\r\n")] + #[case::out_of_memory("-OOM command not allowed when used memory > 'maxmemory'.\r\n")] + #[case::auth_arity("-ERR wrong number of arguments for 'auth' command\r\n")] + #[case::max_clients_with_suffix("-ERR max number of clients reached, try later\r\n")] + fn rejections_are_server_errors_on_every_deployment( + #[case] resp: &str, + #[values(Deployment::Standalone, Deployment::Cluster, Deployment::Sentinel)] + deployment: Deployment, + ) { + assert!(!is_transport(resp, deployment)); } #[rstest] #[case::loading("-LOADING Redis is loading the dataset in memory\r\n")] - #[case::cluster_down("-CLUSTERDOWN The cluster is down\r\n")] - #[case::master_down("-MASTERDOWN Link with MASTER is down\r\n")] - #[case::try_again("-TRYAGAIN Multiple keys request during rehashing of slot\r\n")] - #[case::read_only("-READONLY You can't write against a read only replica.\r\n")] - fn unavailable_nodes_are_transport_errors(#[case] resp: &str) { - assert!(matches!(Error::from(reply(resp)), Error::Transport(_))); + #[case::no_auth("-NOAUTH Authentication required.\r\n")] + #[case::wrong_pass("-WRONGPASS invalid username-password pair or user is disabled.\r\n")] + #[case::max_clients("-ERR max number of clients reached\r\n")] + #[case::invalid_password("-ERR invalid password\r\n")] + #[case::no_password_set("-ERR Client sent AUTH, but no password is set\r\n")] + #[case::no_default_password( + "-ERR AUTH called without any password configured for the default user. Are you sure your configuration is correct?\r\n" + )] + fn unavailable_nodes_are_transport_errors_on_every_deployment( + #[case] resp: &str, + #[values(Deployment::Standalone, Deployment::Cluster, Deployment::Sentinel)] + deployment: Deployment, + ) { + assert!(is_transport(resp, deployment)); + } + + #[rstest] + #[case::master_down_on_standalone( + "-MASTERDOWN Link with MASTER is down\r\n", + Deployment::Standalone, + false + )] + #[case::master_down_on_cluster( + "-MASTERDOWN Link with MASTER is down\r\n", + Deployment::Cluster, + true + )] + #[case::cluster_down_on_standalone( + "-CLUSTERDOWN The cluster is down\r\n", + Deployment::Standalone, + false + )] + #[case::cluster_down_on_cluster( + "-CLUSTERDOWN The cluster is down\r\n", + Deployment::Cluster, + true + )] + #[case::read_only_on_standalone( + "-READONLY You can't write against a read only replica.\r\n", + Deployment::Standalone, + false + )] + #[case::read_only_on_cluster( + "-READONLY You can't write against a read only replica.\r\n", + Deployment::Cluster, + false + )] + #[case::master_down_on_sentinel( + "-MASTERDOWN Link with MASTER is down\r\n", + Deployment::Sentinel, + false + )] + #[case::cluster_down_on_sentinel( + "-CLUSTERDOWN The cluster is down\r\n", + Deployment::Sentinel, + false + )] + #[case::read_only_on_sentinel( + "-READONLY You can't write against a read only replica.\r\n", + Deployment::Sentinel, + true + )] + fn topology_decides_like_redis_py( + #[case] resp: &str, + #[case] deployment: Deployment, + #[case] transport: bool, + ) { + assert_eq!(is_transport(resp, deployment), transport); } } diff --git a/litellm-rust/crates/redis-client/src/events.rs b/litellm-rust/crates/redis-client/src/events.rs new file mode 100644 index 00000000000..9ea9ff4defa --- /dev/null +++ b/litellm-rust/crates/redis-client/src/events.rs @@ -0,0 +1,78 @@ +use std::{any::Any, future::Future, sync::Arc, time::Duration}; + +use litellm_circuit_breaker::{State, Transition}; + +use crate::Error; + +/// What the host attached to the calls made inside `tagged`, opaque to the client. +pub type CallTag = Arc; + +tokio::task_local! { + static TAG: CallTag; +} + +pub async fn tagged(tag: CallTag, operation: F) -> F::Output { + TAG.scope(tag, operation).await +} + +/// The tag of the `tagged` scope this task runs in. `RedisEvents::call` runs on the calling task, +/// so a sink reads its call's tag here. +pub fn current_tag() -> Option { + TAG.try_with(Arc::clone).ok() +} + +pub trait RedisEvents: Send + Sync { + fn call(&self, event: &CallEvent); + + fn breaker(&self, transition: &Transition); +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Operation { + Command(&'static str), + Script, + Pipeline { ops: usize }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Outcome { + Succeeded, + Failed(FailureKind), +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum FailureKind { + Timeout, + Transport, + Server, + Decode, + BreakerOpen, +} + +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct CallEvent { + pub operation: Operation, + pub duration: Duration, + pub outcome: Outcome, +} + +impl FailureKind { + pub fn of(error: &Error) -> Self { + match error { + Error::Transport(error) if error.is_timeout() => Self::Timeout, + Error::Transport(_) => Self::Transport, + Error::Server(_) => Self::Server, + Error::Decode(_) => Self::Decode, + Error::BreakerOpen => Self::BreakerOpen, + } + } +} + +impl Outcome { + pub(crate) fn of(result: &Result) -> Self { + match result { + Ok(_) => Self::Succeeded, + Err(error) => Self::Failed(FailureKind::of(error)), + } + } +} diff --git a/litellm-rust/crates/redis-client/src/lib.rs b/litellm-rust/crates/redis-client/src/lib.rs index 47ff8677e06..2754666c329 100644 --- a/litellm-rust/crates/redis-client/src/lib.rs +++ b/litellm-rust/crates/redis-client/src/lib.rs @@ -1,20 +1,30 @@ +mod batch; mod client; -mod commands; +pub mod commands; mod config; mod connection; mod error; +mod events; mod namespace; mod pipeline; +mod pubsub; mod script; -#[cfg(test)] -mod test_support; -pub use client::{RedisClient, Ttl}; -pub use config::{ - ClusterEndpoint, ClusterNode, DEFAULT_TIMEOUT, Endpoint, HostEndpoint, RedisConfig, +pub use batch::{Declared, RedisBatch}; +pub use client::{RedisClient, Ttl, WeakRedisClient}; +pub use commands::{Amount, Command, Expiry}; +pub use config::{Credentials, CredentialsProvider, DEFAULT_TIMEOUT, RedisConfig, Timeouts}; +pub use error::{ConnectError, Error, KeyError}; +pub use events::{ + CallEvent, CallTag, FailureKind, Operation, Outcome, RedisEvents, current_tag, tagged, +}; +pub use litellm_redis_types::{ + Cluster, Node, RedisUrl, Sentinel, Standalone, StaticAuth, Target, TlsOptions, UnixSocket, }; -pub use error::{ConnectError, Error}; pub use namespace::Namespace; pub use pipeline::Pipeline; -pub use redis::Value; +pub use redis::{ + BasicAuth, ErrorKind as RedisErrorKind, RedisError, StreamingCredentialsProvider, Value, + cluster_routing::Slot, +}; pub use script::Script; diff --git a/litellm-rust/crates/redis-client/src/pipeline.rs b/litellm-rust/crates/redis-client/src/pipeline.rs index e54c65e9605..0b6bb6a6673 100644 --- a/litellm-rust/crates/redis-client/src/pipeline.rs +++ b/litellm-rust/crates/redis-client/src/pipeline.rs @@ -1,30 +1,20 @@ -use std::{collections::HashMap, time::Duration}; +use std::collections::HashMap; use futures_util::future::join_all; -use redis::{ - Cmd, ErrorKind, RedisResult, ServerErrorKind, ToRedisArgs, ToSingleRedisArg, Value, - cluster_routing::Slot, -}; +use redis::{ErrorKind, RedisError, RedisResult, ServerErrorKind, Value, cluster_routing::Slot}; -use crate::{Error, Namespace, Script, commands, connection::Connection}; +use crate::{ + Command, Error, Namespace, + commands::{Op, Plan}, + connection::Connection, + script::ScriptCall, +}; pub struct Pipeline { namespace: Namespace, ops: Vec, } -struct Op { - command: Cmd, - key: String, - script: Option, -} - -struct ScriptCall { - script: Script, - keys: Vec, - args: Vec>, -} - type Settled = (usize, Result); impl Pipeline { @@ -35,60 +25,13 @@ impl Pipeline { } } - pub fn get(self, key: &str) -> Self { - let key = self.namespace.apply(key); - self.command(commands::get(key.clone()), key) + pub(crate) fn of(namespace: Namespace, ops: Vec) -> Self { + Self { namespace, ops } } - pub fn set(self, key: &str, value: impl ToSingleRedisArg, ttl: Duration) -> Self { - let key = self.namespace.apply(key); - self.command(commands::set(key.clone(), value, ttl), key) - } - - pub fn del(self, key: &str) -> Self { - let key = self.namespace.apply(key); - self.command(commands::del(vec![key.clone()]), key) - } - - pub fn expire(self, key: &str, ttl: Duration) -> Self { - let key = self.namespace.apply(key); - self.command(commands::expire(key.clone(), ttl), key) - } - - pub fn incr_by(self, key: &str, delta: i64) -> Self { - let key = self.namespace.apply(key); - self.command(commands::incr_by(key.clone(), delta), key) - } - - pub fn incr_by_float(self, key: &str, delta: f64) -> Self { - let key = self.namespace.apply(key); - self.command(commands::incr_by_float(key.clone(), delta), key) - } - - pub fn eval(self, script: &Script, keys: &[&str], args: impl ToRedisArgs) -> Self { - let keys: Vec = keys.iter().map(|key| self.namespace.apply(key)).collect(); - let args = args.to_redis_args(); - let op = Op { - command: script.evalsha(&keys, &args), - key: keys.first().cloned().unwrap_or_default(), - script: Some(ScriptCall { - script: script.clone(), - keys, - args, - }), - }; - self.push(op) - } - - fn command(self, command: Cmd, key: String) -> Self { - self.push(Op { - command, - key, - script: None, - }) - } - - fn push(mut self, op: Op) -> Self { + /// Appends one command in its pipelined form; `RedisClient::run` answers it in order, undecoded. + pub fn with(mut self, command: Command) -> Self { + let (op, _) = command.prepare(&self.namespace); self.ops.push(op); self } @@ -102,7 +45,7 @@ impl Pipeline { let settled = join_all( groups .iter() - .map(|group| self.run_group(connection.clone(), group)), + .map(|batches| self.run_in_order(connection, batches)), ) .await; let mut settled: Vec = settled.into_iter().flatten().collect(); @@ -110,183 +53,131 @@ impl Pipeline { settled.into_iter().map(|(_, result)| result).collect() } - fn groups(&self, connection: &Connection) -> Vec> { + async fn run_in_order(&self, connection: &Connection, batches: &[Vec]) -> Vec { + let mut settled = Vec::with_capacity(batches.iter().map(Vec::len).sum()); + for batch in batches { + settled.extend(self.run_group(connection.clone(), batch).await); + } + settled + } + + fn groups(&self, connection: &Connection) -> Vec>> { match connection { _ if self.ops.is_empty() => Vec::new(), - Connection::Standalone(_) => vec![(0..self.ops.len()).collect()], + Connection::Standalone(_) | Connection::Sentinel(_) => { + vec![vec![(0..self.ops.len()).collect()]] + } Connection::Cluster(_) => { let mut by_slot: HashMap> = HashMap::new(); - for (index, op) in self.ops.iter().enumerate() { + for index in 0..self.ops.len() { by_slot - .entry(Slot::for_key(&op.key)) + .entry(Slot::for_key(&self.ops[index].key)) .or_default() .push(index); } - by_slot.into_values().collect() + by_slot + .into_values() + .map(|indices| self.retry_safe_batches(indices)) + .collect() } } } + fn retry_safe_batches(&self, indices: Vec) -> Vec> { + indices + .chunk_by(|&left, &right| self.ops[left].idempotent && self.ops[right].idempotent) + .map(<[usize]>::to_vec) + .collect() + } + + /// A script the server had not cached ran nothing, so it alone runs again, the way litellm's + /// `RedisBatch` reruns it through redis-py's `Script`: `EVALSHA`, then `SCRIPT LOAD` and + /// `EVALSHA` on `NOSCRIPT`. + async fn settle_script( + &self, + connection: Connection, + index: usize, + reply: RedisResult, + ) -> (usize, RedisResult) { + match (&self.ops[index].plan, reply) { + (Plan::Script(call), Err(error)) if is_missing_script(&error) => { + (index, rerun_missing_script(connection, call).await) + } + (_, reply) => (index, reply), + } + } + async fn run_group(&self, mut connection: Connection, indices: &[usize]) -> Vec { + let deployment = connection.deployment(); let mut pipeline = redis::pipe(); pipeline.ignore_errors(); - for &index in indices { - pipeline.add_command(self.ops[index].command.clone()); - } - let replies: RedisResult>> = - pipeline.query_async(&mut connection).await; + let widths: Vec = indices + .iter() + .map(|&index| { + let commands = self.ops[index].plan.pipelined(); + let width = commands.len(); + for command in commands { + pipeline.add_command(command); + } + width + }) + .collect(); + let replies: RedisResult>> = match pipeline.is_empty() { + true => Ok(Vec::new()), + false => pipeline.query_async(&mut connection).await, + }; let replies = match replies { Ok(replies) => replies, Err(error) => { return indices .iter() - .map(|&index| (index, Err(Error::from(error.clone())))) + .map(|&index| (index, Err(Error::classify(error.clone(), deployment)))) .collect(); } }; - let mut settled = Vec::with_capacity(indices.len()); - for (&index, reply) in indices.iter().zip(replies) { - settled.push((index, self.settle(&mut connection, index, reply).await)); - } - settled - } - - async fn settle( - &self, - connection: &mut Connection, - index: usize, - reply: RedisResult, - ) -> Result { - match (reply, &self.ops[index].script) { - (Err(error), Some(call)) - if error.kind() == ErrorKind::Server(ServerErrorKind::NoScript) => - { - Ok(call - .script - .invocation(&call.keys, &call.args) - .invoke_async(connection) - .await?) - } - (reply, _) => Ok(reply?), - } + let mut replies = replies.into_iter(); + let per_op: Vec<(usize, RedisResult)> = indices + .iter() + .zip(widths) + .map(|(&index, width)| match &self.ops[index].plan { + Plan::Reply(value) => (index, Ok(value.clone())), + _ => (index, first_reply(replies.by_ref().take(width))), + }) + .collect(); + join_all( + per_op + .into_iter() + .map(|(index, reply)| self.settle_script(connection.clone(), index, reply)), + ) + .await + .into_iter() + .map(|(index, reply)| { + ( + index, + reply.map_err(|error| Error::classify(error, deployment)), + ) + }) + .collect() } } -#[cfg(test)] -mod tests { - use std::time::{Duration, Instant}; - - use redis::Value; - use rstest::rstest; - - use crate::{ - DEFAULT_TIMEOUT, Error, Script, - test_support::{Server, Topology, flush_scripts}, - }; - - #[rstest] - #[tokio::test] - async fn each_op_settles_on_its_own_in_input_order( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - client - .set("text", "not a number", Duration::from_secs(60)) - .await - .unwrap(); - let pipeline = client - .pipeline() - .incr_by("a", 1) - .incr_by("text", 1) - .incr_by("c", 5) - .set("b", "2", Duration::from_secs(60)) - .get("b") - .incr_by("a", 1) - .get("missing"); - - let results = client.run(&pipeline).await; - - assert_eq!(results.len(), 7); - assert_eq!(results[0].as_ref().unwrap(), &Value::Int(1)); - assert!( - matches!(results[1], Err(Error::Server(_))), - "{:?}", - results[1] - ); - assert_eq!(results[2].as_ref().unwrap(), &Value::Int(5)); - assert_eq!(results[3].as_ref().unwrap(), &Value::Okay); - assert_eq!( - results[4].as_ref().unwrap(), - &Value::BulkString(b"2".to_vec()) - ); - assert_eq!(results[5].as_ref().unwrap(), &Value::Int(2)); - assert_eq!(results[6].as_ref().unwrap(), &Value::Nil); - } - - #[rstest] - #[tokio::test] - async fn a_flushed_script_in_a_pipeline_reruns_alone( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("ns", DEFAULT_TIMEOUT).await; - let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); - client.eval::(&incr, &["counter"], 1).await.unwrap(); - flush_scripts(&client).await; - let pipeline = client - .pipeline() - .eval(&incr, &["counter"], 2) - .incr_by("other", 7); - - let results = client.run(&pipeline).await; - - assert_eq!(results[0].as_ref().unwrap(), &Value::Int(3)); - assert_eq!(results[1].as_ref().unwrap(), &Value::Int(7)); - assert_eq!(client.get::("counter").await.unwrap(), Some(3)); - } - - #[rstest] - #[tokio::test] - async fn an_unreachable_server_fails_every_op_as_transport( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", Duration::from_millis(500)).await; - let pipeline = client.pipeline().incr_by("a", 1).incr_by("c", 1); - - server.freeze().await; - let results = client.run(&pipeline).await; - server.thaw().await; - - assert_eq!(results.len(), 2); - assert!( - matches!(results[0], Err(Error::Transport(_))), - "{:?}", - results[0] - ); - assert!( - matches!(results[1], Err(Error::Transport(_))), - "{:?}", - results[1] - ); - } - - #[rstest] - #[tokio::test] - async fn an_empty_pipeline_returns_without_asking_redis( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", Duration::from_secs(2)).await; - - server.freeze().await; - let started = Instant::now(); - let results = client.run(&client.pipeline()).await; - let elapsed = started.elapsed(); - server.thaw().await; - - assert!(results.is_empty()); - assert!(elapsed < Duration::from_secs(1), "{elapsed:?}"); - } +async fn rerun_missing_script(mut connection: Connection, call: &ScriptCall) -> RedisResult { + call.script + .invocation(&call.keys, &call.args) + .invoke_async(&mut connection) + .await +} + +fn is_missing_script(error: &RedisError) -> bool { + error.kind() == ErrorKind::Server(ServerErrorKind::NoScript) +} + +// RedisBatch settles an op from its first reply and fails it on any reply that failed +fn first_reply(replies: impl Iterator>) -> RedisResult { + replies + .reduce(|first, next| match (first, next) { + (Err(error), _) | (Ok(_), Err(error)) => Err(error), + (first, Ok(_)) => first, + }) + .unwrap_or(Ok(Value::Nil)) } diff --git a/litellm-rust/crates/redis-client/src/pubsub.rs b/litellm-rust/crates/redis-client/src/pubsub.rs new file mode 100644 index 00000000000..b6b18d3e481 --- /dev/null +++ b/litellm-rust/crates/redis-client/src/pubsub.rs @@ -0,0 +1,225 @@ +use std::{sync::Arc, time::Duration}; + +use futures_util::{StreamExt, stream::BoxStream}; +use redis::{ConnectionInfo, RedisResult, aio::PubSubStream}; + +use crate::{ + ConnectError, Credentials, Standalone, Target, Timeouts, config, + connection::{Connection, SentinelConnection}, +}; + +// litellm/proxy/common_utils/auth_cache_invalidation_pubsub.py: _BACKOFF_INITIAL_SECONDS, _BACKOFF_MAX_SECONDS +const BACKOFF_INITIAL: Duration = Duration::from_secs(5); +const BACKOFF_MAX: Duration = Duration::from_secs(60); + +/// A subscription the way redis-py's `PubSub` holds one: a dedicated RESP2 connection that sends +/// `SUBSCRIBE`, authenticated once when it connects. +#[derive(Clone)] +pub(crate) struct Subscriber { + source: Source, + credentials: Credentials, + connect_timeout: Duration, +} + +#[derive(Clone)] +enum Source { + Nodes(Arc<[ConnectionInfo]>), + Sentinel(SentinelConnection), +} + +impl Subscriber { + pub(crate) fn new( + endpoint: &Target, + credentials: &Credentials, + timeouts: Timeouts, + keepalive: bool, + connection: &Connection, + ) -> Result { + let source = match (endpoint, connection) { + (_, Connection::Sentinel(sentinel)) => Source::Sentinel(sentinel.clone()), + (Target::Cluster(cluster), _) => Source::Nodes( + cluster + .nodes + .iter() + .map(|node| { + let node = Target::Standalone(Standalone { + host: node.host.clone(), + port: node.port, + tls: cluster.tls.clone(), + db: 0, + }); + config::connection_info(&node, credentials, keepalive) + }) + .collect::>()?, + ), + _ => Source::Nodes(Arc::new([config::connection_info( + endpoint, + credentials, + keepalive, + )?])), + }; + Ok(Self { + source, + credentials: credentials.clone(), + connect_timeout: timeouts.connect, + }) + } + + pub(crate) fn messages(self, channel: String) -> BoxStream<'static, Vec> { + futures_util::stream::unfold( + (self, channel, None::, BACKOFF_INITIAL), + |(subscriber, channel, live, backoff)| async move { + let mut live = live; + let mut backoff = backoff; + loop { + let Some(current) = live.as_mut() else { + match subscriber.subscribe(&channel).await { + Ok(subscribed) => { + live = Some(subscribed); + backoff = BACKOFF_INITIAL; + } + Err(error) => { + tracing::warn!(%error, ?backoff, "Redis subscribe failed, retrying"); + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(BACKOFF_MAX); + } + } + continue; + }; + match current.next().await { + Some(message) => { + let payload = message.get_payload_bytes().to_vec(); + return Some((payload, (subscriber, channel, live, backoff))); + } + None => live = None, + } + tracing::warn!(?backoff, "Redis subscription connection lost, resubscribing"); + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(BACKOFF_MAX); + } + }, + ) + .boxed() + } + + async fn subscribe(&self, channel: &str) -> RedisResult { + match &self.source { + Source::Sentinel(sentinel) => { + let master = sentinel.master().await?; + self.subscribe_on(master.get_connection_info().clone(), channel) + .await + } + Source::Nodes(nodes) => { + let mut last = None; + for node in nodes.iter() { + match self.subscribe_on(node.clone(), channel).await { + Ok(live) => return Ok(live), + Err(error) => last = Some(error), + } + } + Err(last.expect("an endpoint always has at least one node")) + } + } + } + + async fn subscribe_on(&self, info: ConnectionInfo, channel: &str) -> RedisResult { + let settings = + config::authenticated(info.redis_settings().clone(), &self.credentials).await?; + let client = redis::Client::open(info.set_redis_settings(settings))?; + let mut pubsub = tokio::time::timeout(self.connect_timeout, client.get_async_pubsub()) + .await + .map_err(|_| { + redis::RedisError::from(std::io::Error::new( + std::io::ErrorKind::TimedOut, + "connecting the Redis subscription timed out", + )) + })??; + subscribe_connected(&mut pubsub, channel, self.connect_timeout).await?; + Ok(pubsub.into_on_message()) + } +} + +async fn subscribe_connected( + pubsub: &mut redis::aio::PubSub, + channel: &str, + connect_timeout: Duration, +) -> RedisResult<()> { + tokio::time::timeout(connect_timeout, pubsub.subscribe(channel)) + .await + .map_err(|_| { + redis::RedisError::from(std::io::Error::new( + std::io::ErrorKind::TimedOut, + "subscribing to the Redis channel timed out", + )) + })? +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use futures_util::StreamExt; + use litellm_redis_testing::{RedisServer, TcpProxy}; + use rstest::rstest; + + use super::subscribe_connected; + + #[rstest] + #[tokio::test] + async fn a_stalled_pubsub_subscribe_times_out_and_a_fresh_pubsub_recovers() { + let server = RedisServer::start().await.unwrap(); + let proxy = TcpProxy::start(server.port()).await.unwrap(); + let client = redis::Client::open(format!("redis://127.0.0.1:{}", proxy.port())).unwrap(); + let mut pubsub = tokio::time::timeout(Duration::from_secs(2), client.get_async_pubsub()) + .await + .expect("connecting the initial pubsub should finish") + .unwrap(); + tokio::time::timeout( + Duration::from_secs(2), + subscribe_connected(&mut pubsub, "existing", Duration::from_secs(1)), + ) + .await + .expect("the initial subscription acknowledgement should arrive") + .unwrap(); + + proxy.blackhole_open_connections(); + let result = tokio::time::timeout( + Duration::from_secs(1), + subscribe_connected(&mut pubsub, "stalled", Duration::from_millis(100)), + ) + .await + .expect("the stalled subscription should return before the safety deadline"); + assert!( + matches!(&result, Err(error) if error.is_timeout()), + "{result:?}" + ); + drop(pubsub); + + let mut recovered = tokio::time::timeout(Duration::from_secs(2), client.get_async_pubsub()) + .await + .expect("a fresh pubsub connection should finish") + .unwrap(); + tokio::time::timeout( + Duration::from_secs(2), + subscribe_connected(&mut recovered, "recovered", Duration::from_secs(1)), + ) + .await + .expect("the fresh subscription acknowledgement should arrive") + .unwrap(); + + let publisher = redis::Client::open(server.url()).unwrap(); + let mut connection = publisher.get_multiplexed_async_connection().await.unwrap(); + let subscriber_count: i64 = redis::cmd("PUBLISH") + .arg("recovered") + .arg("payload") + .query_async(&mut connection) + .await + .unwrap(); + assert_eq!(subscriber_count, 1); + let message = tokio::time::timeout(Duration::from_secs(2), recovered.on_message().next()) + .await + .expect("the fresh subscription should receive the published payload") + .unwrap(); + assert_eq!(message.get_payload::().unwrap(), "payload"); + } +} diff --git a/litellm-rust/crates/redis-client/src/script.rs b/litellm-rust/crates/redis-client/src/script.rs index c5d73133ecc..6dbdbafe9c9 100644 --- a/litellm-rust/crates/redis-client/src/script.rs +++ b/litellm-rust/crates/redis-client/src/script.rs @@ -1,13 +1,18 @@ +use std::sync::Arc; + use redis::{Cmd, ScriptInvocation}; +/// A Lua script, cheap to clone into every call that runs it. #[derive(Clone)] -pub struct Script(redis::Script); +pub struct Script(Arc); impl Script { pub fn new(source: &str) -> Self { - Self(redis::Script::new(source)) + Self(Arc::new(redis::Script::new(source))) } + /// The `EVALSHA` a pipeline sends, like redis-py. A `NOSCRIPT` reply is settled by + /// [`Script::invocation`] alone, which loads the script and sends it again. pub(crate) fn evalsha(&self, keys: &[String], args: &[Vec]) -> Cmd { redis::cmd("EVALSHA") .arg(self.0.get_hash()) @@ -24,65 +29,10 @@ impl Script { } } -#[cfg(test)] -mod tests { - use std::time::Duration; - - use rstest::rstest; - - use super::Script; - use crate::{ - DEFAULT_TIMEOUT, Error, - test_support::{Server, Topology, flush_scripts}, - }; - - #[rstest] - #[tokio::test] - async fn script_keys_are_namespaced( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("a", DEFAULT_TIMEOUT).await; - let bare = server.client("", DEFAULT_TIMEOUT).await; - let read = Script::new("return redis.call('GET', KEYS[1])"); - bare.set("a:x", "1", Duration::from_secs(60)).await.unwrap(); - - let value: Option = client - .eval(&read, &["x"], Vec::::new()) - .await - .unwrap(); - - assert_eq!(value.as_deref(), Some("1")); - } - - #[rstest] - #[tokio::test] - async fn a_failing_script_is_a_server_error_and_keeps_the_connection( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); - client.rpush("list", "item").await.unwrap(); - - let failed = client.eval::(&incr, &["list"], 1).await; - - assert!(matches!(failed, Err(Error::Server(_))), "{failed:?}"); - assert_eq!(client.eval::(&incr, &["counter"], 3).await.unwrap(), 3); - } - - #[rstest] - #[tokio::test] - async fn a_flushed_script_is_loaded_again( - #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, - ) { - let server = Server::start(topology).await; - let client = server.client("", DEFAULT_TIMEOUT).await; - let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); - client.eval::(&incr, &["counter"], 1).await.unwrap(); - - flush_scripts(&client).await; - - assert_eq!(client.eval::(&incr, &["counter"], 1).await.unwrap(), 2); - } +/// A script call in a pipeline, kept apart from plain commands because it is the one entry a +/// pipeline may send again. +pub(crate) struct ScriptCall { + pub(crate) script: Script, + pub(crate) keys: Vec, + pub(crate) args: Vec>, } diff --git a/litellm-rust/crates/redis-client/src/test_support.rs b/litellm-rust/crates/redis-client/src/test_support.rs deleted file mode 100644 index 64a174b1543..00000000000 --- a/litellm-rust/crates/redis-client/src/test_support.rs +++ /dev/null @@ -1,125 +0,0 @@ -use std::time::Duration; - -use litellm_circuit_breaker::Breaker; -use litellm_redis_testing::{RedisCluster, RedisServer}; -use redis::cluster_routing::{Route, RoutingInfo, SingleNodeRoutingInfo, SlotAddr}; - -use crate::{ - ClusterEndpoint, ClusterNode, Endpoint, Namespace, RedisClient, RedisConfig, - connection::Connection, -}; - -#[derive(Clone, Copy, Debug)] -pub(crate) enum Topology { - Standalone, - Cluster, -} - -pub(crate) enum Server { - Standalone(RedisServer), - Cluster(RedisCluster), -} - -impl Server { - pub(crate) async fn start(topology: Topology) -> Self { - match topology { - Topology::Standalone => Self::Standalone(RedisServer::start().await.unwrap()), - Topology::Cluster => Self::Cluster(RedisCluster::start().await.unwrap()), - } - } - - pub(crate) fn endpoint(&self) -> Endpoint { - match self { - Self::Standalone(server) => Endpoint::Url(server.url().to_owned()), - Self::Cluster(cluster) => Endpoint::Cluster(ClusterEndpoint { - nodes: cluster - .nodes() - .map(|(host, port)| ClusterNode { - host: host.to_owned(), - port, - }) - .collect(), - username: None, - password: None, - tls: false, - }), - } - } - - pub(crate) async fn client(&self, namespace: &str, timeout: Duration) -> RedisClient { - self.guarded_client(namespace, timeout, None).await - } - - pub(crate) async fn guarded_client( - &self, - namespace: &str, - timeout: Duration, - breaker: Option, - ) -> RedisClient { - RedisClient::new(RedisConfig { - endpoint: self.endpoint(), - namespace: Namespace::new(namespace), - timeout, - breaker, - }) - .unwrap() - } - - pub(crate) async fn restart(&self) { - match self { - Self::Standalone(server) => server.restart().await.unwrap(), - Self::Cluster(cluster) => cluster.restart().await.unwrap(), - } - } - - pub(crate) async fn stop(&self) { - match self { - Self::Standalone(server) => server.stop().await.unwrap(), - Self::Cluster(cluster) => cluster.stop().await.unwrap(), - } - } - - pub(crate) async fn resume(&self) { - match self { - Self::Standalone(server) => server.resume().await.unwrap(), - Self::Cluster(cluster) => cluster.resume().await.unwrap(), - } - } - - pub(crate) async fn freeze(&self) { - match self { - Self::Standalone(server) => server.freeze().await.unwrap(), - Self::Cluster(cluster) => cluster.freeze().await.unwrap(), - } - } - - pub(crate) async fn thaw(&self) { - match self { - Self::Standalone(server) => server.thaw().await.unwrap(), - Self::Cluster(cluster) => cluster.thaw().await.unwrap(), - } - } -} - -pub(crate) async fn client_id(client: &RedisClient, key: &str) -> i64 { - let command = redis::cmd("CLIENT").arg("ID").to_owned(); - match client.connection.clone() { - Connection::Standalone(mut connection) => { - command.query_async(&mut connection).await.unwrap() - } - Connection::Cluster(mut connection) => { - let node = Route::with_key(key, SlotAddr::Master); - let routing = RoutingInfo::SingleNode(SingleNodeRoutingInfo::SpecificNode(node)); - redis::from_redis_value(connection.route_command(command, routing).await.unwrap()) - .unwrap() - } - } -} - -pub(crate) async fn flush_scripts(client: &RedisClient) { - redis::cmd("SCRIPT") - .arg("FLUSH") - .query_async::<()>(&mut client.connection.clone()) - .await - .unwrap(); -} diff --git a/litellm-rust/crates/redis-client/tests/batch.rs b/litellm-rust/crates/redis-client/tests/batch.rs new file mode 100644 index 00000000000..45ca419951e --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/batch.rs @@ -0,0 +1,330 @@ +mod support; + +use std::{ + sync::{Arc, Mutex}, + time::Duration, +}; + +use litellm_circuit_breaker::{State, Transition}; +use litellm_redis_client::commands; +use litellm_redis_client::{ + Amount, CallEvent, Credentials, DEFAULT_TIMEOUT, Declared, Error, Expiry, Namespace, Operation, + RedisBatch, RedisClient, RedisConfig, RedisEvents, Script, Timeouts, Ttl, +}; +use litellm_redis_testing::{RedisDeployment, Topology}; +use rstest::rstest; +use support::{connect, endpoint}; + +const MINUTE: Option = Some(Expiry::KeepExisting(Duration::from_secs(60))); + +// RedisBatch pipelines a standalone flush and runs each op on its own on a cluster +fn expected_trips(topology: Topology, alone: &[Operation]) -> Vec { + match topology { + Topology::Standalone => vec![Operation::Pipeline { ops: alone.len() }], + Topology::Cluster => alone.to_vec(), + } +} + +#[derive(Default)] +struct RoundTrips(Mutex>); + +impl RoundTrips { + fn seen(&self) -> Vec { + self.0.lock().unwrap().clone() + } +} + +impl RedisEvents for RoundTrips { + fn call(&self, event: &CallEvent) { + self.0.lock().unwrap().push(event.operation); + } + + fn breaker(&self, _transition: &Transition) {} +} + +fn observed(server: &RedisDeployment) -> (RedisClient, Arc) { + let trips = Arc::new(RoundTrips::default()); + let client = RedisClient::new(RedisConfig { + target: endpoint(server), + credentials: Credentials::NONE, + namespace: Namespace::new("ns"), + timeouts: Timeouts::default(), + breaker: None, + events: Some(trips.clone()), + health_check: None, + keepalive: true, + }) + .unwrap(); + (client, trips) +} + +#[rstest] +#[tokio::test] +async fn declared_ops_wait_for_the_flush_and_share_one_round_trip( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let (client, trips) = observed(&server); + let batch = client.batch(); + let counter = batch.add(commands::incr_by_float("counter", Amount::Int(3), MINUTE)); + let total = batch.add(commands::incr_by_float("total", Amount::Float(1.5), MINUTE)); + let written = batch.add(commands::set("other", "v", Duration::from_secs(60))); + let read = batch.add(commands::get::("value")); + + let outside = connect(&server, "ns", DEFAULT_TIMEOUT); + assert_eq!( + outside.send(commands::get::("counter")).await.unwrap(), + None + ); + assert!(trips.seen().is_empty()); + + batch.flush().await; + + let expected = expected_trips( + topology, + &[ + Operation::Command("INCRBYFLOAT"), + Operation::Command("INCRBYFLOAT"), + Operation::Command("SET"), + Operation::Command("GET"), + ], + ); + let mut seen = trips.seen(); + seen.sort_by_key(|operation| format!("{operation:?}")); + let mut sorted = expected.clone(); + sorted.sort_by_key(|operation| format!("{operation:?}")); + assert_eq!(seen, sorted); + assert_eq!(counter.resolve().await.unwrap(), 3.0); + assert_eq!(total.resolve().await.unwrap(), 1.5); + written.resolve().await.unwrap(); + assert_eq!(read.resolve().await.unwrap(), None); + assert_eq!(trips.seen().len(), expected.len()); +} + +#[rstest] +#[tokio::test] +async fn resolving_before_a_flush_sends_everything_declared_so_far( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let (client, trips) = observed(&server); + let batch = client.batch(); + let first = batch.add(commands::incr_by_float("a", Amount::Int(1), MINUTE)); + let second = batch.add(commands::incr_by_float("b", Amount::Int(2), MINUTE)); + let increment = Operation::Command("INCRBYFLOAT"); + + assert_eq!(second.resolve().await.unwrap(), 2.0); + let late = batch.add(commands::incr_by_float("a", Amount::Int(10), MINUTE)); + assert_eq!(first.resolve().await.unwrap(), 1.0); + assert_eq!( + trips.seen(), + expected_trips(topology, &[increment, increment]) + ); + + assert_eq!(late.resolve().await.unwrap(), 11.0); + let both: Vec = expected_trips(topology, &[increment, increment]) + .into_iter() + .chain(expected_trips(topology, &[increment])) + .collect(); + assert_eq!(trips.seen(), both); + batch.flush().await; + assert_eq!(trips.seen().len(), both.len()); +} + +#[rstest] +#[tokio::test] +async fn a_pipelined_batch_is_one_pipeline_on_every_topology( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let (client, trips) = observed(&server); + let batch = RedisBatch::pipelined(client); + let declared = + ["a", "b", "c"].map(|key| batch.add(commands::incr_by_float(key, Amount::Int(2), MINUTE))); + + let totals = Declared::all(declared).await; + + assert_eq!( + totals.into_iter().map(Result::unwrap).collect::>(), + [2.0; 3] + ); + assert_eq!(trips.seen(), [Operation::Pipeline { ops: 3 }]); +} + +#[rstest] +#[tokio::test] +async fn each_declared_op_gets_its_own_typed_result( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set( + "text", + "not a number", + Duration::from_secs(60), + )) + .await + .unwrap(); + let batch = client.batch(); + let rejected = batch.add(commands::incr_by_float("text", Amount::Int(1), MINUTE)); + let undecodable = batch.add(commands::get::("text")); + let missing = batch.add(commands::get::("missing")); + let script = batch.add(commands::eval::>( + &Script::new("return {KEYS[1], ARGV[1]}"), + &["k"], + "arg", + )); + let removed = batch.add(commands::del(&["text"])); + + batch.flush().await; + + assert!(matches!(rejected.resolve().await, Err(Error::Server(_)))); + assert!(matches!(undecodable.resolve().await, Err(Error::Decode(_)))); + assert_eq!(missing.resolve().await.unwrap(), None); + assert_eq!(script.resolve().await.unwrap(), ["k", "arg"]); + assert_eq!(removed.resolve().await.unwrap(), 1); +} + +#[rstest] +#[tokio::test] +async fn an_op_whose_flush_was_cancelled_resolves_as_transport( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_secs(5)); + let batch = client.batch(); + let counter = batch.add(commands::incr_by_float("counter", Amount::Int(1), MINUTE)); + + server.freeze().await.unwrap(); + let cancelled = tokio::time::timeout(Duration::from_millis(200), batch.flush()).await; + server.thaw().await.unwrap(); + + assert!(cancelled.is_err()); + assert!(matches!(counter.resolve().await, Err(Error::Transport(_)))); +} + +#[rstest] +#[tokio::test] +async fn a_settled_op_waits_for_someone_else_to_flush_and_sends_nothing_itself( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let (client, trips) = observed(&server); + let batch = client.batch(); + let counter = tokio::spawn( + batch + .add(commands::incr_by_float("counter", Amount::Int(4), MINUTE)) + .settled(), + ); + + tokio::time::sleep(Duration::from_millis(200)).await; + assert!(!counter.is_finished()); + assert!(trips.seen().is_empty()); + + batch.flush().await; + + assert_eq!(counter.await.unwrap().unwrap(), 4.0); + assert_eq!( + trips.seen(), + expected_trips(topology, &[Operation::Command("INCRBYFLOAT")]) + ); +} + +#[rstest] +#[tokio::test] +async fn a_settled_op_whose_flush_was_cancelled_resolves_as_transport() { + let server = RedisDeployment::start(Topology::Standalone).await.unwrap(); + let client = connect(&server, "", Duration::from_secs(5)); + let batch = client.batch(); + let counter = batch.add(commands::incr_by_float("counter", Amount::Int(1), MINUTE)); + + server.freeze().await.unwrap(); + let cancelled = tokio::time::timeout(Duration::from_millis(200), batch.flush()).await; + server.thaw().await.unwrap(); + + assert!(cancelled.is_err()); + assert!(matches!(counter.settled().await, Err(Error::Transport(_)))); +} + +#[rstest] +#[tokio::test] +async fn clones_are_the_same_client_and_a_second_connection_is_not() { + let server = RedisDeployment::start(Topology::Standalone).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let twin = connect(&server, "", DEFAULT_TIMEOUT); + + assert!(client.same_client(&client.clone())); + assert!(!client.same_client(&twin)); +} + +#[rstest] +#[tokio::test] +async fn mget_reads_each_key_once_and_answers_every_key_in_order( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("a", "1", Duration::from_secs(60))) + .await + .unwrap(); + client + .send(commands::set("c", "3", Duration::from_secs(60))) + .await + .unwrap(); + server + .cli_on_every_node(&["CONFIG", "RESETSTAT"]) + .await + .unwrap(); + let batch = client.batch(); + + let values = batch.add(commands::mget::(&["c", "missing", "a", "c"])); + batch.flush().await; + + assert_eq!( + values.resolve().await.unwrap(), + [ + Some("3".to_owned()), + None, + Some("1".to_owned()), + Some("3".to_owned()) + ] + ); + let stats = support::command_stats(&server).await; + assert!(support::calls(&stats, "mget") > 0, "{stats}"); + assert_eq!(support::calls(&stats, "get"), 0, "{stats}"); + assert_eq!(support::keys_read(&server).await, 3); +} + +// RedisBatch's _Increment refreshes the TTL in a pipeline, while its cluster path calls +// async_increment, which keeps a TTL the key already has +#[rstest] +#[case::pipelined(Topology::Standalone, false)] +#[case::alone_on_a_cluster(Topology::Cluster, true)] +#[tokio::test] +async fn a_batched_increment_treats_the_ttl_like_redis_batch( + #[case] topology: Topology, + #[case] keeps_the_ttl: bool, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("counter", "1", Duration::from_secs(30))) + .await + .unwrap(); + let batch = client.batch(); + + let counter = batch.add(commands::incr_by_float( + "counter", + Amount::Int(1), + Some(Expiry::KeepExisting(Duration::from_secs(600))), + )); + batch.flush().await; + + assert_eq!(counter.resolve().await.unwrap(), 2.0); + let Ttl::Expires(left) = client.send(commands::ttl("counter")).await.unwrap() else { + panic!("expected an expiring counter") + }; + assert_eq!(left <= Duration::from_secs(30), keeps_the_ttl, "{left:?}"); +} diff --git a/litellm-rust/crates/redis-client/tests/client.rs b/litellm-rust/crates/redis-client/tests/client.rs new file mode 100644 index 00000000000..f95023ca6db --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/client.rs @@ -0,0 +1,1039 @@ +mod support; + +use std::{ + sync::{Arc, Mutex}, + time::{Duration, Instant}, +}; + +use litellm_redis_testing::{RedisCluster, RedisDeployment, RedisServer, Topology, eventually}; +use rstest::rstest; + +use litellm_circuit_breaker::{Breaker, Config, State}; + +use litellm_redis_client::commands; +use litellm_redis_client::{ + Amount, CallEvent, Credentials, DEFAULT_TIMEOUT, Error, Expiry, FailureKind, Namespace, + Operation, Outcome, RedisClient, RedisConfig, RedisEvents, RedisUrl, Script, Standalone, + Target, Timeouts, Ttl, Value, +}; +use litellm_secret_value::SecretValue; +use support::{KEEP, connect, connect_guarded, connections, endpoint}; + +async fn persist(client: &RedisClient, key: &str) { + let set = Script::new("return redis.call('SET', KEYS[1], '0')"); + client + .send(commands::eval::<()>(&set, &[key], Vec::::new())) + .await + .unwrap(); +} + +#[derive(Clone, Copy, Debug)] +enum Counter { + IncrBy, + IncrByFloat, +} + +const MINUTE: Duration = Duration::from_secs(60); + +async fn write(client: &RedisClient, counter: Counter, key: &str, expiry: Expiry) { + match (counter, expiry) { + (Counter::IncrBy, Expiry::KeepExisting(ttl)) => { + client.send(commands::incr_by(key, 1, ttl)).await.unwrap(); + } + (Counter::IncrBy, Expiry::Refresh(_)) => unreachable!("increment_cache never refreshes"), + (Counter::IncrByFloat, expiry) => { + client + .send(commands::incr_by_float( + key, + Amount::Float(0.5), + Some(expiry), + )) + .await + .unwrap(); + } + } +} + +async fn persist_as(client: &RedisClient, key: &str) { + let source = "return redis.call('SET', KEYS[1], '1')"; + client + .send(commands::eval::( + &Script::new(source), + &[key], + Vec::::new(), + )) + .await + .unwrap(); +} + +fn remaining(ttl: Ttl) -> Duration { + match ttl { + Ttl::Expires(left) => left, + other => panic!("expected an expiring key, got {other:?}"), + } +} + +#[rstest] +#[case::increment_cache(Counter::IncrBy, Expiry::KeepExisting(MINUTE))] +#[case::async_increment(Counter::IncrByFloat, Expiry::KeepExisting(MINUTE))] +#[case::async_increment_refreshing(Counter::IncrByFloat, Expiry::Refresh(MINUTE))] +#[tokio::test] +async fn a_counter_write_expires_a_new_key( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, + #[case] counter: Counter, + #[case] expiry: Expiry, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + + write(&client, counter, "fresh", expiry).await; + + let left = remaining(client.send(commands::ttl("fresh")).await.unwrap()); + assert!(left > Duration::from_secs(50) && left <= MINUTE, "{left:?}"); +} + +#[rstest] +#[tokio::test] +async fn keep_existing_leaves_a_running_ttl_and_refresh_resets_it( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let short = Expiry::Refresh(Duration::from_secs(30)); + let long = Duration::from_secs(600); + write(&client, Counter::IncrByFloat, "kept", short).await; + write(&client, Counter::IncrByFloat, "refreshed", short).await; + write( + &client, + Counter::IncrBy, + "kept_int", + Expiry::KeepExisting(Duration::from_secs(30)), + ) + .await; + + write( + &client, + Counter::IncrByFloat, + "kept", + Expiry::KeepExisting(long), + ) + .await; + write( + &client, + Counter::IncrByFloat, + "refreshed", + Expiry::Refresh(long), + ) + .await; + write( + &client, + Counter::IncrBy, + "kept_int", + Expiry::KeepExisting(long), + ) + .await; + + assert!( + remaining(client.send(commands::ttl("kept")).await.unwrap()) <= Duration::from_secs(30) + ); + assert!( + remaining(client.send(commands::ttl("kept_int")).await.unwrap()) <= Duration::from_secs(30) + ); + assert!( + remaining(client.send(commands::ttl("refreshed")).await.unwrap()) + > Duration::from_secs(500) + ); +} + +#[rstest] +#[tokio::test] +async fn keep_existing_expires_a_key_that_had_no_ttl( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, + #[values(Counter::IncrBy, Counter::IncrByFloat)] counter: Counter, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "ns", DEFAULT_TIMEOUT); + persist_as(&client, "persistent").await; + + write(&client, counter, "persistent", KEEP).await; + + let left = remaining(client.send(commands::ttl("persistent")).await.unwrap()); + assert!(left > Duration::from_secs(50), "{left:?}"); +} + +#[rstest] +#[tokio::test] +async fn counters_send_the_commands_redis_cache_sends( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + server + .cli_on_every_node(&["CONFIG", "RESETSTAT"]) + .await + .unwrap(); + + client + .send(commands::incr_by("int", 1, MINUTE)) + .await + .unwrap(); + client + .send(commands::incr_by_float("kept", Amount::Int(1), Some(KEEP))) + .await + .unwrap(); + client + .send(commands::incr_by_float( + "refreshed", + Amount::Float(0.5), + Some(Expiry::Refresh(MINUTE)), + )) + .await + .unwrap(); + client + .run( + &client + .pipeline() + .with(commands::incr_by_float( + "piped", + Amount::Float(0.25), + Some(Expiry::Refresh(MINUTE)), + )) + .with(commands::incr_by_float("bare", Amount::Float(0.25), None)), + ) + .await; + + let stats = support::command_stats(&server).await; + let calls = |command: &str| support::calls(&stats, command); + assert_eq!(calls("incrby"), 1); + assert_eq!(calls("incrbyfloat"), 4); + assert_eq!(calls("ttl"), 2); + assert_eq!(calls("expire"), 4); + assert_eq!( + calls("eval") + calls("evalsha") + calls("pttl") + calls("pexpire"), + 0 + ); +} + +#[derive(Default)] +struct Recorded(Mutex>); + +#[rstest] +#[case::beyond_double_precision(9_007_199_254_740_992_i64, 1_i64)] +#[case::negative_beyond_double_precision(-9_007_199_254_740_992_i64, -1_i64)] +#[case::maximum_integer(i64::MAX - 1, 1_i64)] +#[case::minimum_integer(i64::MIN + 1, -1_i64)] +#[tokio::test] +async fn integer_increments_preserve_every_bit( + #[case] initial: i64, + #[case] delta: i64, + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("integer", initial, Duration::from_secs(60))) + .await + .unwrap(); + let expected = initial + delta; + let result = client + .send(commands::incr_by("integer", delta, MINUTE)) + .await + .unwrap(); + assert_eq!(result, expected); + assert_eq!( + client.send(commands::get::("integer")).await.unwrap(), + Some(expected) + ); +} + +#[rstest] +#[tokio::test] +async fn outage_events_report_the_failure_kind( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, + #[values(false, true)] timeout: bool, + #[values(false, true)] pipeline: bool, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let recorded = Arc::new(Recorded::default()); + let client = RedisClient::new(RedisConfig { + target: endpoint(&server), + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_millis(300), None), + breaker: None, + events: Some(recorded.clone()), + credentials: Credentials::NONE, + health_check: None, + keepalive: true, + }) + .unwrap(); + client.ping().await.unwrap(); + if timeout { + server.freeze().await.unwrap(); + } else { + server.stop().await.unwrap(); + } + let result = if pipeline { + client + .run(&client.pipeline().with(commands::get::("missing"))) + .await + .remove(0) + .map(drop) + } else { + client + .send(commands::get::("missing")) + .await + .map(drop) + }; + if timeout { + server.thaw().await.unwrap(); + } else { + server.resume().await.unwrap(); + } + assert!(matches!(result, Err(Error::Transport(_)))); + let expected = if timeout { + FailureKind::Timeout + } else { + FailureKind::Transport + }; + assert_eq!( + recorded.0.lock().unwrap().last().unwrap().outcome, + Outcome::Failed(expected) + ); +} + +impl RedisEvents for Recorded { + fn call(&self, event: &CallEvent) { + self.0.lock().unwrap().push(*event); + } + + fn breaker(&self, _transition: &litellm_circuit_breaker::Transition) {} +} + +#[rstest] +#[tokio::test] +async fn every_call_reports_its_operation_and_outcome( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let recorded = Arc::new(Recorded::default()); + let client = RedisClient::new(RedisConfig { + target: endpoint(&server), + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::default(), + breaker: None, + events: Some(recorded.clone()), + health_check: None, + keepalive: true, + }) + .unwrap(); + client + .send(commands::set("list", "item", Duration::from_secs(60))) + .await + .unwrap(); + + let _ = client + .send(commands::incr_by("list", 1, Duration::from_secs(60))) + .await; + let _ = client.send(commands::get::("missing")).await; + let _ = client + .run( + &client + .pipeline() + .with(commands::get::("a")) + .with(commands::get::("b")), + ) + .await; + let failures = client + .run(&client.pipeline().with(commands::incr_by_float( + "list", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + ))) + .await; + assert!(matches!(failures[0], Err(Error::Server(_)))); + + let seen: Vec<(Operation, Outcome)> = recorded + .0 + .lock() + .unwrap() + .iter() + .map(|event| (event.operation, event.outcome)) + .collect(); + assert_eq!( + seen, + vec![ + (Operation::Command("SET"), Outcome::Succeeded), + ( + Operation::Command("INCRBY"), + Outcome::Failed(FailureKind::Server) + ), + (Operation::Command("GET"), Outcome::Succeeded), + (Operation::Pipeline { ops: 2 }, Outcome::Succeeded), + ( + Operation::Pipeline { ops: 1 }, + Outcome::Failed(FailureKind::Server) + ), + ] + ); +} + +#[rstest] +#[tokio::test] +async fn namespaces_isolate_keys_on_the_server( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let a = connect(&server, "a", DEFAULT_TIMEOUT); + let b = connect(&server, "b", DEFAULT_TIMEOUT); + let bare = connect(&server, "", DEFAULT_TIMEOUT); + + a.send(commands::set("x", "1", Duration::from_secs(60))) + .await + .unwrap(); + + assert_eq!( + a.send(commands::get::("x")) + .await + .unwrap() + .as_deref(), + Some("1") + ); + assert_eq!( + a.send(commands::get::("a:x")) + .await + .unwrap() + .as_deref(), + Some("1") + ); + assert_eq!( + bare.send(commands::get::("a:x")) + .await + .unwrap() + .as_deref(), + Some("1") + ); + assert_eq!(b.send(commands::get::("x")).await.unwrap(), None); +} + +#[rstest] +#[tokio::test] +async fn a_rejected_command_keeps_the_connection( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("list", "item", Duration::from_secs(60))) + .await + .unwrap(); + let before = connections(&server).await; + + let rejected = client + .send(commands::incr_by("list", 1, Duration::from_secs(60))) + .await; + + assert!(matches!(rejected, Err(Error::Server(_))), "{rejected:?}"); + assert_eq!( + client + .send(commands::get::("list")) + .await + .unwrap() + .as_deref(), + Some("item") + ); + assert_eq!(connections(&server).await, before); +} + +#[rstest] +#[tokio::test] +async fn a_reply_of_another_type_is_a_decode_error_and_a_missing_key_is_none( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("float", "15.0", Duration::from_secs(60))) + .await + .unwrap(); + + let decoded = client.send(commands::get::("float")).await; + + assert!(matches!(decoded, Err(Error::Decode(_))), "{decoded:?}"); + assert_eq!( + client.send(commands::get::("missing")).await.unwrap(), + None + ); +} + +#[rstest] +#[tokio::test] +async fn a_timed_out_command_does_not_leak_its_reply_into_the_next_one( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_millis(500)); + client + .send(commands::set("first", "1", Duration::from_secs(60))) + .await + .unwrap(); + client + .send(commands::set("second", "2", Duration::from_secs(60))) + .await + .unwrap(); + + server.freeze().await.unwrap(); + let timed_out = client.send(commands::get::("first")).await; + server.thaw().await.unwrap(); + + assert!( + matches!(timed_out, Err(Error::Transport(_))), + "{timed_out:?}" + ); + assert_eq!( + client + .send(commands::get::("second")) + .await + .unwrap() + .as_deref(), + Some("2") + ); +} + +#[rstest] +#[tokio::test] +async fn the_same_client_recovers_after_redis_restarts( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_millis(500)); + + server.restart().await.unwrap(); + + let recovered = eventually(async || { + client + .send(commands::incr_by( + "after_restart", + 1, + Duration::from_secs(60), + )) + .await + .ok() + }) + .await; + assert_eq!(recovered, 1); +} + +#[rstest] +#[tokio::test] +async fn a_client_created_while_redis_is_down_works_once_redis_is_up( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + server.stop().await.unwrap(); + + let client = connect(&server, "", Duration::from_millis(500)); + let started = Instant::now(); + let while_down = client.ping().await; + let failed_in = started.elapsed(); + server.resume().await.unwrap(); + + assert!(failed_in < Duration::from_secs(2), "{failed_in:?}"); + + assert!( + matches!(while_down, Err(Error::Transport(_))), + "{while_down:?}" + ); + let recovered = eventually(async || client.ping().await.ok()).await; + assert_eq!(recovered, ()); +} + +#[rstest] +#[tokio::test] +async fn empty_batches_succeed_without_asking_redis( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + + assert_eq!( + client.send(commands::mget::(&[])).await.unwrap(), + Vec::>::new() + ); + assert_eq!(client.send(commands::del(&[])).await.unwrap(), 0); +} + +#[rstest] +#[tokio::test] +async fn mget_and_del_span_slots_in_input_order( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set("a", "1", Duration::from_secs(60))) + .await + .unwrap(); + client + .send(commands::set("c", "3", Duration::from_secs(60))) + .await + .unwrap(); + + let values = client + .send(commands::mget::(&["c", "b", "a"])) + .await + .unwrap(); + let deleted = client.send(commands::del(&["a", "b", "c"])).await.unwrap(); + + assert_eq!( + values, + vec![Some("3".to_owned()), None, Some("1".to_owned())] + ); + assert_eq!(deleted, 2); + assert_eq!( + client + .send(commands::mget::(&["a", "c"])) + .await + .unwrap(), + vec![None, None] + ); +} + +#[rstest] +#[tokio::test] +async fn set_nx_only_writes_a_missing_key( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + + let first = client + .send(commands::set_nx("lock", "owner-1", Duration::from_secs(60))) + .await + .unwrap(); + let second = client + .send(commands::set_nx("lock", "owner-2", Duration::from_secs(60))) + .await + .unwrap(); + + assert!(first); + assert!(!second); + assert_eq!( + client + .send(commands::get::("lock")) + .await + .unwrap() + .as_deref(), + Some("owner-1") + ); +} + +#[rstest] +#[tokio::test] +async fn ttl_reports_missing_persistent_and_remaining_time( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + persist(&client, "persistent").await; + client + .send(commands::set("expiring", "1", Duration::from_secs(60))) + .await + .unwrap(); + client + .send(commands::incr_by_float( + "expired_later", + Amount::Float(0.5), + Some(KEEP), + )) + .await + .unwrap(); + assert!( + client + .send(commands::expire("expired_later", Duration::from_secs(30))) + .await + .unwrap() + ); + + assert_eq!( + client.send(commands::ttl("missing")).await.unwrap(), + Ttl::Missing + ); + assert_eq!( + client.send(commands::ttl("persistent")).await.unwrap(), + Ttl::Persistent + ); + assert!(matches!( + client.send(commands::ttl("expiring")).await.unwrap(), + Ttl::Expires(left) if left > Duration::from_secs(50) && left <= Duration::from_secs(60) + )); + assert!(matches!( + client.send(commands::ttl("expired_later")).await.unwrap(), + Ttl::Expires(left) if left > Duration::from_secs(20) && left <= Duration::from_secs(30) + )); + assert!( + !client + .send(commands::expire("missing", Duration::from_secs(30))) + .await + .unwrap() + ); +} + +#[rstest] +#[tokio::test] +async fn incr_by_float_accumulates_fractions( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + + client + .send(commands::incr_by_float( + "spend", + Amount::Float(0.25), + Some(KEEP), + )) + .await + .unwrap(); + + assert_eq!( + client + .send(commands::incr_by_float( + "spend", + Amount::Float(0.5), + Some(KEEP) + )) + .await + .unwrap(), + 0.75 + ); +} + +#[rstest] +#[tokio::test] +async fn concurrent_commands_on_one_cloned_client_all_apply( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + + let tasks: Vec<_> = (0..100) + .map(|_| { + let client = client.clone(); + tokio::spawn(async move { + for _ in 0..10 { + client + .send(commands::incr_by("shared", 1, Duration::from_secs(60))) + .await + .unwrap(); + } + }) + }) + .collect(); + for task in tasks { + task.await.unwrap(); + } + + assert_eq!( + client.send(commands::get::("shared")).await.unwrap(), + Some(1000) + ); +} + +fn breaker(failure_threshold: u32) -> Breaker { + Breaker::new( + Config { + failure_threshold, + recovery_timeout: Duration::from_millis(200), + timeout_min_duration: Duration::ZERO, + }, + Box::new(|_| {}), + ) +} + +#[rstest] +#[tokio::test] +async fn an_open_breaker_rejects_without_waiting_on_redis_and_a_probe_closes_it( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let breaker = breaker(2); + let client = connect_guarded( + &server, + "", + Duration::from_millis(300), + Some(breaker.clone()), + ); + + server.freeze().await.unwrap(); + let first = client.send(commands::get::("k")).await; + let second = client.send(commands::get::("k")).await; + let started = Instant::now(); + let third = client.send(commands::get::("k")).await; + let rejected_in = started.elapsed(); + server.thaw().await.unwrap(); + + assert!(matches!(first, Err(Error::Transport(_))), "{first:?}"); + assert!(matches!(second, Err(Error::Transport(_))), "{second:?}"); + assert!(matches!(third, Err(Error::BreakerOpen)), "{third:?}"); + assert!(rejected_in < Duration::from_millis(50), "{rejected_in:?}"); + + let recovered = eventually(async || client.send(commands::get::("k")).await.ok()).await; + assert_eq!(recovered, None); + assert!( + matches!(breaker.state(), State::Closed { .. }), + "{:?}", + breaker.state() + ); +} + +#[rstest] +#[tokio::test] +async fn rejected_commands_never_open_the_breaker( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let breaker = breaker(2); + let client = connect_guarded(&server, "", DEFAULT_TIMEOUT, Some(breaker.clone())); + client + .send(commands::set("list", "item", Duration::from_secs(60))) + .await + .unwrap(); + + let rejected = [ + client + .send(commands::incr_by("list", 1, Duration::from_secs(60))) + .await, + client + .send(commands::incr_by("list", 1, Duration::from_secs(60))) + .await, + client + .send(commands::incr_by("list", 1, Duration::from_secs(60))) + .await, + ]; + + assert!( + matches!(rejected[2], Err(Error::Server(_))), + "{:?}", + rejected[2] + ); + assert!( + matches!(breaker.state(), State::Closed { hard: 0, .. }), + "{:?}", + breaker.state() + ); +} + +#[rstest] +#[tokio::test] +async fn an_open_breaker_fails_every_pipeline_op_without_sending( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect_guarded(&server, "", Duration::from_millis(300), Some(breaker(1))); + let pipeline = client + .pipeline() + .with(commands::incr_by_float( + "a", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::incr_by_float( + "c", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )); + + server.freeze().await.unwrap(); + let failed = client.run(&pipeline).await; + let rejected = client.run(&pipeline).await; + server.thaw().await.unwrap(); + + assert!( + matches!(failed[0], Err(Error::Transport(_))), + "{:?}", + failed[0] + ); + assert!( + matches!(rejected[0], Err(Error::BreakerOpen)), + "{:?}", + rejected[0] + ); + assert!( + matches!(rejected[1], Err(Error::BreakerOpen)), + "{:?}", + rejected[1] + ); +} + +#[rstest] +#[tokio::test] +async fn a_host_endpoint_connects_to_its_database() { + let server = RedisServer::start().await.unwrap(); + let port = server.url().rsplit(':').next().unwrap().parse().unwrap(); + let connect = |target| { + RedisClient::new(RedisConfig { + target, + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + }; + let url = |url: String| { + Target::Url(RedisUrl { + url: SecretValue::new(url), + tls: None, + db: None, + username: None, + password: None, + }) + }; + let in_db_3 = connect(Target::Standalone(Standalone { + host: "127.0.0.1".to_owned(), + port, + tls: None, + db: 3, + })) + .unwrap(); + let in_db_0 = connect(url(server.url().to_owned())).unwrap(); + let via_url = connect(url(format!("{}/3", server.url()))).unwrap(); + + in_db_3 + .send(commands::set("k", "3", Duration::from_secs(60))) + .await + .unwrap(); + + assert_eq!( + in_db_0.send(commands::get::("k")).await.unwrap(), + None + ); + assert_eq!( + via_url + .send(commands::get::("k")) + .await + .unwrap() + .as_deref(), + Some("3") + ); +} + +async fn ping_calls(cluster: &RedisCluster) -> Vec { + cluster + .cli_outputs(&["INFO", "commandstats"]) + .await + .unwrap() + .iter() + .map(|info| { + info.lines() + .find_map(|line| line.strip_prefix("cmdstat_ping:calls=")) + .and_then(|rest| rest.split(',').next()) + .and_then(|calls| calls.trim().parse().ok()) + .unwrap_or(0) + }) + .collect() +} + +fn health_checked( + server: &RedisDeployment, + interval: Duration, + timeout: Duration, + breaker: Option, + events: Option>, +) -> RedisClient { + RedisClient::new(RedisConfig { + target: endpoint(server), + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::new(timeout, None), + breaker, + events: events.map(|events| events as Arc), + health_check: Some(interval), + keepalive: true, + }) + .unwrap() +} + +// redis-py 5.3.1 redis/asyncio/connection.py check_health: a connection idle longer than +// health_check_interval gets a PING before the next command, a busy one does not +#[rstest] +#[tokio::test] +async fn an_idle_client_pings_every_node_before_the_next_command_like_redis_py() { + let server = RedisDeployment::start(Topology::Cluster).await.unwrap(); + let RedisDeployment::Cluster(cluster) = &server else { + unreachable!("started a cluster") + }; + let client = health_checked(&server, Duration::from_secs(3), DEFAULT_TIMEOUT, None, None); + client + .send(commands::set("k", "1", Duration::from_secs(60))) + .await + .unwrap(); + let before = ping_calls(cluster).await; + + client + .send(commands::set("k", "2", Duration::from_secs(60))) + .await + .unwrap(); + let busy = ping_calls(cluster).await; + tokio::time::sleep(Duration::from_millis(3500)).await; + client + .send(commands::set("k", "3", Duration::from_secs(60))) + .await + .unwrap(); + let idle = ping_calls(cluster).await; + + assert_eq!(busy, before); + assert_eq!( + idle, + before.iter().map(|calls| calls + 1).collect::>(), + "before {before:?}" + ); +} + +#[rstest] +#[tokio::test] +async fn a_failed_health_check_is_neither_a_call_nor_a_breaker_failure() { + let server = RedisDeployment::start(Topology::Cluster).await.unwrap(); + let recorded = Arc::new(Recorded::default()); + let breaker = breaker(2); + let client = health_checked( + &server, + Duration::from_millis(200), + Duration::from_millis(300), + Some(breaker.clone()), + Some(recorded.clone()), + ); + client + .send(commands::set("k", "1", Duration::from_secs(60))) + .await + .unwrap(); + + server.freeze().await.unwrap(); + tokio::time::sleep(Duration::from_millis(300)).await; + let failed = client.send(commands::get::("k")).await; + server.thaw().await.unwrap(); + + assert!(matches!(failed, Err(Error::Transport(_))), "{failed:?}"); + assert!( + matches!(breaker.state(), State::Closed { timeouts: 1, .. }), + "{:?}", + breaker.state() + ); + assert_eq!(recorded.0.lock().unwrap().len(), 2); +} + +#[rstest] +#[case::int(Amount::Int(1), "1")] +#[case::float_keeps_its_point(Amount::Float(1.0), "1.0")] +#[case::small_float(Amount::Float(1e-5), "1e-05")] +#[case::large_float(Amount::Float(1e16), "1e+16")] +#[case::fraction(Amount::Float(0.1 + 0.2), "0.30000000000000004")] +fn amounts_are_written_like_redis_py_encodes_them(#[case] amount: Amount, #[case] expected: &str) { + assert_eq!( + redis::ToRedisArgs::to_redis_args(&amount), + vec![expected.as_bytes().to_vec()] + ); +} diff --git a/litellm-rust/crates/redis-client/tests/config.rs b/litellm-rust/crates/redis-client/tests/config.rs new file mode 100644 index 00000000000..101c6529e94 --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/config.rs @@ -0,0 +1,290 @@ +use rstest::rstest; + +use litellm_redis_client::{Credentials, RedisUrl, Sentinel, StaticAuth, Target, TlsOptions}; +use litellm_secret_value::SecretValue; + +#[rstest] +#[case::url(format!("{:?}", Target::Url(RedisUrl { url: SecretValue::new("redis://user:credential-value@cache:6379/0"), tls: None, db: None, username: None, password: None })))] +#[case::static_password(format!("{:?}", Credentials::Static { username: None, password: Some(SecretValue::new("credential-value")) }))] +#[case::sentinel_password(format!("{:?}", Target::Sentinel(Sentinel { + monitors: Vec::new(), + service_name: "primary".to_owned(), + monitor_auth: Some(StaticAuth { username: None, password: Some(SecretValue::new("credential-value")) }), + tls: None, + db: 0, +})))] +fn debug_output_never_shows_the_secret(#[case] debug: String) { + assert!(!debug.contains("credential-value"), "{debug}"); + assert!(debug.contains("[REDACTED]"), "{debug}"); +} + +#[derive(Clone, Copy, Debug)] +enum Ca { + Trusted, + TrustedData, + TrustedDirectory, + SystemRoots, +} + +fn directory_holding(ca: &std::path::Path) -> std::path::PathBuf { + let directory = std::env::temp_dir().join(format!("litellm-ca-dir-{}", std::process::id())); + std::fs::create_dir_all(&directory).unwrap(); + std::fs::copy(ca, directory.join("ca.pem")).unwrap(); + directory +} + +#[derive(Clone, Copy, Debug)] +enum ClientKey { + Absent, + Plain, + Encrypted { password: Option<&'static str> }, +} + +async fn tls_server(require_client_certificate: bool) -> litellm_redis_testing::RedisTlsServer { + if rustls::crypto::CryptoProvider::get_default().is_none() { + let _ = rustls::crypto::ring::default_provider().install_default(); + } + litellm_redis_testing::RedisTlsServer::start(require_client_certificate) + .await + .unwrap() +} + +fn tls_client( + server: &litellm_redis_testing::RedisTlsServer, + host: &str, + tls: TlsOptions, +) -> Result { + litellm_redis_client::RedisClient::new(litellm_redis_client::RedisConfig { + target: Target::Standalone(litellm_redis_client::Standalone { + host: host.to_owned(), + port: server.port(), + tls: Some(tls), + db: 0, + }), + credentials: Credentials::NONE, + namespace: litellm_redis_client::Namespace::default(), + timeouts: litellm_redis_client::Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) +} + +fn tls_options( + files: &litellm_redis_testing::TlsFiles, + ca: Ca, + verify_certificate: bool, + check_hostname: bool, + client_key: ClientKey, +) -> TlsOptions { + let (keyfile, key_password) = match client_key { + ClientKey::Absent => (None, None), + ClientKey::Plain => (Some(files.client_key.clone()), None), + ClientKey::Encrypted { password } => ( + Some(files.encrypted_client_key.clone()), + password.map(SecretValue::new), + ), + }; + TlsOptions { + ca_certs: matches!(ca, Ca::Trusted).then(|| files.ca.clone()), + ca_data: matches!(ca, Ca::TrustedData).then(|| std::fs::read_to_string(&files.ca).unwrap()), + ca_path: matches!(ca, Ca::TrustedDirectory).then(|| directory_holding(&files.ca)), + certfile: keyfile.is_some().then(|| files.client_cert.clone()), + keyfile, + key_password, + verify_certificate, + check_hostname, + } +} + +async fn ping_over_tls( + require_client_certificate: bool, + host: &str, + ca: Ca, + verify_certificate: bool, + check_hostname: bool, + client_key: ClientKey, +) -> Result<(), litellm_redis_client::Error> { + let server = tls_server(require_client_certificate).await; + let tls = tls_options( + server.files(), + ca, + verify_certificate, + check_hostname, + client_key, + ); + tls_client(&server, host, tls).unwrap().ping().await +} + +#[rstest] +#[case::trusted_ca_without_hostname_check_like_redis_py( + "127.0.0.1", + Ca::Trusted, + true, + false, + true +)] +#[case::trusted_ca_as_pem_text_like_ssl_ca_data("127.0.0.1", Ca::TrustedData, true, false, true)] +#[case::trusted_ca_in_a_directory_like_ssl_ca_path( + "127.0.0.1", + Ca::TrustedDirectory, + true, + false, + true +)] +#[case::hostname_check_with_a_matching_name("localhost", Ca::Trusted, true, true, true)] +#[case::hostname_check_against_an_unlisted_address("127.0.0.1", Ca::Trusted, true, true, false)] +#[case::an_untrusted_ca_is_refused("127.0.0.1", Ca::SystemRoots, true, false, false)] +#[case::cert_reqs_none_skips_verification("127.0.0.1", Ca::SystemRoots, false, false, true)] +#[tokio::test] +async fn tls_options_decide_which_servers_are_trusted( + #[case] host: &str, + #[case] ca: Ca, + #[case] verify_certificate: bool, + #[case] check_hostname: bool, + #[case] connects: bool, +) { + let result = ping_over_tls( + false, + host, + ca, + verify_certificate, + check_hostname, + ClientKey::Absent, + ) + .await; + + assert_eq!(result.is_ok(), connects, "{result:?}"); + if let Err(error) = result { + assert!( + matches!(error, litellm_redis_client::Error::Transport(_)), + "{error:?}" + ); + } +} + +#[rstest] +#[case::with_the_client_certificate(ClientKey::Plain, true)] +#[case::with_an_encrypted_key_and_its_ssl_password( + ClientKey::Encrypted { password: Some(litellm_redis_testing::CLIENT_KEY_PASSWORD) }, + true +)] +#[case::without_a_client_certificate(ClientKey::Absent, false)] +#[tokio::test] +async fn mutual_tls_sends_the_configured_client_certificate( + #[case] client_key: ClientKey, + #[case] connects: bool, +) { + let result = ping_over_tls(true, "127.0.0.1", Ca::Trusted, true, false, client_key).await; + + assert_eq!(result.is_ok(), connects, "{result:?}"); +} + +#[rstest] +#[case::without_ssl_password(None, "ssl_password is required")] +#[case::with_the_wrong_ssl_password(Some("wrong-password"), "does not decrypt the key")] +#[tokio::test] +async fn an_encrypted_key_that_cannot_be_decrypted_fails_before_connecting( + #[case] password: Option<&'static str>, + #[case] reason: &str, +) { + let server = tls_server(true).await; + let tls = tls_options( + server.files(), + Ca::Trusted, + true, + false, + ClientKey::Encrypted { password }, + ); + + let error = tls_client(&server, "127.0.0.1", tls).err().unwrap(); + + let litellm_redis_client::ConnectError::TlsKey { + path, + reason: cause, + } = &error + else { + panic!("expected a TLS key error, got {error:?}"); + }; + assert_eq!( + *path, + server.files().encrypted_client_key.display().to_string() + ); + assert!(cause.to_string().contains(reason), "{cause}"); +} + +#[rstest] +fn a_missing_tls_file_names_the_path() { + let tls = TlsOptions { + ca_certs: Some("/nonexistent/ca.pem".into()), + ..TlsOptions::verified() + }; + + let error = litellm_redis_client::RedisClient::new(litellm_redis_client::RedisConfig { + target: Target::Standalone(litellm_redis_client::Standalone { + host: "127.0.0.1".to_owned(), + port: 6379, + tls: Some(tls), + db: 0, + }), + credentials: Credentials::NONE, + namespace: litellm_redis_client::Namespace::default(), + timeouts: litellm_redis_client::Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .err() + .unwrap(); + + assert!(error.to_string().contains("/nonexistent/ca.pem"), "{error}"); +} + +#[rstest] +fn a_legacy_encrypted_pem_key_is_refused_with_a_conversion_hint() { + let directory = std::env::temp_dir().join(format!("litellm-legacy-key-{}", std::process::id())); + std::fs::create_dir_all(&directory).unwrap(); + let keyfile = directory.join("client.key"); + std::fs::write( + &keyfile, + "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-256-CBC,00112233445566778899AABBCCDDEEFF\n\nAAAA\n-----END RSA PRIVATE KEY-----\n", + ) + .unwrap(); + let tls = TlsOptions { + certfile: Some(keyfile.clone()), + keyfile: Some(keyfile), + key_password: Some(SecretValue::new("legacy-password")), + ..TlsOptions::verified() + }; + + let error = litellm_redis_client::RedisClient::new(litellm_redis_client::RedisConfig { + target: Target::Standalone(litellm_redis_client::Standalone { + host: "127.0.0.1".to_owned(), + port: 6379, + tls: Some(tls), + db: 0, + }), + credentials: Credentials::NONE, + namespace: litellm_redis_client::Namespace::default(), + timeouts: litellm_redis_client::Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .err() + .unwrap(); + + assert!( + matches!( + &error, + litellm_redis_client::ConnectError::TlsKey { + reason: litellm_redis_client::KeyError::LegacyEncryption, + .. + } + ), + "{error:?}" + ); +} diff --git a/litellm-rust/crates/redis-client/tests/connection.rs b/litellm-rust/crates/redis-client/tests/connection.rs new file mode 100644 index 00000000000..4efa6038cf0 --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/connection.rs @@ -0,0 +1,329 @@ +mod support; + +use std::{collections::BTreeSet, sync::Arc, time::Duration}; + +use litellm_redis_testing::{ + ClientConnection, RedisDeployment, RedisSentinel, RedisServer, SERVICE_NAME, Topology, + eventually, +}; +use rstest::rstest; + +use litellm_redis_client::commands; +use litellm_redis_client::{ + Credentials, CredentialsProvider, Error, Namespace, Node, RedisClient, RedisConfig, Sentinel, + Standalone, Target, Timeouts, +}; +use litellm_secret_value::SecretValue; +use support::{Rotating, connect_as, connections, proxied}; + +#[rstest] +#[tokio::test] +async fn a_pushed_credential_reauthenticates_the_live_connection_and_a_reconnect_uses_the_latest( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + server + .cli_on_every_node(&[ + "ACL", "SETUSER", "alice", "on", ">first", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + server + .cli_on_every_node(&[ + "ACL", "SETUSER", "bob", "on", ">second", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + let provider = Arc::new(Rotating::new("alice", "first")); + let client = connect_as( + &server, + Credentials::Provider(CredentialsProvider(provider.clone())), + ); + client + .send(commands::set("k", "1", Duration::from_secs(60))) + .await + .unwrap(); + let before = connections(&server).await; + + provider.rotate("bob", "second"); + let rotated = eventually(async || { + let now = connections(&server).await; + (users(&now) == BTreeSet::from(["bob"])).then_some(now) + }) + .await; + server + .cli_on_every_node(&["ACL", "DELUSER", "alice"]) + .await + .unwrap(); + server + .cli_on_every_node(&["CLIENT", "KILL", "USER", "bob"]) + .await + .unwrap(); + let after_reconnect = eventually(async || { + client + .send(commands::get::("k")) + .await + .ok() + .flatten() + }) + .await; + + let reconnected = connections(&server).await; + + assert_eq!(users(&before), BTreeSet::from(["alice"])); + assert_eq!(ids(&rotated), ids(&before)); + assert_eq!(after_reconnect, "1"); + assert!(!reconnected.is_empty()); + assert_eq!(users(&reconnected), BTreeSet::from(["bob"])); +} + +fn users(connections: &[ClientConnection]) -> BTreeSet<&str> { + connections + .iter() + .map(|connection| connection.user.as_str()) + .collect() +} + +fn ids(connections: &[ClientConnection]) -> Vec<(u16, u64)> { + connections + .iter() + .map(|connection| (connection.node, connection.id)) + .collect() +} + +#[rstest] +#[tokio::test] +async fn an_out_of_range_database_is_a_server_rejection() { + let server = RedisServer::start().await.unwrap(); + let port = server.url().rsplit(':').next().unwrap().parse().unwrap(); + let client = RedisClient::new(RedisConfig { + target: Target::Standalone(Standalone { + host: "127.0.0.1".into(), + port, + tls: None, + db: 999, + }), + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_secs(2), None), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap(); + let result = client.ping().await; + assert!(matches!(result, Err(Error::Server(_))), "{result:?}"); +} + +#[rstest] +#[tokio::test] +async fn a_rejected_credential_is_a_transport_failure() { + let server = RedisDeployment::start(Topology::Standalone).await.unwrap(); + server + .cli_on_every_node(&["ACL", "SETUSER", "default", "on", ">right"]) + .await + .unwrap(); + let with_password = |password: &str| { + connect_as( + &server, + Credentials::Static { + username: None, + password: Some(SecretValue::new(password)), + }, + ) + }; + with_password("right").ping().await.unwrap(); + + let rejected = with_password("wrong").ping().await; + + assert!(matches!(rejected, Err(Error::Transport(_))), "{rejected:?}"); +} + +fn sentinel_client(sentinel: &RedisSentinel, credentials: Credentials) -> RedisClient { + let (host, port) = sentinel.sentinel(); + RedisClient::new(RedisConfig { + target: Target::Sentinel(Sentinel { + monitors: vec![Node { + host: host.to_owned(), + port, + }], + service_name: SERVICE_NAME.to_owned(), + monitor_auth: None, + tls: None, + db: 0, + }), + credentials, + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_secs(1), None), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap() +} + +#[rstest] +#[tokio::test] +async fn sentinel_writes_follow_the_master_through_a_failover() { + let sentinel = RedisSentinel::start().await.unwrap(); + let client = sentinel_client(&sentinel, Credentials::NONE); + client + .send(commands::set("before", "1", Duration::from_secs(600))) + .await + .unwrap(); + eventually(async || { + let replica = + redis::Client::open(format!("redis://127.0.0.1:{}", sentinel.replica_port())).ok()?; + let mut replica = replica.get_multiplexed_async_connection().await.ok()?; + let value: Option = redis::cmd("GET") + .arg("before") + .query_async(&mut replica) + .await + .ok()?; + value + }) + .await; + + let promoted = sentinel.failover().await.unwrap(); + eventually(async || { + client + .send(commands::set("after", "2", Duration::from_secs(600))) + .await + .ok() + }) + .await; + + assert_eq!(promoted, sentinel.replica_port()); + assert_eq!( + client + .send(commands::get::("before")) + .await + .unwrap() + .as_deref(), + Some("1") + ); + let promoted = redis::Client::open(format!("redis://127.0.0.1:{promoted}")).unwrap(); + let mut promoted = promoted.get_multiplexed_async_connection().await.unwrap(); + let after: Option = redis::cmd("GET") + .arg("after") + .query_async(&mut promoted) + .await + .unwrap(); + assert_eq!(after.as_deref(), Some("2")); +} + +const DATA_PASSWORD: &str = "data-secret"; + +#[rstest] +#[tokio::test] +async fn sentinel_reaches_a_master_that_requires_a_password() { + let sentinel = RedisSentinel::start_with_password(DATA_PASSWORD) + .await + .unwrap(); + let client = sentinel_client( + &sentinel, + Credentials::Static { + username: None, + password: Some(SecretValue::new(DATA_PASSWORD)), + }, + ); + + client + .send(commands::set("k", "1", Duration::from_secs(60))) + .await + .unwrap(); + + assert_eq!( + client + .send(commands::get::("k")) + .await + .unwrap() + .as_deref(), + Some("1") + ); +} + +#[rstest] +#[tokio::test] +async fn sentinel_rediscovers_with_the_credentials_the_provider_holds_now() { + let sentinel = RedisSentinel::start_with_password(DATA_PASSWORD) + .await + .unwrap(); + sentinel + .cli_on_data_nodes(&[ + "ACL", "SETUSER", "alice", "on", ">first", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + sentinel + .cli_on_data_nodes(&[ + "ACL", "SETUSER", "bob", "on", ">second", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + let provider = Arc::new(Rotating::new("alice", "first")); + let client = sentinel_client( + &sentinel, + Credentials::Provider(CredentialsProvider(provider.clone())), + ); + client + .send(commands::set("k", "1", Duration::from_secs(600))) + .await + .unwrap(); + + provider.rotate("bob", "second"); + sentinel + .cli_on_data_nodes(&["ACL", "DELUSER", "alice"]) + .await + .unwrap(); + sentinel + .cli_on_data_nodes(&["CLIENT", "KILL", "USER", "bob"]) + .await + .unwrap(); + let after_rediscovery = eventually(async || { + client + .send(commands::get::("k")) + .await + .ok() + .flatten() + }) + .await; + + assert_eq!(after_rediscovery, "1"); +} + +#[rstest] +#[tokio::test] +async fn a_timed_out_command_replaces_a_connection_that_stopped_answering( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let proxies = proxied(&server).await; + let client = RedisClient::new(RedisConfig { + target: proxies.target.clone(), + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_millis(500), None), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap(); + client + .send(commands::set("k", "1", Duration::from_secs(600))) + .await + .unwrap(); + + proxies.blackhole_open_connections(); + let stalled = client.send(commands::get::("k")).await; + let next = client.send(commands::get::("k")).await; + + assert!( + matches!(&stalled, Err(Error::Transport(error)) if error.is_timeout()), + "{stalled:?}" + ); + assert_eq!(next.unwrap().as_deref(), Some("1")); +} diff --git a/litellm-rust/crates/redis-client/tests/pipeline.rs b/litellm-rust/crates/redis-client/tests/pipeline.rs new file mode 100644 index 00000000000..5aed70b42dd --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/pipeline.rs @@ -0,0 +1,362 @@ +mod support; + +use std::time::{Duration, Instant}; + +use litellm_redis_client::Value; +use litellm_redis_client::commands; +use litellm_redis_testing::{RedisDeployment, Topology}; +use rstest::rstest; + +use litellm_redis_client::{Amount, DEFAULT_TIMEOUT, Error, Expiry, Script}; +use support::connect; + +async fn begin_migration(cluster: &litellm_redis_testing::RedisCluster, key: &str) { + let (_, first) = cluster.nodes().next().unwrap(); + let slot = cluster + .cli_on_node(first, &["CLUSTER", "KEYSLOT", key]) + .await + .unwrap(); + let slot = slot.trim(); + let number: u16 = slot.parse().unwrap(); + let nodes = cluster + .cli_on_node(first, &["CLUSTER", "NODES"]) + .await + .unwrap(); + let owner = nodes.lines().find(|line| line.split_whitespace().skip(8).any(|range| { + let (start, end) = range.split_once('-').unwrap_or((range, range)); + matches!((start.parse::(), end.parse::()), (Ok(start), Ok(end)) if (start..=end).contains(&number)) + })).unwrap(); + let target = nodes.lines().find(|line| *line != owner).unwrap(); + let address = |line: &str| { + let value = line.split_whitespace().nth(1).unwrap(); + value + .split_once('@') + .unwrap() + .0 + .rsplit_once(':') + .unwrap() + .1 + .parse::() + .unwrap() + }; + let source_port = address(owner); + let target_port = address(target); + let source_id = owner.split_whitespace().next().unwrap(); + let target_id = target.split_whitespace().next().unwrap(); + assert_eq!( + cluster + .cli_on_node(source_port, &["GET", key]) + .await + .unwrap() + .trim(), + "0" + ); + assert_eq!( + cluster + .cli_on_node( + target_port, + &["CLUSTER", "SETSLOT", slot, "IMPORTING", source_id] + ) + .await + .unwrap() + .trim(), + "OK" + ); + assert_eq!( + cluster + .cli_on_node( + source_port, + &["CLUSTER", "SETSLOT", slot, "MIGRATING", target_id] + ) + .await + .unwrap() + .trim(), + "OK" + ); + let state = cluster + .cli_on_node(source_port, &["CLUSTER", "NODES"]) + .await + .unwrap(); + assert!( + state.contains(&format!("[{slot}->-{target_id}]")), + "{state}" + ); +} + +#[rstest] +#[tokio::test] +async fn a_retrying_neighbor_cannot_reapply_a_cluster_counter() { + let cluster = litellm_redis_testing::RedisCluster::start().await.unwrap(); + let server = RedisDeployment::Cluster(cluster); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let counter = "{migration}:counter"; + client + .send(commands::set(counter, 0, Duration::from_secs(60))) + .await + .unwrap(); + let RedisDeployment::Cluster(cluster) = &server else { + unreachable!() + }; + begin_migration(cluster, counter).await; + let both_keys = Script::new("return redis.call('MGET', KEYS[1], KEYS[2])"); + + let results = client + .run( + &client + .pipeline() + .with(commands::incr_by_float( + counter, + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::eval::( + &both_keys, + &[counter, "{migration}:missing"], + Vec::::new(), + )), + ) + .await; + + assert_eq!( + client.send(commands::get::(counter)).await.unwrap(), + Some(1) + ); + assert_eq!( + results[0].as_ref().unwrap(), + &Value::BulkString(b"1".to_vec()) + ); + assert!( + matches!(&results[1], Err(Error::Server(error)) if error.code() == Some("TRYAGAIN")), + "{:?}", + results[1] + ); +} + +#[rstest] +#[tokio::test] +async fn each_op_settles_on_its_own_in_input_order( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + client + .send(commands::set( + "text", + "not a number", + Duration::from_secs(60), + )) + .await + .unwrap(); + let pipeline = client + .pipeline() + .with(commands::incr_by_float( + "a", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::incr_by_float( + "text", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::incr_by_float( + "c", + Amount::Int(5), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::set("b", "2", Duration::from_secs(60))) + .with(commands::get::("b")) + .with(commands::incr_by_float( + "a", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::get::("missing")); + + let results = client.run(&pipeline).await; + + assert_eq!(results.len(), 7); + assert_eq!( + results[0].as_ref().unwrap(), + &Value::BulkString(b"1".to_vec()) + ); + assert!( + matches!(results[1], Err(Error::Server(_))), + "{:?}", + results[1] + ); + assert_eq!( + results[2].as_ref().unwrap(), + &Value::BulkString(b"5".to_vec()) + ); + assert_eq!(results[3].as_ref().unwrap(), &Value::Okay); + assert_eq!( + results[4].as_ref().unwrap(), + &Value::BulkString(b"2".to_vec()) + ); + assert_eq!( + results[5].as_ref().unwrap(), + &Value::BulkString(b"2".to_vec()) + ); + assert_eq!(results[6].as_ref().unwrap(), &Value::Nil); +} + +#[rstest] +#[case::cached(true, Value::BulkString(b"2".to_vec()))] +#[case::lost_by_the_server(false, Value::Nil)] +#[tokio::test] +async fn a_lost_script_reruns_after_the_rest_of_its_pipeline_like_redis_py( + #[case] cached: bool, + #[case] read_in_the_same_pipeline: Value, +) { + let server = RedisDeployment::start(Topology::Standalone).await.unwrap(); + let client = connect(&server, "ns", DEFAULT_TIMEOUT); + let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); + client + .send(commands::eval::(&incr, &["warmup"], 1)) + .await + .unwrap(); + if !cached { + server + .cli_on_every_node(&["SCRIPT", "FLUSH"]) + .await + .unwrap(); + } + let pipeline = client + .pipeline() + .with(commands::eval::(&incr, &["counter"], 2)) + .with(commands::get::("counter")); + + let results = client.run(&pipeline).await; + + assert_eq!(results[0].as_ref().unwrap(), &Value::Int(2)); + assert_eq!(results[1].as_ref().unwrap(), &read_in_the_same_pipeline); + assert_eq!( + client.send(commands::get::("counter")).await.unwrap(), + Some(2) + ); +} + +#[rstest] +#[tokio::test] +async fn an_unreachable_server_fails_every_op_as_transport( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_millis(500)); + let pipeline = client + .pipeline() + .with(commands::incr_by_float( + "a", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::incr_by_float( + "c", + Amount::Int(1), + Some(Expiry::Refresh(Duration::from_secs(60))), + )); + + server.freeze().await.unwrap(); + let results = client.run(&pipeline).await; + server.thaw().await.unwrap(); + + assert_eq!(results.len(), 2); + assert!( + matches!(results[0], Err(Error::Transport(_))), + "{:?}", + results[0] + ); + assert!( + matches!(results[1], Err(Error::Transport(_))), + "{:?}", + results[1] + ); +} + +#[rstest] +#[tokio::test] +async fn an_empty_pipeline_returns_without_asking_redis( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_secs(2)); + + server.freeze().await.unwrap(); + let started = Instant::now(); + let results = client.run(&client.pipeline()).await; + let elapsed = started.elapsed(); + server.thaw().await.unwrap(); + + assert!(results.is_empty()); + assert!(elapsed < Duration::from_secs(1), "{elapsed:?}"); +} + +#[rstest] +#[tokio::test] +async fn a_pipeline_sends_scripts_by_sha_and_reruns_only_the_ones_the_server_lost( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let deployment = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&deployment, "scripts", DEFAULT_TIMEOUT); + let count = Script::new("return redis.call('INCR', KEYS[1])"); + deployment + .cli_on_every_node(&["SCRIPT", "FLUSH"]) + .await + .unwrap(); + deployment + .cli_on_every_node(&["CONFIG", "RESETSTAT"]) + .await + .unwrap(); + + let first = client + .run( + &client + .pipeline() + .with(commands::incr_by_float( + "{a}spend", + Amount::Float(0.5), + Some(Expiry::Refresh(Duration::from_secs(60))), + )) + .with(commands::eval::( + &count, + &["{a}"], + Vec::::new(), + )) + .with(commands::eval::( + &count, + &["b"], + Vec::::new(), + )), + ) + .await; + let second = client + .run(&client.pipeline().with(commands::eval::( + &count, + &["{a}"], + Vec::::new(), + ))) + .await; + + let replies: Vec = first + .into_iter() + .chain(second) + .map(Result::unwrap) + .collect(); + assert_eq!( + replies, + [ + Value::BulkString(b"0.5".to_vec()), + Value::Int(1), + Value::Int(1), + Value::Int(2) + ] + ); + assert_eq!( + client.send(commands::get::("{a}spend")).await.unwrap(), + Some(0.5) + ); + let stats = support::command_stats(&deployment).await; + assert!(support::calls(&stats, "evalsha") > 0, "{stats}"); + assert_eq!(support::calls(&stats, "eval"), 0, "{stats}"); +} diff --git a/litellm-rust/crates/redis-client/tests/pubsub.rs b/litellm-rust/crates/redis-client/tests/pubsub.rs new file mode 100644 index 00000000000..2ad7cb38a5a --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/pubsub.rs @@ -0,0 +1,189 @@ +mod support; + +use std::time::{Duration, Instant}; + +use futures_util::StreamExt; +use litellm_circuit_breaker::{Breaker, Config}; +use litellm_redis_client::commands; +use litellm_redis_client::{ + Credentials, DEFAULT_TIMEOUT, Error, Namespace, Node, RedisClient, RedisConfig, Sentinel, + Target, Timeouts, +}; +use litellm_redis_testing::{RedisDeployment, RedisSentinel, SERVICE_NAME, Topology, eventually}; +use rstest::rstest; +use support::{connect, connect_guarded}; +use tokio::sync::mpsc::{UnboundedReceiver, unbounded_channel}; + +const CHANNEL: &str = "litellm_proxy.auth_cache_invalidation"; + +fn listen(client: &RedisClient, channel: &str) -> UnboundedReceiver> { + let (sender, receiver) = unbounded_channel(); + let mut messages = client.subscribe(channel); + tokio::spawn(async move { + while let Some(message) = messages.next().await { + if sender.send(message).is_err() { + return; + } + } + }); + receiver +} + +async fn delivered( + publisher: &RedisClient, + channel: &str, + received: &mut UnboundedReceiver>, + payload: &[u8], +) { + eventually(async || { + publisher + .send(commands::publish(channel, payload)) + .await + .ok()?; + let deadline = tokio::time::Instant::now() + Duration::from_millis(200); + while let Ok(Some(message)) = tokio::time::timeout_at(deadline, received.recv()).await { + if message == payload { + return Some(()); + } + } + None + }) + .await; +} + +#[rstest] +#[tokio::test] +async fn a_subscriber_receives_raw_bytes_on_the_channel_name_without_the_namespace( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let publisher = connect(&server, "ns", DEFAULT_TIMEOUT); + let plain = connect(&server, "", DEFAULT_TIMEOUT); + let mut received = listen(&plain, CHANNEL); + + delivered( + &publisher, + CHANNEL, + &mut received, + b"{\"cache_key\":\"k\"}\xff", + ) + .await; +} + +#[rstest] +#[tokio::test] +async fn a_subscriber_resubscribes_after_redis_restarts( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", Duration::from_millis(500)); + let mut received = listen(&client, CHANNEL); + delivered(&client, CHANNEL, &mut received, b"before").await; + + server.restart().await.unwrap(); + + delivered(&client, CHANNEL, &mut received, b"after").await; +} + +#[rstest] +#[tokio::test] +async fn a_sentinel_subscriber_receives_from_the_master() { + let server = RedisSentinel::start().await.unwrap(); + let (host, port) = server.sentinel(); + let client = RedisClient::new(RedisConfig { + target: Target::Sentinel(Sentinel { + monitors: vec![Node { + host: host.to_owned(), + port, + }], + service_name: SERVICE_NAME.to_owned(), + monitor_auth: None, + tls: None, + db: 0, + }), + credentials: Credentials::NONE, + namespace: Namespace::default(), + timeouts: Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap(); + let mut received = listen(&client, CHANNEL); + + delivered(&client, CHANNEL, &mut received, b"payload").await; +} + +async fn publish_calls(server: &RedisDeployment) -> Vec { + let replies = match server { + RedisDeployment::Standalone(server) => { + vec![server.cli(&["INFO", "commandstats"]).await.unwrap()] + } + RedisDeployment::Cluster(cluster) => cluster + .cli_outputs(&["INFO", "commandstats"]) + .await + .unwrap(), + }; + replies + .iter() + .flat_map(|reply| reply.lines()) + .filter(|line| line.starts_with("cmdstat_publish:")) + .map(str::to_owned) + .collect() +} + +#[rstest] +#[tokio::test] +async fn an_open_breaker_rejects_a_publish_without_sending_it( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let breaker = Breaker::new( + Config { + failure_threshold: 2, + recovery_timeout: Duration::from_secs(60), + timeout_min_duration: Duration::ZERO, + }, + Box::new(|_| {}), + ); + let client = connect_guarded(&server, "", Duration::from_millis(300), Some(breaker)); + + server.freeze().await.unwrap(); + let opened = [client.ping().await, client.ping().await]; + let started = Instant::now(); + let rejected = client.send(commands::publish(CHANNEL, b"payload")).await; + let rejected_in = started.elapsed(); + server.thaw().await.unwrap(); + + assert!( + opened + .iter() + .all(|ping| matches!(ping, Err(Error::Transport(_)))), + "{opened:?}" + ); + assert!(matches!(rejected, Err(Error::BreakerOpen)), "{rejected:?}"); + assert!(rejected_in < Duration::from_millis(50), "{rejected_in:?}"); + assert_eq!(publish_calls(&server).await, Vec::::new()); +} + +// redis-py's PubSub subscribes on a RESP2 connection, so it never sends HELLO +#[rstest] +#[tokio::test] +async fn a_subscription_speaks_resp2_like_redis_py( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + server + .cli_on_every_node(&["CONFIG", "RESETSTAT"]) + .await + .unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let mut received = listen(&client, CHANNEL); + + delivered(&client, CHANNEL, &mut received, b"evict").await; + + let stats = support::command_stats(&server).await; + assert!(support::calls(&stats, "subscribe") > 0, "{stats}"); + assert_eq!(support::calls(&stats, "hello"), 0, "{stats}"); +} diff --git a/litellm-rust/crates/redis-client/tests/script.rs b/litellm-rust/crates/redis-client/tests/script.rs new file mode 100644 index 00000000000..ea6d1c5ff78 --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/script.rs @@ -0,0 +1,133 @@ +mod support; + +use std::time::Duration; + +use litellm_redis_testing::{RedisDeployment, Topology}; +use rstest::rstest; + +use litellm_redis_client::Script; +use litellm_redis_client::commands; +use litellm_redis_client::{DEFAULT_TIMEOUT, Error, Ttl}; +use support::connect; + +#[rstest] +#[tokio::test] +async fn script_keys_are_namespaced( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "a", DEFAULT_TIMEOUT); + let bare = connect(&server, "", DEFAULT_TIMEOUT); + let read = Script::new("return redis.call('GET', KEYS[1])"); + bare.send(commands::set("a:x", "1", Duration::from_secs(60))) + .await + .unwrap(); + + let value: Option = client + .send(commands::eval(&read, &["x"], Vec::::new())) + .await + .unwrap(); + + assert_eq!(value.as_deref(), Some("1")); +} + +#[rstest] +#[tokio::test] +async fn a_failing_script_is_a_server_error_and_keeps_the_connection( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); + client + .send(commands::set("list", "item", Duration::from_secs(60))) + .await + .unwrap(); + + let failed = client + .send(commands::eval::(&incr, &["list"], 1)) + .await; + + assert!(matches!(failed, Err(Error::Server(_))), "{failed:?}"); + assert_eq!( + client + .send(commands::eval::(&incr, &["counter"], 3)) + .await + .unwrap(), + 3 + ); +} + +#[rstest] +#[tokio::test] +async fn a_flushed_script_is_loaded_again( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "", DEFAULT_TIMEOUT); + let incr = Script::new("return redis.call('INCRBY', KEYS[1], ARGV[1])"); + client + .send(commands::eval::(&incr, &["counter"], 1)) + .await + .unwrap(); + + server + .cli_on_every_node(&["SCRIPT", "FLUSH"]) + .await + .unwrap(); + + assert_eq!( + client + .send(commands::eval::(&incr, &["counter"], 1)) + .await + .unwrap(), + 2 + ); +} + +const INCREMENT_WITH_FLOOR: &str = "local count = redis.call('INCRBY', KEYS[1], ARGV[1]) \ +if count < 0 then count = redis.call('INCRBY', KEYS[1], -count) end \ +if redis.call('TTL', KEYS[1]) < 0 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end \ +return count"; + +#[rstest] +#[tokio::test] +async fn pythons_increment_with_floor_script_floors_at_zero_and_expires_in_one_call( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, + #[values(false, true)] pipelined: bool, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + let client = connect(&server, "ns", DEFAULT_TIMEOUT); + let floor = Script::new(INCREMENT_WITH_FLOOR); + server + .cli_on_every_node(&["SCRIPT", "FLUSH"]) + .await + .unwrap(); + + let count = match pipelined { + false => client + .send(commands::eval::(&floor, &["slots"], (-5, 60))) + .await + .unwrap(), + true => { + let results = client + .run(&client.pipeline().with(commands::eval::( + &floor, + &["slots"], + (-5, 60), + ))) + .await; + redis::from_redis_value(results[0].as_ref().unwrap().clone()).unwrap() + } + }; + + assert_eq!(count, 0); + assert_eq!( + client.send(commands::get::("slots")).await.unwrap(), + Some(0) + ); + assert!(matches!( + client.send(commands::ttl("slots")).await.unwrap(), + Ttl::Expires(left) if left > Duration::from_secs(50) && left <= Duration::from_secs(60) + )); +} diff --git a/litellm-rust/crates/redis-client/tests/support/mod.rs b/litellm-rust/crates/redis-client/tests/support/mod.rs new file mode 100644 index 00000000000..9cf209dce4a --- /dev/null +++ b/litellm-rust/crates/redis-client/tests/support/mod.rs @@ -0,0 +1,206 @@ +#![allow(dead_code)] // each test binary compiles this module on its own and uses a different subset + +use std::{pin::Pin, time::Duration}; + +use futures_util::Stream; +use litellm_circuit_breaker::Breaker; +use litellm_redis_client::{ + BasicAuth, Cluster, Credentials, Expiry, Namespace, Node, RedisClient, RedisConfig, RedisError, + RedisUrl, Standalone, StreamingCredentialsProvider, Target, Timeouts, +}; +use litellm_redis_testing::{ClientConnection, RedisDeployment, TcpProxy}; +use litellm_secret_value::SecretValue; +use tokio::sync::watch; + +pub const KEEP: Expiry = Expiry::KeepExisting(Duration::from_secs(60)); + +pub fn endpoint(deployment: &RedisDeployment) -> Target { + match deployment { + RedisDeployment::Standalone(server) => Target::Url(RedisUrl { + url: SecretValue::new(server.url()), + tls: None, + db: None, + username: None, + password: None, + }), + RedisDeployment::Cluster(cluster) => Target::Cluster(Cluster { + nodes: cluster + .nodes() + .map(|(host, port)| Node { + host: host.to_owned(), + port, + }) + .collect(), + tls: None, + }), + } +} + +pub fn connect(deployment: &RedisDeployment, namespace: &str, timeout: Duration) -> RedisClient { + connect_guarded(deployment, namespace, timeout, None) +} + +pub fn connect_guarded( + deployment: &RedisDeployment, + namespace: &str, + timeout: Duration, + breaker: Option, +) -> RedisClient { + RedisClient::new(RedisConfig { + target: endpoint(deployment), + credentials: Credentials::NONE, + namespace: Namespace::new(namespace), + timeouts: Timeouts::new(timeout, None), + breaker, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap() +} + +pub fn connect_as(deployment: &RedisDeployment, credentials: Credentials) -> RedisClient { + RedisClient::new(RedisConfig { + target: endpoint(deployment), + credentials, + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_secs(2), None), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap() +} + +pub async fn connections(deployment: &RedisDeployment) -> Vec { + deployment.clients().await.unwrap() +} + +/// A `TcpProxy` in front of every node, and a target that reaches the nodes only through them. A +/// cluster announces the proxy ports, so its slot map and redirects lead there too. +pub struct Proxied { + proxies: Vec, + pub target: Target, +} + +impl Proxied { + pub fn blackhole_open_connections(&self) { + self.proxies + .iter() + .for_each(TcpProxy::blackhole_open_connections); + } +} + +pub async fn proxied(deployment: &RedisDeployment) -> Proxied { + match deployment { + RedisDeployment::Standalone(server) => { + let proxy = TcpProxy::start(server.port()).await.unwrap(); + let target = Target::Standalone(Standalone { + host: "127.0.0.1".to_owned(), + port: proxy.port(), + tls: None, + db: 0, + }); + Proxied { + proxies: vec![proxy], + target, + } + } + RedisDeployment::Cluster(cluster) => { + let mut proxies = Vec::new(); + for (_, port) in cluster.nodes() { + proxies.push(TcpProxy::start(port).await.unwrap()); + } + let ports: [u16; 3] = std::array::from_fn(|node| proxies[node].port()); + cluster.announce_ports(ports).await.unwrap(); + let target = Target::Cluster(Cluster { + nodes: ports + .iter() + .map(|&port| Node { + host: "127.0.0.1".to_owned(), + port, + }) + .collect(), + tls: None, + }); + Proxied { proxies, target } + } + } +} + +/// `INFO

` from every node, concatenated. +async fn info(deployment: &RedisDeployment, section: &str) -> String { + match deployment { + RedisDeployment::Standalone(server) => server.cli(&["INFO", section]).await.unwrap(), + RedisDeployment::Cluster(cluster) => { + let mut stats = String::new(); + for (_, port) in cluster.nodes() { + stats.push_str(&cluster.cli_on_node(port, &["INFO", section]).await.unwrap()); + } + stats + } + } +} + +/// `INFO commandstats` from every node, so a test can count what was actually sent. +pub async fn command_stats(deployment: &RedisDeployment) -> String { + info(deployment, "commandstats").await +} + +/// Keys looked up on every node since the last `CONFIG RESETSTAT`, found or not. +pub async fn keys_read(deployment: &RedisDeployment) -> u64 { + let stats = info(deployment, "stats").await; + ["keyspace_hits:", "keyspace_misses:"] + .iter() + .flat_map(|field| { + stats + .lines() + .filter_map(move |line| line.trim().strip_prefix(field)?.parse::().ok()) + }) + .sum() +} + +/// How many times `command` reached a server, summed across the nodes in `stats`. +pub fn calls(stats: &str, command: &str) -> u64 { + let prefix = format!("cmdstat_{command}:calls="); + stats + .lines() + .filter_map(|line| line.trim().strip_prefix(&prefix)) + .filter_map(|rest| rest.split(',').next()?.parse::().ok()) + .sum() +} + +pub struct Rotating(watch::Sender); + +impl Rotating { + pub fn new(username: &str, password: &str) -> Self { + Self(watch::Sender::new(BasicAuth::new( + username.to_owned(), + password.to_owned(), + ))) + } + + pub fn rotate(&self, username: &str, password: &str) { + self.0 + .send_replace(BasicAuth::new(username.to_owned(), password.to_owned())); + } +} + +impl StreamingCredentialsProvider for Rotating { + fn subscribe( + &self, + ) -> Pin> + Send + 'static>> { + let receiver = self.0.subscribe(); + Box::pin(futures_util::stream::unfold( + (receiver, true), + |(mut receiver, first)| async move { + if !first { + receiver.changed().await.ok()?; + } + let credentials = receiver.borrow_and_update().clone(); + Some((Ok(credentials), (receiver, false))) + }, + )) + } +} diff --git a/litellm-rust/crates/redis-connect/AGENTS.md b/litellm-rust/crates/redis-connect/AGENTS.md new file mode 100644 index 00000000000..1b47804ddd8 --- /dev/null +++ b/litellm-rust/crates/redis-connect/AGENTS.md @@ -0,0 +1,10 @@ +- Turns a `litellm_redis_types::Resolution` into a `RedisClient`, the same way for every host. It depends on `litellm-redis-types`, never on `litellm-config`, so a host that resolves Redis some other way connects the same way. `redis-client` stays cloud-agnostic, so credential providers and cloud token sources are built here +- The host passes only what it alone can build in `HostServices`: the event sink, the HTTP client and the environment lookup. The breaker config comes from `Resolution::breaker`, which `config` reads from `REDIS_CIRCUIT_BREAKER_*` with Python's fallback, so a bad value keeps the default instead of failing connect. Each cloud source asks its auth crate for one explicit identity, the way the Python SDK builds it: `AwsAuthService::default_chain_credentials` (botocore's default chain), `CredentialSource::application_default` (`google.auth.default`) and `AzureIdentity` (the class `_build_azure_credential` picks). LiteLLM's provider settings such as `AWS_ROLE_NAME`, `AZURE_AD_TOKEN` or `VERTEXAI_CREDENTIALS` belong to Bedrock, Azure OpenAI and Vertex and never reach those constructors. `Auth::Provider` (a handle for a Python `credential_provider` object) fails at connect with an error naming the key, since no host can supply one yet +- Every expiring credential goes through the one refresh loop in `refresh.rs`. A cloud source only fetches. Never add a second loop or let a source sleep. The expiry deadline lives only in the published `Latest::Ready`, so expiring it leaves `Ready` and cannot fire again. Never keep a copy of the deadline in the loop +- AWS SDK identity settings must be in the process environment. `aws-config` cannot take the host lookup, so config-only overrides for profiles, role/web-identity settings, credentials, files and metadata sources fail before connecting rather than silently using another identity. Equal process/config values are allowed. Bedrock-only settings remain unrelated to Redis +- redis-rs subscribes once for the connection handshake and again for re-authentication, so a new connection sends a redundant initial AUTH. Unchanged refreshes do not trigger another AUTH. Do not skip a provider's first or alternating subscriptions, because concurrent connections and retries each need the current token. Removing the initial duplicate requires an upstream connection-level fix +- The loop starts on the first `subscribe()`, which redis-rs calls inside its runtime, so `connect` is safe off a runtime +- A token-only credential (Azure without `REDIS_USERNAME`, GCP) is sent as `AUTH default `, because redis-rs always sends a username with a credentials provider. On Redis 6+ that is the same as Python's `AUTH ` +- Tests for the refresh loop use a fake source and real Redis ACL users. Cloud sources are proven against the live service, unit tests only pin request shapes we own +- Rotation tests must retain the same server-side connection IDs across re-AUTH, then force a reconnect after removing the old user and verify the new identity still works +- Live cloud tests live in `tests/live/`. Set `LITELLM_REDIS_AWS_LIVE_CONFIG`, `LITELLM_REDIS_AZURE_LIVE_CONFIG` or `LITELLM_REDIS_GCP_LIVE_CONFIG` to a JSON file containing `redis` settings and `hold_seconds`. Set the waiting period from the deployment's token lifetime when checking expiry. Optional fixtures return early when unset, and configured tests run under plain `cargo test`. An unconfigured run never counts as live proof diff --git a/litellm-rust/crates/redis-connect/Cargo.toml b/litellm-rust/crates/redis-connect/Cargo.toml new file mode 100644 index 00000000000..72a453d36de --- /dev/null +++ b/litellm-rust/crates/redis-connect/Cargo.toml @@ -0,0 +1,39 @@ +[package] +name = "litellm-redis-connect" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +litellm-auth-aws.workspace = true +litellm-auth-azure.workspace = true +litellm-auth-gcp.workspace = true +litellm-auth-types.workspace = true +litellm-error.workspace = true +litellm-secret-value.workspace = true +strum.workspace = true +litellm-circuit-breaker.workspace = true +litellm-redis-types.workspace = true +litellm-core-utils.workspace = true +litellm-http.workspace = true +rustls.workspace = true +litellm-redis-client.workspace = true +futures-util.workspace = true +serde.workspace = true +serde_json.workspace = true +thiserror.workspace = true +veil.workspace = true +time.workspace = true +url.workspace = true +tokio = { workspace = true, features = ["sync", "time", "macros"] } +tracing.workspace = true + +[dev-dependencies] +litellm-redis-resolve.workspace = true +litellm-http = { workspace = true, features = ["test-support"] } +litellm-redis-testing.workspace = true +rustls.workspace = true +rstest.workspace = true +serde_json.workspace = true +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "test-util"] } diff --git a/litellm-rust/crates/redis-connect/src/connect.rs b/litellm-rust/crates/redis-connect/src/connect.rs new file mode 100644 index 00000000000..305aa181f2b --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/connect.rs @@ -0,0 +1,154 @@ +use std::{sync::Arc, time::Duration}; + +use litellm_circuit_breaker::{Breaker, Config as BreakerConfig}; +use litellm_core_utils::settings::Lookup; +use litellm_redis_client::{ + Credentials, CredentialsProvider, DEFAULT_TIMEOUT, Namespace, RedisClient, RedisConfig, + RedisEvents, Target, Timeouts, +}; +use litellm_redis_types::{Auth, HealthCheck, Resolution, StaticAuth}; + +use crate::{ + CredentialSource, Error, RefreshingCredentials, + sources::{AzureAd, ElastiCacheIam, GcpIam}, +}; + +pub type SharedEnvironment = Arc; + +pub struct HostServices { + pub events: Option>, + pub http: litellm_http::Client, + pub environment: SharedEnvironment, + pub credential_provider: Option>, +} + +// litellm/constants.py REDIS_CLUSTER_HEALTH_CHECK_INTERVAL, the default _redis.py sets for cluster clients only +const CLUSTER_HEALTH_CHECK_INTERVAL: Duration = Duration::from_secs(25); + +pub fn connect( + resolved: Resolution, + host: HostServices, + breaker: Option, +) -> Result { + if rustls::crypto::CryptoProvider::get_default().is_none() { + let _ = rustls::crypto::ring::default_provider().install_default(); + } + tracing::info!(target = ?resolved.target, namespace = ?resolved.namespace, "redis client configured"); + let credentials = credentials(resolved.auth, &host)?; + let events = host.events; + let breaker = breaker.map(|config| { + let observer = events.clone(); + Breaker::new( + config, + Box::new(move |transition| { + if let Some(events) = &observer { + events.breaker(transition); + } + }), + ) + }); + let health_check = health_check(&resolved.target, resolved.health_check); + Ok(RedisClient::new(RedisConfig { + target: resolved.target, + credentials, + namespace: resolved.namespace.map(Namespace::new).unwrap_or_default(), + timeouts: Timeouts::new( + resolved.timeout.unwrap_or(DEFAULT_TIMEOUT), + resolved.connect_timeout, + ), + breaker, + events, + health_check, + keepalive: resolved.keepalive, + })?) +} + +fn health_check(target: &Target, configured: HealthCheck) -> Option { + match (target, configured) { + (_, HealthCheck::Every(interval)) => Some(interval), + (_, HealthCheck::Disabled) => None, + (Target::Cluster(_), HealthCheck::Unset) => Some(CLUSTER_HEALTH_CHECK_INTERVAL), + (_, HealthCheck::Unset) => None, + } +} + +fn credentials(auth: Auth, host: &HostServices) -> Result { + let refreshing = |provider: RefreshingCredentials| { + Credentials::Provider(CredentialsProvider(Arc::new(provider))) + }; + Ok(match auth { + Auth::Static(StaticAuth { username, password }) => { + Credentials::Static { username, password } + } + Auth::Provider(_) => match &host.credential_provider { + Some(source) => refreshing(RefreshingCredentials::new(Arc::clone(source))), + None => return Err(Error::NoHostCredentialProvider), + }, + Auth::AzureAd(auth) => refreshing(RefreshingCredentials::new(AzureAd::new( + auth, + host.environment.clone(), + ))), + Auth::AwsIam(auth) => refreshing(RefreshingCredentials::new(ElastiCacheIam::new( + auth, + &*host.environment, + )?)), + Auth::GcpIam { service_account } => refreshing(RefreshingCredentials::new(GcpIam::new( + host.http.clone(), + service_account, + host.environment.clone(), + ))), + }) +} + +#[cfg(test)] +mod tests { + use litellm_redis_client::{Cluster, Node, Standalone}; + use rstest::rstest; + + use super::*; + + fn cluster() -> Target { + Target::Cluster(Cluster { + nodes: vec![Node { + host: "node".to_owned(), + port: 7000, + }], + tls: None, + }) + } + + fn standalone() -> Target { + Target::Standalone(Standalone { + host: "redis".to_owned(), + port: 6379, + tls: None, + db: 0, + }) + } + + #[rstest] + #[case::an_unset_cluster_takes_the_python_default( + cluster(), + HealthCheck::Unset, + Some(CLUSTER_HEALTH_CHECK_INTERVAL) + )] + #[case::zero_keeps_a_cluster_off(cluster(), HealthCheck::Disabled, None)] + #[case::an_unset_standalone_stays_off(standalone(), HealthCheck::Unset, None)] + #[case::a_configured_interval_wins( + cluster(), + HealthCheck::Every(Duration::from_secs(5)), + Some(Duration::from_secs(5)) + )] + #[case::a_negative_interval_checks_every_command( + standalone(), + HealthCheck::Every(Duration::ZERO), + Some(Duration::ZERO) + )] + fn health_checks_follow_redis_py( + #[case] target: Target, + #[case] configured: HealthCheck, + #[case] expected: Option, + ) { + assert_eq!(health_check(&target, configured), expected); + } +} diff --git a/litellm-rust/crates/redis-connect/src/error.rs b/litellm-rust/crates/redis-connect/src/error.rs new file mode 100644 index 00000000000..98343a03fb8 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/error.rs @@ -0,0 +1,102 @@ +#[derive(veil::Redact, thiserror::Error)] +pub enum Error { + #[error(transparent)] + Connect(#[from] litellm_redis_client::ConnectError), + #[error( + "Redis setting credential_provider names a Python object, which this host cannot supply" + )] + NoHostCredentialProvider, + #[error( + "Redis credential setting {setting} must be set in the process environment, not only environment_variables" + )] + ProcessEnvironmentRequired { setting: &'static str }, + #[error("{stage} failed{}", http_status_suffix(*http_status))] + Credential { + stage: CredentialStage, + http_status: Option, + #[source] + #[redact(with = "[REDACTED]")] + cause: litellm_error::Source, + }, + #[error("credential acquisition exceeded {timeout:?}")] + CredentialTimeout { + timeout: std::time::Duration, + #[source] + cause: tokio::time::error::Elapsed, + }, + #[error("Redis credential has expired")] + ExpiredCredential, + #[error("no {provider} credential is available for Redis")] + MissingCredential { provider: &'static str }, +} + +#[derive(Clone, Copy, Debug, strum::Display, strum::VariantArray)] +pub enum CredentialStage { + #[strum(serialize = "caller identity")] + Identity, + #[strum(serialize = "Redis token signing")] + Signing, + #[strum(serialize = "Redis token exchange")] + Exchange, + #[strum(serialize = "Redis token response")] + Response, + #[strum(serialize = "Redis credential_provider")] + Provider, +} + +impl Error { + pub fn credential( + stage: CredentialStage, + ) -> impl FnOnce(E) -> Self { + move |cause| Self::Credential { + stage, + http_status: None, + cause: litellm_error::Source::new(cause), + } + } + + pub(crate) fn credential_response( + stage: CredentialStage, + status: u16, + ) -> impl FnOnce(litellm_http::Error) -> Self { + move |cause| Self::Credential { + stage, + http_status: Some(status), + cause: litellm_error::Source::new(cause), + } + } + + pub(crate) fn upstream_status(&self) -> Option { + match self { + Self::Credential { http_status, .. } => *http_status, + Self::Connect(_) + | Self::NoHostCredentialProvider + | Self::ProcessEnvironmentRequired { .. } + | Self::CredentialTimeout { .. } + | Self::ExpiredCredential + | Self::MissingCredential { .. } => None, + } + } + + pub(crate) fn kind(&self) -> &'static str { + match self { + Self::Connect(_) => "connection", + Self::NoHostCredentialProvider => "host-credential-provider", + Self::ProcessEnvironmentRequired { .. } => "process-environment-required", + Self::Credential { stage, .. } => match stage { + CredentialStage::Identity => "caller-identity", + CredentialStage::Signing => "token-signing", + CredentialStage::Exchange => "token-exchange", + CredentialStage::Response => "token-response", + CredentialStage::Provider => "credential-provider", + }, + Self::CredentialTimeout { .. } => "acquisition-timeout", + Self::ExpiredCredential => "expired-credential", + Self::MissingCredential { .. } => "missing-credential", + } + } +} + +fn http_status_suffix(status: Option) -> String { + status.map_or_else(String::new, |status| format!(" (HTTP {status})")) +} diff --git a/litellm-rust/crates/redis-connect/src/events.rs b/litellm-rust/crates/redis-connect/src/events.rs new file mode 100644 index 00000000000..a556b7d370b --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/events.rs @@ -0,0 +1,129 @@ +use std::{ + sync::Mutex, + time::{Duration, Instant}, +}; + +use litellm_circuit_breaker::{State, Transition}; +use litellm_core_utils::settings::Lookup; +use litellm_redis_client::{CallEvent, FailureKind, Outcome, RedisEvents}; + +const DEFAULT_TIMEOUT_LOG_INTERVAL: Duration = Duration::from_secs(5); + +pub struct TracingEvents { + throttle: TimeoutLogThrottle, +} + +impl TracingEvents { + pub fn new(lookup: &dyn Lookup) -> Self { + let interval = lookup + .get("REDIS_TIMEOUT_LOG_INTERVAL") + .and_then(|seconds| seconds.parse::().ok()) + .and_then(|seconds| Duration::try_from_secs_f64(seconds).ok()) + .unwrap_or(DEFAULT_TIMEOUT_LOG_INTERVAL); + Self { + throttle: TimeoutLogThrottle::new(interval), + } + } +} + +impl RedisEvents for TracingEvents { + fn call(&self, event: &CallEvent) { + let Outcome::Failed(kind) = event.outcome else { + return; + }; + match kind { + FailureKind::BreakerOpen => { + tracing::debug!(operation = ?event.operation, "redis call rejected by the open circuit breaker") + } + FailureKind::Timeout => match self.throttle.admit(Instant::now()) { + None => { + tracing::debug!(operation = ?event.operation, duration = ?event.duration, "redis call timed out") + } + Some(0) => { + tracing::error!(operation = ?event.operation, duration = ?event.duration, "redis call timed out") + } + Some(suppressed) => tracing::error!( + operation = ?event.operation, + duration = ?event.duration, + suppressed, + "redis call timed out, more redis timeouts since the previous line were logged at debug" + ), + }, + FailureKind::Transport | FailureKind::Server | FailureKind::Decode => { + tracing::error!(operation = ?event.operation, failure = ?kind, "redis call failed") + } + } + } + + fn breaker(&self, transition: &Transition) { + match transition.to { + State::Open { .. } => tracing::warn!( + generation = transition.generation, + "redis circuit breaker opened, fast-failing redis calls" + ), + State::HalfOpen => tracing::info!( + generation = transition.generation, + "redis circuit breaker half open, probing redis" + ), + State::Closed { .. } => tracing::info!( + generation = transition.generation, + "redis circuit breaker closed, redis recovered" + ), + } + } +} + +struct TimeoutLogThrottle { + interval: Duration, + state: Mutex<(Option, u64)>, +} + +impl TimeoutLogThrottle { + fn new(interval: Duration) -> Self { + Self { + interval, + state: Mutex::new((None, 0)), + } + } + + fn admit(&self, now: Instant) -> Option { + let mut state = self + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let (last_logged_at, suppressed) = &mut *state; + if last_logged_at.is_some_and(|last| now.duration_since(last) < self.interval) { + *suppressed += 1; + return None; + } + *last_logged_at = Some(now); + Some(std::mem::take(suppressed)) + } +} + +#[cfg(test)] +mod tests { + use std::time::{Duration, Instant}; + + use rstest::rstest; + + use super::TimeoutLogThrottle; + + #[rstest] + fn admits_one_line_per_interval_and_counts_the_rest() { + let throttle = TimeoutLogThrottle::new(Duration::from_secs(5)); + let start = Instant::now(); + let at = |seconds: u64| start + Duration::from_secs(seconds); + + let admitted = [ + throttle.admit(at(0)), + throttle.admit(at(1)), + throttle.admit(at(4)), + throttle.admit(at(5)), + throttle.admit(at(6)), + throttle.admit(at(20)), + ]; + + assert_eq!(admitted, [Some(0), None, None, Some(2), None, Some(1)]); + } +} diff --git a/litellm-rust/crates/redis-connect/src/lib.rs b/litellm-rust/crates/redis-connect/src/lib.rs new file mode 100644 index 00000000000..b3d1d411eed --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/lib.rs @@ -0,0 +1,11 @@ +mod connect; +mod error; +mod events; +mod refresh; +mod sources; + +pub use connect::{HostServices, SharedEnvironment, connect}; +pub use error::{CredentialStage, Error}; +pub use events::TracingEvents; +pub(crate) use refresh::RefreshingCredentials; +pub use refresh::{CredentialSource, Expiring, SourceFuture}; diff --git a/litellm-rust/crates/redis-connect/src/refresh.rs b/litellm-rust/crates/redis-connect/src/refresh.rs new file mode 100644 index 00000000000..1e05d3d6289 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/refresh.rs @@ -0,0 +1,783 @@ +use std::{ + future::{Future, pending}, + pin::Pin, + sync::{Arc, Mutex}, + time::{Duration, SystemTime}, +}; + +use futures_util::{Stream, stream}; +use litellm_redis_client::{BasicAuth, RedisError, StreamingCredentialsProvider}; +use litellm_secret_value::SecretValue; +use tokio::{sync::watch, time::Instant}; + +use crate::Error; + +const TOKEN_ONLY_USERNAME: &str = "default"; +const UNKNOWN_LIFETIME: Duration = Duration::from_secs(300); +const MIN_REFRESH: Duration = Duration::from_secs(1); +const ACQUISITION_TIMEOUT: Duration = Duration::from_secs(30); +const FIRST_RETRY: Duration = Duration::from_secs(1); +const MAX_RETRY: Duration = Duration::from_secs(60); + +#[derive(Clone, PartialEq, Eq)] +pub struct Expiring { + pub username: Option, + pub password: SecretValue, + pub expires_at: Option, +} + +pub type SourceFuture<'a> = Pin> + Send + 'a>>; + +pub trait CredentialSource: Send + Sync + 'static { + fn fetch(&self) -> SourceFuture<'_>; +} + +impl CredentialSource for Arc { + fn fetch(&self) -> SourceFuture<'_> { + (**self).fetch() + } +} + +#[derive(Clone)] +enum Latest { + Pending, + Ready { + username: String, + password: SecretValue, + expires_at: Option, + }, + Unavailable(Arc), +} + +impl Latest { + fn is_same_token(&self, other: &Self) -> bool { + matches!( + (self, other), + ( + Self::Ready { username, password, .. }, + Self::Ready { username: other_username, password: other_password, .. }, + ) if username == other_username && password == other_password + ) + } + + fn expires_at(&self) -> Option { + match self { + Self::Ready { expires_at, .. } => *expires_at, + Self::Pending | Self::Unavailable(_) => None, + } + } +} + +type Start = Box; + +#[derive(Clone)] +pub struct RefreshingCredentials { + latest: watch::Receiver, + start: Arc>>, +} + +impl RefreshingCredentials { + pub fn new(source: impl CredentialSource) -> Self { + let (sender, latest) = watch::channel(Latest::Pending); + let start: Start = Box::new(move || drop(tokio::spawn(refresh(source, sender)))); + Self { + latest, + start: Arc::new(Mutex::new(Some(start))), + } + } + + fn start_once(&self) { + let start = self + .start + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .take(); + if let Some(start) = start { + start(); + } + } +} + +impl StreamingCredentialsProvider for RefreshingCredentials { + fn subscribe( + &self, + ) -> Pin> + Send + 'static>> { + self.start_once(); + Box::pin(stream::unfold( + (self.latest.clone(), true), + |(mut receiver, first)| async move { + if !first { + receiver.changed().await.ok()?; + } + let latest = receiver + .wait_for(|latest| !matches!(latest, Latest::Pending)) + .await + .ok()? + .clone(); + let item = match latest { + Latest::Ready { + username, password, .. + } => Ok(BasicAuth::new(username, password.expose().to_owned())), + Latest::Unavailable(error) => Err(public_error(&error)), + Latest::Pending => unreachable!("wait_for skips Pending"), + }; + Some((item, (receiver, false))) + }, + )) + } +} + +fn public_error(error: &Error) -> RedisError { + RedisError::from(( + litellm_redis_client::RedisErrorKind::AuthenticationFailed, + "fetching Redis credentials failed", + error.to_string(), + )) +} + +struct Fresh { + latest: Latest, + refresh_in: Duration, +} + +fn fresh(credentials: Expiring, now: SystemTime) -> Result { + let remaining = credentials + .expires_at + .map(|expires_at| expires_at.duration_since(now).unwrap_or_default()); + if remaining == Some(Duration::ZERO) { + return Err(Error::ExpiredCredential); + } + Ok(Fresh { + latest: Latest::Ready { + username: credentials + .username + .unwrap_or_else(|| TOKEN_ONLY_USERNAME.to_owned()), + password: credentials.password, + expires_at: remaining.map(|remaining| Instant::now() + remaining), + }, + refresh_in: refresh_after(credentials.expires_at, now), + }) +} + +async fn until_expiry( + future: impl Future, + sender: &watch::Sender, +) -> Option { + tokio::pin!(future); + loop { + let expires_at = sender.borrow().expires_at(); + let expiration = async { + match expires_at { + Some(deadline) => tokio::time::sleep_until(deadline).await, + None => pending().await, + } + }; + tokio::select! { + biased; + () = sender.closed() => return None, + () = expiration => { + sender.send_replace(Latest::Unavailable(Arc::new(Error::ExpiredCredential))); + } + result = &mut future => return Some(result), + } + } +} + +async fn refresh(source: impl CredentialSource, sender: watch::Sender) { + let mut retry = FIRST_RETRY; + loop { + let acquisition = async { + tokio::time::timeout(ACQUISITION_TIMEOUT, source.fetch()) + .await + .map_err(|cause| Error::CredentialTimeout { + timeout: ACQUISITION_TIMEOUT, + cause, + })? + .and_then(|credentials| fresh(credentials, SystemTime::now())) + }; + let Some(result) = until_expiry(acquisition, &sender).await else { + return; + }; + let wait = match result { + Ok(Fresh { latest, refresh_in }) => { + retry = FIRST_RETRY; + sender.send_if_modified(|current| { + let changed = !current.is_same_token(&latest); + *current = latest; + changed + }); + refresh_in + } + Err(error) => { + tracing::warn!(failure = error.kind(), upstream_status = error.upstream_status(), retry_in = ?retry, "fetching Redis credentials failed"); + if !matches!(*sender.borrow(), Latest::Ready { .. }) { + sender.send_replace(Latest::Unavailable(Arc::new(error))); + } + let wait = retry; + retry = (retry * 2).min(MAX_RETRY); + wait + } + }; + if until_expiry(tokio::time::sleep(wait), &sender) + .await + .is_none() + { + return; + } + } +} + +fn refresh_after(expires_at: Option, now: SystemTime) -> Duration { + let remaining = expires_at + .map(|expires_at| expires_at.duration_since(now).unwrap_or_default()) + .unwrap_or(UNKNOWN_LIFETIME); + (remaining * 3 / 4).max(MIN_REFRESH) +} + +#[cfg(test)] +mod tests { + use litellm_redis_client::commands; + use std::{ + collections::BTreeSet, + sync::{ + Arc, Mutex, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, SystemTime}, + }; + + use futures_util::StreamExt; + use litellm_redis_client::StreamingCredentialsProvider; + use litellm_secret_value::SecretValue; + use rstest::rstest; + + use super::{CredentialSource, Expiring, RefreshingCredentials, SourceFuture, refresh_after}; + use litellm_redis_client::{ + Cluster, Credentials, CredentialsProvider, Namespace, Node, RedisClient, RedisConfig, + RedisUrl, Sentinel, Target, Timeouts, + }; + use litellm_redis_testing::{ + ClientConnection, RedisDeployment, RedisSentinel, SERVICE_NAME, Topology, eventually, + }; + + #[rstest] + #[case::three_quarters_of_the_lifetime( + Some(Duration::from_secs(900)), + Duration::from_secs(675) + )] + #[case::already_expired(Some(Duration::ZERO), Duration::from_secs(1))] + #[case::unknown_expiry(None, Duration::from_secs(225))] + fn refreshes_well_before_expiry( + #[case] lifetime: Option, + #[case] expected: Duration, + ) { + let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000); + + assert_eq!( + refresh_after(lifetime.map(|lifetime| now + lifetime), now), + expected + ); + } + + struct Scripted { + replies: Mutex>>, + calls: Arc, + } + + impl CredentialSource for Scripted { + fn fetch(&self) -> SourceFuture<'_> { + self.calls.fetch_add(1, Ordering::SeqCst); + let reply = self.replies.lock().unwrap().remove(0); + Box::pin(async move { + reply.map_err(|reason| { + crate::Error::credential(crate::error::CredentialStage::Identity)( + std::io::Error::other(reason), + ) + }) + }) + } + } + + fn token(password: &str, lifetime: Duration) -> Result { + Ok(Expiring { + username: None, + password: SecretValue::new(password), + expires_at: Some(SystemTime::now() + lifetime), + }) + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn pushes_each_refresh_and_keeps_the_last_token_through_failures() { + let calls = Arc::new(AtomicUsize::new(0)); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + token("first", Duration::from_secs(8)), + Err("token endpoint down"), + token("second", Duration::from_secs(3600)), + ]), + calls: calls.clone(), + }); + let mut stream = provider.subscribe(); + + let first = stream.next().await.unwrap().unwrap(); + let second = stream.next().await.unwrap().unwrap(); + let late = provider.subscribe().next().await.unwrap().unwrap(); + + assert_eq!((first.username(), first.password()), ("default", "first")); + assert_eq!(second.password(), "second"); + assert_eq!(late.password(), "second"); + assert_eq!(calls.load(Ordering::SeqCst), 3); + } + + #[rstest] + fn building_outside_a_runtime_fetches_nothing_until_a_connection_subscribes() { + let calls = Arc::new(AtomicUsize::new(0)); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![token("first", Duration::from_secs(3600))]), + calls: calls.clone(), + }); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let before = calls.load(Ordering::SeqCst); + let first = runtime.block_on(async { provider.subscribe().next().await.unwrap().unwrap() }); + + assert_eq!(before, 0); + assert_eq!(first.password(), "first"); + assert_eq!(calls.load(Ordering::SeqCst), 1); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn a_failure_before_any_token_fails_the_connection_instead_of_hanging() { + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + Err("no identity"), + token("late", Duration::from_secs(3600)), + ]), + calls: Arc::new(AtomicUsize::new(0)), + }); + let mut stream = provider.subscribe(); + + assert!(stream.next().await.unwrap().is_err()); + assert_eq!(stream.next().await.unwrap().unwrap().password(), "late"); + } + + struct Blocking { + first: Mutex>, + started: Arc, + cancelled: Arc, + } + + struct Cancellation(Arc); + + impl Drop for Cancellation { + fn drop(&mut self) { + self.0.fetch_add(1, Ordering::SeqCst); + } + } + + impl CredentialSource for Blocking { + fn fetch(&self) -> SourceFuture<'_> { + let first = self.first.lock().unwrap().take(); + Box::pin(async move { + if let Some(first) = first { + return Ok(first); + } + let _guard = Cancellation(self.cancelled.clone()); + self.started.notify_one(); + std::future::pending().await + }) + } + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn a_refresh_that_returns_the_same_token_is_not_pushed_again() { + let calls = Arc::new(AtomicUsize::new(0)); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + token("same", Duration::from_secs(8)), + token("same", Duration::from_secs(3600)), + ]), + calls: calls.clone(), + }); + let mut stream = provider.subscribe(); + assert_eq!(stream.next().await.unwrap().unwrap().password(), "same"); + + let pushed = tokio::time::timeout(Duration::from_secs(60), stream.next()).await; + + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert!(pushed.is_err(), "an unchanged token was pushed again"); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn expiration_is_published_while_refresh_is_stalled() { + let started = Arc::new(tokio::sync::Notify::new()); + let provider = RefreshingCredentials::new(Blocking { + first: Mutex::new(Some(token("first", Duration::from_secs(4)).unwrap())), + started: started.clone(), + cancelled: Arc::new(AtomicUsize::new(0)), + }); + let mut stream = provider.subscribe(); + assert_eq!(stream.next().await.unwrap().unwrap().password(), "first"); + let start = tokio::time::Instant::now(); + started.notified().await; + let expired = stream.next().await.unwrap().unwrap_err(); + assert!(start.elapsed() <= Duration::from_secs(5)); + assert!(expired.to_string().contains("expired")); + let fresh = provider.subscribe().next().await.unwrap().unwrap_err(); + assert!(fresh.to_string().contains("expired")); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn expiration_is_published_once_and_later_failures_keep_their_cause() { + let replies = std::iter::once(token("first", Duration::from_secs(8))) + .chain(std::iter::repeat_n(Err("token endpoint down"), 16)) + .collect(); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(replies), + calls: Arc::new(AtomicUsize::new(0)), + }); + let mut stream = provider.subscribe(); + assert_eq!(stream.next().await.unwrap().unwrap().password(), "first"); + + let failures: Vec = stream + .take(5) + .map(|item| item.unwrap_err().to_string()) + .collect() + .await; + + assert!(failures[0].contains("expired"), "{failures:?}"); + assert!( + failures[1..] + .iter() + .all(|failure| failure.contains("caller identity") && !failure.contains("expired")), + "{failures:?}" + ); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn initial_acquisition_has_a_bounded_wait() { + let provider = RefreshingCredentials::new(Blocking { + first: Mutex::new(None), + started: Arc::new(tokio::sync::Notify::new()), + cancelled: Arc::new(AtomicUsize::new(0)), + }); + let error = provider.subscribe().next().await.unwrap().unwrap_err(); + assert!(error.to_string().contains("exceeded")); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn dropping_all_subscribers_cancels_inflight_acquisition() { + let started = Arc::new(tokio::sync::Notify::new()); + let cancelled = Arc::new(AtomicUsize::new(0)); + let provider = RefreshingCredentials::new(Blocking { + first: Mutex::new(None), + started: started.clone(), + cancelled: cancelled.clone(), + }); + let stream = provider.subscribe(); + started.notified().await; + drop(stream); + drop(provider); + tokio::task::yield_now().await; + assert_eq!(cancelled.load(Ordering::SeqCst), 1); + } + + #[rstest] + #[tokio::test(start_paused = true)] + async fn an_expired_fetch_is_rejected_and_a_later_valid_token_recovers() { + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + Ok(Expiring { + username: None, + password: SecretValue::new("expired"), + expires_at: Some(SystemTime::UNIX_EPOCH), + }), + token("recovered", Duration::from_secs(3600)), + ]), + calls: Arc::new(AtomicUsize::new(0)), + }); + let mut stream = provider.subscribe(); + assert!( + stream + .next() + .await + .unwrap() + .unwrap_err() + .to_string() + .contains("expired") + ); + assert_eq!( + stream.next().await.unwrap().unwrap().password(), + "recovered" + ); + } + + #[rstest] + fn credential_failure_keeps_its_typed_cause_but_hides_it_at_the_redis_boundary() { + let error = crate::Error::credential(crate::error::CredentialStage::Identity)( + std::io::Error::new(std::io::ErrorKind::PermissionDenied, "credential-value"), + ); + assert_eq!( + litellm_error::find::(&error) + .unwrap() + .kind(), + std::io::ErrorKind::PermissionDenied + ); + let debug = format!("{error:?}"); + assert!(!debug.contains("credential-value")); + assert!(debug.contains("[REDACTED]")); + let public = super::public_error(&error).to_string(); + assert!(public.contains("caller identity")); + assert!(!public.contains("credential-value")); + } + + #[rstest] + fn credential_http_failures_expose_status_without_exposing_the_response_body() { + let error = crate::Error::credential_response(crate::CredentialStage::Exchange, 503)( + litellm_http::Error::upstream(503, "credential-value".into()), + ); + let public = super::public_error(&error).to_string(); + assert!(public.contains("HTTP 503")); + assert!(!public.contains("credential-value")); + assert_eq!( + litellm_error::find::(&error) + .unwrap() + .upstream_response() + .unwrap() + .0, + 503 + ); + } + + fn endpoint(deployment: &RedisDeployment) -> Target { + match deployment { + RedisDeployment::Standalone(server) => Target::Url(RedisUrl { + url: SecretValue::new(server.url()), + tls: None, + db: None, + username: None, + password: None, + }), + RedisDeployment::Cluster(cluster) => Target::Cluster(Cluster { + nodes: cluster + .nodes() + .map(|(host, port)| Node { + host: host.to_owned(), + port, + }) + .collect(), + tls: None, + }), + } + } + + fn user(username: &str, password: &str, lifetime: Duration) -> Result { + Ok(Expiring { + username: Some(username.to_owned()), + password: SecretValue::new(password), + expires_at: Some(SystemTime::now() + lifetime), + }) + } + + fn connection_users(connections: &[ClientConnection]) -> BTreeSet<&str> { + connections + .iter() + .map(|connection| connection.user.as_str()) + .collect() + } + + fn connection_ids(connections: &[ClientConnection]) -> BTreeSet<(u16, u64)> { + connections + .iter() + .map(|connection| (connection.node, connection.id)) + .collect() + } + + #[rstest] + #[tokio::test] + async fn a_refreshed_token_reauthenticates_live_connections_before_the_old_one_expires( + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, + ) { + let server = RedisDeployment::start(topology).await.unwrap(); + server + .cli_on_every_node(&[ + "ACL", "SETUSER", "alice", "on", ">first", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + server + .cli_on_every_node(&[ + "ACL", "SETUSER", "bob", "on", ">second", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + user("alice", "first", Duration::from_secs(2)), + user("bob", "second", Duration::from_secs(3600)), + ]), + calls: Arc::new(AtomicUsize::new(0)), + }); + let client = RedisClient::new(RedisConfig { + target: endpoint(&server), + credentials: Credentials::Provider(CredentialsProvider(Arc::new(provider))), + namespace: Namespace::default(), + timeouts: Timeouts::default(), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap(); + client + .send(commands::set("k", "1", Duration::from_secs(60))) + .await + .unwrap(); + let before = server.clients().await.unwrap(); + + let after = eventually(async || { + let connections = server.clients().await.ok()?; + (!connections.is_empty() && connection_users(&connections) == BTreeSet::from(["bob"])) + .then_some(connections) + }) + .await; + server + .cli_on_every_node(&["ACL", "DELUSER", "alice"]) + .await + .unwrap(); + + assert!(!before.is_empty()); + assert_eq!(connection_users(&before), BTreeSet::from(["alice"])); + assert_eq!(connection_ids(&after), connection_ids(&before)); + assert_eq!( + client + .send(commands::get::("k")) + .await + .unwrap() + .as_deref(), + Some("1") + ); + server + .cli_on_every_node(&["CLIENT", "KILL", "USER", "bob"]) + .await + .unwrap(); + let value = eventually(async || { + client + .send(commands::get::("k")) + .await + .ok() + .flatten() + }) + .await; + let reconnected = server.clients().await.unwrap(); + + assert_eq!(value, "1"); + assert!(!reconnected.is_empty()); + assert_eq!(connection_users(&reconnected), BTreeSet::from(["bob"])); + assert!(connection_ids(&reconnected).is_disjoint(&connection_ids(&after))); + } + + /// Cloud credentials rotate while Sentinel fails over: the refreshed user must be what the + /// client presents to the master Sentinel promotes, not the one it connected with first. + #[rstest] + #[tokio::test] + async fn a_refreshed_user_reaches_the_master_sentinel_promotes() { + let sentinel = RedisSentinel::start().await.unwrap(); + sentinel + .cli_on_data_nodes(&[ + "ACL", "SETUSER", "alice", "on", ">first", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + sentinel + .cli_on_data_nodes(&[ + "ACL", "SETUSER", "bob", "on", ">second", "~*", "&*", "+@all", + ]) + .await + .unwrap(); + let provider = RefreshingCredentials::new(Scripted { + replies: Mutex::new(vec![ + user("alice", "first", Duration::from_secs(2)), + user("bob", "second", Duration::from_secs(3600)), + ]), + calls: Arc::new(AtomicUsize::new(0)), + }); + let (host, port) = sentinel.sentinel(); + let client = RedisClient::new(RedisConfig { + target: Target::Sentinel(Sentinel { + monitors: vec![Node { + host: host.to_owned(), + port, + }], + service_name: SERVICE_NAME.to_owned(), + monitor_auth: None, + tls: None, + db: 0, + }), + credentials: Credentials::Provider(CredentialsProvider(Arc::new(provider))), + namespace: Namespace::default(), + timeouts: Timeouts::new(Duration::from_secs(1), None), + breaker: None, + events: None, + health_check: None, + keepalive: true, + }) + .unwrap(); + client + .send(commands::set("before", "1", Duration::from_secs(60))) + .await + .unwrap(); + let master = sentinel.master_port().await.unwrap(); + let app_users = |connections: &[ClientConnection]| -> BTreeSet { + connections + .iter() + .filter(|connection| connection.user != "default") + .map(|connection| connection.user.clone()) + .collect() + }; + eventually(async || { + let connections = sentinel.clients_on(master).await.ok()?; + (app_users(&connections) == BTreeSet::from(["bob".to_owned()])).then_some(()) + }) + .await; + sentinel + .cli_on_data_nodes(&["ACL", "DELUSER", "alice"]) + .await + .unwrap(); + + let promoted = sentinel.failover().await.unwrap(); + eventually(async || { + client + .send(commands::set("after", "2", Duration::from_secs(60))) + .await + .ok() + }) + .await; + let on_promoted = sentinel.clients_on(promoted).await.unwrap(); + + assert_eq!(promoted, sentinel.replica_port()); + assert_eq!( + app_users(&on_promoted), + BTreeSet::from(["bob".to_owned()]), + "{on_promoted:?}" + ); + assert_eq!( + client + .send(commands::get::("after")) + .await + .unwrap() + .as_deref(), + Some("2") + ); + } +} diff --git a/litellm-rust/crates/redis-connect/src/sources/aws.rs b/litellm-rust/crates/redis-connect/src/sources/aws.rs new file mode 100644 index 00000000000..1e87bdd1b12 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/sources/aws.rs @@ -0,0 +1,188 @@ +use std::time::{Duration, SystemTime}; + +use litellm_auth_aws::{AwsAuthService, SigningScope, presign_url}; +use litellm_core_utils::settings::Lookup; +use litellm_redis_types::AwsIamAuth; + +use crate::Error; +use crate::error::CredentialStage; +use litellm_secret_value::SecretValue; + +use crate::{CredentialSource, Expiring, SourceFuture}; + +const SERVICE: &str = "elasticache"; +const TOKEN_LIFETIME: Duration = Duration::from_secs(900); + +const SDK_IDENTITY_ENVIRONMENT: &[&str] = &[ + "AWS_ACCESS_KEY_ID", + "AWS_SECRET_ACCESS_KEY", + "AWS_SESSION_TOKEN", + "AWS_PROFILE", + "AWS_DEFAULT_PROFILE", + "AWS_SHARED_CREDENTIALS_FILE", + "AWS_CONFIG_FILE", + "AWS_WEB_IDENTITY_TOKEN_FILE", + "AWS_ROLE_ARN", + "AWS_ROLE_SESSION_NAME", + "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI", + "AWS_CONTAINER_CREDENTIALS_FULL_URI", + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", + "AWS_EC2_METADATA_DISABLED", + "AWS_EC2_METADATA_SERVICE_ENDPOINT", + "AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE", + "AWS_ENDPOINT_URL", + "AWS_ENDPOINT_URL_STS", +]; + +pub(crate) struct ElastiCacheIam { + service: AwsAuthService, + auth: AwsIamAuth, +} + +impl ElastiCacheIam { + pub(crate) fn new(auth: AwsIamAuth, environment: &dyn Lookup) -> Result { + validate_environment(environment, &|name: &str| std::env::var(name).ok())?; + Ok(Self { + service: AwsAuthService::default(), + auth, + }) + } +} + +fn validate_environment(environment: &dyn Lookup, process: &dyn Lookup) -> Result<(), Error> { + match SDK_IDENTITY_ENVIRONMENT.iter().find(|name| { + environment.get(name).is_some_and(|value| { + Some(SecretValue::new(value)) != process.get(name).map(SecretValue::new) + }) + }) { + Some(setting) => Err(Error::ProcessEnvironmentRequired { setting }), + None => Ok(()), + } +} + +pub(crate) fn connect_url(auth: &AwsIamAuth) -> String { + let query = [ + Some(("Action", "connect")), + Some(("User", auth.user_name.as_str())), + auth.serverless + .then_some(("ResourceType", "ServerlessCache")), + ]; + let mut url = String::from("https://"); + url.push_str(&auth.cache_name.to_lowercase()); + url.push_str("/?"); + url.push_str( + &url::form_urlencoded::Serializer::new(String::new()) + .extend_pairs(query.into_iter().flatten()) + .finish(), + ); + url +} + +impl CredentialSource for ElastiCacheIam { + fn fetch(&self) -> SourceFuture<'_> { + Box::pin(async move { + let credentials = self + .service + .default_chain_credentials(&self.auth.region) + .await + .map_err(Error::credential(CredentialStage::Identity))?; + let now = SystemTime::now(); + let signed = presign_url( + &connect_url(&self.auth), + TOKEN_LIFETIME, + SigningScope { + region: &self.auth.region, + service: SERVICE, + credentials: &credentials, + time: now, + }, + ) + .map_err(Error::credential(CredentialStage::Signing))?; + let token_expiry = now + TOKEN_LIFETIME; + Ok(Expiring { + username: Some(self.auth.user_name.clone()), + password: SecretValue::new(signed.trim_start_matches("https://")), + expires_at: Some( + credentials + .expiry() + .map_or(token_expiry, |expiry| expiry.min(token_expiry)), + ), + }) + }) + } +} + +#[cfg(test)] +mod tests { + use litellm_redis_types::AwsIamAuth; + use rstest::rstest; + + use super::{connect_url, validate_environment}; + + #[rstest] + #[case::profile("AWS_PROFILE")] + #[case::role("AWS_ROLE_ARN")] + #[case::web_identity("AWS_WEB_IDENTITY_TOKEN_FILE")] + #[case::source_key("AWS_ACCESS_KEY_ID")] + #[case::source_secret("AWS_SECRET_ACCESS_KEY")] + #[case::session("AWS_SESSION_TOKEN")] + #[case::profile_path("AWS_SHARED_CREDENTIALS_FILE")] + #[case::container("AWS_CONTAINER_CREDENTIALS_FULL_URI")] + fn configured_identity_overrides_cannot_silently_use_an_ambient_identity( + #[case] setting: &'static str, + ) { + let configured = |name: &str| (name == setting).then(|| "credential-value".to_owned()); + let process = |name: &str| (name == setting).then(|| "ambient".to_owned()); + let error = validate_environment(&configured, &process).unwrap_err(); + + assert!(matches!( + error, + crate::Error::ProcessEnvironmentRequired { setting: rejected } if rejected == setting + )); + assert!(!error.to_string().contains("credential-value")); + assert!(!format!("{error:?}").contains("credential-value")); + } + + #[rstest] + #[case::process_environment("AWS_PROFILE", Some("profile"), Some("profile"))] + #[case::ambient_only("AWS_PROFILE", None, Some("profile"))] + #[case::unconfigured("AWS_PROFILE", None, None)] + #[case::unrelated_bedrock_setting("AWS_ROLE_NAME", Some("bedrock"), None)] + fn unchanged_or_unrelated_settings_preserve_the_sdk_chain( + #[case] setting: &str, + #[case] configured: Option<&str>, + #[case] ambient: Option<&str>, + ) { + let configured = |name: &str| { + (name == setting) + .then_some(configured) + .flatten() + .map(str::to_owned) + }; + let ambient = |name: &str| { + (name == setting) + .then_some(ambient) + .flatten() + .map(str::to_owned) + }; + assert!(validate_environment(&configured, &ambient).is_ok()); + } + + #[rstest] + #[case::provisioned(false, "https://cache-1/?Action=connect&User=iam+user%2F1")] + #[case::serverless( + true, + "https://cache-1/?Action=connect&User=iam+user%2F1&ResourceType=ServerlessCache" + )] + fn the_connect_url_matches_python_urlencode(#[case] serverless: bool, #[case] expected: &str) { + let auth = AwsIamAuth { + user_name: "iam user/1".to_owned(), + cache_name: "Cache-1".to_owned(), + region: "us-east-1".to_owned(), + serverless, + }; + + assert_eq!(connect_url(&auth), expected); + } +} diff --git a/litellm-rust/crates/redis-connect/src/sources/azure.rs b/litellm-rust/crates/redis-connect/src/sources/azure.rs new file mode 100644 index 00000000000..6dc72e6ce72 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/sources/azure.rs @@ -0,0 +1,64 @@ +use litellm_auth_azure::{AzureAuthService, AzureIdentity}; +use litellm_auth_types::ResolvedCredential; +use litellm_redis_types::AzureAdAuth; + +use crate::Error; +use crate::error::CredentialStage; + +use crate::{CredentialSource, Expiring, SharedEnvironment, SourceFuture}; + +const AZURE_REDIS_SCOPE: &str = "https://redis.azure.com/.default"; + +pub(crate) struct AzureAd { + service: AzureAuthService, + identity: AzureIdentity, + username: Option, + lookup: SharedEnvironment, +} + +impl AzureAd { + pub(crate) fn new(auth: AzureAdAuth, lookup: SharedEnvironment) -> Self { + // _build_azure_credential in litellm/_redis.py picks the credential class the same way + let identity = match (auth.client_id, auth.tenant_id, auth.client_secret) { + (Some(client_id), Some(tenant_id), Some(client_secret)) => { + AzureIdentity::ClientSecret { + tenant_id, + client_id, + client_secret, + } + } + (Some(client_id), _, _) => AzureIdentity::ManagedIdentity { client_id }, + (None, _, _) => AzureIdentity::Default, + }; + Self { + service: AzureAuthService::default(), + identity, + username: auth.username, + lookup, + } + } +} + +impl CredentialSource for AzureAd { + fn fetch(&self) -> SourceFuture<'_> { + Box::pin(async move { + let credential = self + .service + .identity_token(&self.identity, AZURE_REDIS_SCOPE, &|name| { + self.lookup.get(name) + }) + .await + .map_err(Error::credential(CredentialStage::Identity))? + .ok_or(Error::MissingCredential { provider: "Azure" })?; + let (token, expires_at) = match credential { + ResolvedCredential::AccessToken { token, expires_on } => (token, expires_on), + ResolvedCredential::Static(token) => (token, None), + }; + Ok(Expiring { + username: self.username.clone(), + password: token, + expires_at, + }) + }) + } +} diff --git a/litellm-rust/crates/redis-connect/src/sources/gcp.rs b/litellm-rust/crates/redis-connect/src/sources/gcp.rs new file mode 100644 index 00000000000..9da60fe9c08 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/sources/gcp.rs @@ -0,0 +1,126 @@ +use std::time::SystemTime; + +use litellm_auth_gcp::{CredentialSource as GcpCredentialSource, GcpAuth}; +use serde::Deserialize; + +use crate::Error; +use crate::error::CredentialStage; +use litellm_secret_value::SecretValue; + +use crate::{CredentialSource, Expiring, SharedEnvironment, SourceFuture}; + +const IAM_CREDENTIALS: &str = "https://iamcredentials.googleapis.com/v1"; +const CLOUD_PLATFORM_SCOPE: &str = "https://www.googleapis.com/auth/cloud-platform"; + +pub(crate) struct GcpIam { + http: litellm_http::Client, + identity: GcpAuth, + service_account: String, + lookup: SharedEnvironment, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct AccessToken { + access_token: String, + expire_time: String, +} + +impl GcpIam { + pub(crate) fn new( + http: litellm_http::Client, + service_account: String, + lookup: SharedEnvironment, + ) -> Self { + Self { + http, + identity: GcpAuth::default(), + service_account, + lookup, + } + } +} + +impl CredentialSource for GcpIam { + fn fetch(&self) -> SourceFuture<'_> { + Box::pin(async move { + let lookup = |name: &str| self.lookup.get(name); + let caller = self + .identity + .access_token(GcpCredentialSource::application_default(&lookup)) + .await + .map_err(Error::credential(CredentialStage::Identity))?; + let response = self + .http + .post(format!( + "{IAM_CREDENTIALS}/{}:generateAccessToken", + self.service_account + )) + .bearer_auth(caller.expose()) + .json(&serde_json::json!({ "scope": [CLOUD_PLATFORM_SCOPE] })) + .send() + .await + .map_err(Error::credential(CredentialStage::Exchange))?; + let status = response.status().as_u16(); + let reply: AccessToken = response + .error_for_status() + .map_err(litellm_http::Error::from) + .map_err(Error::credential_response( + CredentialStage::Exchange, + status, + ))? + .json() + .await + .map_err(litellm_http::Error::from) + .map_err(Error::credential(CredentialStage::Response))?; + let expires_at = time::OffsetDateTime::parse( + &reply.expire_time, + &time::format_description::well_known::Rfc3339, + ) + .map(SystemTime::from) + .map_err(Error::credential(CredentialStage::Response))?; + Ok(Expiring { + username: None, + password: SecretValue::new(reply.access_token), + expires_at: Some(expires_at), + }) + }) + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::GcpIam; + use crate::CredentialSource; + + #[rstest] + #[tokio::test] + async fn the_caller_identity_comes_from_application_default_credentials_not_vertex_settings() { + let environment = |name: &str| { + match name { + "VERTEXAI_CREDENTIALS" => { + Some(r#"{"type":"service_account","project_id":"vertex"}"#) + } + "GOOGLE_APPLICATION_CREDENTIALS" => Some("/nonexistent/redis-adc.json"), + _ => None, + } + .map(str::to_owned) + }; + let source = GcpIam::new( + litellm_http::Client::for_test(), + "projects/-/serviceAccounts/redis@project.iam.gserviceaccount.com".to_owned(), + std::sync::Arc::new(environment), + ); + + let Err(error) = source.fetch().await else { + panic!("a missing credentials file must fail"); + }; + + assert_eq!( + litellm_error::find::(&error).map(std::io::Error::kind), + Some(std::io::ErrorKind::NotFound) + ); + } +} diff --git a/litellm-rust/crates/redis-connect/src/sources/mod.rs b/litellm-rust/crates/redis-connect/src/sources/mod.rs new file mode 100644 index 00000000000..988ad3e1d23 --- /dev/null +++ b/litellm-rust/crates/redis-connect/src/sources/mod.rs @@ -0,0 +1,7 @@ +mod aws; +mod azure; +mod gcp; + +pub(crate) use aws::ElastiCacheIam; +pub(crate) use azure::AzureAd; +pub(crate) use gcp::GcpIam; diff --git a/litellm-rust/crates/redis-connect/tests/connect.rs b/litellm-rust/crates/redis-connect/tests/connect.rs new file mode 100644 index 00000000000..b1d541f4a5e --- /dev/null +++ b/litellm-rust/crates/redis-connect/tests/connect.rs @@ -0,0 +1,230 @@ +use litellm_redis_client::commands; +use std::{collections::BTreeMap, sync::Arc, time::Duration}; + +use litellm_redis_connect::{ + CredentialSource, Error, Expiring, HostServices, SourceFuture, connect, +}; +use litellm_redis_testing::{ + RedisDeployment, RedisSentinel, RedisServer, RedisTlsServer, SERVICE_NAME, Topology, eventually, +}; +use litellm_redis_types::{ProviderHandle, Resolution}; +use litellm_secret_value::SecretValue; +use rstest::rstest; + +// a host hands its credential_provider object over as a handle, so these tests do the same +fn resolved(kwargs: serde_json::Value) -> Resolution { + let provider = kwargs + .get("credential_provider") + .map(|_| ProviderHandle::new(1)); + let kwargs = serde_json::from_value::>(kwargs).unwrap(); + litellm_redis_resolve::interpret(&litellm_redis_resolve::kwargs(kwargs), provider, None) + .unwrap() +} + +fn services() -> HostServices { + HostServices { + events: None, + http: litellm_http::Client::for_test(), + environment: Arc::new(|_: &str| None), + credential_provider: None, + } +} + +struct HostPassword(&'static str); + +impl CredentialSource for HostPassword { + fn fetch(&self) -> SourceFuture<'_> { + Box::pin(async move { + Ok(Expiring { + username: None, + password: SecretValue::new(self.0), + expires_at: None, + }) + }) + } +} + +fn port(url: &str) -> u16 { + url.rsplit(':').next().unwrap().parse().unwrap() +} + +#[rstest] +#[case::trusted_ca(true, true)] +#[case::system_roots_only(false, false)] +#[tokio::test] +async fn python_tls_kwargs_reach_the_server(#[case] with_ca: bool, #[case] connects: bool) { + if rustls::crypto::CryptoProvider::get_default().is_none() { + let _ = rustls::crypto::ring::default_provider().install_default(); + } + let server = RedisTlsServer::start(false).await.unwrap(); + let kwargs = serde_json::json!({ + "host": server.host(), + "port": server.port(), + "ssl": true, + "ssl_ca_certs": with_ca.then(|| server.files().ca.display().to_string()), + "socket_timeout": 2, + }); + + let pinged = connect(resolved(kwargs), services(), None) + .unwrap() + .ping() + .await; + + assert_eq!(pinged.is_ok(), connects, "{pinged:?}"); +} + +#[rstest] +#[case::right_password("right", true)] +#[case::wrong_password("wrong", false)] +#[tokio::test] +async fn static_credentials_authenticate( + #[case] password: &str, + #[case] connects: bool, + #[values(Topology::Standalone, Topology::Cluster)] topology: Topology, +) { + let server = RedisDeployment::start(topology).await.unwrap(); + server + .cli_on_every_node(&["ACL", "SETUSER", "default", "on", ">right"]) + .await + .unwrap(); + let kwargs = match &server { + RedisDeployment::Standalone(server) => serde_json::json!({ + "host": "127.0.0.1", + "port": port(server.url()), + "password": password, + "socket_timeout": 2, + }), + RedisDeployment::Cluster(cluster) => serde_json::json!({ + "startup_nodes": cluster.nodes().map(|(host, port)| serde_json::json!({ + "host": host, + "port": port, + })).collect::>(), + "password": password, + "socket_timeout": 2, + }), + }; + + let pinged = connect(resolved(kwargs), services(), None) + .unwrap() + .ping() + .await; + + assert_eq!(pinged.is_ok(), connects, "{pinged:?}"); +} + +#[rstest] +#[tokio::test] +async fn python_sentinel_kwargs_follow_the_master_through_a_failover() { + let sentinel = RedisSentinel::start().await.unwrap(); + let (host, sentinel_port) = sentinel.sentinel(); + let kwargs = serde_json::json!({ + "sentinel_nodes": [[host, sentinel_port]], + "service_name": SERVICE_NAME, + "socket_timeout": 1, + }); + let client = connect(resolved(kwargs), services(), None).unwrap(); + client + .send(commands::set("before", "1", Duration::from_secs(600))) + .await + .unwrap(); + + let promoted = sentinel.failover().await.unwrap(); + eventually(async || { + client + .send(commands::set("after", "2", Duration::from_secs(600))) + .await + .ok() + }) + .await; + + assert_eq!(promoted, sentinel.replica_port()); + assert_eq!(sentinel.master_port().await.unwrap(), promoted); + assert_eq!( + client + .send(commands::get::("after")) + .await + .unwrap() + .as_deref(), + Some("2") + ); +} + +#[rstest] +fn a_python_credential_provider_without_a_host_source_fails_at_boot() { + let error = connect( + resolved(serde_json::json!({"host": "redis", "credential_provider": true})), + services(), + None, + ) + .err() + .unwrap(); + + assert!( + matches!(error, Error::NoHostCredentialProvider), + "{error:?}" + ); +} + +#[rstest] +#[case::profile("AWS_PROFILE")] +#[case::role("AWS_ROLE_ARN")] +#[case::web_identity("AWS_WEB_IDENTITY_TOKEN_FILE")] +fn config_only_aws_identities_fail_before_connecting(#[case] setting: &'static str) { + let configured = format!("{}-override", std::env::var(setting).unwrap_or_default()); + let host = HostServices { + environment: Arc::new(move |name: &str| (name == setting).then(|| configured.clone())), + ..services() + }; + let error = connect( + resolved(serde_json::json!({ + "host": "redis", "ssl": true, "aws_iam_auth": true, + "aws_iam_user_name": "user", "aws_iam_cache_name": "cache", + "aws_iam_region": "us-east-1", + })), + host, + None, + ) + .err() + .expect("must not select ambient credentials"); + + assert!(matches!( + error, + Error::ProcessEnvironmentRequired { setting: rejected } if rejected == setting + )); +} + +#[rstest] +#[case::right_password("right", true)] +#[case::wrong_password("wrong", false)] +#[tokio::test] +async fn a_python_credential_provider_authenticates_through_the_host_source( + #[case] password: &'static str, + #[case] connects: bool, +) { + let server = RedisServer::start().await.unwrap(); + server + .cli(&["ACL", "SETUSER", "default", "on", ">right"]) + .await + .unwrap(); + let kwargs = serde_json::json!({ + "host": "127.0.0.1", + "port": port(server.url()), + "password": "ignored", + "credential_provider": true, + "socket_timeout": 2, + }); + + let pinged = connect( + resolved(kwargs), + HostServices { + credential_provider: Some(Arc::new(HostPassword(password))), + ..services() + }, + None, + ) + .unwrap() + .ping() + .await; + + assert_eq!(pinged.is_ok(), connects, "{pinged:?}"); +} diff --git a/litellm-rust/crates/redis-connect/tests/live/aws.rs b/litellm-rust/crates/redis-connect/tests/live/aws.rs new file mode 100644 index 00000000000..da5ec323392 --- /dev/null +++ b/litellm-rust/crates/redis-connect/tests/live/aws.rs @@ -0,0 +1,17 @@ +use rstest::{fixture, rstest}; + +use super::{LiveConfig, config, roundtrip}; + +#[fixture] +fn live_config() -> Option { + config("LITELLM_REDIS_AWS_LIVE_CONFIG") +} + +#[rstest] +#[tokio::test] +async fn credentials_authenticate_existing_and_fresh_connections(live_config: Option) { + let Some(config) = live_config else { + return; + }; + roundtrip(config).await; +} diff --git a/litellm-rust/crates/redis-connect/tests/live/azure.rs b/litellm-rust/crates/redis-connect/tests/live/azure.rs new file mode 100644 index 00000000000..a7f678edef4 --- /dev/null +++ b/litellm-rust/crates/redis-connect/tests/live/azure.rs @@ -0,0 +1,17 @@ +use rstest::{fixture, rstest}; + +use super::{LiveConfig, config, roundtrip}; + +#[fixture] +fn live_config() -> Option { + config("LITELLM_REDIS_AZURE_LIVE_CONFIG") +} + +#[rstest] +#[tokio::test] +async fn credentials_authenticate_existing_and_fresh_connections(live_config: Option) { + let Some(config) = live_config else { + return; + }; + roundtrip(config).await; +} diff --git a/litellm-rust/crates/redis-connect/tests/live/gcp.rs b/litellm-rust/crates/redis-connect/tests/live/gcp.rs new file mode 100644 index 00000000000..0965a7a9e9a --- /dev/null +++ b/litellm-rust/crates/redis-connect/tests/live/gcp.rs @@ -0,0 +1,17 @@ +use rstest::{fixture, rstest}; + +use super::{LiveConfig, config, roundtrip}; + +#[fixture] +fn live_config() -> Option { + config("LITELLM_REDIS_GCP_LIVE_CONFIG") +} + +#[rstest] +#[tokio::test] +async fn credentials_authenticate_existing_and_fresh_connections(live_config: Option) { + let Some(config) = live_config else { + return; + }; + roundtrip(config).await; +} diff --git a/litellm-rust/crates/redis-connect/tests/live/main.rs b/litellm-rust/crates/redis-connect/tests/live/main.rs new file mode 100644 index 00000000000..90e7637530e --- /dev/null +++ b/litellm-rust/crates/redis-connect/tests/live/main.rs @@ -0,0 +1,111 @@ +mod aws; +mod azure; +mod gcp; + +use std::{collections::BTreeMap, path::PathBuf, sync::Arc, time::Duration}; + +use litellm_http::{ + ClientVariant, HttpClientPool, HttpSettings, HttpSettingsLayer, Resolution, + media::PublicDnsResolver, +}; +use litellm_redis_client::commands; +use litellm_redis_connect::{HostServices, connect}; +use serde::Deserialize; + +#[derive(Deserialize)] +struct LiveConfig { + redis: BTreeMap, + hold_seconds: u64, +} + +fn config(variable: &str) -> Option { + let path = PathBuf::from(std::env::var_os(variable)?); + Some( + serde_json::from_slice(&std::fs::read(path).expect("read live Redis config")) + .expect("parse live Redis config"), + ) +} + +async fn roundtrip(config: LiveConfig) { + if rustls::crypto::CryptoProvider::get_default().is_none() { + let _ = rustls::crypto::ring::default_provider().install_default(); + } + let lookup = |name: &str| std::env::var(name).ok(); + let resolved = litellm_redis_resolve::interpret( + &litellm_redis_resolve::kwargs(config.redis), + None, + lookup("REDIS_USERNAME").as_deref(), + ) + .expect("resolve live Redis config"); + let http_settings = HttpSettings::from_layers([HttpSettingsLayer::from_environment(&lookup)]); + let pool = HttpClientPool::new(Arc::new(PublicDnsResolver)); + let http = pool + .client( + &Resolution::from(&http_settings).config, + ClientVariant::Provider, + ) + .expect("build production HTTP client"); + let services = || HostServices { + events: None, + http: http.clone(), + environment: Arc::new(|name: &str| std::env::var(name).ok()), + credential_provider: None, + }; + let client = connect(resolved.clone(), services(), None).expect("build live Redis client"); + let marker = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let key = format!("litellm:redis-connect:live:{}:{marker}", std::process::id()); + let ttl = Duration::from_secs(config.hold_seconds + 120); + + client.ping().await.expect("authenticate and ping"); + client + .send(commands::set(&key, "before-refresh", ttl)) + .await + .expect("set before refresh"); + assert_eq!( + client + .send(commands::get::(&key)) + .await + .unwrap() + .as_deref(), + Some("before-refresh") + ); + + tokio::time::sleep(Duration::from_secs(config.hold_seconds)).await; + + assert_eq!( + client + .send(commands::get::(&key)) + .await + .unwrap() + .as_deref(), + Some("before-refresh") + ); + let fresh = connect(resolved, services(), None).expect("build a fresh live Redis client"); + assert_eq!( + fresh + .send(commands::get::(&key)) + .await + .unwrap() + .as_deref(), + Some("before-refresh") + ); + fresh + .send(commands::set(&key, "after-refresh", ttl)) + .await + .expect("set after refresh"); + assert_eq!( + client + .send(commands::get::(&key)) + .await + .unwrap() + .as_deref(), + Some("after-refresh") + ); + client + .send(commands::del(&[&key])) + .await + .expect("delete live test key"); +} diff --git a/litellm-rust/crates/redis-resolve/AGENTS.md b/litellm-rust/crates/redis-resolve/AGENTS.md new file mode 100644 index 00000000000..b7ce8278975 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/AGENTS.md @@ -0,0 +1,31 @@ +## Interpreting merged kwargs + +- The boundary is merged redis-py kwargs in, `litellm_redis_types::Resolution` out. `interpret` takes the dict a host already merged the way `_merge_redis_kwargs` in `litellm/_redis.py` does, with its `REDIS_*` values and `get_secret` fallbacks filled, and stands in for redis-py's constructors and `from_url`. It never reads the environment or a secret, so both hosts share it. Every `Entry` keeps the source a warning or error names, which `kwargs` sets to the kwarg name and the gateway merge sets to `REDIS_` or the fallback name +- The `Resolution` holds the target, the auth and the timeouts Rust models, and every setting the Rust client ignores as an `Ignored` with its source and consequence, which the host logs. A host passes its `credential_provider` object as a `ProviderHandle`, and a `credential_provider` key without one is `Error::ProviderWithoutHandle`. Python's Azure AD auth reads `REDIS_USERNAME` from `os.environ` instead of the merged kwargs, so it is a required argument of `interpret` that each host reads from its own process environment, the gateway through its `Lookup` and python-bridge through Python's `os.environ`. An empty one is unset, like `or None` +- `options.rs` reads the kwargs into the typed `Options` and collects the ignored ones. `endpoint.rs`, `auth.rs` and `tls.rs` pick the target, the credentials and TLS from `Options` alone +- Rust never introspects redis-py. Every Redis kwarg redis-py or `litellm/_redis.py` accepts is one row in `SETTINGS` in `src/keys.rs`, and that row is the whole answer for the kwarg: its `Handling` (`Read`, `Warn` with the consequence the warning names, or `Reject`), whether a url query carries it (`FromUrl`), whether `_get_redis_env_kwarg_mapping` reads it as `REDIS_`, and the names `_merge_redis_kwargs` falls back to. The last two columns are only read by the gateway merge, and they stay on the row so there is one table. A `Read` row is a field of `Options`, which an inline test pins. A kwarg with no row is not a Redis setting, so it is dropped without a warning +- `redis-py-kwargs.json` holds the Python facts each row answers: which redis-py versions name the kwarg in which signature, and whether Python reads its `REDIS_*` name under the locked redis-py. Never edit it by hand. `redis_py_kwargs.py` next to it regenerates it, and its docstring says how to run it. A kwarg litellm reads that no signature or `REDIS_*` name covers goes in its `LITELLM_ONLY`, and a new redis-py version in its `VERSIONS` and the `redis-compat` CI matrix together. `test_redis_py_kwargs_catalog_matches_the_installed_redis_py` in `tests/unit/test_redis.py` checks it against each version the `redis-compat` CI matrix installs, and `tests/catalog.rs` fails until `SETTINGS` has exactly one row per kwarg in it with the same `environment`. A redis-py bump that adds a kwarg therefore cannot land without a row deciding how Rust handles it +- The contract is `resolves_like_python_redis` in `tests/interpret.rs` and its mirror `test_redis_settings_resolve_like_the_rust_resolver_expects` in `tests/unit/test_redis.py`. Each case holds the merged dict, the sources the merge gave it, the `REDIS_USERNAME` the host passes, and the expected outcome written next to it. `Resolves` is the view both sides build. `Refuses` is a config Rust refuses and Python connects with, a disagreement on purpose. `Fails` is a config redis-py 5.3.1 already raises on, building the client or its first connection, so Rust failing changes no deployment that works today. Its Rust side names the class `Error::raised_building_the_python_client` answers, `None` when Python only raises once a command connects, and its Python mirror `_Fails` asserts the class and message redis-py raises. The Python side asserts that `_merge_redis_kwargs` returns that merged dict and reads the effective redis-py 5.3.1 connection objects without connecting, and `merges_like_python_redis` in `gateway/src/redis/surface_tests.rs` pins the same merged dicts from the Rust merge. A new rule is a new case in every file, never a generated fixture. Python and Rust pods share one Redis during the switch, so a change here is a change to a wire contract +- A url carries its own TLS. redis-py's url allowlist is the plain `Connection` signature, so every `ssl_*` setting outside the url is dropped, and a credential provider strips the url's userinfo. A `rediss://` url's `ssl_cert_reqs`, `ssl_ca_certs`, `ssl_ca_data`, `ssl_ca_path`, `ssl_certfile`, `ssl_keyfile`, `ssl_password` and `ssl_check_hostname` query parameters are read like `parse_url` reads them +- TLS options and `db` are only read for the endpoint that uses them, so a bad `ssl_cert_reqs` never fails a url target. A cluster with a non-zero `db`, and Sentinel with a `host` or `port`, are errors because redis-py raises building those clients +- An empty value (`""`, `[]`, `{}`) is the same as an unset one, like Python's `x or ...`. A null is unset too, except `ssl_cert_reqs`, where redis-py reads `None` as `CERT_NONE`, so a null skips certificate verification +- `ssl_cert_reqs` takes `none`, `optional` and `required` case-sensitively like redis-py's lookup, so `None` (what `REDIS_SSL_CERT_REQS=None` gives) is `Error::Invalid`, and an `ssl.VerifyMode` int, where 0 skips verification and 1 or 2 verifies. `optional` verifies, because rustls has no optional mode +- Precedence matches Python. The target is cluster, then URL, then Sentinel (`sentinel_nodes` plus `service_name`), then host. Auth is a `credential_provider` handle, then GCP, then Azure, then AWS, and the first one wins with a warning +- Numbers are read with `litellm_python_compat::number`, so whitespace and digit underscores parse like Python's `int()` and `float()`. A value they cannot parse (`port: abc`, `socket_timeout: slow`) is dropped with a warning and the default is used, like `_coerce_redis_kwargs_types`. A number that parses but is out of range (negative seconds, a port above 65535) fails with `Error::Invalid` +- Flags (`ssl`, `ssl_check_hostname`, `aws_iam_auth`, `aws_iam_serverless`) are read one way on every surface. `true`, `1` and `yes` are true and `false`, `0`, `no`, `off`, `f` and `n` are false, the text redis-py's `_str_to_bool` and the proxy's pydantic models agree on. Any other text is `Error::Invalid`, so a TLS setting is never guessed +- The resolver fails on a setting it cannot honor only when that setting is a security check. A `Warn` row is dropped and returned as an `Ignored` naming its source (`ssl_ciphers` or `REDIS_SSL_CIPHERS`) and the consequence on its row, because a Python deployment that worked must keep working when its stores move to Rust. Only a value that is malformed (`Error::Invalid` or `Error::UrlQuery`), a target that is missing or a `Reject` row that asks for its check fails. Promoting a `Warn` row to `Read` is then a pure improvement with no behavior cliff, while moving one to `Reject` can fail a config that connects today + - A `Warn` row with a `met` check is dropped silently when its value asks for nothing more than Rust does, so `ssl_min_version` of TLS 1.2 or lower never warns + - A `Reject` row is a security check rustls cannot perform that Python's locked async client already refuses with a `TypeError`, so failing closed breaks no proxy that runs today. `tests/catalog.rs` fails if the locked redis-py's async, cluster or connection signature accepts a `Reject` row, so a redis-py bump forces the decision again. A row's `Asked` says which values ask for the check, from the kwargs, `REDIS_*` or a url query, and those are `Error::Unenforceable` + - `ssl_validate_ocsp` and `ssl_validate_ocsp_stapled` ask when true, because rustls has no client-side OCSP. False asks for nothing and is dropped without a warning + - `ssl_include_verify_flags` and `ssl_exclude_verify_flags` ask when set, because rustls takes no X.509 verify flags such as CRL checking. An empty list asks for nothing + - `ssl_ca_data` and `ssl_ca_path` are read into `TlsOptions` and trusted next to `ssl_ca_certs`. `ssl_password` is read into `TlsOptions::key_password`, which `redis-client` uses to decrypt a PKCS#8 key like the sync client's `load_cert_chain(password=...)`. redis-py 5.3.1's async client refuses it, so this is a superset of the path the proxy uses, like `ssl_ca_path` + - `unix_socket_path` resolves to `Target::Unix`, which like redis-py's `UnixDomainSocketConnection` replaces host, port and TLS. Python still requires `host` next to it, so Rust does too + - A url's `?db=`, `?username=`, `?password=`, `?socket_timeout=`, `?socket_connect_timeout=` and `?health_check_interval=` are read like `parse_url` reads them, so `?db=` wins over the path, the userinfo wins over `?username=` and `?password=`, and the timeouts and health check win over the kwargs like `from_url`. A repeated parameter takes its first value like `parse_qs`. A number `int()` or `float()` cannot read is `Error::UrlQuery`, as `parse_url` raises `ValueError`. Any other query parameter, including one with no row, is ignored naming `url ?`, because redis-py turns each one into a client kwarg. Which ones a url carries is the `FromUrl` column + - `socket_keepalive` is a flag that defaults to true on every target, like `_get_redis_cluster_kwargs` defaults it for a cluster. Python leaves it off for a standalone or Sentinel client, so an unset value there is a difference on purpose, because without keepalive a load balancer drops an idle connection silently. False turns it off. `socket_keepalive_options` is warned, and `redis-client` probes after its own idle time instead + - `health_check_interval` is read as an `int` of seconds like `_coerce_redis_kwargs_types` into `HealthCheck`. Zero is `Disabled` and stays off on every target, a negative interval checks before every command like redis-py, and only `Unset` takes a default. `redis-connect` gives an unset cluster Python's 25s default and `redis-client` pings every node before the next command once the client sat idle that long, outside the breaker and the call events +- `Error::raised_building_the_python_client` is how python-bridge treats a config Rust cannot use. Where Python raises building its client, the bridge raises the same class from `NativeRedisClient.connect`. Where Python builds the client and only fails once a command connects, which `RedisCache` catches and logs on every call, the stores stay on memory and the bridge logs an error +- Cases where Rust and Python disagree on purpose are cases in both contract tests with a different expected outcome on each side + - `ssl_ca_path`: redis-py 5.3.1's async `Redis` has no such parameter and raises `TypeError` building the client, while Rust trusts the directory like `load_verify_locations(capath=...)` + - `on`, `t` and `y` for a flag are `Error::Invalid` on every surface. pydantic reads them as true for `coordination_redis` and `_str_to_bool` reads them as false everywhere else, so Rust refuses to pick one. Other unrecognized text is false in Python and an error here + - `ssl_cert_reqs` as an `ssl.VerifyMode` int: redis-py 5.3.1's sync client takes it, while its async `RedisSSLContext` leaves `cert_reqs` unset and raises `AttributeError` once a command connects. Rust reads it like the sync client + - Empty `sentinel_nodes` with no `REDIS_SENTINEL_NODES` is no target at all, so it fails as "either host or url must be specified". redis-py keeps the empty list and fails building the Sentinel client diff --git a/litellm-rust/crates/redis-resolve/Cargo.toml b/litellm-rust/crates/redis-resolve/Cargo.toml new file mode 100644 index 00000000000..b3d3fc01e44 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/Cargo.toml @@ -0,0 +1,20 @@ +[package] +name = "litellm-redis-resolve" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +litellm-python-compat.workspace = true +litellm-redis-types.workspace = true +litellm-secret-value.workspace = true +serde_json.workspace = true +strum.workspace = true +thiserror.workspace = true +tracing.workspace = true +url.workspace = true + +[dev-dependencies] +rstest.workspace = true +serde.workspace = true diff --git a/litellm-rust/crates/redis-resolve/redis-py-kwargs.json b/litellm-rust/crates/redis-resolve/redis-py-kwargs.json new file mode 100644 index 00000000000..22c4495818b --- /dev/null +++ b/litellm-rust/crates/redis-resolve/redis-py-kwargs.json @@ -0,0 +1,118 @@ +{ + "meta": { + "description": "What redis-py and litellm/_redis.py accept for every Redis kwarg the Rust resolver can be handed. SETTINGS in src/keys.rs has exactly one row per arg here", + "redis_py_versions": [ + "5.3.1", + "6.4.0", + "7.4.1", + "8.0.1" + ], + "python": "3.12", + "method": "python redis_py_kwargs.py next to this file, see its docstring. It reads inspect.getfullargspec(inspect.unwrap(cls.__init__)) args and kwonlyargs, minus self", + "signatures": { + "sync": "redis.Redis.__init__. _get_redis_kwargs uses it as the config allowlist and the REDIS_ environment mapping", + "async": "redis.asyncio.Redis.__init__, which the proxy constructs. A kwarg in sync but not async raises TypeError when the client is built", + "connection": "redis.Connection or redis.asyncio.Connection MRO, the url allowlist from _get_redis_url_kwargs", + "cluster": "redis.asyncio.RedisCluster.__init__, the allowlist from _get_redis_cluster_kwargs" + }, + "environment": "true when _get_redis_env_kwarg_mapping reads the arg as REDIS_ under the locked redis-py", + "redis_py": "each redis-py version whose signatures name the arg, with which ones. An empty object means only litellm reads it", + "checked_by": "tests/unit/test_redis.py checks the installed redis-py's column and the REDIS_* names against this file, and tests/catalog.rs checks SETTINGS against it", + "redis_py_locked": "5.3.1" + }, + "args": { + "address_remap": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "auto_close_connection_pool": {"environment": false, "redis_py": {"5.3.1": ["async"], "6.4.0": ["async"], "7.4.1": ["async"], "8.0.1": ["async"]}}, + "aws_iam_auth": {"environment": true, "redis_py": {}}, + "aws_iam_cache_name": {"environment": true, "redis_py": {}}, + "aws_iam_region": {"environment": true, "redis_py": {}}, + "aws_iam_serverless": {"environment": true, "redis_py": {}}, + "aws_iam_user_name": {"environment": true, "redis_py": {}}, + "azure_client_id": {"environment": true, "redis_py": {}}, + "azure_client_secret": {"environment": true, "redis_py": {}}, + "azure_redis_ad_token": {"environment": true, "redis_py": {}}, + "azure_tenant_id": {"environment": true, "redis_py": {}}, + "cache": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "cache_config": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "charset": {"environment": true, "redis_py": {"5.3.1": ["sync"]}}, + "client_name": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "cluster_error_retry_attempts": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "command_packer": {"environment": false, "redis_py": {"5.3.1": ["connection"], "6.4.0": ["connection"], "7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "connection_class": {"environment": false, "redis_py": {}}, + "connection_error_retry_attempts": {"environment": false, "redis_py": {"5.3.1": ["cluster"]}}, + "connection_pool": {"environment": false, "redis_py": {"5.3.1": ["sync", "async"], "6.4.0": ["sync", "async"], "7.4.1": ["sync", "async"], "8.0.1": ["sync", "async"]}}, + "credential_provider": {"environment": false, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "db": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "decode_responses": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "driver_info": {"environment": false, "redis_py": {"7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "dynamic_startup_nodes": {"environment": false, "redis_py": {"6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "encoder_class": {"environment": false, "redis_py": {"5.3.1": ["connection"], "6.4.0": ["connection"], "7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "encoding": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "encoding_errors": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "errors": {"environment": true, "redis_py": {"5.3.1": ["sync"]}}, + "event_dispatcher": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "gcp_service_account": {"environment": true, "redis_py": {}}, + "gcp_ssl_ca_certs": {"environment": true, "redis_py": {}}, + "health_check_interval": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "host": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "legacy_responses": {"environment": false, "redis_py": {"8.0.1": ["sync", "async", "connection", "cluster"]}}, + "lib_name": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "lib_version": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "load_balancing_strategy": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "maint_notifications_config": {"environment": false, "redis_py": {"7.4.1": ["sync", "connection"], "8.0.1": ["sync", "connection"]}}, + "maint_notifications_pool_handler": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "maintenance_notification_hash": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "maintenance_state": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "max_connections": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "namespace": {"environment": false, "redis_py": {}}, + "orig_host_address": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "orig_socket_connect_timeout": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "orig_socket_timeout": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "oss_cluster_maint_notifications_handler": {"environment": false, "redis_py": {"7.4.1": ["sync", "connection"], "8.0.1": ["sync", "connection"]}}, + "parser": {"environment": false, "redis_py": {"7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "parser_class": {"environment": false, "redis_py": {"5.3.1": ["connection"], "6.4.0": ["connection"], "7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "password": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "path": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "policy_resolver": {"environment": false, "redis_py": {"7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "port": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "protocol": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "read_from_replicas": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "redis_connect_func": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection"], "6.4.0": ["sync", "async", "connection"], "7.4.1": ["sync", "async", "connection"], "8.0.1": ["sync", "async", "connection"]}}, + "reinitialize_steps": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "require_full_coverage": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "retry": {"environment": false, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "retry_on_error": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "retry_on_timeout": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection"], "6.4.0": ["sync", "async", "connection"], "7.4.1": ["sync", "async", "connection"], "8.0.1": ["sync", "async", "connection"]}}, + "sentinel_nodes": {"environment": false, "redis_py": {}}, + "sentinel_password": {"environment": false, "redis_py": {}}, + "service_name": {"environment": false, "redis_py": {}}, + "single_connection_client": {"environment": true, "redis_py": {"5.3.1": ["sync", "async"], "6.4.0": ["sync", "async"], "7.4.1": ["sync", "async"], "8.0.1": ["sync", "async"]}}, + "socket_connect_timeout": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "socket_keepalive": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "socket_keepalive_options": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "socket_read_size": {"environment": false, "redis_py": {"5.3.1": ["connection"], "6.4.0": ["connection"], "7.4.1": ["connection"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "socket_timeout": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}}, + "socket_type": {"environment": false, "redis_py": {"5.3.1": ["connection"], "6.4.0": ["connection"], "7.4.1": ["connection"], "8.0.1": ["connection"]}}, + "ssl": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_ca_certs": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_ca_data": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_ca_path": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync", "async"], "8.0.1": ["sync", "async"]}}, + "ssl_cert_reqs": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_certfile": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_check_hostname": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_ciphers": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_exclude_verify_flags": {"environment": false, "redis_py": {"7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_include_verify_flags": {"environment": false, "redis_py": {"7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_keyfile": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_min_version": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "cluster"], "6.4.0": ["sync", "async", "cluster"], "7.4.1": ["sync", "async", "cluster"], "8.0.1": ["sync", "async", "cluster"]}}, + "ssl_ocsp_context": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "ssl_ocsp_expected_cert": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "ssl_password": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync", "async"], "8.0.1": ["sync", "async"]}}, + "ssl_validate_ocsp": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "ssl_validate_ocsp_stapled": {"environment": true, "redis_py": {"5.3.1": ["sync"], "6.4.0": ["sync"], "7.4.1": ["sync"], "8.0.1": ["sync"]}}, + "startup_nodes": {"environment": false, "redis_py": {"5.3.1": ["cluster"], "6.4.0": ["cluster"], "7.4.1": ["cluster"], "8.0.1": ["cluster"]}}, + "unix_socket_path": {"environment": true, "redis_py": {"5.3.1": ["sync", "async"], "6.4.0": ["sync", "async"], "7.4.1": ["sync", "async"], "8.0.1": ["sync", "async"]}}, + "url": {"environment": true, "redis_py": {}}, + "username": {"environment": true, "redis_py": {"5.3.1": ["sync", "async", "connection", "cluster"], "6.4.0": ["sync", "async", "connection", "cluster"], "7.4.1": ["sync", "async", "connection", "cluster"], "8.0.1": ["sync", "async", "connection", "cluster"]}} + } +} diff --git a/litellm-rust/crates/redis-resolve/redis_py_kwargs.py b/litellm-rust/crates/redis-resolve/redis_py_kwargs.py new file mode 100644 index 00000000000..9459cfd3761 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/redis_py_kwargs.py @@ -0,0 +1,110 @@ +"""Regenerates redis-py-kwargs.json, the Python facts SETTINGS in src/keys.rs answers. + +Run it from the repository root in the project environment, which has the locked redis-py and litellm: + + uv run python litellm-rust/crates/redis-resolve/redis_py_kwargs.py + +It reads every other redis-py version's signatures in a throwaway environment through +`uv run --no-project --with redis==`, so nothing else is installed. Then update SETTINGS until +tests/catalog.rs passes. +""" + +import inspect +import json +import subprocess +import sys +from collections.abc import Mapping +from pathlib import Path +from typing import Final + +VERSIONS: Final = ("5.3.1", "6.4.0", "7.4.1", "8.0.1") +PYTHON: Final = "3.12" +CATALOG: Final = Path(__file__).with_name("redis-py-kwargs.json") +# Keys litellm/_redis.py or its callers read that neither a redis-py signature nor the REDIS_* mapping names +LITELLM_ONLY: Final = frozenset( + {"connection_class", "namespace", "sentinel_nodes", "sentinel_password", "service_name"} +) + +META: Final = { + "description": "What redis-py and litellm/_redis.py accept for every Redis kwarg the Rust resolver can be handed. SETTINGS in src/keys.rs has exactly one row per arg here", + "redis_py_versions": list(VERSIONS), + "python": PYTHON, + "method": "python redis_py_kwargs.py next to this file, see its docstring. It reads inspect.getfullargspec(inspect.unwrap(cls.__init__)) args and kwonlyargs, minus self", + "signatures": { + "sync": "redis.Redis.__init__. _get_redis_kwargs uses it as the config allowlist and the REDIS_ environment mapping", + "async": "redis.asyncio.Redis.__init__, which the proxy constructs. A kwarg in sync but not async raises TypeError when the client is built", + "connection": "redis.Connection or redis.asyncio.Connection MRO, the url allowlist from _get_redis_url_kwargs", + "cluster": "redis.asyncio.RedisCluster.__init__, the allowlist from _get_redis_cluster_kwargs", + }, + "environment": "true when _get_redis_env_kwarg_mapping reads the arg as REDIS_ under the locked redis-py", + "redis_py": "each redis-py version whose signatures name the arg, with which ones. An empty object means only litellm reads it", + "checked_by": "tests/unit/test_redis.py checks the installed redis-py's column and the REDIS_* names against this file, and tests/catalog.rs checks SETTINGS against it", +} + + +def _named_args(*classes: type) -> frozenset[str]: + return frozenset( + name + for cls in classes + for spec in (inspect.getfullargspec(inspect.unwrap(cls.__init__)),) + for name in spec.args + spec.kwonlyargs + if name != "self" + ) + + +def arg_signatures() -> dict[str, list[str]]: + """Each kwarg the installed redis-py names, with the signatures that name it.""" + import redis + import redis.asyncio as async_redis + + connection_classes: Final = (*redis.Connection.__mro__, *async_redis.Connection.__mro__) + signatures: Final = { + "sync": _named_args(redis.Redis), + "async": _named_args(async_redis.Redis), + "connection": _named_args(*(cls for cls in connection_classes if cls is not object)), + "cluster": _named_args(async_redis.RedisCluster), + } + names: Final = sorted(frozenset().union(*signatures.values())) + return {name: [kind for kind, named in signatures.items() if name in named] for name in names} + + +def _arg_signatures_of(version: str) -> Mapping[str, list[str]]: + command: Final = ("uv", "run", "--no-project", "--python", PYTHON, "--with", f"redis=={version}") + script: Final = ("python", "-I", __file__, "--arg-signatures") + return json.loads(subprocess.run((*command, *script), check=True, capture_output=True, text=True).stdout) + + +def catalog() -> dict[str, object]: + import redis + + from litellm._redis import _get_redis_env_kwarg_mapping + + environment: Final = frozenset(_get_redis_env_kwarg_mapping().values()) + by_version: Final = {version: _arg_signatures_of(version) for version in VERSIONS} + names: Final = sorted(LITELLM_ONLY | environment | frozenset().union(*by_version.values())) + return { + "meta": {**META, "redis_py_locked": redis.__version__}, + "args": { + name: { + "environment": name in environment, + "redis_py": {version: args[name] for version, args in by_version.items() if name in args}, + } + for name in names + }, + } + + +# one line per arg, so a version bump reads as a diff of the args it touched +def render(catalog: Mapping[str, object]) -> str: + meta: Final = json.dumps(catalog["meta"], indent=2).replace("\n", "\n ") + args: Final = catalog["args"] + assert isinstance(args, dict) + rows: Final = ",\n".join(f" {json.dumps(name)}: {json.dumps(arg)}" for name, arg in args.items()) + return f'{{\n "meta": {meta},\n "args": {{\n{rows}\n }}\n}}\n' + + +if __name__ == "__main__": + if sys.argv[1:] == ["--arg-signatures"]: + sys.stdout.write(json.dumps(arg_signatures())) + else: + CATALOG.write_text(render(catalog())) diff --git a/litellm-rust/crates/redis-resolve/src/auth.rs b/litellm-rust/crates/redis-resolve/src/auth.rs new file mode 100644 index 00000000000..620f1a2e11b --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/auth.rs @@ -0,0 +1,95 @@ +use litellm_redis_types::{Auth, AwsIamAuth, AzureAdAuth, ProviderHandle, StaticAuth}; + +use crate::{ + Error, + options::{self, AwsOptions, Options, flag, secret, text}, +}; + +pub(super) fn redis_auth( + options: &Options, + provider: Option, + redis_username: Option<&str>, +) -> Result { + match (provider, &options.credential_provider) { + (Some(handle), _) => return Ok(Auth::Provider(handle)), + (None, Some(_)) => return Err(Error::ProviderWithoutHandle), + (None, None) => {} + } + let gcp = text(options.gcp.service_account.as_ref()); + let azure = text(options.azure.ad_token.as_ref()) + .is_some_and(|enabled| enabled.eq_ignore_ascii_case("true")); + let aws = flag(options.aws.auth.as_ref())?; + match (gcp, azure, aws) { + (Some(service_account), azure, aws) => { + if azure { + tracing::warn!( + "Both GCP IAM (gcp_service_account) and Azure AD (azure_redis_ad_token) are configured for Redis. Using GCP IAM. Remove one to avoid misconfiguration." + ); + } + if aws { + tracing::warn!( + "Both GCP IAM (gcp_service_account) and AWS ElastiCache IAM (aws_iam_auth) are configured for Redis. Using GCP IAM. Remove one to avoid misconfiguration." + ); + } + Ok(Auth::GcpIam { service_account }) + } + (None, true, aws) => { + if aws { + tracing::warn!( + "Both Azure AD (azure_redis_ad_token) and AWS ElastiCache IAM (aws_iam_auth) are configured for Redis. Using Azure AD. Remove one to avoid misconfiguration." + ); + } + Ok(Auth::AzureAd(AzureAdAuth { + client_id: text(options.azure.client_id.as_ref()), + tenant_id: text(options.azure.tenant_id.as_ref()), + client_secret: secret(options.azure.client_secret.as_ref()), + username: redis_username + .filter(|username| !username.is_empty()) + .map(str::to_owned), + })) + } + (None, false, true) => aws_iam(&options.aws, uses_tls(options)?).map(Auth::AwsIam), + (None, false, false) => Ok(Auth::Static(StaticAuth { + username: secret(options.username.as_ref()), + password: secret(options.password.as_ref()), + })), + } +} + +// _uses_tls: a url's scheme decides unless cluster nodes are set, otherwise the ssl kwarg does +fn uses_tls(options: &Options) -> Result { + match (&options.startup_nodes, text(options.url.as_ref())) { + (None, Some(url)) => Ok(url + .split_once("://") + .is_some_and(|(scheme, _)| scheme.eq_ignore_ascii_case("rediss"))), + _ => flag(options.ssl.as_ref()), + } +} + +fn aws_iam(aws: &AwsOptions, uses_tls: bool) -> Result { + if !uses_tls { + return Err(Error::AwsIamWithoutTls); + } + match ( + text(aws.user_name.as_ref()), + text(aws.cache_name.as_ref()), + text(aws.region.as_ref()), + ) { + (Some(user_name), Some(cache_name), Some(region)) => Ok(AwsIamAuth { + user_name, + cache_name, + region, + serverless: options::flag(aws.serverless.as_ref())?, + }), + (user_name, cache_name, region) => Err(Error::AwsIamIncomplete { + missing: [ + ("aws_iam_user_name", user_name.is_none()), + ("aws_iam_cache_name", cache_name.is_none()), + ("aws_iam_region", region.is_none()), + ] + .into_iter() + .filter_map(|(name, missing)| missing.then_some(name)) + .collect(), + }), + } +} diff --git a/litellm-rust/crates/redis-resolve/src/endpoint.rs b/litellm-rust/crates/redis-resolve/src/endpoint.rs new file mode 100644 index 00000000000..e3722b1b7cc --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/endpoint.rs @@ -0,0 +1,180 @@ +use litellm_redis_types::{ + Auth, Cluster, Node, RedisUrl, Sentinel, Standalone, StaticAuth, Target, TlsOptions, UnixSocket, +}; +use litellm_secret_value::SecretValue; + +use serde_json::Value; + +use crate::{ + Entry, Error, Expected, + options::{Options, as_text, flag, integer, invalid, secret, text}, + tls::{self, QuerySettings}, +}; + +/// The target, first match wins like `get_redis_client`: cluster, url, Sentinel, then host. A +/// url also returns the settings its query carries. +pub(super) fn target(options: &Options, auth: &Auth) -> Result<(Target, QuerySettings), Error> { + if let Some(nodes) = cluster_nodes(options.startup_nodes.as_ref())? { + reject_cluster_db(options.db.as_ref())?; + let cluster = Cluster { + nodes, + tls: tls_for(options, auth)?, + }; + return Ok((Target::Cluster(cluster), QuerySettings::default())); + } + if let Some(url) = text(options.url.as_ref()) { + let query = tls::url_query(&url)?; + let url = RedisUrl { + url: SecretValue::new(match auth { + Auth::Static(_) => url, + _ => without_userinfo(&url), + }), + tls: query.tls, + db: query.db, + username: query.username, + password: query.password, + }; + return Ok((Target::Url(url), query.settings)); + } + if let Some(sentinels) = options.sentinel.nodes.as_ref() { + let sentinel = sentinel(options, auth, sentinels)?; + return Ok((Target::Sentinel(sentinel), QuerySettings::default())); + } + let host = text(options.host.as_ref()).ok_or(Error::NoTarget)?; + let target = match text(options.unix_socket_path.as_ref()) { + Some(path) => Target::Unix(UnixSocket { + path: path.into(), + db: db(options)?, + }), + None => Target::Standalone(Standalone { + host, + port: integer(options.port.as_ref())?.unwrap_or(6379), + tls: tls_for(options, auth)?, + db: db(options)?, + }), + }; + Ok((target, QuerySettings::default())) +} + +fn tls_for(options: &Options, auth: &Auth) -> Result, Error> { + if !flag(options.ssl.as_ref())? { + return Ok(None); + } + let gcp_ca_certs = match auth { + Auth::GcpIam { .. } => options.gcp.ssl_ca_certs.as_ref(), + _ => None, + }; + tls::tls_options(&options.tls, gcp_ca_certs).map(Some) +} + +fn db(options: &Options) -> Result { + integer(options.db.as_ref()).map(|db| db.unwrap_or(0)) +} + +fn sentinel(options: &Options, auth: &Auth, sentinels: &Entry) -> Result { + let monitors = sentinel_nodes(&sentinels.value) + .ok_or_else(|| invalid(&sentinels.source, Expected::SentinelNodes))?; + let service_name = + text(options.sentinel.service_name.as_ref()).ok_or(Error::NoSentinelService)?; + // redis.asyncio.Sentinel passes the connection kwargs to each monitor's Redis(host=..., port=...), so a + // host or port setting raises TypeError when litellm builds the client + if let Some(entry) = options.host.as_ref().or(options.port.as_ref()) { + return Err(Error::SentinelWithHost { + setting: entry.source.clone(), + }); + } + Ok(Sentinel { + monitors, + service_name, + monitor_auth: monitor_auth(options, auth), + tls: tls_for(options, auth)?, + db: db(options)?, + }) +} + +// _sentinel_auth_kwargs gives the monitors the data node's connection kwargs with sentinel_password as the +// password, so a static username reaches them too +fn monitor_auth(options: &Options, auth: &Auth) -> Option { + let password = secret(options.sentinel.password.as_ref())?; + let username = match auth { + Auth::Static(StaticAuth { username, .. }) => username.clone(), + _ => None, + }; + Some(StaticAuth { + username, + password: Some(password), + }) +} + +fn cluster_nodes(entry: Option<&Entry>) -> Result>, Error> { + let Some(entry) = entry else { + return Ok(None); + }; + let nodes = nodes(&entry.value, |node| { + node_address( + node.get("host").and_then(as_text), + node.get("port").and_then(as_text), + ) + }) + .ok_or_else(|| invalid(&entry.source, Expected::StartupNodes))?; + Ok((!nodes.is_empty()).then_some(nodes)) +} + +fn sentinel_nodes(value: &Value) -> Option> { + let nodes = nodes(value, |pair| match pair { + Value::Array(pair) if pair.len() == 2 => node_address(as_text(&pair[0]), as_text(&pair[1])), + _ => None, + })?; + (!nodes.is_empty()).then_some(nodes) +} + +fn nodes(value: &Value, parse: impl Fn(&Value) -> Option) -> Option> { + match value { + Value::String(json) => serde_json::from_str::>(json) + .ok()? + .iter() + .map(parse) + .collect(), + Value::Array(nodes) => nodes.iter().map(parse).collect(), + _ => None, + } +} + +fn node_address(host: Option, port: Option) -> Option { + Some(Node { + host: host?, + port: port?.parse().ok()?, + }) +} + +// redis-py's RedisCluster raises unless db is 0 or None +fn reject_cluster_db(db: Option<&Entry>) -> Result<(), Error> { + let Some(entry) = db else { + return Ok(()); + }; + let is_zero = match &entry.value { + Value::String(_) => matches!(integer::(Some(entry)), Ok(None | Some(0))), + Value::Bool(flag) => !flag, + Value::Number(number) => number.as_f64() == Some(0.0), + _ => false, + }; + match is_zero { + true => Ok(()), + false => Err(Error::DbInCluster { + setting: entry.source.clone(), + }), + } +} + +// _get_redis_url_kwargs keeps only plain Connection kwargs, so redis-py drops every ssl_* setting for a url +// and a credential provider replaces the url's userinfo, which _url_without_userinfo strips +fn without_userinfo(url: &str) -> String { + let Some((scheme, rest)) = url.split_once("://") else { + return url.to_owned(); + }; + let (authority, tail) = rest.split_at(rest.find(['/', '?', '#']).unwrap_or(rest.len())); + let host = authority + .rsplit_once('@') + .map_or(authority, |(_, host)| host); + format!("{scheme}://{host}{tail}") +} diff --git a/litellm-rust/crates/redis-resolve/src/error.rs b/litellm-rust/crates/redis-resolve/src/error.rs new file mode 100644 index 00000000000..b13b65edfb5 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/error.rs @@ -0,0 +1,80 @@ +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error("invalid Redis setting {setting}: expected {expected}")] + Invalid { setting: String, expected: Expected }, + #[error("invalid Redis setting {setting}: redis-py's parse_url cannot read it as a number")] + UrlQuery { setting: String }, + #[error("either host or url must be specified for redis")] + NoTarget, + #[error("service_name is required with sentinel_nodes")] + NoSentinelService, + #[error("{setting} cannot be combined with sentinel_nodes, which redis-py rejects at startup")] + SentinelWithHost { setting: String }, + #[error("{setting} must be 0 in cluster mode")] + DbInCluster { setting: String }, + #[error("credential_provider is a host object, and the host passed no handle for it")] + ProviderWithoutHandle, + #[error( + "{setting} asks for {check}, which rustls cannot perform, so Redis is not connected without them" + )] + Unenforceable { + setting: String, + check: &'static str, + }, + #[error("AWS ElastiCache IAM Redis authentication requires TLS")] + AwsIamWithoutTls, + #[error("AWS ElastiCache IAM Redis authentication requires: {}", missing.join(", "))] + AwsIamIncomplete { missing: Vec<&'static str> }, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::Display)] +pub enum Expected { + #[strum(serialize = "an integer")] + Integer, + #[strum(serialize = "a boolean")] + Boolean, + #[strum(serialize = "a non-negative number of seconds")] + Seconds, + #[strum(serialize = "none, optional, required, null or an ssl.VerifyMode value")] + CertRequirement, + #[strum(serialize = "a list of objects with host and port")] + StartupNodes, + #[strum(serialize = "a non-empty list of [host, port] pairs")] + SentinelNodes, +} + +/// The exception class redis-py 5.3.1 raises building its client. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum PythonException { + ValueError, + TypeError, + RedisClusterException, +} + +impl Error { + /// What `_get_redis_client_logic` and redis-py's async client constructors raise on the same kwargs, or `None` + /// when Python builds the client and fails only once a command opens a connection, or never fails. + pub fn raised_building_the_python_client(&self) -> Option { + match self { + Self::Invalid { expected, .. } => match expected { + Expected::StartupNodes | Expected::SentinelNodes => { + Some(PythonException::ValueError) + } + Expected::Integer + | Expected::Boolean + | Expected::Seconds + | Expected::CertRequirement => None, + }, + Self::UrlQuery { .. } + | Self::NoTarget + | Self::NoSentinelService + | Self::AwsIamWithoutTls + | Self::AwsIamIncomplete { .. } => Some(PythonException::ValueError), + Self::SentinelWithHost { .. } | Self::Unenforceable { .. } => { + Some(PythonException::TypeError) + } + Self::DbInCluster { .. } => Some(PythonException::RedisClusterException), + Self::ProviderWithoutHandle => None, + } + } +} diff --git a/litellm-rust/crates/redis-resolve/src/keys.rs b/litellm-rust/crates/redis-resolve/src/keys.rs new file mode 100644 index 00000000000..c4897701d6f --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/keys.rs @@ -0,0 +1,304 @@ +use litellm_redis_types::Ignored; +use serde_json::Value; + +/// How the Rust client honors a redis-py kwarg. +#[derive(Clone, Copy, Debug)] +pub enum Handling { + /// Modeled by the resolver into a field of `Options`. + Read, + /// Dropped and returned as an `Ignored` naming `consequence`, unless `met` says the value asks for nothing the + /// Rust client does not already do. + Warn { + consequence: &'static str, + met: Option bool>, + }, + /// A security check the Rust client cannot perform and Python's async client already refuses, so asking for it + /// fails instead of connecting without it. + Reject { check: &'static str, asked: Asked }, +} + +/// Which values of a `Reject` row ask for its check. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Asked { + /// A flag read as true. False asks for nothing. + WhenTrue, + /// Any present value, like a list of flags. An empty one asks for nothing. + WhenSet, +} + +/// Whether redis-py's `parse_url` reads the setting from a url's query and the Rust resolver does too. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum FromUrl { + No, + Any, + Rediss, +} + +#[derive(Clone, Copy, Debug)] +pub struct Setting { + pub kwarg: &'static str, + pub handling: Handling, + pub from_url: FromUrl, + /// `_get_redis_env_kwarg_mapping` reads it as `REDIS_`. + pub environment: bool, + /// The names `_merge_redis_kwargs` reads with `get_secret` when the merged kwargs lack the setting. + pub fallback: &'static [&'static str], +} + +impl Setting { + const fn new(kwarg: &'static str, handling: Handling) -> Self { + Self { + kwarg, + handling, + from_url: FromUrl::No, + environment: false, + fallback: &[], + } + } + + const fn env(self) -> Self { + Self { + environment: true, + ..self + } + } + + const fn in_url(self, from_url: FromUrl) -> Self { + Self { from_url, ..self } + } + + const fn or_secrets(self, fallback: &'static [&'static str]) -> Self { + Self { fallback, ..self } + } + + pub fn environment_name(&self) -> Option { + self.environment + .then(|| format!("REDIS_{}", self.kwarg.to_ascii_uppercase())) + } + + pub fn is_read(&self) -> bool { + matches!(self.handling, Handling::Read) + } + + /// The warning a set value earns, or `None` when the setting is read, refused or already met. + pub(crate) fn warning(&self, value: &Value) -> Option<&'static str> { + match self.handling { + Handling::Warn { consequence, met } if !met.is_some_and(|met| met(value)) => { + Some(consequence) + } + _ => None, + } + } +} + +const fn read(kwarg: &'static str) -> Setting { + Setting::new(kwarg, Handling::Read) +} + +const fn warn(kwarg: &'static str, consequence: &'static str) -> Setting { + Setting::new( + kwarg, + Handling::Warn { + consequence, + met: None, + }, + ) +} + +const fn warn_unless( + kwarg: &'static str, + consequence: &'static str, + met: fn(&Value) -> bool, +) -> Setting { + Setting::new( + kwarg, + Handling::Warn { + consequence, + met: Some(met), + }, + ) +} + +const fn reject(kwarg: &'static str, check: &'static str, asked: Asked) -> Setting { + Setting::new(kwarg, Handling::Reject { check, asked }) +} + +const NO_EQUIVALENT: &str = "the Rust client has no equivalent"; +const OWN_STACK: &str = "the Rust client uses its own pool, retries and events"; +const OWN_CONNECTION: &str = "it tunes redis-py's own connection, and the Rust client has its own"; +const OWN_CLUSTER: &str = "the Rust cluster client discovers, routes and retries on its own"; +const MAINTENANCE: &str = "the Rust client does not follow server maintenance notifications"; +const RUSTLS: &str = "rustls negotiates TLS 1.2+ with its own ECDHE AEAD suites"; +const REVOCATION: &str = "OCSP revocation checks"; +const OCSP: &str = "it only configures ssl_validate_ocsp, which the Rust client refuses when set"; +const VERIFY_FLAGS: &str = "X.509 verify flags such as CRL checking"; + +/// Every Redis kwarg redis-py or `litellm/_redis.py` accepts, one row each. `redis-py-kwargs.json` records which +/// redis-py versions and `REDIS_*` names accept them, and `tests/catalog.rs` keeps the two in step. +pub const SETTINGS: &[Setting] = &[ + warn("address_remap", OWN_CLUSTER), + warn("auto_close_connection_pool", OWN_STACK), + read("aws_iam_auth").env(), + read("aws_iam_cache_name").env(), + read("aws_iam_region") + .env() + .or_secrets(&["AWS_REGION", "AWS_DEFAULT_REGION"]), + read("aws_iam_serverless").env(), + read("aws_iam_user_name").env(), + read("azure_client_id") + .env() + .or_secrets(&["AZURE_CLIENT_ID"]), + read("azure_client_secret") + .env() + .or_secrets(&["AZURE_CLIENT_SECRET"]), + read("azure_redis_ad_token") + .env() + .or_secrets(&["REDIS_AZURE_AD_TOKEN"]), + read("azure_tenant_id") + .env() + .or_secrets(&["AZURE_TENANT_ID"]), + warn("cache", NO_EQUIVALENT).env(), + warn("cache_config", NO_EQUIVALENT).env(), + warn("charset", NO_EQUIVALENT).env(), + warn("client_name", NO_EQUIVALENT).env(), + warn("cluster_error_retry_attempts", OWN_CLUSTER), + warn("command_packer", OWN_CONNECTION), + warn("connection_class", OWN_STACK), + warn("connection_error_retry_attempts", OWN_CLUSTER), + warn("connection_pool", OWN_STACK), + read("credential_provider"), + read("db").env().in_url(FromUrl::Any), + warn("decode_responses", NO_EQUIVALENT).env(), + warn("driver_info", NO_EQUIVALENT), + warn("dynamic_startup_nodes", OWN_CLUSTER), + warn("encoder_class", OWN_CONNECTION), + warn("encoding", NO_EQUIVALENT).env(), + warn("encoding_errors", NO_EQUIVALENT).env(), + warn("errors", NO_EQUIVALENT).env(), + warn("event_dispatcher", OWN_STACK).env(), + read("gcp_service_account") + .env() + .or_secrets(&["REDIS_GCP_SERVICE_ACCOUNT"]), + read("gcp_ssl_ca_certs") + .env() + .or_secrets(&["REDIS_GCP_SSL_CA_CERTS"]), + read("health_check_interval").env().in_url(FromUrl::Any), + read("host").env(), + warn("legacy_responses", NO_EQUIVALENT), + warn("lib_name", NO_EQUIVALENT).env(), + warn("lib_version", NO_EQUIVALENT).env(), + warn("load_balancing_strategy", OWN_CLUSTER), + warn("maint_notifications_config", MAINTENANCE), + warn("maint_notifications_pool_handler", MAINTENANCE), + warn("maintenance_notification_hash", MAINTENANCE), + warn("maintenance_state", MAINTENANCE), + warn("max_connections", "the Rust client has no connection cap").env(), + read("namespace"), + warn("orig_host_address", MAINTENANCE), + warn("orig_socket_connect_timeout", MAINTENANCE), + warn("orig_socket_timeout", MAINTENANCE), + warn("oss_cluster_maint_notifications_handler", MAINTENANCE), + warn("parser", OWN_CONNECTION), + warn("parser_class", OWN_CONNECTION), + read("password").env().in_url(FromUrl::Any), + warn("path", OWN_CLUSTER), + warn("policy_resolver", OWN_CLUSTER), + read("port").env(), + warn("protocol", NO_EQUIVALENT).env(), + warn("read_from_replicas", OWN_CLUSTER), + warn( + "redis_connect_func", + "the Python object is not callable from Rust, so static username and password are used", + ) + .env(), + warn("reinitialize_steps", OWN_CLUSTER), + warn("require_full_coverage", OWN_CLUSTER), + warn("retry", OWN_STACK), + warn("retry_on_error", NO_EQUIVALENT).env(), + warn("retry_on_timeout", NO_EQUIVALENT).env(), + read("sentinel_nodes").or_secrets(&["REDIS_SENTINEL_NODES"]), + read("sentinel_password").or_secrets(&["REDIS_SENTINEL_PASSWORD"]), + read("service_name").or_secrets(&["REDIS_SERVICE_NAME"]), + warn("single_connection_client", NO_EQUIVALENT).env(), + read("socket_connect_timeout").env().in_url(FromUrl::Any), + read("socket_keepalive").env(), + warn("socket_keepalive_options", NO_EQUIVALENT).env(), + warn("socket_read_size", OWN_CONNECTION), + read("socket_timeout").env().in_url(FromUrl::Any), + warn("socket_type", OWN_CONNECTION), + read("ssl").env(), + read("ssl_ca_certs").env().in_url(FromUrl::Rediss), + read("ssl_ca_data").env().in_url(FromUrl::Rediss), + read("ssl_ca_path").env().in_url(FromUrl::Rediss), + read("ssl_cert_reqs").env().in_url(FromUrl::Rediss), + read("ssl_certfile").env().in_url(FromUrl::Rediss), + read("ssl_check_hostname").env().in_url(FromUrl::Rediss), + warn("ssl_ciphers", RUSTLS).env(), + reject("ssl_exclude_verify_flags", VERIFY_FLAGS, Asked::WhenSet), + reject("ssl_include_verify_flags", VERIFY_FLAGS, Asked::WhenSet), + read("ssl_keyfile").env().in_url(FromUrl::Rediss), + warn_unless("ssl_min_version", RUSTLS, is_met_by_rustls).env(), + warn("ssl_ocsp_context", OCSP).env(), + warn("ssl_ocsp_expected_cert", OCSP).env(), + read("ssl_password").env().in_url(FromUrl::Rediss), + reject("ssl_validate_ocsp", REVOCATION, Asked::WhenTrue).env(), + reject("ssl_validate_ocsp_stapled", REVOCATION, Asked::WhenTrue).env(), + read("startup_nodes").or_secrets(&["REDIS_CLUSTER_NODES"]), + read("unix_socket_path").env(), + read("url").env(), + read("username").env().in_url(FromUrl::Any), +]; + +// rustls only speaks TLS 1.2 and 1.3, so a minimum of 1.2 or lower is already met. redis-py takes an +// ssl.TLSVersion, which reaches Rust as its name or its value +fn is_met_by_rustls(value: &Value) -> bool { + match value { + Value::Number(number) => matches!(number.as_i64(), Some(-2 | 769 | 770 | 771)), + Value::String(text) => matches!( + text.strip_prefix("TLSVersion.") + .unwrap_or(text) + .to_ascii_uppercase() + .as_str(), + "MINIMUM_SUPPORTED" | "TLSV1" | "TLSV1_1" | "TLSV1_2" + ), + _ => false, + } +} + +pub fn setting(kwarg: &str) -> Option<&'static Setting> { + SETTINGS.iter().find(|setting| setting.kwarg == kwarg) +} + +/// The check a `Reject` row stands for and which values ask for it. +pub(crate) fn refusal(kwarg: &str) -> Option<(&'static str, Asked)> { + match setting(kwarg)?.handling { + Handling::Reject { check, asked } => Some((check, asked)), + Handling::Read | Handling::Warn { .. } => None, + } +} + +/// A warned setting the kwargs set, with what dropping it means. A kwarg with no row is not a Redis setting, so it +/// is dropped without a warning. +pub(crate) fn ignored(source: &str, kwarg: &str, value: &Value) -> Option { + let consequence = setting(kwarg)?.warning(value)?; + Some(Ignored { + setting: source.to_owned(), + consequence, + }) +} + +/// A url query parameter the resolver does not read. redis-py turns every one into a client kwarg, so even one with +/// no row is reported. +pub(crate) fn ignored_query(source: &str, name: &str, value: &Value) -> Option { + let consequence = match setting(name) { + None => NO_EQUIVALENT, + Some(setting) if setting.is_read() => { + "the Rust client does not read it from this url's query" + } + Some(setting) => setting.warning(value)?, + }; + Some(Ignored { + setting: source.to_owned(), + consequence, + }) +} diff --git a/litellm-rust/crates/redis-resolve/src/lib.rs b/litellm-rust/crates/redis-resolve/src/lib.rs new file mode 100644 index 00000000000..b7aa5d18f64 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/lib.rs @@ -0,0 +1,95 @@ +use std::{collections::BTreeMap, time::Duration}; + +use litellm_redis_types::{Auth, HealthCheck, ProviderHandle, Resolution, StaticAuth, Target}; +use serde_json::Value; + +mod auth; +mod endpoint; +mod error; +mod keys; +mod options; +mod tls; + +pub use error::{Error, Expected, PythonException}; +pub use keys::{Asked, FromUrl, Handling, SETTINGS, Setting, setting}; +pub use options::is_present; + +use options::Options; + +/// One merged redis-py kwarg and the name a warning or error reports it under. +#[derive(Clone, Debug, PartialEq)] +pub struct Entry { + pub value: Value, + pub source: String, +} + +pub type Kwargs = BTreeMap; + +/// Kwargs whose every entry is reported under its own kwarg name. +pub fn kwargs(fields: impl IntoIterator) -> Kwargs { + fields + .into_iter() + .map(|(kwarg, value)| { + let entry = Entry { + value, + source: kwarg.clone(), + }; + (kwarg, entry) + }) + .collect() +} + +/// What redis-py's constructors make of kwargs a host already merged the way `_get_redis_client_logic` does. A +/// host passes its `credential_provider` object as `provider`, and `REDIS_USERNAME` as its process environment +/// holds it, which Python's Azure AD auth reads from `os.environ` instead of the merged kwargs. +pub fn interpret( + kwargs: &Kwargs, + provider: Option, + redis_username: Option<&str>, +) -> Result { + let (options, ignored) = Options::read(kwargs)?; + let auth = auth::redis_auth(&options, provider, redis_username)?; + let (target, query) = endpoint::target(&options, &auth)?; + let auth = match (&target, auth) { + (Target::Url(_), Auth::Static(StaticAuth { username, .. })) => Auth::Static(StaticAuth { + username, + password: None, + }), + (_, auth) => auth, + }; + let timeout = query.timeout.map_or_else( + || options::seconds(options.socket_timeout.as_ref()), + |timeout| Ok(Some(timeout)), + )?; + let connect_timeout = query.connect_timeout.map_or_else( + || options::seconds(options.socket_connect_timeout.as_ref()), + |timeout| Ok(Some(timeout)), + )?; + let health_check_interval = query.health_check_interval.map_or_else( + || options::integer::(options.health_check_interval.as_ref()), + |interval| Ok(Some(interval)), + )?; + Ok(Resolution { + target, + auth, + namespace: options::text(options.namespace.as_ref()), + timeout, + connect_timeout, + health_check: health_check(health_check_interval), + keepalive: options + .socket_keepalive + .as_ref() + .map_or(Ok(true), |entry| options::flag(Some(entry)))?, + ignored: ignored.into_iter().chain(query.ignored).collect(), + }) +} + +// redis-py checks health only when the interval is truthy, so zero turns it off and a negative interval checks +// before every command +fn health_check(seconds: Option) -> HealthCheck { + match seconds { + None => HealthCheck::Unset, + Some(0) => HealthCheck::Disabled, + Some(seconds) => HealthCheck::Every(Duration::from_secs(seconds.try_into().unwrap_or(0))), + } +} diff --git a/litellm-rust/crates/redis-resolve/src/options.rs b/litellm-rust/crates/redis-resolve/src/options.rs new file mode 100644 index 00000000000..f2539e46ca5 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/options.rs @@ -0,0 +1,281 @@ +use std::time::Duration; + +use litellm_python_compat::{BigInt, number}; +use litellm_redis_types::Ignored; +use litellm_secret_value::SecretValue; +use serde_json::Value; + +use crate::{ + Entry, Error, Expected, Kwargs, + keys::{self, Asked}, +}; + +/// Every setting the resolver reads. A setting is `None` when it is unset, null or empty, except that a null +/// `tls.cert_reqs` is kept because redis-py reads `ssl_cert_reqs=None` as `CERT_NONE`. +pub(super) struct Options { + pub(super) url: Option, + pub(super) host: Option, + pub(super) port: Option, + pub(super) db: Option, + pub(super) unix_socket_path: Option, + pub(super) startup_nodes: Option, + pub(super) sentinel: SentinelOptions, + pub(super) username: Option, + pub(super) password: Option, + pub(super) credential_provider: Option, + pub(super) gcp: GcpOptions, + pub(super) azure: AzureOptions, + pub(super) aws: AwsOptions, + pub(super) ssl: Option, + pub(super) tls: TlsInput, + pub(super) namespace: Option, + pub(super) socket_timeout: Option, + pub(super) socket_connect_timeout: Option, + pub(super) health_check_interval: Option, + pub(super) socket_keepalive: Option, +} + +pub(super) struct SentinelOptions { + pub(super) nodes: Option, + pub(super) service_name: Option, + pub(super) password: Option, +} + +pub(super) struct GcpOptions { + pub(super) service_account: Option, + pub(super) ssl_ca_certs: Option, +} + +pub(super) struct AzureOptions { + pub(super) ad_token: Option, + pub(super) client_id: Option, + pub(super) tenant_id: Option, + pub(super) client_secret: Option, +} + +pub(super) struct AwsOptions { + pub(super) auth: Option, + pub(super) user_name: Option, + pub(super) cache_name: Option, + pub(super) region: Option, + pub(super) serverless: Option, +} + +/// The TLS settings, read from the kwargs or from a `rediss://` url's query. +pub(super) struct TlsInput { + pub(super) cert_reqs: Option, + pub(super) ca_certs: Option, + pub(super) ca_data: Option, + pub(super) ca_path: Option, + pub(super) certfile: Option, + pub(super) keyfile: Option, + pub(super) key_password: Option, + pub(super) check_hostname: Option, +} + +/// The settings the kwargs set that the Rust client drops, once none of them asks for a check it cannot perform. +fn ignored(kwargs: &Kwargs) -> Result, Error> { + refuse_unenforceable(kwargs.iter())?; + Ok(kwargs + .iter() + .filter_map(|(key, entry)| keys::ignored(&entry.source, key, &entry.value)) + .collect()) +} + +impl Options { + /// The typed settings and the ones the Rust client drops. + pub(super) fn read(kwargs: &Kwargs) -> Result<(Self, Vec), Error> { + let ignored = ignored(kwargs)?; + let options = Self::from_lookup(|key| kwargs.get(key).cloned()); + Ok((options, ignored)) + } + + fn from_lookup(lookup: impl Fn(&'static str) -> Option) -> Self { + let get = |key| lookup(key).filter(|entry| is_present(&entry.value)); + Self { + url: get("url"), + host: get("host"), + port: get("port"), + db: get("db"), + unix_socket_path: get("unix_socket_path"), + startup_nodes: get("startup_nodes"), + sentinel: SentinelOptions { + nodes: get("sentinel_nodes"), + service_name: get("service_name"), + password: get("sentinel_password"), + }, + username: get("username"), + password: get("password"), + credential_provider: get("credential_provider"), + gcp: GcpOptions { + service_account: get("gcp_service_account"), + ssl_ca_certs: get("gcp_ssl_ca_certs"), + }, + azure: AzureOptions { + ad_token: get("azure_redis_ad_token"), + client_id: get("azure_client_id"), + tenant_id: get("azure_tenant_id"), + client_secret: get("azure_client_secret"), + }, + aws: AwsOptions { + auth: get("aws_iam_auth"), + user_name: get("aws_iam_user_name"), + cache_name: get("aws_iam_cache_name"), + region: get("aws_iam_region"), + serverless: get("aws_iam_serverless"), + }, + ssl: get("ssl"), + tls: TlsInput { + cert_reqs: lookup("ssl_cert_reqs") + .filter(|entry| entry.value.is_null() || is_present(&entry.value)), + ca_certs: get("ssl_ca_certs"), + ca_data: get("ssl_ca_data"), + ca_path: get("ssl_ca_path"), + certfile: get("ssl_certfile"), + keyfile: get("ssl_keyfile"), + key_password: get("ssl_password"), + check_hostname: get("ssl_check_hostname"), + }, + namespace: get("namespace"), + socket_timeout: get("socket_timeout"), + socket_connect_timeout: get("socket_connect_timeout"), + health_check_interval: get("health_check_interval"), + socket_keepalive: get("socket_keepalive"), + } + } +} + +/// A `Reject` row whose value asks for its check fails, because Rust cannot perform it. +pub(super) fn refuse_unenforceable<'a>( + entries: impl IntoIterator, +) -> Result<(), Error> { + entries + .into_iter() + .filter_map(|(key, entry)| Some((keys::refusal(key)?, entry))) + .try_for_each(|((check, asked), entry)| { + let asks = match asked { + Asked::WhenTrue => flag(Some(entry))?, + Asked::WhenSet => is_present(&entry.value), + }; + match asks { + true => Err(Error::Unenforceable { + setting: entry.source.clone(), + check, + }), + false => Ok(()), + } + }) +} + +// Python reads most of these with `x or fallback`, so an empty value counts as unset everywhere +pub fn is_present(value: &Value) -> bool { + match value { + Value::Null => false, + Value::String(text) => !text.is_empty(), + Value::Array(items) => !items.is_empty(), + Value::Object(entries) => !entries.is_empty(), + Value::Bool(_) | Value::Number(_) => true, + } +} + +pub(super) fn invalid(setting: &str, expected: Expected) -> Error { + Error::Invalid { + setting: setting.to_owned(), + expected, + } +} + +pub(super) fn as_text(value: &Value) -> Option { + match value { + Value::String(text) => Some(text.clone()), + Value::Number(number) => Some(number.to_string()), + Value::Bool(flag) => Some(flag.to_string()), + _ => None, + } +} + +pub(super) fn text(entry: Option<&Entry>) -> Option { + entry.and_then(|entry| as_text(&entry.value)) +} + +pub(super) fn secret(entry: Option<&Entry>) -> Option { + text(entry).map(SecretValue::new) +} + +/// A flag every Python surface reads the same way. `_str_to_bool` takes only `true`, `1` and `yes` +/// as true while the proxy's pydantic models also take `on`, `t` and `y`, so text they disagree on, +/// or text neither accepts, is an error instead of a guess. +pub(super) fn flag(entry: Option<&Entry>) -> Result { + let Some(entry) = entry else { + return Ok(false); + }; + match &entry.value { + Value::Bool(flag) => Ok(*flag), + Value::Number(number) => Ok(number.as_f64().is_some_and(|number| number != 0.0)), + Value::String(text) => match text.to_ascii_lowercase().as_str() { + "true" | "1" | "yes" => Ok(true), + "false" | "0" | "no" | "off" | "f" | "n" => Ok(false), + _ => Err(invalid(&entry.source, Expected::Boolean)), + }, + _ => Err(invalid(&entry.source, Expected::Boolean)), + } +} + +pub(super) fn integer>(entry: Option<&Entry>) -> Result, Error> { + coerced(entry, Expected::Integer, number::int, |number| { + T::try_from(number).ok() + }) +} + +pub(super) fn seconds(entry: Option<&Entry>) -> Result, Error> { + coerced(entry, Expected::Seconds, number::float, |seconds| { + Duration::try_from_secs_f64(seconds).ok() + }) +} + +fn coerced( + entry: Option<&Entry>, + expected: Expected, + parse: impl Fn(&str) -> Option, + convert: impl Fn(N) -> Option, +) -> Result, Error> { + let Some((entry, text)) = entry.and_then(|entry| Some((entry, as_text(&entry.value)?))) else { + return Ok(None); + }; + match parse(&text) { + Some(number) => convert(number) + .map(Some) + .ok_or_else(|| invalid(&entry.source, expected)), + None => { + tracing::warn!(setting = %entry.source, "Redis setting is not a number, so the default is used like redis-py"); + Ok(None) + } + } +} + +#[cfg(test)] +mod tests { + use std::{cell::RefCell, collections::BTreeSet}; + + use rstest::rstest; + + use super::Options; + use crate::SETTINGS; + + // a Read row that Options never asks for would be neither modeled nor passed through nor warned about + #[rstest] + fn options_ask_for_exactly_the_read_rows() { + let asked = RefCell::new(BTreeSet::new()); + let _ = Options::from_lookup(|kwarg| { + asked.borrow_mut().insert(kwarg); + None + }); + + let read: BTreeSet<&str> = SETTINGS + .iter() + .filter(|setting| setting.is_read()) + .map(|setting| setting.kwarg) + .collect(); + assert_eq!(asked.into_inner(), read); + } +} diff --git a/litellm-rust/crates/redis-resolve/src/tls.rs b/litellm-rust/crates/redis-resolve/src/tls.rs new file mode 100644 index 00000000000..a46a6ccc650 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/src/tls.rs @@ -0,0 +1,189 @@ +use std::time::Duration; + +use litellm_python_compat::number; +use litellm_redis_types::{Ignored, TlsOptions}; +use litellm_secret_value::SecretValue; + +use serde_json::Value; + +use crate::{ + Entry, Error, Expected, + keys::{self, FromUrl}, + options::{self, TlsInput, flag, invalid, secret, text}, +}; + +/// `gcp_ca_certs` replaces `ssl_ca_certs` when GCP IAM is on, as `_get_redis_client_logic` does. +pub(super) fn tls_options( + input: &TlsInput, + gcp_ca_certs: Option<&Entry>, +) -> Result { + Ok(TlsOptions { + ca_certs: text(gcp_ca_certs.or(input.ca_certs.as_ref())).map(Into::into), + ca_data: text(input.ca_data.as_ref()), + ca_path: text(input.ca_path.as_ref()).map(Into::into), + certfile: text(input.certfile.as_ref()).map(Into::into), + keyfile: text(input.keyfile.as_ref()).map(Into::into), + key_password: secret(input.key_password.as_ref()), + verify_certificate: verify_certificate(input.cert_reqs.as_ref())?, + check_hostname: flag(input.check_hostname.as_ref())?, + }) +} + +// RedisSSLContext looks text up case-sensitively and takes an ssl.VerifyMode as is +fn verify_certificate(cert_reqs: Option<&Entry>) -> Result { + let Some(entry) = cert_reqs else { + return Ok(true); + }; + match &entry.value { + // Explicit None intentionally means CERT_NONE, matching redis-py rather than the unset default. + Value::Null => Ok(false), + Value::String(text) if text == "none" => Ok(false), + Value::String(text) if text == "optional" || text == "required" => Ok(true), + Value::Number(mode) => match mode.as_i64() { + Some(0) => Ok(false), + Some(1 | 2) => Ok(true), + _ => Err(invalid(&entry.source, Expected::CertRequirement)), + }, + _ => Err(invalid(&entry.source, Expected::CertRequirement)), + } +} + +/// What redis-py's `parse_url` reads from a url's query on top of the url itself. +pub(super) struct UrlQuery { + pub(super) tls: Option, + pub(super) db: Option, + pub(super) username: Option, + pub(super) password: Option, + pub(super) settings: QuerySettings, +} + +/// The connection settings a url's query sets, which win over the kwargs because `from_url` applies them last. +#[derive(Default)] +pub(super) struct QuerySettings { + pub(super) timeout: Option, + pub(super) connect_timeout: Option, + pub(super) health_check_interval: Option, + pub(super) ignored: Vec, +} + +pub(super) fn url_query(url: &str) -> Result { + let Ok(url) = url::Url::parse(url) else { + return Ok(UrlQuery { + tls: None, + db: None, + username: None, + password: None, + settings: QuerySettings::default(), + }); + }; + let tls_scheme = url.scheme().eq_ignore_ascii_case("rediss"); + let pairs: Vec<(String, Entry)> = url + .query_pairs() + .filter(|(_, value)| !value.is_empty()) + .map(|(name, value)| { + ( + name.to_string(), + Entry { + value: Value::String(value.into_owned()), + source: format!("url ?{name}"), + }, + ) + }) + .collect(); + // parse_qs keeps every value of a repeated parameter and parse_url reads the first + let query: Vec<(String, Entry)> = pairs + .iter() + .enumerate() + .filter(|(index, (name, _))| !pairs[..*index].iter().any(|(earlier, _)| earlier == name)) + .map(|(_, pair)| pair.clone()) + .collect(); + options::refuse_unenforceable(query.iter().map(|(name, entry)| (name, entry)))?; + let (query, ignored): (Vec<_>, Vec<_>) = query.into_iter().partition(|(name, _)| { + keys::setting(name).is_some_and(|setting| match setting.from_url { + FromUrl::Any => true, + FromUrl::Rediss => tls_scheme, + FromUrl::No => false, + }) + }); + let get = |key: &str| { + query + .iter() + .find(|(name, _)| name == key) + .map(|(_, entry)| entry.clone()) + }; + let seconds = |key: &str| { + query_number( + get(key).as_ref(), + number::float, + Expected::Seconds, + |seconds| Duration::try_from_secs_f64(seconds).ok(), + ) + }; + let integer = |key: &str| { + query_number( + get(key).as_ref(), + number::int, + Expected::Integer, + |number| i64::try_from(number).ok(), + ) + }; + let db = integer("db")?; + let settings = QuerySettings { + timeout: seconds("socket_timeout")?, + connect_timeout: seconds("socket_connect_timeout")?, + health_check_interval: integer("health_check_interval")?, + ignored: ignored + .iter() + .filter_map(|(name, entry)| keys::ignored_query(&entry.source, name, &entry.value)) + .collect(), + }; + let tls = match tls_scheme { + true => Some(TlsOptions { + ca_certs: text(get("ssl_ca_certs").as_ref()).map(Into::into), + ca_data: text(get("ssl_ca_data").as_ref()), + ca_path: text(get("ssl_ca_path").as_ref()).map(Into::into), + certfile: text(get("ssl_certfile").as_ref()).map(Into::into), + keyfile: text(get("ssl_keyfile").as_ref()).map(Into::into), + key_password: secret(get("ssl_password").as_ref()), + verify_certificate: verify_certificate(get("ssl_cert_reqs").as_ref())?, + check_hostname: text(get("ssl_check_hostname").as_ref()) + .is_some_and(|text| url_bool(&text)), + }), + false => None, + }; + Ok(UrlQuery { + tls, + db, + username: secret(get("username").as_ref()), + password: secret(get("password").as_ref()), + settings, + }) +} + +// parse_url reads these with int() or float() and raises ValueError building the client on anything else +fn query_number( + entry: Option<&Entry>, + parse: impl Fn(&str) -> Option, + expected: Expected, + convert: impl Fn(N) -> Option, +) -> Result, Error> { + let Some(entry) = entry else { + return Ok(None); + }; + let number = text(Some(entry)) + .and_then(|text| parse(&text)) + .ok_or_else(|| Error::UrlQuery { + setting: entry.source.clone(), + })?; + convert(number) + .map(Some) + .ok_or_else(|| invalid(&entry.source, expected)) +} + +// redis-py's to_bool: only 0, F, FALSE, N and NO (any case) are false +fn url_bool(text: &str) -> bool { + !matches!( + text.to_ascii_uppercase().as_str(), + "" | "0" | "F" | "FALSE" | "N" | "NO" + ) +} diff --git a/litellm-rust/crates/redis-resolve/tests/catalog.rs b/litellm-rust/crates/redis-resolve/tests/catalog.rs new file mode 100644 index 00000000000..b26ad0eebfb --- /dev/null +++ b/litellm-rust/crates/redis-resolve/tests/catalog.rs @@ -0,0 +1,129 @@ +use std::collections::{BTreeMap, BTreeSet}; + +use litellm_redis_resolve::{Asked, Error, Handling, SETTINGS, Setting, interpret, kwargs}; +use rstest::rstest; +use serde::Deserialize; +use serde_json::json; + +#[derive(Deserialize)] +struct Catalog { + meta: Meta, + args: BTreeMap, +} + +#[derive(Deserialize)] +struct Meta { + redis_py_locked: String, +} + +#[derive(Deserialize)] +struct Arg { + environment: bool, + redis_py: BTreeMap>, +} + +// the signatures of the clients the proxy builds, where an unknown kwarg raises TypeError +const PROXY_SIGNATURES: &[&str] = &["async", "cluster", "connection"]; + +fn rejected() -> impl Iterator { + SETTINGS + .iter() + .filter_map(|setting| match setting.handling { + Handling::Reject { asked, .. } => Some((setting, asked)), + Handling::Read | Handling::Warn { .. } => None, + }) +} + +fn catalog() -> Catalog { + serde_json::from_str(include_str!("../redis-py-kwargs.json")).unwrap() +} + +fn host_with(kwarg: &str, value: serde_json::Value) -> litellm_redis_resolve::Kwargs { + kwargs([ + ("host".to_owned(), json!("redis")), + (kwarg.to_owned(), value), + ]) +} + +#[rstest] +fn every_catalogued_kwarg_has_exactly_one_settings_row() { + let rows: Vec<&str> = SETTINGS.iter().map(|setting| setting.kwarg).collect(); + let unique: BTreeSet<&str> = rows.iter().copied().collect(); + assert_eq!(rows.len(), unique.len(), "a kwarg has two SETTINGS rows"); + + let catalog = catalog(); + let catalogued: BTreeSet<&str> = catalog.args.keys().map(String::as_str).collect(); + assert_eq!(unique, catalogued); +} + +#[rstest] +fn settings_read_redis_environment_names_exactly_where_python_does() { + let catalog = catalog(); + + let differing: Vec<&str> = SETTINGS + .iter() + .filter(|setting| { + catalog + .args + .get(setting.kwarg) + .is_some_and(|arg| arg.environment != setting.environment) + }) + .map(|setting| setting.kwarg) + .collect(); + assert!(differing.is_empty(), "{differing:?}"); +} + +#[rstest] +fn every_warned_setting_is_reported() { + let warned = SETTINGS + .iter() + .filter(|setting| matches!(setting.handling, Handling::Warn { met: None, .. })); + + for setting in warned { + let resolved = interpret(&host_with(setting.kwarg, json!("set")), None, None).unwrap(); + + let reported: Vec<&str> = resolved + .ignored + .iter() + .map(|ignored| ignored.setting.as_str()) + .collect(); + assert_eq!(reported, vec![setting.kwarg]); + } +} + +#[rstest] +fn every_rejected_setting_fails_when_asked_for_and_not_otherwise() { + for (setting, asked) in rejected() { + let (asking, not_asking) = match asked { + Asked::WhenTrue => (json!(true), json!(false)), + Asked::WhenSet => (json!(["VERIFY_CRL_CHECK_LEAF"]), json!([])), + }; + + let error = interpret(&host_with(setting.kwarg, asking), None, None).unwrap_err(); + assert!( + matches!(&error, Error::Unenforceable { setting: source, .. } if source == setting.kwarg), + "{error:?}" + ); + assert!(interpret(&host_with(setting.kwarg, not_asking), None, None).is_ok()); + } +} + +#[rstest] +fn python_already_refuses_every_rejected_setting() { + let catalog = catalog(); + + let accepted: Vec<&str> = rejected() + .map(|(setting, _)| setting.kwarg) + .filter(|kwarg| { + catalog.args[*kwarg] + .redis_py + .get(&catalog.meta.redis_py_locked) + .is_some_and(|signatures| { + signatures + .iter() + .any(|signature| PROXY_SIGNATURES.contains(&signature.as_str())) + }) + }) + .collect(); + assert!(accepted.is_empty(), "{accepted:?}"); +} diff --git a/litellm-rust/crates/redis-resolve/tests/interpret.rs b/litellm-rust/crates/redis-resolve/tests/interpret.rs new file mode 100644 index 00000000000..fd59ec2acb1 --- /dev/null +++ b/litellm-rust/crates/redis-resolve/tests/interpret.rs @@ -0,0 +1,790 @@ +use std::collections::BTreeMap; + +use litellm_redis_resolve::{Entry, Error, Expected, Kwargs, PythonException, interpret, kwargs}; +use litellm_redis_types::{ + Auth, HealthCheck, Node, ProviderHandle, Resolution, StaticAuth, Target, TlsOptions, +}; +use rstest::rstest; +use serde_json::json; + +use Outcome::{Fails, Refuses, Resolves}; +use PythonException::{RedisClusterException, TypeError, ValueError}; + +const PROVIDER: ProviderHandle = ProviderHandle::new(7); + +fn merged(fields: serde_json::Value, sources: &[(&str, &str)]) -> Kwargs { + let fields = serde_json::from_value::>(fields).unwrap(); + kwargs(fields) + .into_iter() + .map(|(kwarg, entry)| { + let source = sources + .iter() + .find(|(named, _)| *named == kwarg) + .map_or(entry.source, |(_, source)| (*source).to_owned()); + (kwarg, Entry { source, ..entry }) + }) + .collect() +} + +// a host hands its credential_provider object over as a handle, so the cases do the same +fn interpret_merged( + fields: serde_json::Value, + sources: &[(&str, &str)], + redis_username: Option<&str>, +) -> Result { + let has_provider = fields + .get("credential_provider") + .is_some_and(|provider| !provider.is_null()); + interpret( + &merged(fields, sources), + has_provider.then_some(PROVIDER), + redis_username, + ) +} + +fn interpret_kwargs(fields: serde_json::Value) -> Result { + interpret_merged(fields, &[], None) +} + +/// What a contract case expects of the resolver, next to what its Python mirror expects of redis-py. +enum Outcome { + Resolves(serde_json::Value), + /// Rust refuses a config Python connects with, a disagreement on purpose. + Refuses(&'static str), + /// redis-py 5.3.1 already raises on the config, building the client or its first connection, so failing + /// here breaks no deployment that works today. The class is what it raises building the client, and `None` + /// when it only raises once a command connects. + Fails(&'static str, Option), +} + +#[rstest] +fn a_provider_handle_beats_every_cloud_auth_and_static_credentials() { + let resolved = interpret_kwargs(json!({ + "host": "redis", "password": "pass", "credential_provider": "host-object", + "gcp_service_account": "account", "aws_iam_auth": true, "azure_redis_ad_token": "true", + })) + .unwrap(); + + assert_eq!(resolved.auth, Auth::Provider(PROVIDER)); +} + +#[rstest] +fn a_rediss_url_reads_ssl_password_from_its_query() { + let resolved = + interpret_kwargs(json!({"url": "rediss://cache:6380/0?ssl_password=query-key-secret"})) + .unwrap(); + + let Target::Url(parsed) = resolved.target else { + panic!("expected a url target, got {:?}", resolved.target); + }; + let key_password = parsed.tls.and_then(|tls| tls.key_password); + assert_eq!(expose(key_password.as_ref()), Some("query-key-secret")); + assert!(resolved.ignored.is_empty(), "{:?}", resolved.ignored); +} + +#[rstest] +#[case::read_on_a_plain_url( + "redis://cache:6379/1?db=3&username=query-user&password=query-secret", + Some(3), + Some("query-user"), + Some("query-secret") +)] +#[case::absent("rediss://cache:6380/1", None, None, None)] +fn a_url_query_carries_db_and_credentials_like_parse_url( + #[case] url: &str, + #[case] db: Option, + #[case] username: Option<&str>, + #[case] password: Option<&str>, +) { + let resolved = interpret_kwargs(json!({"url": url})).unwrap(); + + let Target::Url(parsed) = resolved.target else { + panic!("expected a url target, got {:?}", resolved.target); + }; + assert_eq!(parsed.db, db); + assert_eq!(expose(parsed.username.as_ref()), username); + assert_eq!(expose(parsed.password.as_ref()), password); + assert!(resolved.ignored.is_empty(), "{:?}", resolved.ignored); +} + +#[rstest] +fn a_url_query_db_that_is_not_an_integer_fails_like_parse_url() { + let error = interpret_kwargs(json!({"url": "redis://cache:6379?db=first"})).unwrap_err(); + + assert!(matches!(error, Error::UrlQuery { setting } if setting == "url ?db")); +} + +#[rstest] +#[case::tls_1_2_is_already_met(json!("TLSv1_2"), false)] +#[case::the_enum_name(json!("TLSVersion.TLSv1_2"), false)] +#[case::the_enum_value(json!(771), false)] +#[case::tls_1_3_is_not_enforced(json!("TLSv1_3"), true)] +fn ssl_min_version_warns_only_when_rustls_cannot_meet_it( + #[case] version: serde_json::Value, + #[case] warns: bool, +) { + let resolved = + interpret_kwargs(json!({"host": "redis", "ssl": true, "ssl_min_version": version})) + .unwrap(); + + assert_eq!( + !resolved.ignored.is_empty(), + warns, + "{:?}", + resolved.ignored + ); +} + +#[rstest] +#[case::unset(json!({"host": "redis"}), true)] +#[case::empty_like_unset(json!({"host": "redis", "socket_keepalive": ""}), true)] +#[case::enabled(json!({"host": "redis", "socket_keepalive": "yes"}), true)] +#[case::disabled(json!({"host": "redis", "socket_keepalive": false}), false)] +#[case::disabled_as_text(json!({"host": "redis", "socket_keepalive": "0"}), false)] +#[case::disabled_on_a_cluster(json!({"startup_nodes": [{"host": "node-a", "port": 7000}], "socket_keepalive": "false"}), false)] +fn keepalive_stays_on_unless_socket_keepalive_turns_it_off( + #[case] kwargs: serde_json::Value, + #[case] keepalive: bool, +) { + let resolved = interpret_kwargs(kwargs).unwrap(); + + assert_eq!(resolved.keepalive, keepalive); + assert!(resolved.ignored.is_empty(), "{:?}", resolved.ignored); +} + +#[rstest] +fn a_socket_keepalive_no_python_surface_agrees_on_is_invalid() { + let error = interpret_kwargs(json!({"host": "redis", "socket_keepalive": "on"})).unwrap_err(); + + assert!( + matches!(&error, Error::Invalid { setting, expected: Expected::Boolean } if setting == "socket_keepalive"), + "{error:?}" + ); +} + +fn tls_view(tls: Option<&TlsOptions>) -> serde_json::Value { + tls.map_or(serde_json::Value::Null, |tls| { + serde_json::json!({ + "ca_certs": tls.ca_certs, + "ca_data": tls.ca_data, + "ca_path": tls.ca_path, + "certfile": tls.certfile, + "keyfile": tls.keyfile, + "verify_certificate": tls.verify_certificate, + "check_hostname": tls.check_hostname, + }) + }) +} + +fn nodes_view(nodes: &[Node]) -> Vec { + nodes + .iter() + .map(|node| format!("{}:{}", node.host, node.port)) + .collect() +} + +fn expose(value: Option<&litellm_secret_value::SecretValue>) -> Option<&str> { + value.map(litellm_secret_value::SecretValue::expose) +} + +fn view(resolved: &Resolution) -> serde_json::Value { + let target = match &resolved.target { + Target::Unix(socket) => serde_json::json!({ + "mode": "unix", "path": socket.path, "tls": null, + }), + Target::Standalone(host) => serde_json::json!({ + "mode": "host", + "host": host.host, + "port": host.port, + "db": host.db, + "tls": tls_view(host.tls.as_ref()), + }), + Target::Url(url) => serde_json::json!({ + "mode": "url", + "url": url.url.expose(), + "tls": tls_view(url.tls.as_ref()), + }), + Target::Cluster(cluster) => serde_json::json!({ + "mode": "cluster", + "nodes": nodes_view(&cluster.nodes), + "tls": tls_view(cluster.tls.as_ref()), + }), + Target::Sentinel(sentinel) => serde_json::json!({ + "mode": "sentinel", + "nodes": nodes_view(&sentinel.monitors), + "service_name": sentinel.service_name, + "sentinel_password": expose(sentinel.monitor_auth.as_ref().and_then(|auth| auth.password.as_ref())), + "db": sentinel.db, + "tls": tls_view(sentinel.tls.as_ref()), + }), + }; + let auth = match &resolved.auth { + Auth::Static(StaticAuth { username, password }) => serde_json::json!({ + "kind": "static", + "username": expose(username.as_ref()), + "password": expose(password.as_ref()), + }), + Auth::Provider(_) => serde_json::json!({"kind": "host"}), + Auth::GcpIam { service_account } => { + serde_json::json!({"kind": "gcp_iam", "service_account": service_account}) + } + Auth::AzureAd(azure) => { + serde_json::json!({"kind": "azure_ad", "username": azure.username}) + } + Auth::AwsIam(aws) => serde_json::json!({ + "kind": "aws_iam", + "user_name": aws.user_name, + "cache_name": aws.cache_name, + "region": aws.region, + "serverless": aws.serverless, + }), + }; + let serde_json::Value::Object(target) = target else { + unreachable!("every target view is an object") + }; + let rest = serde_json::json!({ + "auth": auth, + "socket_timeout": resolved.timeout.map_or(5.0, |timeout| timeout.as_secs_f64()), + "socket_connect_timeout": resolved.connect_timeout.or(resolved.timeout).map_or(5.0, |timeout| timeout.as_secs_f64()), + "health_check_interval": match (&resolved.target, resolved.health_check) { + (_, HealthCheck::Every(interval)) => Some(interval.as_secs_f64()), + (Target::Cluster(_), HealthCheck::Unset) => Some(25.0), + (_, HealthCheck::Unset | HealthCheck::Disabled) => None, + }, + }); + let serde_json::Value::Object(rest) = rest else { + unreachable!("the view is an object") + }; + serde_json::Value::Object(target.into_iter().chain(rest).collect()) +} + +fn expected_view(fields: serde_json::Value) -> serde_json::Value { + let serde_json::Value::Object(fields) = fields else { + unreachable!("expected views are objects") + }; + let defaults = json!({ + "auth": {"kind": "static", "username": null, "password": null}, + "tls": null, + "socket_timeout": 5.0, + "socket_connect_timeout": 5.0, + "health_check_interval": null, + }); + let serde_json::Value::Object(defaults) = defaults else { + unreachable!("the defaults are an object") + }; + serde_json::Value::Object(defaults.into_iter().chain(fields).collect()) +} + +#[rstest] +#[case::host_defaults( + json!({"host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::empty_cluster_nodes_use_the_environment( + json!({"startup_nodes": "[{\"host\": \"node-a\", \"port\": 7000}]"}), + &[("startup_nodes", "REDIS_CLUSTER_NODES")], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "health_check_interval": 25.0}))), +)] +#[case::empty_sentinel_nodes_use_the_environment( + json!({"sentinel_nodes": "[[\"sentinel-a\", 26379]]", "service_name": "primary"}), + &[("sentinel_nodes", "REDIS_SENTINEL_NODES")], + None, + Resolves(expected_view(json!({"mode": "sentinel", "nodes": ["sentinel-a:26379"], "service_name": "primary", "sentinel_password": null, "db": 0}))), +)] +#[case::empty_service_name_uses_the_environment( + json!({"sentinel_nodes": [["sentinel-a", 26379]], "service_name": "primary"}), + &[("service_name", "REDIS_SERVICE_NAME")], + None, + Resolves(expected_view(json!({"mode": "sentinel", "nodes": ["sentinel-a:26379"], "service_name": "primary", "sentinel_password": null, "db": 0}))), +)] +#[case::plaintext_ignores_tls_requirements( + json!({"host": "redis", "ssl_cert_reqs": "strict"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::an_ambiguous_flag_is_refused( + json!({"host": "redis", "ssl": "on"}), + &[], + None, + Refuses("invalid Redis setting ssl: expected a boolean"), +)] +#[case::empty_sentinel_nodes_without_the_environment_name_no_target( + json!({"sentinel_nodes": [], "service_name": "primary"}), + &[], + None, + Fails("either host or url must be specified for redis", Some(ValueError)), +)] +#[case::ssl_ca_path_is_unsupported( + json!({"host": "redis", "ssl": true, "ssl_ca_path": "/ca"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": "/ca", "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::ssl_validate_ocsp_is_unsupported( + json!({"host": "redis", "ssl": true, "ssl_validate_ocsp": true}), + &[], + None, + Fails("ssl_validate_ocsp asks for OCSP revocation checks, which rustls cannot perform, so Redis is not connected without them", Some(TypeError)), +)] +#[case::ssl_include_verify_flags_is_unsupported( + json!({"host": "redis", "ssl": true, "ssl_include_verify_flags": ["VERIFY_CRL_CHECK_LEAF"]}), + &[], + None, + Fails("ssl_include_verify_flags asks for X.509 verify flags such as CRL checking, which rustls cannot perform, so Redis is not connected without them", Some(TypeError)), +)] +#[case::ssl_password_is_unsupported( + json!({"host": "redis", "ssl": true, "ssl_certfile": "/client.pem", "ssl_password": "key-secret"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": "/client.pem", "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::host_with_every_basic_setting( + json!({"db": 2, "host": "redis", "password": "pass", "port": 6380, "socket_timeout": 1.5, "ssl": true, "username": "user"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6380, "db": 2, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "static", "username": "user", "password": "pass"}, "socket_timeout": 1.5, "socket_connect_timeout": 1.5}))), +)] +#[case::environment_host_port_and_password( + json!({"port": "6390", "ssl": true, "host": "environment", "password": "environment-secret"}), + &[("host", "REDIS_HOST"), ("password", "REDIS_PASSWORD"), ("port", "REDIS_PORT"), ("ssl", "REDIS_SSL")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "environment", "port": 6390, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "static", "username": null, "password": "environment-secret"}}))), +)] +#[case::string_numbers_are_coerced( + json!({"db": "3", "host": "redis", "port": "6380", "socket_timeout": "2.5"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6380, "db": 3, "socket_timeout": 2.5, "socket_connect_timeout": 2.5}))), +)] +#[case::padded_and_underscored_numbers_parse_like_python_int_and_float( + json!({"host": "redis", "port": "6380\n", "db": " 2 ", "socket_timeout": "1_0.5\n"}), + &[], + None, + Resolves(expected_view(json!({"db": 2, "host": "redis", "mode": "host", "port": 6380, "socket_timeout": 10.5, "socket_connect_timeout": 10.5}))), +)] +#[case::padded_environment_numbers_parse_like_python_int( + json!({"port": "6_380", "host": "redis", "db": "3\n"}), + &[("db", "REDIS_DB"), ("host", "REDIS_HOST"), ("port", "REDIS_PORT")], + None, + Resolves(expected_view(json!({"db": 3, "host": "redis", "mode": "host", "port": 6380}))), +)] +#[case::socket_timeout_from_the_environment( + json!({"socket_timeout": "0.75", "host": "redis"}), + &[("socket_timeout", "REDIS_SOCKET_TIMEOUT")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "socket_timeout": 0.75, "socket_connect_timeout": 0.75}))), +)] +#[case::environment_url( + json!({"url": "redis://environment:6379/1"}), + &[("url", "REDIS_URL")], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://environment:6379/1"}))), +)] +#[case::environment_url_beats_the_environment_host( + json!({"host": "ignored", "url": "redis://from-url:6379"}), + &[("host", "REDIS_HOST"), ("url", "REDIS_URL")], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://from-url:6379"}))), +)] +#[case::an_explicit_host_drops_the_environment_url( + json!({"host": "configured"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "configured", "port": 6379, "db": 0}))), +)] +#[case::a_url_drops_host_db_and_password_and_keeps_the_username( + json!({"db": 4, "host": "dropped", "password": "dropped", "url": "rediss://cache:6380", "username": "user"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "rediss://cache:6380", "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "static", "username": "user", "password": null}}))), +)] +#[case::a_url_ignores_tls_settings_outside_the_url( + json!({"ssl_ca_certs": "/ca.pem", "ssl_cert_reqs": "none", "ssl_check_hostname": true, "url": "rediss://cache:6380"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "rediss://cache:6380", "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::startup_nodes_beat_a_url( + json!({"password": "cluster-secret", "startup_nodes": [{"host": "node-a", "port": 7000}], "url": "redis://single:6379"}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "auth": {"kind": "static", "username": null, "password": "cluster-secret"}, "health_check_interval": 25.0}))), +)] +#[case::environment_cluster_nodes( + json!({"startup_nodes": "[{\"host\": \"node-a\", \"port\": \"7000\"}, {\"host\": \"node-b\", \"port\": 7001}]"}), + &[("startup_nodes", "REDIS_CLUSTER_NODES")], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000", "node-b:7001"], "health_check_interval": 25.0}))), +)] +#[case::cluster_tls_options( + json!({"ssl": "true", "ssl_ca_certs": "/ca.pem", "ssl_check_hostname": true, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "tls": {"ca_certs": "/ca.pem", "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": true}, "health_check_interval": 25.0}))), +)] +#[case::sentinel_from_settings( + json!({"password": "data-secret", "sentinel_nodes": [["sentinel-a", 26379], ["sentinel-b", 26380]], "sentinel_password": "sentinel-secret", "service_name": "primary"}), + &[], + None, + Resolves(expected_view(json!({"mode": "sentinel", "nodes": ["sentinel-a:26379", "sentinel-b:26380"], "service_name": "primary", "sentinel_password": "sentinel-secret", "db": 0, "auth": {"kind": "static", "username": null, "password": "data-secret"}}))), +)] +#[case::sentinel_from_the_environment( + json!({"sentinel_nodes": "[[\"sentinel-a\", 26379]]", "sentinel_password": "sentinel-secret", "service_name": "primary"}), + &[("sentinel_nodes", "REDIS_SENTINEL_NODES"), ("sentinel_password", "REDIS_SENTINEL_PASSWORD"), ("service_name", "REDIS_SERVICE_NAME")], + None, + Resolves(expected_view(json!({"mode": "sentinel", "nodes": ["sentinel-a:26379"], "service_name": "primary", "sentinel_password": "sentinel-secret", "db": 0}))), +)] +#[case::a_host_breaks_sentinel( + json!({"host": "ignored", "sentinel_nodes": [["sentinel-a", 26379]], "service_name": "primary"}), + &[], + None, + Fails("host cannot be combined with sentinel_nodes, which redis-py rejects at startup", Some(TypeError)), +)] +#[case::a_url_beats_sentinels( + json!({"sentinel_nodes": [["sentinel-a", 26379]], "service_name": "primary", "url": "redis://single:6379"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://single:6379"}))), +)] +#[case::an_explicit_credential_provider_beats_cloud_auth( + json!({"aws_iam_auth": true, "credential_provider": true, "gcp_service_account": "account", "host": "redis", "password": "dropped"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "auth": {"kind": "host"}}))), +)] +#[case::gcp_beats_azure_and_aws( + json!({"gcp_service_account": "projects/-/serviceAccounts/sa@project.iam.gserviceaccount.com", "aws_iam_auth": true, "azure_redis_ad_token": "True", "host": "redis", "password": "dropped"}), + &[("gcp_service_account", "REDIS_GCP_SERVICE_ACCOUNT")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "auth": {"kind": "gcp_iam", "service_account": "projects/-/serviceAccounts/sa@project.iam.gserviceaccount.com"}}))), +)] +#[case::gcp_ca_certs_replace_the_configured_ones_under_ssl( + json!({"gcp_service_account": "account", "gcp_ssl_ca_certs": "/gcp-ca.pem", "host": "redis", "ssl": true, "ssl_ca_certs": "/configured-ca.pem"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": "/gcp-ca.pem", "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "gcp_iam", "service_account": "account"}}))), +)] +#[case::azure_beats_aws_and_takes_the_environment_username( + json!({"aws_iam_auth": true, "azure_client_id": "client", "azure_client_secret": "secret", "azure_redis_ad_token": true, "azure_tenant_id": "tenant", "host": "redis", "username": "dropped"}), + &[], + Some("environment-user"), + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "auth": {"kind": "azure_ad", "username": "environment-user"}}))), +)] +#[case::an_empty_environment_username_sends_only_the_token( + json!({"azure_client_id": "client", "azure_client_secret": "secret", "azure_redis_ad_token": true, "azure_tenant_id": "tenant", "host": "redis", "username": ""}), + &[("username", "REDIS_USERNAME")], + Some(""), + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "auth": {"kind": "azure_ad", "username": null}}))), +)] +#[case::azure_without_a_username_sends_only_the_token( + json!({"azure_client_id": "client", "azure_client_secret": "secret", "azure_redis_ad_token": true, "azure_tenant_id": "tenant", "host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "auth": {"kind": "azure_ad", "username": null}}))), +)] +#[case::azure_needs_the_literal_true( + json!({"azure_client_id": "client", "azure_client_secret": "secret", "azure_redis_ad_token": "1", "azure_tenant_id": "tenant", "host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::aws_iam_over_tls_with_the_default_region( + json!({"aws_iam_auth": "yes", "aws_iam_cache_name": "cache-1", "aws_iam_user_name": "iam-user", "host": "cache-1", "password": "dropped", "ssl": true, "aws_iam_region": "us-east-1"}), + &[("aws_iam_region", "AWS_DEFAULT_REGION")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "cache-1", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "aws_iam", "user_name": "iam-user", "cache_name": "cache-1", "region": "us-east-1", "serverless": false}}))), +)] +#[case::aws_region_beats_the_default_region( + json!({"aws_iam_auth": true, "aws_iam_cache_name": "cache-1", "aws_iam_user_name": "iam-user", "host": "cache-1", "ssl": true, "aws_iam_region": "eu-central-1"}), + &[("aws_iam_region", "AWS_REGION")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "cache-1", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "aws_iam", "user_name": "iam-user", "cache_name": "cache-1", "region": "eu-central-1", "serverless": false}}))), +)] +#[case::aws_iam_serverless_over_a_rediss_url_drops_its_userinfo( + json!({"aws_iam_auth": true, "aws_iam_cache_name": "cache-1", "aws_iam_region": "eu-west-1", "aws_iam_serverless": true, "aws_iam_user_name": "iam-user", "url": "rediss://user:pass@cache-1:6379"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "rediss://cache-1:6379", "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}, "auth": {"kind": "aws_iam", "user_name": "iam-user", "cache_name": "cache-1", "region": "eu-west-1", "serverless": true}}))), +)] +#[case::aws_iam_without_tls_fails( + json!({"aws_iam_auth": true, "aws_iam_cache_name": "cache-1", "aws_iam_region": "us-east-1", "aws_iam_user_name": "iam-user", "host": "cache-1"}), + &[], + None, + Fails("AWS ElastiCache IAM Redis authentication requires TLS", Some(ValueError)), +)] +#[case::aws_iam_without_a_region_fails( + json!({"aws_iam_auth": true, "aws_iam_cache_name": "cache-1", "aws_iam_user_name": "iam-user", "host": "cache-1", "ssl": true}), + &[], + None, + Fails("AWS ElastiCache IAM Redis authentication requires: aws_iam_region", Some(ValueError)), +)] +#[case::tls_options_from_settings( + json!({"host": "redis", "ssl": true, "ssl_ca_certs": "/ca.pem", "ssl_cert_reqs": "none", "ssl_check_hostname": true}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": "/ca.pem", "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": false, "check_hostname": true}}))), +)] +#[case::a_null_cert_requirement_skips_verification( + json!({"host": "redis", "ssl": true, "ssl_cert_reqs": null}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": false, "check_hostname": false}}))), +)] +#[case::a_verify_mode_of_none_skips_verification( + json!({"host": "redis", "ssl": true, "ssl_cert_reqs": 0}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": false, "check_hostname": false}}))), +)] +#[case::a_verify_mode_of_optional_verifies( + json!({"host": "redis", "ssl": true, "ssl_cert_reqs": 1}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::a_cert_requirement_spelled_like_python_none_fails( + json!({"ssl": true, "ssl_cert_reqs": "None", "host": "redis"}), + &[("ssl", "REDIS_SSL"), ("ssl_cert_reqs", "REDIS_SSL_CERT_REQS")], + None, + Fails("invalid Redis setting REDIS_SSL_CERT_REQS: expected none, optional, required, null or an ssl.VerifyMode value", None), +)] +#[case::tls_options_from_the_environment( + json!({"ssl_ca_certs": "/env-ca.pem", "ssl": true, "ssl_cert_reqs": "optional", "host": "redis"}), + &[("ssl", "REDIS_SSL"), ("ssl_ca_certs", "REDIS_SSL_CA_CERTS"), ("ssl_cert_reqs", "REDIS_SSL_CERT_REQS")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": "/env-ca.pem", "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::connect_timeout_is_separate_from_the_socket_timeout( + json!({"host": "redis", "socket_connect_timeout": 0.25, "socket_timeout": 2.5}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "socket_timeout": 2.5, "socket_connect_timeout": 0.25}))), +)] +#[case::environment_references_resolve( + json!({"host": "referenced", "password": "referenced-secret"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "referenced", "port": 6379, "db": 0, "auth": {"kind": "static", "username": null, "password": "referenced-secret"}}))), +)] +#[case::no_target_fails( + json!({"port": 6379}), + &[], + None, + Fails("either host or url must be specified for redis", Some(ValueError)), +)] +#[case::client_certificates( + json!({"host": "redis", "ssl": true, "ssl_certfile": "/client.pem", "ssl_keyfile": "/client.key"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": "/client.pem", "keyfile": "/client.key", "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::a_rediss_url_takes_tls_options_from_its_query( + json!({"url": "rediss://cache:6380/0?ssl_cert_reqs=none&ssl_ca_certs=/ca.pem&ssl_check_hostname=yes"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "rediss://cache:6380/0?ssl_cert_reqs=none&ssl_ca_certs=/ca.pem&ssl_check_hostname=yes", "tls": {"ca_certs": "/ca.pem", "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": false, "check_hostname": true}}))), +)] +#[case::a_url_query_carries_socket_timeout( + json!({"url": "redis://cache:6379/0?socket_timeout=2"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://cache:6379/0?socket_timeout=2", "socket_timeout": 2.0, "socket_connect_timeout": 2.0}))), +)] +#[case::url_query_settings_win_over_the_kwargs( + json!({"health_check_interval": 4, "socket_connect_timeout": 7, "socket_timeout": 1, "url": "redis://cache:6379/0?socket_connect_timeout=0.5&health_check_interval=9&socket_timeout=3"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://cache:6379/0?socket_connect_timeout=0.5&health_check_interval=9&socket_timeout=3", "socket_timeout": 3.0, "socket_connect_timeout": 0.5, "health_check_interval": 9.0}))), +)] +#[case::a_repeated_url_query_parameter_takes_its_first_value( + json!({"url": "redis://cache:6379/0?socket_timeout=2&socket_timeout=9"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "redis://cache:6379/0?socket_timeout=2&socket_timeout=9", "socket_timeout": 2.0, "socket_connect_timeout": 2.0}))), +)] +#[case::a_url_query_number_parse_url_cannot_read( + json!({"url": "redis://cache:6379/0?socket_timeout=slow"}), + &[], + None, + Fails("invalid Redis setting url ?socket_timeout: redis-py's parse_url cannot read it as a number", Some(ValueError)), +)] +#[case::tls_settings_outside_a_url_never_fail_it( + json!({"db": "99999999999999999999", "ssl_cert_reqs": "strict", "url": "rediss://cache:6380"}), + &[], + None, + Resolves(expected_view(json!({"mode": "url", "url": "rediss://cache:6380", "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::db_is_not_read_for_a_cluster( + json!({"db": 99_999_999_999_i64, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Fails("db must be 0 in cluster mode", Some(RedisClusterException)), +)] +#[case::cluster_db_true( + json!({"db": true, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Fails("db must be 0 in cluster mode", Some(RedisClusterException)), +)] +#[case::cluster_db_false( + json!({"db": false, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "health_check_interval": 25.0}))), +)] +#[case::cluster_db_fraction( + json!({"db": 0.5, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Fails("db must be 0 in cluster mode", Some(RedisClusterException)), +)] +#[case::cluster_db_float_zero( + json!({"db": 0.0, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "health_check_interval": 25.0}))), +)] +#[case::cluster_db_nonempty_list( + json!({"db": [1], "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Fails("db must be 0 in cluster mode", Some(RedisClusterException)), +)] +#[case::cluster_db_empty_list( + json!({"db": [], "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"], "health_check_interval": 25.0}))), +)] +#[case::ssl_ciphers( + json!({"host": "redis", "ssl": true, "ssl_ciphers": "HIGH"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::retry_on_timeout( + json!({"host": "redis", "retry_on_timeout": true}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::malformed_port( + json!({"host": "redis", "port": "abc"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::malformed_db( + json!({"db": "first", "host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::socket_timeout_not_a_number( + json!({"host": "redis", "socket_timeout": "slow"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::malformed_port_from_the_environment( + json!({"port": "abc", "host": "redis"}), + &[("host", "REDIS_HOST"), ("port", "REDIS_PORT")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::protocol_3( + json!({"host": "redis", "protocol": 3}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::retry_on_error_from_the_environment( + json!({"retry_on_error": true, "host": "redis"}), + &[("retry_on_error", "REDIS_RETRY_ON_ERROR")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::unix_socket_path( + json!({"host": "redis", "unix_socket_path": "/tmp/redis.sock"}), + &[], + None, + Resolves(expected_view(json!({"mode": "unix", "path": "/tmp/redis.sock"}))), +)] +#[case::unix_socket_path_keeps_tls_settings_out_of_the_target( + json!({"host": "redis", "ssl": true, "ssl_ca_certs": "/ca.pem", "unix_socket_path": "/tmp/redis.sock"}), + &[], + None, + Resolves(expected_view(json!({"mode": "unix", "path": "/tmp/redis.sock"}))), +)] +#[case::ssl_ca_data( + json!({"host": "redis", "ssl": true, "ssl_ca_data": "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----", "ca_path": null, "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::ssl_ca_path_from_the_environment( + json!({"ssl": true, "ssl_ca_path": "/etc/ssl/private-cas", "host": "redis"}), + &[("ssl", "REDIS_SSL"), ("ssl_ca_path", "REDIS_SSL_CA_PATH")], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "tls": {"ca_certs": null, "ca_data": null, "ca_path": "/etc/ssl/private-cas", "certfile": null, "keyfile": null, "verify_certificate": true, "check_hostname": false}}))), +)] +#[case::health_check_interval_is_whole_seconds( + json!({"health_check_interval": "7", "host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0, "health_check_interval": 7.0}))), +)] +#[case::health_check_interval_zero_is_off( + json!({"health_check_interval": 0, "host": "redis"}), + &[], + None, + Resolves(expected_view(json!({"mode": "host", "host": "redis", "port": 6379, "db": 0}))), +)] +#[case::health_check_interval_zero_keeps_a_cluster_off( + json!({"health_check_interval": 0, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Resolves(expected_view(json!({"mode": "cluster", "nodes": ["node-a:7000"]}))), +)] +#[case::a_cluster_rejects_a_nonzero_db( + json!({"db": 3, "startup_nodes": [{"host": "node-a", "port": 7000}]}), + &[], + None, + Fails("db must be 0 in cluster mode", Some(RedisClusterException)), +)] +#[case::unix_socket_path_without_a_host_fails_like_python( + json!({"unix_socket_path": "/tmp/redis.sock"}), + &[], + None, + Fails("either host or url must be specified for redis", Some(ValueError)), +)] +fn resolves_like_python_redis( + #[case] merged: serde_json::Value, + #[case] sources: &[(&str, &str)], + #[case] redis_username: Option<&str>, + #[case] expected: Outcome, +) { + let outcome = interpret_merged(merged, sources, redis_username); + let rendered = outcome + .as_ref() + .map(view) + .map_err(|error| error.to_string()); + + match expected { + Resolves(expected) => assert_eq!(rendered, Ok(expected)), + Refuses(message) => assert_eq!(rendered, Err(message.to_owned())), + Fails(message, raised) => { + assert_eq!(rendered, Err(message.to_owned())); + assert_eq!( + outcome + .err() + .and_then(|error| error.raised_building_the_python_client()), + raised + ); + } + } +} diff --git a/litellm-rust/crates/redis-store/AGENTS.md b/litellm-rust/crates/redis-store/AGENTS.md new file mode 100644 index 00000000000..cdd2bd7ca64 --- /dev/null +++ b/litellm-rust/crates/redis-store/AGENTS.md @@ -0,0 +1,13 @@ +- One folder per Redis family (`router_cooldowns/`, `router_usage/`, `spend_counters/`, `auth_cache/`, `rate_limits/`), each with `mod.rs` (the store and its ops), `keys.rs` (key types and spellings) and `types.rs` (the family's values, with the `FromValue` / `IntoValue` codecs for what it stores in Redis). It is the only place that family's keys, value codec and scripts are written, and it takes domain ids, never keys. Shared pieces live once at the crate root: `Error` and `ErrorKind` (`error.rs`), `Memory` and `Clock` (`memory.rs`), `Pending` (`request.rs`), `Remote` and its attach-once `RemoteSlot`, which knows nothing of memory (`remote.rs`), `DualCache` (`dual_cache/`) and the Python codecs (`codec.rs`, `value.rs`) +- An op declared on a `Send` returns `Pending`, a lazy future built when the op is declared. An op that only means something on Redis (an `MGET` the caller falls back from, a script) returns `None` without one, and an op that replaced a `DualCache` call answers from memory. A post-call method returns `false` when there was no Redis to declare on +- A store that sends a pipeline of its own uses `RedisBatch::pipelined`, never `Pipeline` and raw `Value`s +- Python and Rust pods share these keys during a rolling deploy, so a module writes the bytes `main`'s Python writes and reads everything `main` reads. Python formats go through `litellm-python-compat` (`repr`, `literal_eval`, `json`), never a hand-rolled codec +- A family holds one `DualCache` per `DualCache` its Python held on `main`, and each `DualCache` method is the Python method it names, so a family op that was a `DualCache` call is one call here. Where `main`'s Python reached past `DualCache` to `in_memory_cache` or `redis_cache`, the family does the same through `memory()` and `redis()`, which keeps that reach visible. Caches a family's Python attached to one `RedisCache` share one `Arc`. Only a family that batch-reads builds its cache with `DualCache::batch_reading` from a `TierPolicy` the host passes in, and the Python host fills it from what `main` reads from the environment or `litellm` globals (`litellm/rust_bridge/host/settings.py`, `router_tiers`). Callers pass a `Remote` (or none), never a cache. Redis-only families are plain functions over `&RedisClient` +- Every family works with no Redis, answering from memory. `attach` adds the Redis tier at most once and keeps memory, as `DualCache.attach_redis_cache` does, so a host builds the store at startup and never rebuilds it to connect +- Where `main` swallows a Redis failure and serves memory, the family does the same and logs. It never turns a failure into a miss that `main` would not +- Key suffixes and wire spellings derive from strum on the family's enums. A value the family stores in Redis implements `FromValue` / `IntoValue` over `litellm_python_compat::Value`. An argument or reply is a typed record, never a positional tuple: a frozen Python dataclass read in `python-bridge` by a `#[derive(FromPyObject)]` struct with `#[pyo3(attribute)]` fields and converted into the family's type with `From`, and a reply returned as a frozen `#[pyclass]`. The bridge reads an enum through its strum spelling, never a hand-written match +- Wire contracts are hand-written cases with expected values, inlined as the same literals in this crate's test and in `tests/unit/redis/stores/`, checked here against real Redis and there against Python's own codec. Change both sides together +- `rate_limits.rs` keeps the v3 limiter's Lua byte for byte (`rate_limits/lua.rs`) so pods on either stack share one script SHA; the admission decisions stay in the Python limiter +- `RequestRedis` is one request's batches: a pre-call and a post-call `RedisBatch` per client, the post one flushed by `flush_post` or its deadline. Each op exists once and takes a `Send` (`Now`, `PreCall`, `PostCall`) picked by the caller, and the store calls `send.batch(&client)`. A store never encodes a phase in a method name. A post-call method exists only where a follow-up runs after the write settles (`token_increment_post_call`), as a thin wrapper over the shared op. `Send::Now` is an alone batch, so a script call outside a request goes out alone, like a redis-py `Script` outside a pipeline, so `NOSCRIPT` costs `SCRIPT LOAD` and one more `EVALSHA` the way it does on main. Only what main's Python puts on its request pipeline rides these batches. An op main sends on its own round trip, like a deployment's tpm and rpm increment, stays on its own round trip here +- Every store sends the commands main's `RedisCache` sends for that operation: `MGET` where Python reads several keys at once, `SET ... EX`, and `INCRBYFLOAT` with `EXPIRE` or `TTL`. A `MONITOR` diff against main is a parity check, and any difference is a defect +- Tests run against `litellm-redis-testing` on standalone and cluster. A test is single-topology only for outages diff --git a/litellm-rust/crates/redis-store/Cargo.toml b/litellm-rust/crates/redis-store/Cargo.toml new file mode 100644 index 00000000000..1dd61d76d3c --- /dev/null +++ b/litellm-rust/crates/redis-store/Cargo.toml @@ -0,0 +1,25 @@ +[package] +name = "litellm-redis-store" +version = "0.1.0" +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +futures-util.workspace = true +litellm-cache.workspace = true +litellm-cache-memory.workspace = true +litellm-python-compat.workspace = true +litellm-redis-client.workspace = true +strum.workspace = true +thiserror.workspace = true +tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] } +tracing.workspace = true + +[dev-dependencies] +litellm-circuit-breaker.workspace = true +litellm-redis-testing.workspace = true +litellm-secret-value.workspace = true +litellm-tracing.workspace = true +rstest.workspace = true +tokio.workspace = true diff --git a/litellm-rust/crates/redis-store/src/auth_cache/keys.rs b/litellm-rust/crates/redis-store/src/auth_cache/keys.rs new file mode 100644 index 00000000000..feb87a40b00 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/auth_cache/keys.rs @@ -0,0 +1,8 @@ +/// `is_user_key_cache_key`: user-key objects are cached under a bare `hash_token` digest, every +/// other object under a prefixed key. +pub fn is_key_object(key: &str) -> bool { + key.len() == 64 + && key + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} diff --git a/litellm-rust/crates/redis-store/src/auth_cache/mod.rs b/litellm-rust/crates/redis-store/src/auth_cache/mod.rs new file mode 100644 index 00000000000..b5a83b5aec6 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/auth_cache/mod.rs @@ -0,0 +1,258 @@ +mod keys; +mod types; + +use std::{ + sync::{Arc, Mutex}, + time::Duration, +}; + +use litellm_python_compat::{Value, json}; +use litellm_redis_client::{Declared, RedisBatch, commands}; + +pub use self::keys::is_key_object; +pub use self::types::Settings; +use crate::{ + Error, Number, Pending, RequestRedis, Send, + dual_cache::DualCache, + memory::{Clock, lock, system_clock}, + remote::{Remote, RemoteSlot}, + request::ready, +}; + +/// `RedisCache.async_increment_if_present`, byte for byte, sent as `EVAL` like it. +const INCREMENT_IF_PRESENT: &str = "if redis.call('EXISTS', KEYS[1]) == 0 then return false end return redis.call('INCRBYFLOAT', KEYS[1], ARGV[1])"; + +/// The proxy's auth object cache with `enable_redis_auth_cache` on: `UserApiKeyCache`'s two +/// `DualCache` partitions, so churn in management objects cannot evict user-key objects, in front of +/// one Redis shared by every pod. Keys are the logical cache keys `UserApiKeyCache` callers already +/// use; values are what `json.loads` returns, written as `json.dumps`, like `RedisCache.async_set_cache`. +pub struct AuthCache { + key_objects: DualCache, + objects: DualCache, + redis: Arc, + settings: Mutex, +} + +impl AuthCache { + pub fn new(remote: Option, settings: Settings) -> Self { + Self::with_clock(remote, settings, system_clock()) + } + + pub fn with_clock(remote: Option, settings: Settings, now: Clock) -> Self { + let redis = Arc::new(RemoteSlot::new(remote)); + let partition = || DualCache::new(redis.clone(), settings.memory(), now.clone()); + Self { + key_objects: partition(), + objects: partition(), + redis, + settings: Mutex::new(settings), + } + } + + pub fn has_redis(&self) -> bool { + self.redis.get().is_some() + } + + pub fn settings(&self) -> Settings { + *lock(&self.settings) + } + + /// `UserApiKeyCache.update_cache_ttl` and `update_in_memory_max_size` on both partitions, which + /// keep what they hold. + pub fn reconfigure(&self, settings: Settings) { + *lock(&self.settings) = settings; + self.key_objects.reconfigure(settings.memory()); + self.objects.reconfigure(settings.memory()); + } + + pub fn get_local(&self, key: &str) -> Result, Error> { + Ok(self.partition(key).memory().get(key)?) + } + + /// A memory write with `ttl`, or `default_in_memory_ttl` when it names none. + pub fn set_local(&self, key: &str, value: Value, ttl: Option) -> Result<(), Error> { + let partition = self.partition(key); + Ok(partition + .memory() + .set(key, value, ttl.or(partition.default_in_memory_ttl()))?) + } + + pub fn delete_local(&self, key: &str) -> Result<(), Error> { + Ok(self.partition(key).memory().delete(key)?) + } + + pub fn flush_local(&self) -> Result<(), Error> { + self.key_objects.memory().flush()?; + Ok(self.objects.memory().flush()?) + } + + /// Entries held by the management-object and the user-key partitions. + pub fn len_local(&self) -> Result<(usize, usize), Error> { + Ok(( + self.objects.memory().len()?, + self.key_objects.memory().len()?, + )) + } + + /// `DualCache.async_get_cache`. A key `request` already read as missing from Redis is not read + /// again. + pub async fn get( + &self, + request: Option<&RequestRedis>, + key: &str, + ) -> Result, Error> { + self.partition(key).get(request, key, None).await + } + + /// `DualCache.async_set_cache`, writing `json.dumps(value)`. + pub async fn set(&self, key: &str, value: Value, ttl: Option) -> Result<(), Error> { + self.partition(key).set(key, value, ttl).await + } + + /// `DualCache.set_cache`, whose `RedisCache.set_cache` writes `str(value)`. + pub async fn set_str( + &self, + key: &str, + value: Value, + ttl: Option, + ) -> Result<(), Error> { + self.partition(key).set_str(key, value, ttl).await + } + + /// `DualCache.async_set_cache_pre_call`: memory now, the `SET` on the batch `send` picks. + pub fn declare_set( + &self, + send: Send<'_>, + key: &str, + value: Value, + ttl: Option, + ) -> Result, Error> { + self.partition(key).declare_set(send, key, value, ttl) + } + + /// `DualCache.async_delete_cache_pre_call`, memory now and the `DEL` on the batch `send` picks. + pub fn declare_delete(&self, send: Send<'_>, key: &str) -> Result, Error> { + self.partition(key).declare_delete(send, key) + } + + /// `UserApiKeyCache.async_set_cache_pipeline`: the key-object entries in one pipeline, then + /// the rest in another, each with one TTL. + pub async fn set_many( + &self, + entries: Vec<(String, Value)>, + ttl: Option, + ) -> Result<(), Error> { + let (key_objects, others): (Vec<_>, Vec<_>) = + entries.into_iter().partition(|(key, _)| is_key_object(key)); + self.key_objects.set_pipeline(key_objects, ttl).await?; + self.objects.set_pipeline(others, ttl).await + } + + /// The auth prefetch write-back: memory first, an entry without a TTL taking + /// `default_in_memory_ttl`, then `RedisCache.async_set_cache_pipeline_with_ttls`, which encodes + /// every value before it sends anything and gives an entry without a TTL the connection's + /// default. On a request batch the `SET`s are declared and nothing is left to wait for. + /// [`Send::Now`] is one pipeline of its own rather than one round trip per entry. + pub fn write_back( + &self, + send: Send<'_>, + entries: Vec<(String, Value, Option)>, + ) -> Result, Error> { + let encoded = self + .redis + .get() + .map(|_| { + entries + .iter() + .map(|(_, value, _)| json::dumps(value)) + .collect::, _>>() + }) + .transpose(); + let writes = entries + .into_iter() + .map(|(key, value, ttl)| { + self.set_local(&key, value, ttl)?; + Ok((key, ttl)) + }) + .collect::, Error>>()?; + let (Some(remote), Some(encoded)) = (self.redis.get(), encoded?) else { + return Ok(ready(())); + }; + let sets = writes.iter().zip(encoded).map(|((key, ttl), value)| { + commands::set(key, value, ttl.unwrap_or(remote.default_ttl)) + }); + let Send::Now = send else { + let batch = send.batch(&remote.client); + sets.for_each(|set| drop(batch.add(set))); + return Ok(ready(())); + }; + let pipeline = RedisBatch::pipelined(remote.client.clone()); + let written: Vec<_> = sets.map(|set| pipeline.add(set)).collect(); + Ok(Box::pin(async move { + Declared::all(written) + .await + .into_iter() + .try_for_each(|reply| Ok(reply?)) + })) + } + + /// `UserApiKeyCache.async_delete_cache_keys`: the key-object partition and the rest as two + /// concurrent `DualCache.async_delete_cache_keys`. Both run even when one fails. + pub async fn delete(&self, keys: &[String]) -> Result<(), Error> { + let (key_objects, others): (Vec, Vec) = + keys.iter().cloned().partition(|key| is_key_object(key)); + let (key_objects, others) = tokio::join!( + self.key_objects.delete_keys(&key_objects), + self.objects.delete_keys(&others) + ); + key_objects.and(others) + } + + /// `RedisCache.async_batch_get_cache` for keys the caller found missing in memory, one `MGET` on + /// the batch `send` picks, noting the keys Redis lacks as the request's misses. `None` without + /// Redis. Any failed read or undecodable value fails the whole read, as a failed `MGET` does. + pub fn get_many(&self, send: Send<'_>, keys: &[String]) -> Option>>> { + let client = self.redis.client()?; + let misses = send.misses(client); + let read = send.batch(client).add(commands::mget::>( + &keys.iter().map(String::as_str).collect::>(), + )); + let keys = keys.to_vec(); + Some(Box::pin(async move { + let raw = read.resolve().await?; + if let Some(misses) = misses { + misses.note( + keys.into_iter() + .zip(&raw) + .filter(|(_, raw)| raw.is_none()) + .map(|(key, _)| key), + ); + } + raw.into_iter().map(|raw| Ok(types::decode(raw)?)).collect() + })) + } + + /// `RedisCache.async_increment_if_present`: `INCRBYFLOAT` only an entry Redis still holds. + /// `None` without Redis or for an absent entry. + pub async fn increment_if_present( + &self, + key: &str, + amount: Number, + ) -> Result, Error> { + let Some(client) = self.redis.client() else { + return Ok(None); + }; + let amount = litellm_redis_client::Amount::from(amount); + Ok(client + .send(commands::eval_source(INCREMENT_IF_PRESENT, &[key], amount)) + .await?) + } + + fn partition(&self, key: &str) -> &DualCache { + if is_key_object(key) { + &self.key_objects + } else { + &self.objects + } + } +} diff --git a/litellm-rust/crates/redis-store/src/auth_cache/types.rs b/litellm-rust/crates/redis-store/src/auth_cache/types.rs new file mode 100644 index 00000000000..e933ac19af9 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/auth_cache/types.rs @@ -0,0 +1,35 @@ +use std::time::Duration; + +use litellm_python_compat::Value; + +use crate::{DecodeError, codec::python_value, dual_cache::MemorySettings}; + +/// `UserApiKeyCache`'s memory configuration: each partition's size and `default_in_memory_ttl`. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Settings { + pub memory_entries: usize, + pub memory_ttl: Option, +} + +impl Default for Settings { + /// `InMemoryCache`'s 200 entries. + fn default() -> Self { + Self { + memory_entries: 200, + memory_ttl: None, + } + } +} + +impl Settings { + pub(super) fn memory(self) -> MemorySettings { + MemorySettings { + entries: Some(self.memory_entries), + ttl: self.memory_ttl, + } + } +} + +pub(super) fn decode(raw: Option>) -> Result, DecodeError> { + raw.as_deref().map(python_value).transpose() +} diff --git a/litellm-rust/crates/redis-store/src/codec.rs b/litellm-rust/crates/redis-store/src/codec.rs new file mode 100644 index 00000000000..21cde7d0b81 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/codec.rs @@ -0,0 +1,74 @@ +use litellm_python_compat::{Value, json, literal, repr::to_str}; + +use crate::{Error, FromValue, IntoValue, Number}; + +#[derive(Debug, thiserror::Error)] +pub enum DecodeError { + #[error("Redis value is not UTF-8")] + NotUtf8, + #[error("Redis value is neither JSON nor a Python literal: {0}")] + Unparsed(#[source] litellm_python_compat::Error), + #[error("Redis value has the wrong shape: {0}")] + Shape(&'static str), +} + +/// `RedisCache._get_cache_logic`: `json.loads`, falling back to `ast.literal_eval`. +pub(crate) fn python_value(raw: &[u8]) -> Result { + let text = std::str::from_utf8(raw).map_err(|_| DecodeError::NotUtf8)?; + json::loads(text).or_else(|_| literal::literal_eval(text).map_err(DecodeError::Unparsed)) +} + +/// A cached value as `RedisCache._get_cache_logic` reads it. +pub(crate) fn decode(raw: &[u8]) -> Result { + V::from_value(&python_value(raw)?) +} + +/// `json.dumps(value)`, what `RedisCache.async_set_cache` writes. +pub(crate) fn dumps(value: V) -> Result { + Ok(json::dumps(&value.into_value())?) +} + +/// `str(value)`, what the sync `RedisCache.set_cache` writes. +pub(crate) fn str(value: V) -> String { + to_str(&value.into_value()) +} + +/// A number written by Python, `shape` naming what it was meant to be when it is not one. +pub(crate) fn number(raw: &[u8], shape: &'static str) -> Result { + Number::from_value(&python_value(raw)?).ok_or(DecodeError::Shape(shape)) +} + +#[cfg(test)] +mod tests { + use litellm_python_compat::{Value, json, repr::repr}; + use rstest::rstest; + + use super::{number, python_value}; + use crate::Number; + + // Expected values are the repr of main's RedisCache._get_cache_logic on these bytes + #[rstest] + #[case::nan(b"NaN", "nan")] + #[case::infinity(b"Infinity", "inf")] + #[case::negative_infinity(b"-Infinity", "-inf")] + #[case::nan_inside_a_document(br#"{"spend": NaN}"#, "{'spend': nan}")] + #[case::python_literal_fallback(b"{'a': (1, 2)}", "{'a': (1, 2)}")] + fn reads_what_python_reads(#[case] raw: &[u8], #[case] expected: &str) { + assert_eq!(repr(&python_value(raw).unwrap()), expected); + } + + #[rstest] + #[case::nan(f64::NAN)] + #[case::infinity(f64::INFINITY)] + #[case::negative_infinity(f64::NEG_INFINITY)] + fn a_non_finite_number_reads_back_as_written(#[case] value: f64) { + let written = json::dumps(&Value::Float(value)).unwrap(); + + let read = number(written.as_bytes(), "a number").unwrap(); + + assert_eq!( + repr(&read.to_value()), + repr(&Number::Float(value).to_value()) + ); + } +} diff --git a/litellm-rust/crates/redis-store/src/dual_cache/batch_read.rs b/litellm-rust/crates/redis-store/src/dual_cache/batch_read.rs new file mode 100644 index 00000000000..71dfda49233 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/dual_cache/batch_read.rs @@ -0,0 +1,242 @@ +use std::{ + collections::{HashMap, VecDeque}, + sync::Mutex, + time::Duration, +}; + +use litellm_redis_client::{Declared, Error as RedisError, commands}; + +use super::{DualCache, MemorySettings}; +use crate::{Error, FromValue, Send, codec, memory::lock, request::Misses}; + +/// A batch-reading family's `DualCache` settings, as the host reads them from `litellm` globals. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct TierPolicy { + /// Python's `redis_batch_cache_expiry`: a key memory misses is read from Redis at most this often. + pub refresh_after: Duration, + /// Python's `default_max_redis_batch_cache_size`: how many keys the refresh throttle tracks. + pub tracked_reads: usize, + /// `litellm.default_in_memory_ttl`, which a value read from Redis keeps in memory. + pub default_in_memory_ttl: Option, +} + +impl TierPolicy { + pub(crate) fn memory(&self) -> MemorySettings { + MemorySettings { + entries: None, + ttl: self.default_in_memory_ttl, + } + } +} + +/// `DualCache.last_redis_batch_access_time` with its expiry, which only `async_batch_get_cache` reads. +pub(crate) struct BatchReads { + refresh_after: Duration, + recent: Mutex, +} + +impl BatchReads { + pub(super) fn new(policy: TierPolicy) -> Self { + Self { + refresh_after: policy.refresh_after, + recent: Mutex::new(RecentReads::new(policy.tracked_reads)), + } + } +} + +impl DualCache { + /// `DualCache.async_batch_get_cache`: memory first, then one `MGET` for the misses whose refresh + /// is due, written back to memory. A failed Redis read serves memory, as `RedisCache` swallows + /// it, and only a breaker refusal releases the refresh it reserved. + pub(crate) async fn batch_get(&self, keys: &[String]) -> Result>, Error> { + let declared = self.declare_batch_get(Send::Now, keys)?; + self.resolve_batch_get(declared).await + } + + /// The first half of `batch_get`: reserves the due refreshes and declares their `MGET` on the + /// batch `send` picks, without sending anything. + pub(crate) fn declare_batch_get( + &self, + send: Send<'_>, + keys: &[String], + ) -> Result { + let fetch = match self.redis.client() { + None => None, + Some(client) => { + let reservation = self.reserve(keys, &self.read_memory(keys)?); + (!reservation.keys.is_empty()).then(|| { + let keys: Vec<&str> = reservation.keys.iter().map(String::as_str).collect(); + let replies = send.batch(client).add(commands::mget::>(&keys)); + Fetch { + reservation, + replies, + misses: send.misses(client), + } + }) + } + }; + Ok(DeclaredRead { + keys: keys.to_vec(), + fetch, + }) + } + + /// The second half of `batch_get`. Memory is read again here, so a value written to memory + /// since the declaration wins over what Redis answered. The keys Redis lacks are noted as the + /// request's misses. An undecodable value serves memory for the whole read, where `main`'s + /// `RedisCache` returns the raw bytes and `DualCache` backfills memory with them. + pub(crate) async fn resolve_batch_get( + &self, + declared: DeclaredRead, + ) -> Result>, Error> { + let DeclaredRead { keys, fetch } = declared; + let Some(Fetch { + reservation, + replies, + misses, + }) = fetch + else { + return self.read_memory(&keys); + }; + let fetched = replies.resolve().await; + let local = self.read_memory(&keys)?; + let fetched = match fetched { + Ok(fetched) => fetched, + Err(RedisError::BreakerOpen) => { + self.release(reservation); + return Ok(local); + } + Err(error) => { + crate::failure_log::redis_failure!( + &error, + "Redis read failed, serving the memory tier" + ); + return Ok(local); + } + }; + if let Some(misses) = misses { + misses.note( + reservation + .keys + .iter() + .zip(&fetched) + .filter(|(_, raw)| raw.is_none()) + .map(|(key, _)| key.clone()), + ); + } + let decoded = match fetched + .iter() + .map(|raw| raw.as_deref().map(codec::decode).transpose()) + .collect::, _>>() + { + Ok(decoded) => decoded, + Err(error) => { + tracing::warn!(%error, "a Redis value did not decode, serving the memory tier"); + return Ok(local); + } + }; + let read = reservation + .keys + .into_iter() + .zip(decoded) + .collect::>(); + keys.iter() + .zip(local) + .map(|(key, local)| match local { + Some(value) => Ok(Some(value)), + None => self.backfill(key, read.get(key).cloned().flatten(), None), + }) + .collect() + } + + fn read_memory(&self, keys: &[String]) -> Result>, Error> { + keys.iter().map(|key| Ok(self.memory.get(key)?)).collect() + } + + fn reserve(&self, keys: &[String], local: &[Option]) -> Reservation { + let now = self.now(); + let refresh_after = self.reads.refresh_after; + let mut recent = lock(&self.reads.recent); + let due = keys + .iter() + .zip(local) + .filter(|(_, value)| value.is_none()) + .filter_map(|(key, _)| { + let previous = recent.get(key); + let due = previous.is_none_or(|read| now.saturating_sub(read) >= refresh_after); + due.then(|| { + recent.insert(key.clone(), now); + (key.clone(), previous) + }) + }) + .collect::>(); + let (keys, previous) = due.into_iter().unzip(); + Reservation { keys, previous } + } + + fn release(&self, reservation: Reservation) { + let mut recent = lock(&self.reads.recent); + for (key, previous) in reservation.keys.into_iter().zip(reservation.previous) { + match previous { + Some(read) => recent.insert(key, read), + None => recent.remove(&key), + } + } + } +} + +/// The `MGET` `declare_batch_get` put on a batch, with the refreshes it reserved. +pub(crate) struct DeclaredRead { + keys: Vec, + fetch: Option, +} + +struct Fetch { + reservation: Reservation, + replies: Declared>>>, + misses: Option, +} + +struct Reservation { + keys: Vec, + previous: Vec>, +} + +/// Python's `LimitedSizeOrderedDict`: insertion-ordered, and any write at capacity drops the +/// oldest entry first, even when the written key is already present. +struct RecentReads { + capacity: usize, + order: VecDeque, + times: HashMap, +} + +impl RecentReads { + fn new(capacity: usize) -> Self { + Self { + capacity, + order: VecDeque::new(), + times: HashMap::new(), + } + } + + fn get(&self, key: &str) -> Option { + self.times.get(key).copied() + } + + fn insert(&mut self, key: String, time: Duration) { + if self.times.len() >= self.capacity + && let Some(oldest) = self.order.pop_front() + { + self.times.remove(&oldest); + } + if self.times.insert(key.clone(), time).is_none() { + self.order.push_back(key); + } + } + + fn remove(&mut self, key: &str) { + if self.times.remove(key).is_some() { + self.order.retain(|tracked| tracked != key); + } + } +} diff --git a/litellm-rust/crates/redis-store/src/dual_cache/increment.rs b/litellm-rust/crates/redis-store/src/dual_cache/increment.rs new file mode 100644 index 00000000000..705bc71b2c1 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/dual_cache/increment.rs @@ -0,0 +1,129 @@ +use std::time::Duration; + +use futures_util::future::join_all; +use litellm_redis_client::{Declared, Expiry, RedisBatch, commands}; + +use super::DualCache; +use crate::{Error, Number, RequestRedis, Send, failure_log::redis_failure, memory::Counter}; + +const INCREMENT_FAILED: &str = + "Redis async_increment_cache failed, falling back to in-memory result"; +const PIPELINE_FAILED: &str = + "Redis async_increment_cache_pipeline failed, falling back to in-memory result"; + +/// A `RedisPipelineIncrementOperation`. +pub(crate) struct PipelineIncrement { + pub(crate) key: String, + pub(crate) amount: Number, + pub(crate) ttl: Option, +} + +impl DualCache { + /// `DualCache.async_increment_cache`: memory, then `INCRBYFLOAT` keeping an existing TTL, else + /// setting `ttl` or Redis's default. `None` when memory refuses the increment, which then never + /// reaches Redis, and the memory count when Redis fails. Both only log, as there. + pub(crate) async fn increment( + &self, + key: &str, + amount: Number, + ttl: Option, + ) -> Option { + let local = self + .memory + .increment(key, amount, ttl) + .map_err(|error| redis_failure!(&Error::from(error), INCREMENT_FAILED)) + .ok()?; + let Some(remote) = self.redis.get() else { + return Some(local); + }; + let expiry = Expiry::KeepExisting(ttl.unwrap_or(remote.default_ttl)); + let increment = commands::incr_by_float(key, amount.into(), Some(expiry)); + match remote.client.send(increment).await { + Ok(total) => Some(Number::Float(total)), + Err(error) => { + redis_failure!(&error, INCREMENT_FAILED); + Some(local) + } + } + } + + /// `DualCache.async_increment_cache_pipeline`: memory, then one pipeline of `INCRBYFLOAT` with + /// `EXPIRE` per increment. `None` when memory refuses an increment, which leaves the ones before + /// it applied and sends nothing, and the memory totals when Redis fails. + pub(crate) async fn increment_pipeline( + &self, + increments: &[PipelineIncrement], + ) -> Option> { + let local = increments + .iter() + .map(|increment| { + self.memory + .increment(&increment.key, increment.amount, increment.ttl) + }) + .collect::, _>>() + .map_err(|error| redis_failure!(&Error::from(error), PIPELINE_FAILED)) + .ok()?; + let Some(client) = self.redis.client() else { + return Some(local); + }; + let pipeline = RedisBatch::pipelined(client.clone()); + let totals = increments + .iter() + .map(|increment| { + pipeline.add(commands::incr_by_float( + &increment.key, + increment.amount.into(), + increment.ttl.map(Expiry::Refresh), + )) + }) + .collect::>(); + match Declared::all(totals) + .await + .into_iter() + .collect::, _>>() + { + Ok(totals) => Some(totals.into_iter().map(Number::Float).collect()), + Err(error) => { + redis_failure!(&error, PIPELINE_FAILED); + Some(local) + } + } + } + + /// `DualCache.async_increment_cache_pipeline_post_call`: memory now, stopping at the first + /// increment memory refuses and only logging it, then each increment on the request's post-call + /// batch, whose failure is logged once it settles. + pub(crate) fn increment_pipeline_post_call( + &self, + request: &RequestRedis, + increments: &[PipelineIncrement], + ) { + if let Err(error) = increments.iter().try_for_each(|increment| { + self.memory + .increment(&increment.key, increment.amount, increment.ttl) + .map(drop) + }) { + tracing::warn!(%error, "in-memory increment failed"); + } + let Some(batch) = self.redis.batch(Send::PostCall(request)) else { + return; + }; + let declared = increments + .iter() + .map(|increment| { + batch.add(commands::incr_by_float( + &increment.key, + increment.amount.into(), + increment.ttl.map(Expiry::KeepExisting), + )) + }) + .collect::>(); + request.spawn(async move { + join_all(declared.into_iter().map(Declared::settled)) + .await + .into_iter() + .filter_map(Result::err) + .for_each(|error| redis_failure!(&error, "deferred Redis increment failed")); + }); + } +} diff --git a/litellm-rust/crates/redis-store/src/dual_cache/mod.rs b/litellm-rust/crates/redis-store/src/dual_cache/mod.rs new file mode 100644 index 00000000000..b2c277a118e --- /dev/null +++ b/litellm-rust/crates/redis-store/src/dual_cache/mod.rs @@ -0,0 +1,279 @@ +mod batch_read; +mod increment; +mod read; +mod write; + +use std::{ + sync::{Arc, Mutex}, + time::Duration, +}; + +pub use self::batch_read::TierPolicy; +pub(crate) use self::{batch_read::BatchReads, increment::PipelineIncrement}; +use crate::{ + memory::{Clock, Memory, lock}, + remote::RemoteSlot, +}; + +/// The memory tier's `InMemoryCache` size and the `default_in_memory_ttl` a write or a Redis +/// backfill takes when it names no TTL. `None` keeps `InMemoryCache`'s own default. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct MemorySettings { + pub(crate) entries: Option, + pub(crate) ttl: Option, +} + +/// Python's `DualCache`: a per-worker `InMemoryCache` in front of the Redis every pod shares. Each +/// method is the `DualCache` method it names. A family whose Python reached past `DualCache` to its +/// `in_memory_cache` or `redis_cache` uses `memory` and `redis` the same way. +pub(crate) struct DualCache { + memory: Memory, + redis: Arc, + default_in_memory_ttl: Mutex>, + now: Clock, + reads: R, +} + +impl DualCache { + /// `redis` may be shared with the family's other caches, as one `RedisCache` is attached to each. + pub(crate) fn new(redis: Arc, memory: MemorySettings, now: Clock) -> Self { + Self { + memory: Memory::new(memory.entries, &now), + redis, + default_in_memory_ttl: Mutex::new(memory.ttl), + now, + reads: (), + } + } + + /// A cache that batch-reads, with the `async_batch_get_cache` throttle `policy` configures. + pub(crate) fn batch_reading( + redis: Arc, + policy: TierPolicy, + now: Clock, + ) -> DualCache { + let Self { + memory, + redis, + default_in_memory_ttl, + now, + reads: (), + } = Self::new(redis, policy.memory(), now); + DualCache { + memory, + redis, + default_in_memory_ttl, + now, + reads: BatchReads::new(policy), + } + } +} + +impl DualCache { + pub(crate) fn memory(&self) -> &Memory { + &self.memory + } + + pub(crate) fn redis(&self) -> &RemoteSlot { + &self.redis + } + + pub(crate) fn now(&self) -> Duration { + (self.now)() + } + + pub(crate) fn default_in_memory_ttl(&self) -> Option { + *lock(&self.default_in_memory_ttl) + } + + /// `DualCache.update_cache_ttl` and `update_in_memory_max_size` together: both apply in place, + /// and the entries memory already holds stay. + pub(crate) fn reconfigure(&self, memory: MemorySettings) { + *lock(&self.default_in_memory_ttl) = memory.ttl; + self.memory.update_max_size(memory.entries); + } +} + +#[cfg(test)] +mod tests { + use std::{ + sync::{ + Arc, + atomic::{AtomicU64, Ordering}, + }, + time::Duration, + }; + + use rstest::rstest; + use tokio::runtime::Handle; + + use super::{DualCache, MemorySettings, PipelineIncrement}; + use crate::{Clock, Number, RequestRedis, memory::Counter, remote::RemoteSlot}; + + #[derive(Clone, Debug, PartialEq)] + enum Entry { + Number(Number), + Text, + } + + impl From for Entry { + fn from(number: Number) -> Self { + Self::Number(number) + } + } + + impl Counter for Entry { + fn as_number(&self) -> Option { + match self { + Self::Number(number) => Some(*number), + Self::Text => None, + } + } + } + + fn clock() -> (Clock, Arc) { + let seconds = Arc::new(AtomicU64::new(1_000)); + let read = seconds.clone(); + let clock: Clock = Arc::new(move || Duration::from_secs(read.load(Ordering::SeqCst))); + (clock, seconds) + } + + fn without_redis( + default_in_memory_ttl: Option, + now: Clock, + ) -> DualCache { + DualCache::new( + Arc::new(RemoteSlot::new(None)), + MemorySettings { + entries: None, + ttl: default_in_memory_ttl, + }, + now, + ) + } + + fn increment(key: &str, amount: Number) -> PipelineIncrement { + PipelineIncrement { + key: key.to_owned(), + amount, + ttl: None, + } + } + + // InMemoryCache's own default TTL is 600 s, which an entry outlives here only when the cache + // configured none + #[rstest] + #[case::default_in_memory_ttl(Some(Duration::from_secs(5)), None)] + #[case::in_memory_cache_default(None, Some(Number::Int(1)))] + #[tokio::test] + async fn a_set_without_a_ttl_takes_default_in_memory_ttl( + #[case] default_in_memory_ttl: Option, + #[case] after_six_seconds: Option, + ) { + let (now, seconds) = clock(); + let cache = without_redis(default_in_memory_ttl, now); + + cache.set("k", Number::Int(1), None).await.unwrap(); + seconds.fetch_add(6, Ordering::SeqCst); + + assert_eq!(cache.memory().get("k").unwrap(), after_six_seconds); + } + + #[rstest] + #[case::absent_starts_at_int_zero(None, Number::Int(2), Number::Int(2))] + #[case::int_plus_int_stays_int(Some(Number::Int(1)), Number::Int(2), Number::Int(3))] + #[case::a_float_makes_a_float(Some(Number::Int(1)), Number::Float(0.5), Number::Float(1.5))] + fn a_local_increment_adds_as_python_adds( + #[case] stored: Option, + #[case] amount: Number, + #[case] expected: Number, + ) { + let (now, _) = clock(); + let cache = without_redis::(None, now); + if let Some(stored) = stored { + cache.memory().set("k", stored, None).unwrap(); + } + + let total = cache.memory().increment("k", amount, None).unwrap(); + + assert_eq!(total, expected); + assert_eq!(cache.memory().get("k").unwrap(), Some(expected)); + } + + #[test] + fn a_local_increment_of_a_non_number_fails_and_keeps_the_entry() { + let (now, _) = clock(); + let cache = without_redis::(None, now); + cache.memory().set("k", Entry::Text, None).unwrap(); + + let total = cache.memory().increment("k", Number::Int(1), None); + + assert!(total.is_err()); + assert_eq!(cache.memory().get("k").unwrap(), Some(Entry::Text)); + } + + // async_increment_cache and async_increment_cache_pipeline catch the in-memory failure with the + // Redis one, so the caller gets None and nothing is sent + #[tokio::test] + async fn an_increment_memory_refuses_answers_none() { + let (now, _) = clock(); + let cache = without_redis::(None, now); + cache.memory().set("text", Entry::Text, None).unwrap(); + + let single = cache.increment("text", Number::Int(1), None).await; + let pipelined = cache + .increment_pipeline(&[ + increment("before", Number::Int(1)), + increment("text", Number::Int(1)), + ]) + .await; + + assert_eq!(single, None); + assert_eq!(pipelined, None); + assert_eq!(cache.memory().get("text").unwrap(), Some(Entry::Text)); + } + + // update_cache_ttl and update_in_memory_max_size change the settings in place, so what memory + // already holds survives and the next write takes the new TTL + #[tokio::test] + async fn reconfigure_keeps_entries_and_applies_the_new_ttl() { + let (now, seconds) = clock(); + let cache = without_redis(None, now); + cache.set("kept", Number::Int(1), None).await.unwrap(); + + cache.reconfigure(MemorySettings { + entries: Some(10), + ttl: Some(Duration::from_secs(5)), + }); + cache.set("short", Number::Int(2), None).await.unwrap(); + seconds.fetch_add(6, Ordering::SeqCst); + + assert_eq!(cache.memory().get("kept").unwrap(), Some(Number::Int(1))); + assert_eq!(cache.memory().get("short").unwrap(), None); + } + + // async_increment_cache_pipeline_post_call: InMemoryCache.async_increment_pipeline raises at the + // first entry it cannot add to, so later increments never reach memory, and DualCache only logs + #[tokio::test] + async fn a_post_call_increment_stops_at_the_first_entry_memory_refuses() { + let (now, _) = clock(); + let cache = without_redis::(None, now); + let request = RequestRedis::new(Handle::current(), Duration::from_secs(60)); + cache.memory().set("text", Entry::Text, None).unwrap(); + + cache.increment_pipeline_post_call( + &request, + &[ + increment("before", Number::Int(1)), + increment("text", Number::Int(1)), + increment("after", Number::Int(1)), + ], + ); + + assert_eq!( + cache.memory().get("before").unwrap(), + Some(Entry::Number(Number::Int(1))) + ); + assert_eq!(cache.memory().get("after").unwrap(), None); + } +} diff --git a/litellm-rust/crates/redis-store/src/dual_cache/read.rs b/litellm-rust/crates/redis-store/src/dual_cache/read.rs new file mode 100644 index 00000000000..14156483c6f --- /dev/null +++ b/litellm-rust/crates/redis-store/src/dual_cache/read.rs @@ -0,0 +1,55 @@ +use std::time::Duration; + +use litellm_redis_client::commands; + +use super::DualCache; +use crate::{Error, FromValue, RequestRedis, codec, failure_log::redis_failure}; + +impl DualCache { + /// `DualCache.async_get_cache`: memory, else `None` for a key `request` already read as missing + /// from Redis, else a `GET` written back to memory for `ttl` or `default_in_memory_ttl`. A failed + /// or undecodable Redis read is a miss, as there. + pub(crate) async fn get( + &self, + request: Option<&RequestRedis>, + key: &str, + ttl: Option, + ) -> Result, Error> { + if let Some(value) = self.memory.get(key)? { + return Ok(Some(value)); + } + let Some(client) = self.redis.client() else { + return Ok(None); + }; + if request.is_some_and(|request| request.misses(client).contains(key)) { + return Ok(None); + } + let value = client + .send(commands::get::>(key)) + .await + .map_err(Error::from) + .and_then(|raw| Ok(raw.as_deref().map(codec::decode).transpose()?)) + .unwrap_or_else(|error| { + redis_failure!(&error, "LiteLLM Cache: exception in async_get_cache"); + None + }); + self.backfill(key, value, ttl) + } +} + +impl DualCache { + /// `_backfill_kwargs`: a value read from Redis stays in memory for `ttl` or + /// `default_in_memory_ttl`. + pub(super) fn backfill( + &self, + key: &str, + value: Option, + ttl: Option, + ) -> Result, Error> { + if let Some(value) = &value { + self.memory + .set(key, value.clone(), ttl.or(self.default_in_memory_ttl()))?; + } + Ok(value) + } +} diff --git a/litellm-rust/crates/redis-store/src/dual_cache/write.rs b/litellm-rust/crates/redis-store/src/dual_cache/write.rs new file mode 100644 index 00000000000..6fcc6f32741 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/dual_cache/write.rs @@ -0,0 +1,178 @@ +use std::time::Duration; + +use litellm_redis_client::{Declared, RedisBatch, commands}; + +use super::DualCache; +use crate::{ + Error, IntoValue, Pending, Send, codec, + failure_log::redis_failure, + request::{pending, ready}, +}; + +/// `DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE`, the chunk `async_delete_cache_keys` deletes at once. +const DELETE_CHUNK: usize = 1000; + +impl DualCache { + /// `DualCache.async_set_cache`: memory, then a `SET` of `json.dumps(value)`. A Redis failure, or + /// a value `json.dumps` refuses, only logs. + pub(crate) async fn set( + &self, + key: &str, + value: V, + ttl: Option, + ) -> Result<(), Error> { + self.set_now(key, value, ttl, codec::dumps).await + } + + /// `DualCache.set_cache`, whose sync `RedisCache.set_cache` writes `str(value)`. + pub(crate) async fn set_str( + &self, + key: &str, + value: V, + ttl: Option, + ) -> Result<(), Error> { + self.set_now(key, value, ttl, |value| Ok(codec::str(value))) + .await + } + + /// `DualCache._set_on_batch`: memory now, the `SET` on the batch `send` picks. Memory and Redis + /// both take `ttl`, else `default_in_memory_ttl`, and Redis falls back to its own default. + pub(crate) fn declare_set( + &self, + send: Send<'_>, + key: &str, + value: V, + ttl: Option, + ) -> Result, Error> { + self.declare_encoded(send, key, value, ttl, codec::dumps) + } + + /// `DualCache.async_set_cache_pipeline`: memory, then one pipeline of `SET`s. A value + /// `json.dumps` refuses aborts the whole pipeline, and any failure only logs, as `RedisCache` + /// does. + pub(crate) async fn set_pipeline( + &self, + entries: Vec<(String, V)>, + ttl: Option, + ) -> Result<(), Error> { + let ttl = ttl.or(self.default_in_memory_ttl()); + let encoded = self.redis.get().map(|_| { + entries + .iter() + .map(|(_, value)| codec::dumps(value.clone())) + .collect::, Error>>() + }); + let keys = entries + .into_iter() + .map(|(key, value)| { + self.memory.set(&key, value, ttl)?; + Ok(key) + }) + .collect::, Error>>()?; + let (Some(remote), Some(encoded)) = (self.redis.get(), encoded) else { + return Ok(()); + }; + let failure = match encoded { + Err(error) => Some(error), + Ok(values) => { + let pipeline = RedisBatch::pipelined(remote.client.clone()); + let ttl = ttl.unwrap_or(remote.default_ttl); + let written = keys + .iter() + .zip(values) + .map(|(key, value)| pipeline.add(commands::set(key, value, ttl))) + .collect::>(); + Declared::all(written) + .await + .into_iter() + .find_map(Result::err) + .map(Error::from) + } + }; + if let Some(error) = failure { + redis_failure!( + &error, + "LiteLLM Redis Caching: async set_cache_pipeline() - Got exception from REDIS", + ); + } + Ok(()) + } + + async fn set_now( + &self, + key: &str, + value: V, + ttl: Option, + encode: impl FnOnce(V) -> Result, + ) -> Result<(), Error> { + if let Err(error) = self + .declare_encoded(Send::Now, key, value, ttl, encode)? + .await + { + redis_failure!(&error, "LiteLLM Cache: exception in async add_cache"); + } + Ok(()) + } + + fn declare_encoded( + &self, + send: Send<'_>, + key: &str, + value: V, + ttl: Option, + encode: impl FnOnce(V) -> Result, + ) -> Result, Error> { + let ttl = ttl.or(self.default_in_memory_ttl()); + let encoded = self.redis.get().map(|_| encode(value.clone())); + self.memory.set(key, value, ttl)?; + let (Some(remote), Some(encoded)) = (self.redis.get(), encoded) else { + return Ok(ready(())); + }; + let encoded = match encoded { + Ok(encoded) => encoded, + Err(error) => { + redis_failure!(&error, "a cache value did not encode, kept in memory only"); + return Ok(ready(())); + } + }; + if let Some(misses) = send.misses(&remote.client) { + misses.forget(key); + } + Ok(pending(send.batch(&remote.client).add(commands::set( + key, + encoded, + ttl.unwrap_or(remote.default_ttl), + )))) + } +} + +impl DualCache { + /// `DualCache.async_delete_cache_pre_call`: memory now, the `DEL` on the batch `send` picks, + /// after which the request reads the key as missing. + pub(crate) fn declare_delete(&self, send: Send<'_>, key: &str) -> Result, Error> { + self.memory.delete(key)?; + let Some(client) = self.redis.client() else { + return Ok(ready(())); + }; + if let Some(misses) = send.misses(client) { + misses.note([key.to_owned()]); + } + let deleted = send.batch(client).add(commands::del(&[key])); + Ok(Box::pin( + async move { Ok(deleted.resolve().await.map(drop)?) }, + )) + } + + /// `DualCache.async_delete_cache_keys`: memory, then one multi-key `DEL` per chunk, in order. + pub(crate) async fn delete_keys(&self, keys: &[String]) -> Result<(), Error> { + keys.iter().try_for_each(|key| self.memory.delete(key))?; + let Some(client) = self.redis.client() else { + return Ok(()); + }; + for chunk in keys.chunks(DELETE_CHUNK) { + let chunk: Vec<&str> = chunk.iter().map(String::as_str).collect(); + client.send(commands::del(&chunk)).await?; + } + Ok(()) + } +} diff --git a/litellm-rust/crates/redis-store/src/error.rs b/litellm-rust/crates/redis-store/src/error.rs new file mode 100644 index 00000000000..fdc722855bb --- /dev/null +++ b/litellm-rust/crates/redis-store/src/error.rs @@ -0,0 +1,63 @@ +use crate::DecodeError; + +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error(transparent)] + Redis(#[from] litellm_redis_client::Error), + #[error(transparent)] + Memory(#[from] litellm_cache::Error), + #[error(transparent)] + Decode(#[from] DecodeError), + #[error("the value is not JSON serializable: {0}")] + Encode(#[from] litellm_python_compat::Error), +} + +impl Error { + pub fn kind(&self) -> ErrorKind { + match self { + Self::Redis(error) => error.into(), + Self::Memory(_) => ErrorKind::Memory, + Self::Decode(_) => ErrorKind::Decode, + Self::Encode(_) => ErrorKind::Encode, + } + } +} + +/// The failure class a host reports with a store error, `timeout` and `breaker_open` being the ones a +/// caller may drop. +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::IntoStaticStr)] +pub enum ErrorKind { + #[strum(serialize = "breaker_open")] + BreakerOpen, + #[strum(serialize = "timeout")] + Timeout, + #[strum(serialize = "transport")] + Transport, + #[strum(serialize = "rejected")] + Rejected, + #[strum(serialize = "decode")] + Decode, + #[strum(serialize = "encode")] + Encode, + #[strum(serialize = "memory")] + Memory, +} + +impl From<&Error> for ErrorKind { + fn from(error: &Error) -> Self { + error.kind() + } +} + +impl From<&litellm_redis_client::Error> for ErrorKind { + fn from(error: &litellm_redis_client::Error) -> Self { + use litellm_redis_client::Error; + match error { + Error::BreakerOpen => Self::BreakerOpen, + Error::Transport(error) if error.is_timeout() => Self::Timeout, + Error::Transport(_) => Self::Transport, + Error::Server(_) => Self::Rejected, + Error::Decode(_) => Self::Decode, + } + } +} diff --git a/litellm-rust/crates/redis-store/src/failure_log.rs b/litellm-rust/crates/redis-store/src/failure_log.rs new file mode 100644 index 00000000000..e6bf5187cfd --- /dev/null +++ b/litellm-rust/crates/redis-store/src/failure_log.rs @@ -0,0 +1,18 @@ +/// `log_redis_failure`: an open breaker is DEBUG, anything else WARNING. The event's `redis.failure` +/// field names the class, so the host can rate-limit timeout lines with the throttle redis-py shares. +/// A macro so the event carries its caller's target, file and line, as `stacklevel=2` does in Python. +macro_rules! redis_failure { + ($error:expr, $message:expr $(,)?) => {{ + let error = $error; + let kind = $crate::ErrorKind::from(error); + let failure: &'static str = kind.into(); + match kind { + $crate::ErrorKind::BreakerOpen => { + tracing::debug!(redis.failure = failure, "{}: {}", $message, error) + } + _ => tracing::warn!(redis.failure = failure, "{}: {}", $message, error), + } + }}; +} + +pub(crate) use redis_failure; diff --git a/litellm-rust/crates/redis-store/src/lib.rs b/litellm-rust/crates/redis-store/src/lib.rs new file mode 100644 index 00000000000..69600aac2a6 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/lib.rs @@ -0,0 +1,24 @@ +pub mod auth_cache; +mod codec; +mod dual_cache; +mod error; +mod failure_log; +mod memory; +mod number; +pub mod rate_limits; +mod remote; +mod request; +pub mod router_cooldowns; +pub mod router_usage; +pub mod sensitive_route_pins; +pub mod spend_counters; +mod value; + +pub use codec::DecodeError; +pub use dual_cache::TierPolicy; +pub use error::{Error, ErrorKind}; +pub use memory::{Clock, system_clock}; +pub use number::Number; +pub use remote::Remote; +pub use request::{Pending, RequestRedis, Send}; +pub use value::{FromValue, IntoValue}; diff --git a/litellm-rust/crates/redis-store/src/memory.rs b/litellm-rust/crates/redis-store/src/memory.rs new file mode 100644 index 00000000000..f89d5926760 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/memory.rs @@ -0,0 +1,122 @@ +use std::{ + sync::{Arc, Mutex, MutexGuard, PoisonError}, + time::{Duration, SystemTime, UNIX_EPOCH}, +}; + +use litellm_cache::Error; +use litellm_cache_memory::InMemoryCache; + +use crate::Number; + +pub type Clock = Arc Duration + Send + Sync>; + +pub fn system_clock() -> Clock { + Arc::new(|| { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + }) +} + +/// A store's memory tier: Python's `InMemoryCache`, with its 200 entries and 600 s TTL unless the +/// store names others, plus one lock for every read-modify-write on it. +pub struct Memory { + cache: InMemoryCache, + updates: Mutex<()>, +} + +impl Memory { + pub fn new(entries: Option, now: &Clock) -> Self { + let now = now.clone(); + Self { + cache: InMemoryCache::with_clock(entries, None, move || now()), + updates: Mutex::new(()), + } + } + + pub fn get(&self, key: &str) -> Result, Error> { + self.cache.get_cache(key) + } + + pub fn set(&self, key: &str, value: V, ttl: Option) -> Result<(), Error> { + self.cache.set_cache(key, value, ttl).map(drop) + } + + pub fn delete(&self, key: &str) -> Result<(), Error> { + self.cache.delete_cache(key) + } + + pub fn flush(&self) -> Result<(), Error> { + self.cache.flush_cache() + } + + pub fn len(&self) -> Result { + self.cache.len() + } + + pub fn expires_at(&self, key: &str) -> Result, Error> { + self.cache.expires_at(key) + } + + pub fn entries(&self) -> Result, Error> { + self.cache.entries() + } + + /// `DualCache.update_in_memory_max_size`: `None` restores the 200-entry default. + pub fn update_max_size(&self, entries: Option) { + self.cache.update_max_size_in_memory(entries); + } + + /// Read-modify-write of one entry, atomic against every other update of this tier. + pub fn update( + &self, + key: &str, + ttl: Option, + update: impl FnOnce(Option) -> Result, + ) -> Result { + let _guard = self.guard(); + let value = update(self.get(key)?)?; + self.set(key, value.clone(), ttl)?; + Ok(value) + } + + /// Held across a read and a conditional write that `update` cannot express. + pub fn guard(&self) -> MutexGuard<'_, ()> { + lock(&self.updates) + } +} + +/// A value `InMemoryCache.increment_cache` can add to. +pub(crate) trait Counter: Clone + From { + fn as_number(&self) -> Option; +} + +impl Counter for Number { + fn as_number(&self) -> Option { + Some(*self) + } +} + +impl Memory { + /// `InMemoryCache.increment_cache`: `(current or 0) + amount`, refusing an entry that is not a + /// number and leaving it in place. + pub(crate) fn increment( + &self, + key: &str, + amount: Number, + ttl: Option, + ) -> Result { + let _guard = self.guard(); + let current = match self.get(key)? { + None => Number::Int(0), + Some(value) => value.as_number().ok_or(Error::InvalidEntry(None))?, + }; + let total = current + amount; + self.set(key, V::from(total), ttl)?; + Ok(total) + } +} + +pub(crate) fn lock(mutex: &Mutex) -> MutexGuard<'_, T> { + mutex.lock().unwrap_or_else(PoisonError::into_inner) +} diff --git a/litellm-rust/crates/redis-store/src/number.rs b/litellm-rust/crates/redis-store/src/number.rs new file mode 100644 index 00000000000..7f3e9f4d10b --- /dev/null +++ b/litellm-rust/crates/redis-store/src/number.rs @@ -0,0 +1,83 @@ +use std::{cmp::Ordering, ops::Add}; + +use litellm_python_compat::{BigInt, Value, json, repr}; + +/// A Python `int` or `float`, kept apart because `repr` and `json.dumps` write them differently. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Number { + Int(i64), + Float(f64), +} + +impl From for litellm_redis_client::Amount { + fn from(number: Number) -> Self { + match number { + Number::Int(value) => Self::Int(value), + Number::Float(value) => Self::Float(value), + } + } +} + +impl Number { + pub fn as_f64(self) -> f64 { + match self { + Self::Int(value) => value as f64, + Self::Float(value) => value, + } + } + + /// Python's `int(value)`, which truncates toward zero. + pub fn truncate(self) -> i64 { + match self { + Self::Int(value) => value, + Self::Float(value) => value.trunc() as i64, + } + } + + pub fn to_value(self) -> Value { + match self { + Self::Int(value) => Value::Int(BigInt::from(value)), + Self::Float(value) => Value::Float(value), + } + } + + /// `json.dumps`, which `RedisCache.async_set_cache` writes. + pub fn to_json(self) -> String { + let value = self.to_value(); + json::dumps(&value).unwrap_or_else(|_| repr::to_str(&value)) + } + + pub fn from_value(value: &Value) -> Option { + match value { + Value::Int(value) => i64::try_from(value).ok().map(Self::Int), + Value::Float(value) => Some(Self::Float(*value)), + _ => None, + } + } + + /// Python's `min` over numbers: the first of the smallest, keeping its type. + pub fn min(values: impl IntoIterator) -> Option { + values + .into_iter() + .fold(None, |smallest, value| match smallest { + Some(smallest) + if value.as_f64().partial_cmp(&smallest.as_f64()) != Some(Ordering::Less) => + { + Some(smallest) + } + _ => Some(value), + }) + } +} + +/// Python's `+`: two ints stay an int, anything with a float is a float. +impl Add for Number { + type Output = Self; + + fn add(self, other: Self) -> Self { + match (self, other) { + (Self::Int(left), Self::Int(right)) => Self::Int(left + right), + (left, right) => Self::Float(left.as_f64() + right.as_f64()), + } + } +} diff --git a/litellm-rust/crates/redis-store/src/rate_limits/keys.rs b/litellm-rust/crates/redis-store/src/rate_limits/keys.rs new file mode 100644 index 00000000000..7f5714a19ad --- /dev/null +++ b/litellm-rust/crates/redis-store/src/rate_limits/keys.rs @@ -0,0 +1,74 @@ +/// A rate-limit descriptor, `{descriptor:value}`, the hash tag every key of its window shares. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub struct Scope { + pub descriptor: String, + pub value: String, +} + +impl Scope { + pub(super) fn key(&self, suffix: &str) -> String { + format!("{{{}:{}}}:{suffix}", self.descriptor, self.value) + } + + pub(super) fn window_key(&self) -> String { + self.key(WINDOW) + } + + pub(super) fn gauge_key(&self) -> String { + self.key(GAUGE) + } +} + +const WINDOW: &str = "window"; +const GAUGE: &str = "max_parallel_requests"; + +/// A counter's key suffix, which is also how the Python limiter names it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, strum::EnumString, strum::IntoStaticStr)] +pub enum Kind { + #[strum(serialize = "requests")] + Requests, + #[strum(serialize = "tokens")] + Tokens, +} + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub struct Counter { + pub scope: Scope, + pub kind: Kind, +} + +impl Counter { + pub(super) fn key(&self) -> String { + self.scope.key(self.kind.into()) + } +} + +/// One value of the limiter's keyspace, as both tiers address it. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum Entry { + Window(Scope), + Counter(Counter), + Gauge(Scope), +} + +impl Entry { + /// The entry the Python limiter names by its key suffix: `window`, `max_parallel_requests`, or a + /// counter's `Kind`. + pub fn new(suffix: &str, scope: Scope) -> Result { + match suffix { + WINDOW => Ok(Self::Window(scope)), + GAUGE => Ok(Self::Gauge(scope)), + kind => kind + .parse() + .map(|kind| Self::Counter(Counter { scope, kind })), + } + } + + pub(super) fn key(&self) -> String { + match self { + Self::Window(scope) => scope.window_key(), + Self::Counter(counter) => counter.key(), + Self::Gauge(scope) => scope.gauge_key(), + } + } +} diff --git a/litellm-rust/crates/redis-store/src/rate_limits/lua.rs b/litellm-rust/crates/redis-store/src/rate_limits/lua.rs new file mode 100644 index 00000000000..96451212d35 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/rate_limits/lua.rs @@ -0,0 +1,260 @@ +//! The limiter's Lua, byte for byte as `main`'s Python registers it, so both share one script SHA. + +pub(super) const BATCH_RATE_LIMITER: &str = r#" +local results = {} +local now = tonumber(ARGV[1]) +local window_size = tonumber(ARGV[2]) + +-- Process each window/counter pair +for i = 1, #KEYS, 2 do + local window_key = KEYS[i] + local counter_key = KEYS[i + 1] + local increment_value = 1 + + -- Check if window exists and is valid + local window_start = redis.call('GET', window_key) + if not window_start or (now - tonumber(window_start)) >= window_size then + -- Reset window and counter + local prefix = string.sub(window_key, 1, -(#':window') - 1) + redis.call('DEL', prefix .. ':requests', prefix .. ':tokens') + redis.call('SET', window_key, tostring(now)) + redis.call('SET', counter_key, increment_value) + redis.call('EXPIRE', window_key, window_size) + redis.call('EXPIRE', counter_key, window_size) + table.insert(results, tostring(now)) -- window_start + table.insert(results, increment_value) -- counter + else + local counter = redis.call('INCR', counter_key) + -- This happens when window_key exists but counter_key doesn't (e.g., tokens key + -- created after requests key when both share the same window_key) + local current_ttl = redis.call('TTL', counter_key) + if current_ttl == -1 then + redis.call('EXPIRE', counter_key, window_size) + end + table.insert(results, window_start) -- window_start + table.insert(results, counter) -- counter + end +end + +return results +"#; + +pub(super) const BATCH_COUNTER_READ: &str = r#"return redis.call('MGET', unpack(KEYS))"#; + +pub(super) const CHECK_AND_INCREMENT_BY_N: &str = r#" +-- Atomic check-and-increment-by-N across one or more descriptors. +-- All-or-nothing: if any descriptor would exceed its limit, no counter is +-- modified. +-- +-- Uses Redis server time (`redis.call('TIME')`) instead of a client-supplied +-- timestamp so that window resets are deterministic across replicas with +-- skewed wall-clocks. This prevents a clock-skew-induced reopening of the +-- TOCTOU window across multi-replica deployments. +-- +-- KEYS layout: pairs of (window_key, counter_key), one pair per descriptor. +-- ARGV layout: per-descriptor 4-tuple, starting at ARGV[1]: +-- ARGV[(i-1)*4 + 1] = limit +-- ARGV[(i-1)*4 + 2] = increment +-- ARGV[(i-1)*4 + 3] = ttl_seconds (counter TTL when window resets) +-- ARGV[(i-1)*4 + 4] = window_size_seconds (sliding-window length) +-- +-- Return on success: +-- { 0, new_counter_1, window_start_1, new_counter_2, window_start_2, ... } +-- Return on over-limit: { 1, descriptor_index, current_counter, limit } +local time_reply = redis.call('TIME') +local now = tonumber(time_reply[1]) +local descriptor_count = #KEYS / 2 +local reset_windows = {} + +-- Pass 1: read state, validate. Abort without writing if any over limit. +local descriptor_state = {} +for i = 1, descriptor_count do + local window_key = KEYS[(i - 1) * 2 + 1] + local counter_key = KEYS[(i - 1) * 2 + 2] + local arg_base = (i - 1) * 4 + 1 + local limit = tonumber(ARGV[arg_base]) + local increment = tonumber(ARGV[arg_base + 1]) + local window_size = tonumber(ARGV[arg_base + 3]) + + local window_start = redis.call('GET', window_key) + local window_expired = (not window_start) or + ((now - tonumber(window_start)) >= window_size) + + local current_counter + if window_expired then + current_counter = 0 + else + current_counter = tonumber(redis.call('GET', counter_key) or 0) + end + + local blocked + if increment > 0 then + blocked = current_counter + increment > limit + else + blocked = current_counter >= limit + end + if blocked then + return { 1, i, current_counter, limit } + end + + descriptor_state[i] = { window_expired, current_counter, window_start } +end + +-- Pass 2: all checks passed. Apply increments. +local results = { 0 } +for i = 1, descriptor_count do + local window_key = KEYS[(i - 1) * 2 + 1] + local counter_key = KEYS[(i - 1) * 2 + 2] + local arg_base = (i - 1) * 4 + 1 + local increment = tonumber(ARGV[arg_base + 1]) + local ttl = tonumber(ARGV[arg_base + 2]) + local window_size = tonumber(ARGV[arg_base + 3]) + + local window_expired = descriptor_state[i][1] + local active_window_start + + if window_expired then + active_window_start = now + if not reset_windows[window_key] then + local prefix = string.sub(window_key, 1, -(#':window') - 1) + redis.call('DEL', prefix .. ':requests', prefix .. ':tokens') + reset_windows[window_key] = true + end + redis.call('SET', window_key, tostring(now)) + redis.call('SET', counter_key, increment) + redis.call('EXPIRE', window_key, window_size) + if ttl > 0 then + redis.call('EXPIRE', counter_key, ttl) + end + table.insert(results, increment) + else + active_window_start = tonumber(descriptor_state[i][3]) + local new_counter = redis.call('INCRBY', counter_key, increment) + local current_ttl = redis.call('TTL', counter_key) + if current_ttl == -1 and ttl > 0 then + redis.call('EXPIRE', counter_key, ttl) + end + table.insert(results, new_counter) + end + table.insert(results, active_window_start) +end + +return results +"#; + +pub(super) const WINDOW_GUARDED_TOKEN_INCREMENT: &str = r#" +local results = {} +for i = 1, #KEYS, 2 do + local window_key = KEYS[i] + local counter_key = KEYS[i + 1] + local arg_base = ((i - 1) / 2) * 3 + 1 + local expected_window_start = ARGV[arg_base] + local increment = tonumber(ARGV[arg_base + 1]) + local ttl = tonumber(ARGV[arg_base + 2]) + local active_window_start = redis.call('GET', window_key) + + if active_window_start and active_window_start == expected_window_start then + local new_counter = redis.call('INCRBY', counter_key, increment) + local current_ttl = redis.call('TTL', counter_key) + if current_ttl == -1 and ttl > 0 then + redis.call('EXPIRE', counter_key, ttl) + end + table.insert(results, 1) + table.insert(results, new_counter) + else + table.insert(results, 0) + table.insert(results, tonumber(redis.call('GET', counter_key) or 0)) + end +end +return results +"#; + +pub(super) const PARALLEL_ACQUIRE: &str = r#" +-- Atomic check-and-acquire for the max_parallel_requests concurrency gauge. +-- Each gauge key is a sorted set of per-request slot ids scored by acquire +-- time (Redis server clock). In-flight requests are counted by ZCARD after +-- pruning slots older than the slot TTL, so unlike the windowed RPM/TPM +-- counters the gauge is never reset while requests are in flight, a +-- rejected request never occupies a slot, and a slot leaked by a crashed +-- worker self-heals after the slot TTL even under continuous traffic. +-- +-- KEYS: one gauge zset key per descriptor. +-- ARGV: per-key triples (limit, slot_ttl_seconds, slot_id). +-- Success: { 0, in_flight_1, ... }. Over-limit: { 1, key_index, in_flight, limit }. +local time_reply = redis.call('TIME') +local now = tonumber(time_reply[1]) +for i = 1, #KEYS do + local limit = tonumber(ARGV[(i - 1) * 3 + 1]) + local slot_ttl = tonumber(ARGV[(i - 1) * 3 + 2]) + redis.call('ZREMRANGEBYSCORE', KEYS[i], '-inf', now - slot_ttl) + local in_flight = redis.call('ZCARD', KEYS[i]) + if in_flight + 1 > limit then + return { 1, i, in_flight, limit } + end +end +local results = { 0 } +for i = 1, #KEYS do + local slot_ttl = tonumber(ARGV[(i - 1) * 3 + 2]) + local slot_id = ARGV[(i - 1) * 3 + 3] + redis.call('ZADD', KEYS[i], now, slot_id) + redis.call('EXPIRE', KEYS[i], slot_ttl) + table.insert(results, redis.call('ZCARD', KEYS[i])) +end +return results +"#; + +pub(super) const PARALLEL_RELEASE: &str = r#" +-- Release one slot per gauge key by removing this request's slot id. +-- ZREM of an absent member (or key) is a no-op, so a release without a +-- matching acquire (proxy-side rejection, double-fired callback, slot +-- already expired) can never free a slot owned by another request. +-- KEYS: gauge zset keys. ARGV: per-key slot_id. +-- Returns the remaining in-flight count per key. +local results = {} +for i = 1, #KEYS do + redis.call('ZREM', KEYS[i], ARGV[i]) + table.insert(results, redis.call('ZCARD', KEYS[i])) +end +return results +"#; + +pub(super) const PARALLEL_COUNT: &str = r#" +-- Read the current in-flight count per gauge key (prunes expired slots +-- first so leaked slots do not inflate the reading). +-- KEYS: gauge zset keys. ARGV: per-key slot_ttl_seconds. +local time_reply = redis.call('TIME') +local now = tonumber(time_reply[1]) +local results = {} +for i = 1, #KEYS do + redis.call('ZREMRANGEBYSCORE', KEYS[i], '-inf', now - tonumber(ARGV[i])) + table.insert(results, redis.call('ZCARD', KEYS[i])) +end +return results +"#; + +pub(super) const TOKEN_INCREMENT: &str = r#" +local results = {} + +-- Process each key/increment_value/ttl triplet +for i = 1, #KEYS do + local key = KEYS[i] + local increment_value = tonumber(ARGV[i * 2 - 1]) + local ttl_seconds = tonumber(ARGV[i * 2]) + + -- Increment the value + local new_value = redis.call('INCRBYFLOAT', key, increment_value) + + -- Handle TTL: only set expire if ttl_seconds > 0 and key has no current TTL + -- ttl_seconds can be 0 (no TTL) or positive (set TTL) + if ttl_seconds and ttl_seconds > 0 then + local current_ttl = redis.call('TTL', key) + if current_ttl == -1 then + redis.call('EXPIRE', key, ttl_seconds) + end + end + + table.insert(results, new_value) +end + +return results +"#; diff --git a/litellm-rust/crates/redis-store/src/rate_limits/mod.rs b/litellm-rust/crates/redis-store/src/rate_limits/mod.rs new file mode 100644 index 00000000000..71b67d2ecc0 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/rate_limits/mod.rs @@ -0,0 +1,304 @@ +mod keys; +mod lua; +mod scripts; +mod types; + +use std::{sync::Arc, time::Duration}; + +use litellm_redis_client::{Declared, RedisBatch, RedisClient, Slot}; + +pub use self::keys::{Counter, Entry, Kind, Scope}; +pub use self::types::{ + AcquireReply, Applied, AtomicCounter, AtomicReply, Gauge, GuardedIncrement, GuardedReply, + Increment, Local, WindowCount, +}; +use crate::{ + Error, Number, Pending, RequestRedis, Send, + dual_cache::{DualCache, MemorySettings, PipelineIncrement}, + memory::{Clock, system_clock}, + remote::{Remote, RemoteSlot}, + request::pending, +}; + +/// The v3 rate limiter's keyspace: its memory tier, its Lua scripts and the plain counter writes +/// `DualCache` made for it on `main`. The admission logic stays with the caller. A script needs +/// Redis, so without one it declares nothing and answers `None`. +pub struct RateLimits { + cache: DualCache, +} + +impl RateLimits { + /// `backfill_ttl` is the `default_in_memory_ttl` of the `DualCache` this tier replaces. An + /// increment without a TTL takes the remote's default. + pub fn new(remote: Option, backfill_ttl: Option) -> Self { + Self::with_clock(remote, backfill_ttl, system_clock()) + } + + pub fn with_clock(remote: Option, backfill_ttl: Option, now: Clock) -> Self { + Self { + cache: DualCache::new( + Arc::new(RemoteSlot::new(remote)), + MemorySettings { + entries: None, + ttl: backfill_ttl, + }, + now, + ), + } + } + + pub fn has_redis(&self) -> bool { + self.cache.redis().get().is_some() + } + + /// `_group_keys_by_hash_tag`: indices of `scopes` sharing a cluster slot, in first-seen order, + /// or one group when the client is not a cluster and grouping is not forced. + pub fn slot_groups(&self, scopes: &[Scope], force: bool) -> Vec> { + let by_slot = force + || self + .cache + .redis() + .client() + .is_some_and(RedisClient::is_cluster); + if !by_slot { + return vec![(0..scopes.len()).collect()]; + } + scopes + .iter() + .enumerate() + .fold( + Vec::<(Slot, Vec)>::new(), + |mut groups, (index, scope)| { + let slot = Slot::for_key(scope.window_key()); + match groups.iter_mut().find(|(seen, _)| *seen == slot) { + Some((_, members)) => members.push(index), + None => groups.push((slot, vec![index])), + } + groups + }, + ) + .into_iter() + .map(|(_, members)| members) + .collect() + } + + pub fn get_local(&self, entry: &Entry) -> Result, Error> { + Ok(self.cache.memory().get(&entry.key())?) + } + + pub fn set_local( + &self, + entry: &Entry, + value: Local, + ttl: Option, + ) -> Result<(), Error> { + Ok(self.cache.memory().set(&entry.key(), value, ttl)?) + } + + /// `DualCache.async_get_cache`, backfilling memory for `backfill_ttl` or the store's default. + /// A key `request` already read as missing from Redis is not read again. + pub async fn get( + &self, + request: Option<&RequestRedis>, + entry: &Entry, + backfill_ttl: Option, + ) -> Result, Error> { + self.cache.get(request, &entry.key(), backfill_ttl).await + } + + pub fn sliding_window( + &self, + send: Send<'_>, + counters: &[Counter], + now: i64, + window_size: i64, + ) -> Option>> { + Some(pending(scripts::sliding_window( + &self.batch(send)?, + counters, + now, + window_size, + ))) + } + + pub fn read_counters( + &self, + send: Send<'_>, + counters: &[Counter], + ) -> Option>>> { + Some(pending(scripts::read_counters( + &self.batch(send)?, + counters, + ))) + } + + pub fn check_and_increment( + &self, + send: Send<'_>, + counters: &[AtomicCounter], + ) -> Option> { + Some(pending(scripts::check_and_increment( + &self.batch(send)?, + counters, + ))) + } + + pub fn guarded_increment( + &self, + send: Send<'_>, + increment: &GuardedIncrement, + ) -> Option> { + Some(pending(scripts::guarded_increment( + &self.batch(send)?, + increment, + ))) + } + + pub fn parallel_acquire( + &self, + send: Send<'_>, + gauges: &[Gauge], + slot_ttl: Duration, + slot_id: &str, + ) -> Option> { + Some(pending(scripts::parallel_acquire( + &self.batch(send)?, + gauges, + slot_ttl, + slot_id, + ))) + } + + pub fn parallel_release( + &self, + send: Send<'_>, + gauges: &[Scope], + slot_id: &str, + ) -> Option>> { + Some(pending(self.declare_release(send, gauges, slot_id)?)) + } + + pub fn parallel_count( + &self, + send: Send<'_>, + gauges: &[Scope], + slot_ttl: Duration, + ) -> Option>> { + Some(pending(scripts::parallel_count( + &self.batch(send)?, + gauges, + slot_ttl, + ))) + } + + pub fn token_increment( + &self, + send: Send<'_>, + increments: &[Increment], + ) -> Option>> { + Some(pending(self.declare_token_increment(send, increments)?)) + } + + /// `TOKEN_INCREMENT_SCRIPT` on the request's post-call batch. A failed script falls back to + /// `increment_many` for these increments once the batch has gone out. `false` without Redis. + pub fn token_increment_post_call( + self: &Arc, + request: &RequestRedis, + increments: Vec, + ) -> bool { + let Some(declared) = self.declare_token_increment(Send::PostCall(request), &increments) + else { + return false; + }; + let store = Arc::clone(self); + request.spawn(async move { + let Err(error) = declared.settled().await else { + return; + }; + crate::failure_log::redis_failure!( + &error, + "TTL preservation failed, falling back to regular pipeline" + ); + store.increment_many(&increments).await; + }); + true + } + + /// `PARALLEL_RELEASE_SCRIPT` on the request's post-call batch; the caller has already freed + /// the slot in memory. `false` without Redis. + pub fn parallel_release_post_call( + &self, + request: &RequestRedis, + gauges: &[Scope], + slot_id: &str, + ) -> bool { + let Some(declared) = self.declare_release(Send::PostCall(request), gauges, slot_id) else { + return false; + }; + request.spawn(async move { + if let Err(error) = declared.settled().await { + crate::failure_log::redis_failure!( + &error, + "parallel_release_script failed, the slot stays released in memory only" + ); + } + }); + true + } + + /// `DualCache.async_increment_cache`. A Redis failure answers with the memory count, and an + /// entry memory cannot add to with `None`. + pub async fn increment(&self, increment: &Increment) -> Option { + self.cache + .increment(&increment.counter.key(), increment.amount, increment.ttl) + .await + } + + /// `DualCache.async_increment_cache_pipeline`, whose failures only log. + pub async fn increment_many(&self, increments: &[Increment]) { + self.cache.increment_pipeline(&pipeline(increments)).await; + } + + /// `DualCache.async_increment_cache_pipeline_post_call`: memory now, the Redis increments on + /// the request's post-call batch. + pub fn increment_many_post_call(&self, request: &RequestRedis, increments: &[Increment]) { + self.cache + .increment_pipeline_post_call(request, &pipeline(increments)); + } + + fn declare_release( + &self, + send: Send<'_>, + gauges: &[Scope], + slot_id: &str, + ) -> Option>> { + Some(scripts::parallel_release( + &self.batch(send)?, + gauges, + slot_id, + )) + } + + fn declare_token_increment( + &self, + send: Send<'_>, + increments: &[Increment], + ) -> Option>> { + Some(scripts::token_increment(&self.batch(send)?, increments)) + } + + fn batch(&self, send: Send<'_>) -> Option { + self.cache.redis().batch(send) + } +} + +fn pipeline(increments: &[Increment]) -> Vec { + increments + .iter() + .map(|increment| PipelineIncrement { + key: increment.counter.key(), + amount: increment.amount, + ttl: increment.ttl, + }) + .collect() +} diff --git a/litellm-rust/crates/redis-store/src/rate_limits/scripts.rs b/litellm-rust/crates/redis-store/src/rate_limits/scripts.rs new file mode 100644 index 00000000000..04bdad38268 --- /dev/null +++ b/litellm-rust/crates/redis-store/src/rate_limits/scripts.rs @@ -0,0 +1,303 @@ +use std::{sync::LazyLock, time::Duration}; + +use litellm_python_compat::repr::float_repr; +use litellm_redis_client::{ + Command, Declared, Error, RedisBatch, Script, Value, + commands::{self, unexpected_reply}, +}; + +use super::{ + keys::{Counter, Scope}, + lua, + types::{ + AcquireReply, Applied, AtomicCounter, AtomicReply, Gauge, GuardedIncrement, GuardedReply, + Increment, WindowCount, + }, +}; +use crate::Number; + +static BATCH_RATE_LIMITER: LazyLock', + "deprecation_source": "javascript:alert(1)", + } + ) + email = ModelDeprecationEmail( + model=model, + usage_model=model.litellm_model, + recipient_email="owner@example.com", + milestone=DeprecationEmailMilestone.THIRTY_DAYS, + key_count=3, + requests=12, + team_names=("Research",), + start_date=date(2026, 10, 3), + end_date=NOW.date(), + ) + rendered = render_model_deprecation_email(email, base_url="https://gateway.example") + assert "/public/assets/logos/litellm_logo.png" in rendered.html + assert "litellm-listing.s3.amazonaws.com" not in rendered.html + assert "