scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-13 23:11:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 101
litellm/litellm-rust
History
Exact
yuneng-jiang 2530e32727
chore(release): backport 11 staging PRs onto patch-1.93.0rc2 for the 1.93.0 stable cut (#33847) 
* fix(ci): bump pillow to 12.3.0 to resolve osv-scan CVEs (#33093)

(cherry picked from commit 20e646c49a)

* chore(deps): pin httplib2 and setuptools transitive floors (#33233)

Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected

(cherry picked from commit 8b323202ec)

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244)

* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models

* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models

Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 71dffc1e9a)

* build: raise requires-python cap to <3.15 so Python 3.14 installs current releases (#33438)

* build: drop requires-python upper cap so Python 3.14 resolves to current releases

The <3.14 cap made pip on Python 3.14 fall back to litellm 1.83.7, a
pre-April release whose old auth flow fails with 400s. The cap was added
in d9a460277a because deps lacked 3.14 wheels and uv could not resolve
the 3.14 split; both are fixed now via the existing python_version
markers plus a ddtrace version split (2.x has no cp314 wheels, 3.16+
does). Verified on 3.14.5: uv sync --all-extras installs, litellm and
proxy_server import (rust bridge falls back to pure python), real
provider calls succeed sync/async/streaming, and the core-utils test
suite passes.

* build: cap requires-python at <3.15 and keep ddtrace on one major per python band

Reviewer preference to bound the supported window at the newest tested
minor rather than leaving it open-ended, and Greptile flagged the
ddtrace 3.14+ range spanning two majors; every ddtrace 4.x ships cp314
wheels so the band is now >=4.0,<5.0, matching the single-major
convention of the 2.x band.

(cherry picked from commit c6d49a85b2)

* build(deps): update ddtrace to the 4.x line

A single ddtrace constraint now covers every supported Python version, so this collapses the version split introduced in #33438. Also aligns the build_from_pip image pin and updates the type-only Tracer import to its current module path

(cherry picked from commit edc38eab34)

* fix(docker): restore litellm-proxy-extras source dir in runtime images (#33592)

* fix(docker): restore litellm-proxy-extras source dir in runtime images

#30243 narrowed the runtime stage to an allowlist COPY, which dropped
/app/litellm-proxy-extras from the published images. Downstream
migration jobs point prisma migrate deploy at that path; with the
schema gone (or a schema with no adjacent migrations dir, where prisma
exits 0 without applying anything) those jobs went green while never
migrating the database. Restore the folder in all three runtime stages
and assert in image-scan that the schema and a non-empty migrations dir
ship at the source path

* chore(ci): drop image-scan migration-assets assertion

(cherry picked from commit 111d447e1b)

* fix(model_armor): restore reference attachments via skip_unscannable_attachments and remove the attachment count cap (#33554)

* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through

* fix(model_armor): wire skip_unscannable_attachments through guardrail config

* fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping

* fix(model_armor): remove the per-request attachment count cap and scan all attachments

---------

Co-authored-by: yucheng <yucheng@berri.ai>
(cherry picked from commit 0d7b0f708b)

* build(rust): raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 (#33798)

pyo3 0.23.5 hard-caps the interpreter at Python 3.13, so building the
native bridge against a 3.14 interpreter aborts inside pyo3-ffi's build
script before anything links. This raises pyo3 and pyo3-async-runtimes
to 0.29 (currently the newest line, and the range starting at 0.26 that
supports 3.14) and migrates the three call sites whose APIs were renamed
across that range: Python::with_gil is now Python::attach and
Python::allow_threads is now Python::detach. On a GIL-enabled interpreter
those are pure renames with identical semantics, so behavior on 3.10
through 3.13 is unchanged

Verified by compiling the native module for cp313 and cp314 and driving
it directly on both interpreters: gil_stats reports exactly one GIL
release per sync OCR call and the async path completes, matching the
0.23.5 baseline. cargo fmt, clippy, and the workspace tests pass on both
3.13 and 3.14 with the lockfile locked, and the lock churn is confined to
the pyo3 crates

Part of #26343; addresses the pyo3 build failure reported in #33116

(cherry picked from commit f3d20153b3)

* build(deps): allow redisvl, pypdf, and openapi-core on Python 3.14 (#33801)

Remove the python_version < '3.14' environment markers from redisvl,
pypdf, and openapi-core now that all three install and import cleanly
on 3.14. The relock is marker-only: no package version changed for any
Python branch, and the locked versions (redisvl 0.4.1, pypdf 6.13.3,
openapi-core 0.22.0) now serve 3.14 as well. semantic-router and
aurelio-sdk stay gated because every published release caps
python_requires below 3.14

(cherry picked from commit 967d934484)

* build(deps): bump mcp lock to 1.28.1 to clear image-scan findings (#33803)

* build(deps): bump mcp lock to 1.28.1 to clear image-scan findings

* build(deps): require mcp>=1.28.1

(cherry picked from commit 40e914cfa7)

* fix(proxy): source /v1/models token limits from the cost map instead of Router.get_model_group_info (#33721)

* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info

Resolves the per-model get_model_group_info fan-out on GET /v1/models
(and /models) that pegged the event loop on wildcard listings (#33636).
create_model_info_response now reads max_input_tokens/max_output_tokens
from litellm.get_model_info (the static cost map) rather than the router,
which aggregated and deepcopied every deployment in a group per listed
model.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage

Inject the cost-map lookup (defaulting to litellm.get_model_info) so the
except and max_output_tokens branches are exercised deterministically and
the token-limit tests no longer hardcode mutable cost-map values.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup

Add Router.get_configured_token_limits, an O(1) model-name index lookup that
reads a concrete deployment's configured max_input_tokens/max_output_tokens
without triggering pattern matching or deep copies. create_model_info_response
layers this over the cost map so custom deployments absent from the cost map
still surface their limits, and admin-configured limits override cost-map
defaults, while wildcard-expanded names stay on the fast path.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 8536e3b80e)

---------

Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
2026-07-18 14:19:56 -07:00
..
crates chore(release): backport 11 staging PRs onto patch-1.93.0rc2 for the 1.93.0 stable cut (#33847) 2026-07-18 14:19:56 -07:00
.gitignore feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033) 2026-06-23 13:16:47 -07:00
ADDING_A_PROVIDER.md feat: add openai realtime translation layer to litellm-rust (1/2) (#31129) 2026-06-23 15:48:51 -07:00
AGENTS.md refactor(litellm-rust): dissolve providers into core + ai-gateway (strict 3-crate layers) (#31218) 2026-06-24 12:22:43 -07:00
Cargo.lock chore(release): backport 11 staging PRs onto patch-1.93.0rc2 for the 1.93.0 stable cut (#33847) 2026-07-18 14:19:56 -07:00
Cargo.toml chore(release): backport 11 staging PRs onto patch-1.93.0rc2 for the 1.93.0 stable cut (#33847) 2026-07-18 14:19:56 -07:00
CLAUDE.md feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134) 2026-06-24 15:25:10 -07:00
README.md refactor(litellm-rust): dissolve providers into core + ai-gateway (strict 3-crate layers) (#31218) 2026-06-24 12:22:43 -07:00

README.md

LiteLLM Rust

This workspace contains the staged Rust implementation for LiteLLM.

Rust starts as a pure transform core used by the existing Python host. Python continues to own auth, configuration, network I/O, retries, routing, logging, callbacks, spend tracking, and customer plugins until each Rust path has parity coverage and production evidence.

Crates

Crate Role Pure / I/O
litellm-core Translation layer — types, route contracts (traits), provider transforms (modules under providers/), and the router. Builds requests/responses; no network. Pure
litellm-ai-gateway Routes + host — the only crate that touches the network. HTTP/WebSocket I/O (modules under io/) plus the axum server binary (behind the server feature). I/O
litellm-python-bridge PyO3 cdylib exposing Rust to the litellm Python SDK — a thin adapter over litellm-ai-gateway's I/O. Binding

Dependency direction (acyclic): litellm-core ← litellm-ai-gateway ← litellm-python-bridge.

Layout

crates/
  core/           Route contracts, shared pure types, errors, and templates.
    src/ocr/
  providers/      Provider-specific pure transforms.
    src/mistral/ocr/transformation.rs
  python-bridge/  PyO3 bridge for Python LiteLLM.

The folder shape should follow the Python provider tree: providers/src/<provider>/<route>/transformation.rs. The bridge should expose one function per top-level route, starting with ocr(payload).

Checks

Run these before pushing Rust changes. GitHub Actions runs the same checks for changes under litellm-rust/.

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace