pyo3 0.23.5 hard-caps the interpreter at Python 3.13, so building the
native bridge against a 3.14 interpreter aborts inside pyo3-ffi's build
script before anything links. This raises pyo3 and pyo3-async-runtimes
to 0.29 (currently the newest line, and the range starting at 0.26 that
supports 3.14) and migrates the three call sites whose APIs were renamed
across that range: Python::with_gil is now Python::attach and
Python::allow_threads is now Python::detach. On a GIL-enabled interpreter
those are pure renames with identical semantics, so behavior on 3.10
through 3.13 is unchanged
Verified by compiling the native module for cp313 and cp314 and driving
it directly on both interpreters: gil_stats reports exactly one GIL
release per sync OCR call and the async path completes, matching the
0.23.5 baseline. cargo fmt, clippy, and the workspace tests pass on both
3.13 and 3.14 with the lockfile locked, and the lock churn is confined to
the pyo3 crates
Part of #26343; addresses the pyo3 build failure reported in #33116
* feat(vertex): add shared Rust Google authentication
Mint and refresh Vertex OAuth access tokens in the ai-gateway host layer via the official google-cloud-auth crate, preserving the inline service-account JSON, ADC and GOOGLE_APPLICATION_CREDENTIALS contract with library-managed caching/refresh and no hand-rolled signing. Core stays auth/IO-free: it only classifies the bearer source and rejects Google AIza API keys rather than sending them as OAuth. Also fix Vertex Mistral rawPredict endpoint construction so the global location targets aiplatform.googleapis.com.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(vertex): address review; SHA-256 cache key and env credentials
Move newly introduced Vertex constants into crate-level constants.rs, drop the doc/inline comments added by the auth PR, and replace the DefaultHasher u64 credential-cache key with a collision-resistant SHA-256 digest (sha2 is now a non-optional ai-gateway dependency so the non-server host path can use it).
When no explicit vertex_credentials optional param is supplied, the host now reads VERTEXAI_CREDENTIALS from the environment before falling back to standard ADC/GOOGLE_APPLICATION_CREDENTIALS, without exposing credential content. Adds tests covering env-based inline credential selection and that distinct credential sources cannot collide on one cache entry.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* build(vertex): pin google-cloud-auth to =1.13.0 for dependency-age policy
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(vertex): bound credential cache, reject AIza in auth header, data-minimize auth errors
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* build(vertex): pin google-cloud-auth to =1.9.0 and adopt MSRV-aware resolver for Rust 1.86
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(vertex): require well-formed Bearer scheme on caller Authorization header
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(vertex): require exactly one well-formed Bearer authorization header
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(vertex): key credential cache by content and pin rustls exactly
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(bridge): remove accidentally committed native extension binary
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(vertex): split base auth and shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ocr-errors): preserve public error and timeout contracts
Classify reqwest timeouts as a typed CoreError::Timeout and add
CoreError::public_status_code() as the single exhaustive mapping from a
typed core error to its public HTTP status. The Python bridge raises a
typed RustOcrError carrying that status instead of a generic
RuntimeError, and litellm.ocr()/aocr() translate it into the matching
public exception so AuthenticationError/401, NotFoundError/404,
BadRequestError/4xx, InternalServerError/5xx and Timeout are preserved
end to end instead of collapsing to APIConnectionError/500.
Reject empty or whitespace-only 200 bodies so they fail loudly rather
than becoming an empty OCR success; invalid JSON already fails. Upstream
error bodies stay bounded and sanitized.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ocr-errors): map invalid OCR input to BadRequestError
Invalid caller input (bad document type, non-dict document, unusable
file input) was raised as a plain ValueError inside ocr()/aocr() and
collapsed to APIConnectionError/500 through the generic handler. Route
every OCR failure through one _map_ocr_exception host mapping: typed
RustOcrError keeps its status-based public exception, a plain ValueError
becomes BadRequestError/400, and a pydantic ValidationError (malformed
response, not client input) stays on the generic path.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(ocr-errors): data-minimize public errors and preserve status-specific exceptions
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(ocr-errors): preserve exact unknown status and privatize input error
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(ocr-errors): exhaustive match mapper and typed error tests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ocr-errors): sanitize InvalidRequest public message
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(ocr-errors): raise typed public union and drop NotFound provider miss
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ocr-errors): map malformed provider responses to sanitized 500 and hide input-error detail
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(litellm-rust): move provider transforms into litellm-core + crate allowlist test
* feat(litellm-rust): ai-gateway absorbs route I/O (io/) with lib+server feature split
* refactor(litellm-rust): point python-bridge at litellm-ai-gateway
* build(litellm-rust): macOS pyo3 dynamic_lookup linker flag for cdylib builds
* docs(litellm-rust): 3-crate map in README/AGENTS + refresh CLAUDE boundary
* refactor(litellm-rust): update workspace members to the three crates
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Add litellm rust workspace with mistral ocr bridge
* address greptile rust ocr feedback
* Simplify rust ocr entrypoint
* rust(core): add Auth/Http/Network error variants
* rust: add reqwest (rustls-tls) workspace dependency
* rust(providers): depend on reqwest
* rust(mistral): add complete_url + resolve_api_key helpers
* rust(providers): end-to-end run_ocr orchestrator with shared client + timeout
* rust(bridge): depend on litellm-core
* rust(bridge): add GIL release accounting
* rust(bridge): end-to-end ocr() + gil_stats(), GIL released for HTTP
* ocr: add minimal Rust bridge (use_litellm_rust + rust_ocr)
* ocr: route mistral to Rust when enabled; keep bare-str file rejection
* litellm: export use_litellm_rust()
* test(ocr): cover Rust OCR routing + toggle
* rust: stop ignoring Cargo.lock
* rust: commit Cargo.lock for reproducible builds
* ci(rust): build with --locked to enforce the lockfile
* Potential fix for pull request finding 'CodeQL / Module-level cyclic import'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for pull request finding 'CodeQL / Module-level cyclic import'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* ocr: lazily import rust bridge inside ocr() to break the import cycle the CodeQL autofix mangled
* ocr: guard OCRResponse under TYPE_CHECKING so the annotation resolves
* ocr: modernize rust_bridge typing (PEP 604, drop typing.Any/Dict) to satisfy strict-rule gate
* ci: re-trigger checks
* ci: re-trigger checks
* Potential fix for pull request finding 'CodeQL / Cyclic import'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Potential fix for pull request finding 'CodeQL / Cyclic import'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* ocr: make rust_bridge a leaf (return raw dict, no litellm import) so the CodeQL autofix stops re-breaking it
* ocr: wrap rust bridge dict into OCRResponse at the call site
* test(ocr): assert rust_ocr returns the raw bridge dict
* test(interactions): add budget_exceeded to expected status enum (Google updated the published spec)
* ocr: resolve mistral key via get_secret_str before the rust path (secret-manager parity)
* test(ocr): assert rust path resolves key via secret manager
* rust(mistral): document that secret-manager resolution happens on the Python side
* fix(ocr): honor timeout, logging, and missing-bridge fallback on Rust OCR path
- Forward the caller's timeout into the Rust bridge so the fixed 600s client
ceiling no longer overrides shorter deadlines or the library default.
- Run update_from_kwargs and pre_call before invoking the Rust shortcut so
observability, callbacks, and spend tracking match the Python path.
- Fall back to the Python OCR path when litellm_python_bridge isn't importable
instead of raising ImportError to callers.
- Truncate upstream Mistral OCR error bodies before they cross the host
boundary to avoid leaking document or prompt contents in CoreError::Http.
* fix(ocr): log resolved api_base and headers on Rust path
* refactor(ocr): inject the rust bridge via a typed seam, drop the importlib cycle dodge
The rust OCR path was reached through importlib.import_module both for the
bridge module and for probing the native extension, purely to keep CodeQL from
flagging a cyclic import. rust_bridge has no litellm imports, so it is a leaf
and main.py can import it statically without any cycle; the dance is gone
Bridge selection now goes through a typed RustOcr Protocol and a load_rust_ocr()
seam. use_litellm_rust() takes an optional injected bridge, so an embedder (or a
test) can supply an alternative without reaching into sys.modules. The rust-path
body moves into _run_rust_ocr(), which receives its dependencies (the bridge
callable, the logging object, the key resolver) as arguments and is unit-tested
by passing fakes in rather than monkeypatching class methods or module globals
The tests are rewritten around that injection: the bridge is provided via
use_litellm_rust(ocr=...), pre_call is observed through a spy logging object, and
the missing-extension fallback is covered by load_rust_ocr() returning None when
no wheel is built. Types were tightened along the way (a cast for the logging
object, OCRResponse.model_validate for the bridge result) so no basedpyright
per-rule count increases
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(ocr): preserve injected rust bridge across toggle calls
use_litellm_rust() unconditionally assigned the keyword default of None to
_rust_ocr_impl, so any call without ocr= silently dropped a previously
injected bridge. Use a sentinel default so omission preserves the impl
while ocr=None still clears it explicitly.
* ci: run tests/test_litellm/ocr in the misc unit-test group
The OCR test directory was not wired into any CI test group, so its
coverage never uploaded to Codecov and patch coverage failed for new
OCR lines. Add it to the misc group.
* test(ocr): cover compiled-extension load and Python fallback paths
Adds two tests so the Rust bridge module hits 100% and the ocr()
fallback-to-Python branch is exercised:
- load_rust_ocr() returning the compiled extension's ocr callable
- ocr() degrading to the HTTP handler when no bridge is available
* style(ocr): use PEP 604 X | None annotations in rust_bridge
Converts Optional[X]/Union[...] to the X | None form so the new OCR
code stays under the UP045 strict-rule budget gate (lint job). Safe at
runtime — the module already has 'from __future__ import annotations'.
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
