Extend cache-redis so RedisTopology::Cluster routes single-key commands by
hash slot, groups pipelines by slot while keeping reply order, scans and
scoped-flushes every primary, and fans admin commands out to all nodes.
The bridge projects RedisClusterCache startup_nodes into the typed topology,
accepts the exact RedisClusterCache identity, guards nodes_manager state,
and falls back to Python for anything it cannot project.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add an aws_textract OCR provider on the Rust route, with no Python path. The
detect-document-text model returns plain lines and analyze-document renders
layout and tables as markdown. Both use Textract's synchronous API, so a
multi-page PDF or TIFF is rejected with an error that names the single-page
limit. A call with no region fails instead of falling back to Bedrock's default
SigV4 covers the request body, and host hooks can rewrite that body before it
is sent. litellm-http now has OutboundRequest, which serializes the body once,
shows those bytes to a RequestSigner and is the only thing a route can send.
Chat, audio transcription and OCR build it after their hooks ran, so a callback
that redacts the body still produces a valid Bedrock or Textract signature
ChatCompletionsAuth and AudioTranscriptionAuth are replaced by
litellm_auth::RequestAuth, and one helper in core turns it into a signed or
unsigned request. Audio transcription now signs only the AWS header set and
rejects a forwarded header that SigV4 computes, the same as chat
The OCR catalog routes aws_textract as Rust required, and the dispatch context
reads the provider from the model prefix so a provider scoped rule can match
The OCR route called back into Python's get_secret_str for every env
fallback. With no secret manager configured that is os.environ behind a GIL
hop, and with one configured it blocked a tokio worker on vault I/O and also
sent the Azure and GCP identity variables, which Python reads with os.getenv,
to the vault. The other Rust routes already read the process environment.
Read the process environment here too. When litellm would read secrets from
a secret manager, decline the Rust route so the Python route serves the call
with the vault-backed keys
Python resolves the Vertex project and location as call params, then the
litellm.vertex_project / litellm.vertex_location globals, then env, and
Azure AD token refresh from litellm.enable_azure_ad_token_refresh alone.
Native OCR skipped the globals, so a config.yaml litellm_settings value
silently fell through to the credential's project and us-central1, and a
managed identity setup without an API key failed. The bridge now reads
them through a provider_defaults settings group into OcrSettings, and
VertexConfig / AzureAuthInputs slot them in at Python's precedence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Python reads every provider credential fallback (MISTRAL_API_KEY,
AZURE_AI_API_KEY, AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, Azure AD and
Vertex env, ...) through get_secret_str, which consults the configured
key_management_system before os.environ. Native OCR read std::env
directly, so a key held only in the vault went missing and a stale env
copy silently won. OcrClient now carries an injected secret Lookup that
the connection exposes to providers and auth crates; the bridge backs it
with settings.secret -> get_secret_str, pure Rust keeps the process env.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native OCR hardcoded the default of five Python constants that come from
env vars, so an operator setting them saw no effect:
REQUEST_TIMEOUT (Rust used 600s, Python 6000s), MAX_IMAGE_URL_DOWNLOAD_SIZE_MB
(0 disables document downloads), AZURE_OPERATION_POLLING_TIMEOUT,
AZURE_DOCUMENT_INTELLIGENCE_API_VERSION and
AZURE_DOCUMENT_INTELLIGENCE_DEFAULT_DPI. OcrSettings reads them through
Lookup with Python's parsing, the bridge builds it per call and OcrClient
carries it into the connection. A zero per-call timeout now falls back to
REQUEST_TIMEOUT, matching `timeout or request_timeout`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
custom_httpx mirrored a Python module that mixes transport plumbing with
OCR orchestration. The transport half (media fetcher, transport errors,
request and header helpers) now lives in litellm-http next to the pool,
TLS, proxies and settings, and the OCR request handler moves to
base_llm/ocr/handler.rs. Drops the unused deserialize_optional_param and
stale dead_code allows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Settings sources beyond HTTP (media fetch, Azure Document Intelligence,
Vertex, timeouts) need the same env lookup and precedence merge, so move
them out of litellm-http into core_utils::settings. Lookup readers name the
Python idiom they mirror: get keeps a present empty value like
os.getenv(X, fallback), truthy drops it like an `or` chain, enabled only
switches on for "true". SSL_CERT_FILE now reads through truthy, matching
Python's `if ssl_cert_file and ...` check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each source (per-call kwargs, environment variables, the Python module) now
builds an HttpSettingsLayer, and HttpSettings::from_layers merges them with
explicit precedence. The aiohttp and httpx proxy-env rule is resolved once in
the merge, so HttpSettings carries a single trust_proxy_env flag
HttpClientConfig::resolve becomes From<&HttpSettings> for Resolution and client_builder becomes TryFrom<&HttpClientConfig> for reqwest::ClientBuilder, matching the rustls conversion. The verify decision moves into From<&HttpSettings> for Verify, and the proxy environment rule moves next to its flags as HttpSettings::trusts_proxy_env. The curve and cipher results are read with transpose and a default selection, which removes the tuple destructuring
litellm-http now builds the rustls config itself, so one route-neutral place covers roots, the client certificate, ALPN, ssl_ecdh_curve and ssl_security_level. A curve picks the single key exchange group. A cipher string restricts the TLS 1.2 suites it names, and entries rustls cannot express, such as @SECLEVEL=1, are logged once and skipped.
user_url_validation and user_url_allowed_hosts are applied by the media fetcher. Document downloads honor the environment proxy whenever provider calls do, keeping the per-hop address check, and stay on the pinned resolver when no proxy applies.
AIOHTTP_SO_KEEPALIVE, AIOHTTP_TCP_KEEPIDLE, AIOHTTP_TCP_KEEPINTVL, AIOHTTP_TCP_KEEPCNT and AIOHTTP_KEEPALIVE_TIMEOUT map onto the client. A client= argument and a live SSLContext are ignored
The proxy attaches its shared aiohttp session to every request as shared_session, so declining on it sent every proxy OCR call to Python, which never uses that session for OCR. aclient_session is a litellm global and never a call argument, so that check could not match. The proxy-shaped lifecycle test now asserts the call was served by Rust
litellm.user_url_validation and litellm.user_url_allowed_hosts are only implemented by the Python document fetcher, so an allowlisted internal document was rejected by the Rust route's network policy. The bridge now declines when either is changed from its default
Python's aiohttp transport reads HTTP(S)_PROXY on every request unless disable_aiohttp_trust_env is set, so the Rust clients now do the same instead of requiring aiohttp_trust_env. Transport error messages include reqwest's source chain, so a rejected certificate or refused connection is no longer reported as just 'error sending request'
