Post-call pipeline rewrites on buffered streams failed open on three shapes:
chat streams with n > 1 (the rebuilt response collapsed every choice into
index 0), streams that ended without a finish marker, and Responses streams
whose final event carried no response envelope.
The chat handler now rebuilds the ended stream one choice index at a time and
writes each choice's rewrite back to that choice's buffered deltas. The
Anthropic handler writes an unended stream's rewrite across its text deltas.
The Responses handler spreads an envelope-less rewrite over the buffered
output_text events, still failing open when a scanned event cannot be placed.
Tool-call rewrites on n > 1 chat streams keep failing open.
Applying a database row dropped the runtime layer for every key the row
carried, including keys the config file owns. Those runtime entries hold
the env-resolved config values, so after a reload a key written as
os.environ/<NAME> read back as that literal string. The store now keeps
the runtime entry for a key the config owns and clears only the rest.
Visible as store_model_in_db silently turning itself off: the reload read
the raw reference, coerced it to False, and overwrote the resolved global.
A write into a settings store for a key the config file declares used to
land in the runtime layer and then lose to the config on every read, so
the caller saw success while nothing changed. It now raises
ConfigOwnedKeyError, and the allowed-IP routes turn that into a 400
naming the key instead of reporting success on a list they never changed.
Both allowed-IP routes now build a new list rather than mutating the one
the config layer holds, and the os.environ resolver rebuilds the config
it is given instead of writing back into it, so a reader can no longer
corrupt the raw values the store keeps for provenance.
The database reload leaves a config-owned key alone rather than writing a
normalized copy back over it, which would now raise and abort the rest of
the reconcile pass.
retrieve_batch returned a terminal batch from the DB before checking that the key may use the model encoded in a unified batch id; the grant check now runs right after pre-call processing. The vector store file list helper authorized data["model"] through handle_model_based_routing even when the vector store registry set it server-side and even with no caller, which crashed on a None key; it now authorizes only a caller-supplied hint and resolves credentials directly.
PR #40243 started carrying the upstream error body on InternalServerError so the Responses response.failed event can report the provider's code and message, and openai's APIError.__init__ took the body's type along with it. The proxy then answered an OpenAI-compatible upstream 500 with type server_error while a 502 and a 503 kept internal_server_error, and the integration contract in test_observed_routing.py went red. Pin the type the way RateLimitError pins throttling_error, keeping the body.
Resolves handler.py against main's SearchOutcome refactor: the file is
main's version plus this PR's substantive hunks only (the RichWebSearchInput
import, the rich= wiring at the three _execute_search call sites, the
_rich_search_input and _provider_supports_rich_search helpers, and the
_execute_search forwarding), so the 88-column re-wrap noise the PR carried
is gone and the diff against main is the feature alone. RichWebSearchInput
sits beside main's new SearchSucceeded/SearchFailed types, and main's two
_execute_search test stubs accept the new rich argument.
#41515 added the cost map entry transcribe/StartTranscriptionJob under a
new litellm_provider value "transcribe" without registering that provider
anywhere else, so litellm.models_by_provider had no "transcribe" key.
test_models_by_provider derives its provider set from the cost map itself,
so it went red on main. The user-visible half is that get_provider_models
returned None for the provider, which get_known_models_from_wildcard turns
into an empty list, leaving a transcribe/* key or team grant resolving to
no models.
Mirror the aws_polly registration: an enum member, a model set, an
ingestion branch, and a models_by_provider entry. Amazon Transcribe is
reached through the pass-through route rather than the Add Model form, so
it joins the frozen unlisted set the Add Model drift test tracks.
Python resolves the Vertex project and location as call params, then the
litellm.vertex_project / litellm.vertex_location globals, then env, and
Azure AD token refresh from litellm.enable_azure_ad_token_refresh alone.
Native OCR skipped the globals, so a config.yaml litellm_settings value
silently fell through to the credential's project and us-central1, and a
managed identity setup without an API key failed. The bridge now reads
them through a provider_defaults settings group into OcrSettings, and
VertexConfig / AzureAuthInputs slot them in at Python's precedence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
load_config used to return a local general_settings dict that it had
normalized in place, turning the configured role_permissions entries into
RoleBasedPermissions objects. It now returns the SettingsStore, which never
saw that write, so JWT auth received raw dicts and every request failed with
"'dict' object has no attribute 'role'" whenever role_permissions was set.
Convert the entries in the consumer instead, with a TypeAdapter, so the value
is parsed wherever it comes from. load_config keeps validating at boot, so a
malformed entry still fails startup rather than the first request.
Python reads every provider credential fallback (MISTRAL_API_KEY,
AZURE_AI_API_KEY, AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, Azure AD and
Vertex env, ...) through get_secret_str, which consults the configured
key_management_system before os.environ. Native OCR read std::env
directly, so a key held only in the vault went missing and a stale env
copy silently won. OcrClient now carries an injected secret Lookup that
the connection exposes to providers and auth crates; the bridge backs it
with settings.secret -> get_secret_str, pure Rust keeps the process env.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native OCR hardcoded the default of five Python constants that come from
env vars, so an operator setting them saw no effect:
REQUEST_TIMEOUT (Rust used 600s, Python 6000s), MAX_IMAGE_URL_DOWNLOAD_SIZE_MB
(0 disables document downloads), AZURE_OPERATION_POLLING_TIMEOUT,
AZURE_DOCUMENT_INTELLIGENCE_API_VERSION and
AZURE_DOCUMENT_INTELLIGENCE_DEFAULT_DPI. OcrSettings reads them through
Lookup with Python's parsing, the bridge builds it per call and OcrClient
carries it into the connection. A zero per-call timeout now falls back to
REQUEST_TIMEOUT, matching `timeout or request_timeout`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
custom_httpx mirrored a Python module that mixes transport plumbing with
OCR orchestration. The transport half (media fetcher, transport errors,
request and header helpers) now lives in litellm-http next to the pool,
TLS, proxies and settings, and the OCR request handler moves to
base_llm/ocr/handler.rs. Drops the unused deserialize_optional_param and
stale dead_code allows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reqwest and hyper each read HTTP(S)_PROXY, ALL_PROXY and NO_PROXY from the
process on their own, so tests could not inject them and the pooled client
key ignored proxy changes. EnvironmentProxies now reads them through
Lookup with the same precedence hyper used, the resolved config carries
them (empty when the transport does not trust the env), and both the
provider clients and the media fetcher build from that one value.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Settings sources beyond HTTP (media fetch, Azure Document Intelligence,
Vertex, timeouts) need the same env lookup and precedence merge, so move
them out of litellm-http into core_utils::settings. Lookup readers name the
Python idiom they mirror: get keeps a present empty value like
os.getenv(X, fallback), truthy drops it like an `or` chain, enabled only
switches on for "true". SSL_CERT_FILE now reads through truthy, matching
Python's `if ssl_cert_file and ...` check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each source (per-call kwargs, environment variables, the Python module) now
builds an HttpSettingsLayer, and HttpSettings::from_layers merges them with
explicit precedence. The aiohttp and httpx proxy-env rule is resolved once in
the merge, so HttpSettings carries a single trust_proxy_env flag
A pre-signed batch/file request (Mistral, Bedrock) carries its auth header
inside the transformed request body, which pre_call logs verbatim into
raw_request_typed_dict and raw-request callbacks, leaking the provider key.
Mask the nested headers channel before handing the request to pre_call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HttpClientConfig::resolve becomes From<&HttpSettings> for Resolution and client_builder becomes TryFrom<&HttpClientConfig> for reqwest::ClientBuilder, matching the rustls conversion. The verify decision moves into From<&HttpSettings> for Verify, and the proxy environment rule moves next to its flags as HttpSettings::trusts_proxy_env. The curve and cipher results are read with transpose and a default selection, which removes the tuple destructuring
KeyExchangeGroup and Tls12CipherSuite parse with FromStr and fail with Unsupported, so a setting rustls cannot honor is a typed error instead of a missing value. The cipher string conversions cannot fail and use From. The rustls ClientConfig is built with TryFrom<&HttpClientConfig>, and the built-in root store is constructed in one expression
