When a deployment priced one OCR batch family and the other still needed a
published rate, a failed cost-map lookup returned zero for the whole line and
discarded the deployment rate that was already resolved. Those pages were
billed as free. The lookup failure now only logs, and the families the
deployment prices are billed at the configured rate
LitellmParams mixes every provider config model into one class, so the
Javelin api_version default of "v1" reached the Azure Content Safety
guardrails whenever config.yaml omitted api_version and Azure answered 404.
The shared field now defaults to None, Javelin keeps filling in "v1" itself,
and the Azure guardrails fall back to the documented 2024-09-01 at request
time so a DB update that omits api_version stays on the default too.
POST /v1/rag/ingest authorized the managed vector store the request
named but then handed the raw request options to the ingestion
pipeline, which defaults to OpenAI. A request naming only a registered
store id uploaded the document to OpenAI Files, got an OpenAI 400, and
answered HTTP 200 with status "failed"; naming azure_ai explicitly
escaped as a 500.
The store's provider and litellm_params now merge into the request the
way /v1/rag/query already does (store wins, None values dropped), the
merged provider is checked against the ingestion registry before any
upload so unsupported providers get a 400 naming the supported ones, and
persistence keeps reading the caller's original options so registry
credentials never reach the database. A registry store with no database
row is no longer written as a new row.
The router coverage gate in code-quality flags every router.py function
no router test calls by name, and the two helpers get_credential_deployment
gained (the team public-name lookup and the team-aware wildcard lookup)
were only reached through it. Each now has a test of its own: the
public-name lookup resolves only for the owning team, and the wildcard
lookup prefers the team's own pattern over the shared one and never hands
another team's wildcard deployment to a caller outside that team.
The vector-store file routes accept a model hint through the ?model= query
param and the x-litellm-model header. That hint was authorized with a hand
rolled check that covered only the key allowlist and the team allowlist, so
a key restricted by its project's model grant, a team-member restriction, or a
key config still routed through the hinted deployment. Greptile flagged the
gap as a P1 on the replacement PR.
The hint now goes through the same authorize_model_for_key path the batches
and files routes use, which runs can_key_call_resolved_model with every rule
the proxy enforces elsewhere. Keys those extra rules deny now get a 403 on
these routes. The two remaining behavioral differences are edge cases the
old check tolerated: a key whose team_models is set without a team_id no
longer runs the team allowlist, and a key with a config set skips the key
allowlist, both matching the rest of the proxy.
The regression test caches a project whose grant excludes the hinted model
and asserts the request is refused before any deployment lookup. The two
patch() calls on litellm.proxy.proxy_server carry a test-quality-ok reason
because can_key_call_resolved_model reads prisma_client and
user_api_key_cache through a lazy module import with no injection seam.
A batch retrieved by its model-encoded id takes the direct (non-router) path,
which resolved credentials without stamping the deployment's model_info, so a
completed batch on a deployment with its own per-page pricing was billed at
the published rate with an empty model_id on the spend row.
Extract the router's credential lookup into get_credential_deployment and
stamp the resolved deployment's model_info onto the retrieve call the way the
router does for routed calls.
* fix(proxy): dispatch llm_api_check moderation through during_call_hook
ProxyLogging.during_call_hook only ran async_moderation_hook for CustomGuardrail callbacks, so a
CustomLogger such as the prompt injection detector with llm_api_check enabled never called the
configured moderation model. Dispatch any CustomLogger that overrides async_moderation_hook and hand
the proxy router to every registered prompt injection detector at startup so that call can route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(enterprise): resolve openai_moderations model at call time and default to omni-moderation-latest
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep queued moderation running past a V1 pre_call guardrail
A V1 CustomGuardrail with moderation_check pre_call returned out of
during_call_hook before asyncio.gather, abandoning already-queued
CustomLogger moderation coroutines and skipping every later callback.
Skip only that guardrail instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): skip null tool_calls when formatting prompts for moderation hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Post-call pipeline rewrites on buffered streams failed open on three shapes:
chat streams with n > 1 (the rebuilt response collapsed every choice into
index 0), streams that ended without a finish marker, and Responses streams
whose final event carried no response envelope.
The chat handler now rebuilds the ended stream one choice index at a time and
writes each choice's rewrite back to that choice's buffered deltas. The
Anthropic handler writes an unended stream's rewrite across its text deltas.
The Responses handler spreads an envelope-less rewrite over the buffered
output_text events, still failing open when a scanned event cannot be placed.
Tool-call rewrites on n > 1 chat streams keep failing open.
retrieve_batch returned a terminal batch from the DB before checking that the key may use the model encoded in a unified batch id; the grant check now runs right after pre-call processing. The vector store file list helper authorized data["model"] through handle_model_based_routing even when the vector store registry set it server-side and even with no caller, which crashed on a None key; it now authorizes only a caller-supplied hint and resolves credentials directly.
PR #40243 started carrying the upstream error body on InternalServerError so the Responses response.failed event can report the provider's code and message, and openai's APIError.__init__ took the body's type along with it. The proxy then answered an OpenAI-compatible upstream 500 with type server_error while a 502 and a 503 kept internal_server_error, and the integration contract in test_observed_routing.py went red. Pin the type the way RateLimitError pins throttling_error, keeping the body.
Resolves handler.py against main's SearchOutcome refactor: the file is
main's version plus this PR's substantive hunks only (the RichWebSearchInput
import, the rich= wiring at the three _execute_search call sites, the
_rich_search_input and _provider_supports_rich_search helpers, and the
_execute_search forwarding), so the 88-column re-wrap noise the PR carried
is gone and the diff against main is the feature alone. RichWebSearchInput
sits beside main's new SearchSucceeded/SearchFailed types, and main's two
_execute_search test stubs accept the new rich argument.
#41515 added the cost map entry transcribe/StartTranscriptionJob under a
new litellm_provider value "transcribe" without registering that provider
anywhere else, so litellm.models_by_provider had no "transcribe" key.
test_models_by_provider derives its provider set from the cost map itself,
so it went red on main. The user-visible half is that get_provider_models
returned None for the provider, which get_known_models_from_wildcard turns
into an empty list, leaving a transcribe/* key or team grant resolving to
no models.
Mirror the aws_polly registration: an enum member, a model set, an
ingestion branch, and a models_by_provider entry. Amazon Transcribe is
reached through the pass-through route rather than the Add Model form, so
it joins the frozen unlisted set the Add Model drift test tracks.
load_config used to return a local general_settings dict that it had
normalized in place, turning the configured role_permissions entries into
RoleBasedPermissions objects. It now returns the SettingsStore, which never
saw that write, so JWT auth received raw dicts and every request failed with
"'dict' object has no attribute 'role'" whenever role_permissions was set.
Convert the entries in the consumer instead, with a TypeAdapter, so the value
is parsed wherever it comes from. load_config keeps validating at boot, so a
malformed entry still fails startup rather than the first request.
Each source (per-call kwargs, environment variables, the Python module) now
builds an HttpSettingsLayer, and HttpSettings::from_layers merges them with
explicit precedence. The aiohttp and httpx proxy-env rule is resolved once in
the merge, so HttpSettings carries a single trust_proxy_env flag
A pre-signed batch/file request (Mistral, Bedrock) carries its auth header
inside the transformed request body, which pre_call logs verbatim into
raw_request_typed_dict and raw-request callbacks, leaking the provider key.
Mask the nested headers channel before handing the request to pre_call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HttpClientConfig::resolve becomes From<&HttpSettings> for Resolution and client_builder becomes TryFrom<&HttpClientConfig> for reqwest::ClientBuilder, matching the rustls conversion. The verify decision moves into From<&HttpSettings> for Verify, and the proxy environment rule moves next to its flags as HttpSettings::trusts_proxy_env. The curve and cipher results are read with transpose and a default selection, which removes the tuple destructuring
