The cost tracking callback f-stringed chosen_metadata, litellm_metadata,
and old_metadata into the failed_tracking_spend alert on every failure,
at every log level, so one 250-byte request produced a 23 KB alert
carrying the client's metadata, headers, and key-auth reprs four times
over. The alert now carries the exception, the traceback, the model, and
the call type; the metadata keys are logged once at debug level through
lazy formatting, so nothing is built at warning level
Guardrails created through POST /guardrails on older releases have api_version "v1" saved in the database, because the writer persists every default. Azure Content Safety never accepts that value, so those guardrails kept answering 404 after the default moved to None. The Azure base now resolves "v1" to 2024-09-01 the same way it resolves a missing value. Also restores the OpenAPI snapshot line that a Python 3.14 regeneration had dedented
When a deployment priced one OCR batch family and the other still needed a
published rate, a failed cost-map lookup returned zero for the whole line and
discarded the deployment rate that was already resolved. Those pages were
billed as free. The lookup failure now only logs, and the families the
deployment prices are billed at the configured rate
LitellmParams mixes every provider config model into one class, so the
Javelin api_version default of "v1" reached the Azure Content Safety
guardrails whenever config.yaml omitted api_version and Azure answered 404.
The shared field now defaults to None, Javelin keeps filling in "v1" itself,
and the Azure guardrails fall back to the documented 2024-09-01 at request
time so a DB update that omits api_version stays on the default too.
The router coverage gate in code-quality flags every router.py function
no router test calls by name, and the two helpers get_credential_deployment
gained (the team public-name lookup and the team-aware wildcard lookup)
were only reached through it. Each now has a test of its own: the
public-name lookup resolves only for the owning team, and the wildcard
lookup prefers the team's own pattern over the shared one and never hands
another team's wildcard deployment to a caller outside that team.
The vector-store file routes accept a model hint through the ?model= query
param and the x-litellm-model header. That hint was authorized with a hand
rolled check that covered only the key allowlist and the team allowlist, so
a key restricted by its project's model grant, a team-member restriction, or a
key config still routed through the hinted deployment. Greptile flagged the
gap as a P1 on the replacement PR.
The hint now goes through the same authorize_model_for_key path the batches
and files routes use, which runs can_key_call_resolved_model with every rule
the proxy enforces elsewhere. Keys those extra rules deny now get a 403 on
these routes. The two remaining behavioral differences are edge cases the
old check tolerated: a key whose team_models is set without a team_id no
longer runs the team allowlist, and a key with a config set skips the key
allowlist, both matching the rest of the proxy.
The regression test caches a project whose grant excludes the hinted model
and asserts the request is refused before any deployment lookup. The two
patch() calls on litellm.proxy.proxy_server carry a test-quality-ok reason
because can_key_call_resolved_model reads prisma_client and
user_api_key_cache through a lazy module import with no injection seam.
A batch retrieved by its model-encoded id takes the direct (non-router) path,
which resolved credentials without stamping the deployment's model_info, so a
completed batch on a deployment with its own per-page pricing was billed at
the published rate with an empty model_id on the spend row.
Extract the router's credential lookup into get_credential_deployment and
stamp the resolved deployment's model_info onto the retrieve call the way the
router does for routed calls.
* fix(proxy): dispatch llm_api_check moderation through during_call_hook
ProxyLogging.during_call_hook only ran async_moderation_hook for CustomGuardrail callbacks, so a
CustomLogger such as the prompt injection detector with llm_api_check enabled never called the
configured moderation model. Dispatch any CustomLogger that overrides async_moderation_hook and hand
the proxy router to every registered prompt injection detector at startup so that call can route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(enterprise): resolve openai_moderations model at call time and default to omni-moderation-latest
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep queued moderation running past a V1 pre_call guardrail
A V1 CustomGuardrail with moderation_check pre_call returned out of
during_call_hook before asyncio.gather, abandoning already-queued
CustomLogger moderation coroutines and skipping every later callback.
Skip only that guardrail instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): skip null tool_calls when formatting prompts for moderation hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Post-call pipeline rewrites on buffered streams failed open on three shapes:
chat streams with n > 1 (the rebuilt response collapsed every choice into
index 0), streams that ended without a finish marker, and Responses streams
whose final event carried no response envelope.
The chat handler now rebuilds the ended stream one choice index at a time and
writes each choice's rewrite back to that choice's buffered deltas. The
Anthropic handler writes an unended stream's rewrite across its text deltas.
The Responses handler spreads an envelope-less rewrite over the buffered
output_text events, still failing open when a scanned event cannot be placed.
Tool-call rewrites on n > 1 chat streams keep failing open.
retrieve_batch returned a terminal batch from the DB before checking that the key may use the model encoded in a unified batch id; the grant check now runs right after pre-call processing. The vector store file list helper authorized data["model"] through handle_model_based_routing even when the vector store registry set it server-side and even with no caller, which crashed on a None key; it now authorizes only a caller-supplied hint and resolves credentials directly.
PR #40243 started carrying the upstream error body on InternalServerError so the Responses response.failed event can report the provider's code and message, and openai's APIError.__init__ took the body's type along with it. The proxy then answered an OpenAI-compatible upstream 500 with type server_error while a 502 and a 503 kept internal_server_error, and the integration contract in test_observed_routing.py went red. Pin the type the way RateLimitError pins throttling_error, keeping the body.
