The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
The API Keys route mounted the pre-App-Router UserDashboard component,
whose beforeunload handler cleared sessionStorage on every refresh of
the Virtual Keys page. That wiped the Playground chat history and model,
the logs live-tail preference, and everything else other pages keep in
session storage. The same component also re-decoded the login token,
re-fetched teams, and wrote cache entries nothing read.
ApiKeysDashboard now renders VirtualKeysTable and the Create Key button
directly, taking identity and role from useAuthorized like every other
page. Create Key is hidden for view-only roles, which the proxy already
rejects on /key/generate. The legacy component, its test, the fetch_teams
helper, and their grandfathered eslint suppressions are removed, and the
ProxySettings type moves to useProxySettings.
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued
Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.
Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound
* fix(azure_sentinel): retry undelivered records on the flush timer only
Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.
A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.
* fix(azure_sentinel): decide threshold sends under the flush lock
Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.
* test(azure_sentinel): cover successful threshold waiters
* fix(azure_sentinel): preserve cancelled batches for retry
* fix(azure_sentinel): requeue only the undelivered part of a cancelled split
A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush
The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch
* fix(azure_sentinel): drop batches a permanent rejection will never accept
A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule
The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it
Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error
The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry
* chore(logging): document mutable queue ownership
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
* fix(datadog): preserve non-413 retry behavior
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
* fix(batch_utils): requeue by default and let Sentinel opt into dropping
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections
Also drops an explicit return None the strict ruff gate flags in the test helper
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
