Move the Responses WebSocket transport (connection type, dial, splice)
from ai-gateway io/responses_ws into core responses::connection, with
send/recv locked on separate halves and a Drop that closes the upstream
socket when the last clone is abandoned.
Move the OCR pipeline (prepare, provider call, document fetch, Azure DI
polling) into core ocr, mirroring the audio_transcription split: core
owns prepare/execute plus the ocr()/ocr_with_observer() entrypoints, the
gateway keeps only its lifecycle hooks (custom loggers/guardrails) and
request type, re-exporting through io/ocr unchanged.
Delete litellm-ai-gateway from python-bridge's dependencies and guard
the direction with manifest tests: python-bridge depends only on core +
python-interop, and core carries no pyo3/pythonize/pyo3-async-runtimes.
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
The API Keys route mounted the pre-App-Router UserDashboard component,
whose beforeunload handler cleared sessionStorage on every refresh of
the Virtual Keys page. That wiped the Playground chat history and model,
the logs live-tail preference, and everything else other pages keep in
session storage. The same component also re-decoded the login token,
re-fetched teams, and wrote cache entries nothing read.
ApiKeysDashboard now renders VirtualKeysTable and the Create Key button
directly, taking identity and role from useAuthorized like every other
page. Create Key is hidden for view-only roles, which the proxy already
rejects on /key/generate. The legacy component, its test, the fetch_teams
helper, and their grandfathered eslint suppressions are removed, and the
ProxySettings type moves to useProxySettings.
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
