When the resolved deployment is provider openai and the model is GPT-5.6 or
newer, the cache control hook now writes prompt_cache_breakpoint on the
targeted content block and sets prompt_cache_options to explicit mode unless
the caller already passed one. The /v1/messages bridges carry the marker
through (the Responses bridge moves a marked system prompt into a developer
message, since top-level instructions cannot hold one). Breakpoint counting
and the stand-down check recognise both marker kinds, and client breakpoints
already present in messages are no longer subtracted from the cap twice.
Fixes#37509
Both /v1/messages bridges (Responses API adapter for openai/* and the
chat-completions adapter) now derive prompt_cache_key from the first 64
characters of metadata.user_id, next to the existing user mapping. The
chat bridge only sets it when the resolved provider advertises
prompt_cache_key in its supported params, so providers that reject
unknown params are unaffected. A prompt_cache_key sent explicitly by the
client always wins over the derived value.
Fixes#37508
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
A /v1/realtime connection to a Vertex AI Live model accepted the WebSocket
upgrade and then went silent: a stalled Google OAuth token fetch blocked the
handler before any session event, and the eventual failure closed the socket
with a bare 1011 and no error event, so callers saw an open socket, no frames,
and no reason.
Bound the pre-session token fetch with
REALTIME_CREDENTIAL_RESOLUTION_TIMEOUT_SECONDS (20s default) and, on any
realtime failure, send an OpenAI-style error event before closing with a reason
that names the failure. Close reasons are truncated by bytes, not characters,
since an over-long reason makes the close frame itself fail.
Add gemini-3.5-flash to the placeholder-scoping matrix and a regression test
that a natively signed parallel turn replays with no
skip_thought_signature_validator anywhere in the payload, the shape that was
producing empty text responses on 3.5.
Hoist the repeated placeholder expression into one constant and rewrite the
docstrings that restated their own assertions to say why the case matters
instead.
The proxy pre-creates the logging object before the router picks a deployment,
and the native /v1/messages handler never copied the deployment's
vertex_location into the logging params it updates, so cost resolution fell
back to the environment or the default region and priced every call on this
surface with the regional endpoint uplift. Copy the explicitly configured
location from the request's litellm params, the same source dispatch builds
the request URL from, and register the new regional_endpoint_uplift_multiplier
field in the cost map schema test.
Gemini returns a thoughtSignature on the first function call of a parallel
batch and leaves the siblings bare. When replaying that assistant turn,
litellm gave every unsigned call the skip_thought_signature_validator
placeholder, so a three-call turn went back with three signatures where
Gemini had produced one. Keep the placeholder for the first call only and
forward the siblings with whatever signature they actually carry, which is
usually none.
A trusted hostname over plain http would expose the bearer token or a
replayable SigV4 signature to network observers. Credentials now only ride
https, with localhost exempt so local MCP stubs keep working.
Web-search connector 1.1.0 and later return the machine-readable results in result.structuredContent and may leave the text block as prose, which the parser dropped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Refuse to SigV4-sign requests to hosts that are neither an AgentCore gateway
hostname nor AGENTCORE_GATEWAY_URL's host, match gateway hostnames on the URL
host instead of anywhere in the URL, accept the env token when api_base is a
real gateway, raise on tools/call responses with result.isError, and split
CRLF-framed SSE events.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
* fix(vector_stores): stop leaking stored credentials in direct search debug logs
Direct vector store providers (RESP datastores like Valkey) have no HTTP
request to echo, so both search handlers called `logging_obj.pre_call` with
no `api_base`. The logging helper treats an empty `api_base` as "nothing to
render" and falls back to `str(self.model_call_details)`, which carries the
resolved `litellm_params`: the stored `valkey_password` and the embedding
config's `api_key` among them.
The stdout logger's regex redaction hid this, but `pre_call` also writes the
same string to `litellm_params["metadata"]["raw_request"]`, which ships
unredacted to every logging callback (Langfuse, OTel, etc.).
Pass a synthetic `<provider>://<vector_store_id>` endpoint plus an explicit
`request_str` so the debug output describes the call instead of dumping call
details, and fold the duplicated sync/async blocks into one helper so the
sanitized descriptor cannot drift between them.
* fix(vector_stores): type direct search query as Sequence[str]
The new helper's list[str] annotation pushed LIT001 over its
type-discipline ceiling. Sequence is the read-only shape the helper
actually needs, and list[str] still satisfies it at both call sites.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
On /v1/messages, AnthropicStreamWrapper synthesizes the content_block_start for
the first content block, queues it, then hits a bare `continue` when that same
upstream chunk's translated delta is empty. The queue is only drained at the top
of the next __next__ / __anext__, so the queued content_block_start waits for a
further upstream chunk to arrive.
An empty delta on the opening chunk is the normal tool-call shape: Bedrock
Converse's contentBlockStart carries the tool id and name with no arguments, and
OpenAI-format streams send arguments: "" on the chunk that names the function.
So a client learns a tool call started one upstream event late, and when the
provider delivers argument fragments as a trailing burst it sees nothing at all
after message_start for the whole generation.
Flush the queued event before continuing, in both the sync and async paths. The
sibling block-transition path already returns from the queue, so only the
first-block-open case changed.
resolve_fireworks_resource_name prefixes bare names with
accounts/fireworks/models/ (or routers/ for *-fast). Azure AI Foundry
hosts Fireworks models under deployment ids like FW-Kimi-K3; rewriting
those yields 404 DeploymentNotFound.
Leave names that already start with FW- unchanged. Native Fireworks
short names still get the accounts/ path.
Co-authored-by: Cursor <cursoragent@cursor.com>
