litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
* fix(vector_stores): stop leaking stored credentials in direct search debug logs
Direct vector store providers (RESP datastores like Valkey) have no HTTP
request to echo, so both search handlers called `logging_obj.pre_call` with
no `api_base`. The logging helper treats an empty `api_base` as "nothing to
render" and falls back to `str(self.model_call_details)`, which carries the
resolved `litellm_params`: the stored `valkey_password` and the embedding
config's `api_key` among them.
The stdout logger's regex redaction hid this, but `pre_call` also writes the
same string to `litellm_params["metadata"]["raw_request"]`, which ships
unredacted to every logging callback (Langfuse, OTel, etc.).
Pass a synthetic `<provider>://<vector_store_id>` endpoint plus an explicit
`request_str` so the debug output describes the call instead of dumping call
details, and fold the duplicated sync/async blocks into one helper so the
sanitized descriptor cannot drift between them.
* fix(vector_stores): type direct search query as Sequence[str]
The new helper's list[str] annotation pushed LIT001 over its
type-discipline ceiling. Sequence is the read-only shape the helper
actually needs, and list[str] still satisfies it at both call sites.
* feat(vector_stores): add Valkey as a managed vector store provider
Adds a valkey provider for managed vector stores, searchable via the
valkey-search module over RESP. Introduces BaseDirectVectorStoreConfig
for datastores that execute searches directly instead of building an
HTTP request, and refactors the valkey semantic cache to share the new
connection URL helper. Registered in the provider enum, router params,
proxy config registry, Admin UI Add Vector Store modal, and provider
endpoint support matrix.
* fix(vector_stores): join list queries and bound valkey socket timeouts
Review feedback: multi-string queries are now space-joined like every
other embedding-based provider instead of dropping all but the first,
and the request timeout is threaded through the direct vector store
interface into bounded socket_connect_timeout / socket_timeout values
on both redis clients so an unreachable Valkey host cannot pin proxy
workers until the OS TCP timeout.
* chore(ui): regenerate schema.d.ts for valkey vector store fields
* docs(ui): make the Valkey vector store setup note and field tooltips explicit
* feat(ui): pick the Valkey embedding model from the proxy's models like Milvus
* fix(ui): number the setup steps in the vector store provider alerts
On /v1/messages, AnthropicStreamWrapper synthesizes the content_block_start for
the first content block, queues it, then hits a bare `continue` when that same
upstream chunk's translated delta is empty. The queue is only drained at the top
of the next __next__ / __anext__, so the queued content_block_start waits for a
further upstream chunk to arrive.
An empty delta on the opening chunk is the normal tool-call shape: Bedrock
Converse's contentBlockStart carries the tool id and name with no arguments, and
OpenAI-format streams send arguments: "" on the chunk that names the function.
So a client learns a tool call started one upstream event late, and when the
provider delivers argument fragments as a trailing burst it sees nothing at all
after message_start for the whole generation.
Flush the queued event before continuing, in both the sync and async paths. The
sibling block-transition path already returns from the queue, so only the
first-block-open case changed.
resolve_fireworks_resource_name prefixes bare names with
accounts/fireworks/models/ (or routers/ for *-fast). Azure AI Foundry
hosts Fireworks models under deployment ids like FW-Kimi-K3; rewriting
those yields 404 DeploymentNotFound.
Leave names that already start with FW- unchanged. Native Fireworks
short names still get the accounts/ path.
Co-authored-by: Cursor <cursoragent@cursor.com>
The field itself landed on staging via 0c5c9c79d7; these are the regression tests from PR #31435 for the retrieval-facing half.
(cherry picked from commit a9a322d63f6d4658b1f28d1622335775e94736a4)
Bedrock batch jobs write their results to s3_output_bucket_name when it differs
from the input bucket, but the file-content retrieval path validated the file id
only against the input bucket (s3_bucket_name). A deployment that configures a
separate output bucket therefore could not retrieve its own batch outputs: the
id validated against the input bucket and was rejected as a foreign bucket.
Resolve the trusted output bucket alongside the input bucket from the immutable
credential snapshot (or AWS_S3_OUTPUT_BUCKET_NAME), and try the file id against
each configured bucket, returning the first that validates. The SSRF guard is
preserved: only server-configured buckets are tried, never a request param, and
an id outside both is still rejected.
(cherry picked from commit 1d407c2f26)
Bedrock invoke /v1/messages streaming reports cache_read_input_tokens and
cache_creation_input_tokens on message_stop.usage while attaching
amazon-bedrock-invocationMetrics to the same chunk. The stream decoder
rebuilt that chunk's usage block from inputTokenCount/outputTokenCount
alone, which exclude cache reads and writes, so the cache breakdown was
destroyed before _promote_message_stop_usage could surface it and cache
tokens were billed at $0. Merge instead of replace, and also map
cacheReadInputTokenCount/cacheWriteInputTokenCount when Bedrock reports
the cache itemization inside the invocation metrics.
Co-authored-by: Brian Cox <3924351+brian5021@users.noreply.github.com>
Azure rejects the legacy `max_tokens` key for the whole gpt-5 name family, but
`AzureOpenAIGPT5Config.is_model_gpt_5_model` deliberately excludes `gpt-5-chat*`
so those deployments fall through to `AzureOpenAIConfig`, which sends `max_tokens`
verbatim and gets a 400 back on every request that carries it, `/health` probes
included.
One predicate was answering two independent questions. Split it: the new
`AzureOpenAIConfig.requires_max_completion_tokens` covers the whole gpt-5 name
family and drives only the rename, while `is_model_gpt_5_model` keeps keying
reasoning_effort, the temperature clamp and the dropped penalties off the
reasoning question, so #13781 stays fixed.
Adds an opt-in operator allow-list, litellm_settings::bedrock_request_metadata_fields, that forwards LiteLLM key, team and end-user identity plus client spend_logs_metadata into Bedrock request metadata so Bedrock spend can be grouped in AWS Cost Explorer.
Covers all three Bedrock surfaces: the Converse body requestMetadata field, and a signed X-Amzn-Bedrock-Request-Metadata header on Invoke chat completions and on Invoke /v1/messages, where the header is the only viable leg.
The resolver reads both metadata variable names, reserves the whole user_api_key_ prefix against caller-supplied keys, caps the client slot budget explicitly at 16 minus the reserved count, and drops rather than rejects auto-injected values that violate Bedrock constraints. Caller-supplied requestMetadata keeps its existing 400 semantics.
The request-metadata field and header are proxy-owned whenever forwarding is enabled. A caller-supplied value, reachable through the generic extra_headers passthrough, is dropped unconditionally and compared case-insensitively, and is replaced only by the proxy's own value, so identity in the AWS billing record cannot be forged. Absence of a resolved value still means absence on the wire rather than a fallback to the caller's. The guardrail headers keep their existing no-displace behaviour.
Callers can opt into the provider's raw operation response on /v1/ocr with the x-req-format: native header (or req_format in the body) while page-based cost tracking keeps reading usage_info off the normalized response.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Shape detection and block normalization sat in the generic batch layer, which
let batch and live parsing of the same wire format drift apart. Both now live on
AmazonConverseConfig as is_converse_usage_shape and usage_from_batch_output, so
batch_utils asks the provider adapter rather than knowing Bedrock's field names.
Adds direct coverage for the shape predicate, the completion of an incomplete
block, cache-count inflation, and the streaming usage event that shares the
public transform. Drops the narrative banner from the batch tests.
Every bedrock batch output line went through the Anthropic usage parser, which
reads snake_case input_tokens/output_tokens. Converse-family models (Nova and
friends) report camelCase inputTokens/outputTokens, so their usage came back
0/0/0 and the batch billed $0 despite real token consumption.
Usage is now selected by the shape of the payload: a Converse-shaped block goes
through the same transform the live Converse path uses, so a batch and an
equivalent non-batch call agree on tokens, including cache reads and writes.
Anthropic-shaped bedrock output is unchanged.
A shape neither parser understands (an InvokeModel-native payload from Titan,
Cohere, or Llama, which name their counts differently again) still reads zero,
but now warns with the keys it saw instead of silently billing $0.
Exposes the Converse usage transform as public, since batch parsing is a second
legitimate caller; that also removes the private-member access invoke_handler
was already making.
Resolves the transform_create_file_response conflict by keeping the
_uploaded_object_size handoff over the response Content-Length read,
and adds the rebind-ok justification LIT011 now requires for the
upload-size litellm_params handoff after the base budget ratcheted.
* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5
The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.
Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.
* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias
Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.
* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget
map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.
Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.
* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)
---------
Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
