Streaming chat relays on Azure and azure_ai deployments rebuild the response from
the SSE chunks through the OpenAI passthrough assembler, so the spend log carries
usage. The router relays keep the JSON body when the Content-Type carries a
charset, return the upstream status and body instead of a 500 when the deployment
rejects the call, and fall back to the caller's api-version when the deployment
sets none. Lint budgets ratcheted to the measured totals
* fix(responses): decode JSON-string tool schemas before sending to the provider
A caller that hands a tool schema over already JSON-encoded reached the
Responses API with a string `parameters`, and the provider rejected the
request with a 400 naming the routed model instead of the offending tool.
Decode it at the one place every Responses request converges, and refuse
anything that is neither an object nor a string encoding one.
Collapses the duplicated input/tool sanitization block shared by the
request and compact-request builders into a single owner, so the decode
cannot be wired into one path and not the other.
* test(responses): pin null tool schemas as accepted, and type the parametrized cases
The Responses API serves `parameters: null` and an omitted schema alike, so
neither may raise. Pin both against a future tightening, annotate the
parametrized inputs, and trim the docstrings back to what the code does not
already say.
Every /azure_ai/<router model>/<native path> relay failed with HTTP 500 because
azure_ai had no passthrough config. The new AzureAIPassthroughConfig strips the
router-model prefix from the relayed path, forwards to the deployment's api_base
with its own credential (api-key on Foundry and Azure OpenAI hosts, Bearer
elsewhere, Entra as the fallback), and delegates chat/completions cost logging
to the Azure passthrough config.
The router's provider inference now receives the deployment's api_base so an
OpenAI-family model on a Foundry resource stays azure_ai instead of flipping to
azure through the AZURE_AI_API_BASE env var.
Bedrock passthrough Converse routes flattened every non-empty string under
toolConfig.tools into the guardrail INPUT texts, so tool names, tool
descriptions and JSON-schema strings (object, property names, titles, type
names, enum values) each arrived as a separate guardrail item. A request whose
only prompt was one benign user message could be blocked outright because a
denied term appeared in an app-authored tool definition.
Tool definitions are now excluded from the extracted texts, matching every
other guardrail translation handler, which carries tool definitions in the
structured tools input rather than in texts. Caller content stays scanned:
message text, toolUse.input, toolResult content and json, and
additionalModelRequestFields are unchanged.
Resolves LIT-5797
* fix(anthropic): never carry cache_control on translated thinking blocks
The /v1/messages adapter built every thinking and redacted_thinking block with
cache_control=content.get("cache_control", {}), so a block the client never
marked still came out carrying an empty cache_control. anthropic_messages_pt
replays thinking blocks verbatim and first, so that value landed at content[0]
of the outbound assistant message and Anthropic rejected the request with
messages.N.content.0.thinking.cache_control: Extra inputs are not permitted.
Anthropic's schema has no cache_control on either block type, so there is
nothing to gate or translate here, only to stop copying. Every sibling block
type already routes through _add_cache_control_if_applicable; these two were
the only ones setting the key unconditionally.
This is reachable from any caller that round-trips Anthropic messages through
the OpenAI shape, which is why shadow eval saw it on a majority of sampled
Claude Code turns while the same traffic served natively was fine.
* test(anthropic): assert the outbound wire body for redacted thinking blocks
Async searches reach the sync client through executor threads, so the LRU cache
is shared state. A key could be evicted between the lookup and the reordering
that followed it, and the reordering then raised KeyError and became a 500.
Reproduced at 15 failures per run with 16 threads over 34 keys and a 1ns switch
interval; the regression test is that workload.
litellm only retries 408, 409, 429 and 5xx, so classifying a dropped connection
as a 400 turned one replica set failover into a permanently failed search. It is
a 503 now, with the message still naming the misconfigurations that also close a
connection.
The client cache skipped insertion once it held 32 entries, so any store added
after that rebuilt its client on every search, paying an SRV lookup, a TLS
handshake and topology discovery each time. It evicts the least recently used
entry instead, which only drops the cache's own reference.
Also trims the explanatory comments to the one-line form the repo asks for.
The base vector store interface grew an embedding_executor argument, and
litellm.vector_stores.search now always passes one. MongoDB still carried its
own embedding_fn/aembedding_fn constructor seam, so every search through the
public entry point failed with an unexpected keyword argument.
Drop the local seam in favour of the shared executor: one path instead of two,
and the unit tests now drive the same seam production uses.
tlsCAFile and tlsCertificateKeyFile are how a self-managed deployment presents a
private CA, so they are the options on-prem operators actually set. pymongo opens
those files itself during TLS setup and lets OSError out, which is neither a
PyMongoError nor a ValueError, so it missed every branch of the translator and
litellm.exception_type turned it into a 500 with a traceback in the body. A
mistyped path, or one that exists on the host but not inside the container, is a
routine mistake and has to read as a 400 naming the file.
Matched on the exception carrying a filename so a socket-level OSError still falls
through to the branches that handle it. Verified against a self-managed mongod with
a missing CA file, a CA path that is a directory, and a missing client certificate.
mongod serves $vectorSearch identically whether mongot runs under Atlas or beside
a self-managed deployment, so the provider already worked against on-prem. The
guidance did not: a refused connection told the operator to check their project's
IP access list and whether the cluster was paused, neither of which exists outside
Atlas, and the index errors claimed an "Atlas Vector Search index" they do not have.
Every message now names a remedy for both, keeping the Atlas-specific hint labelled
as such.
Also diagnoses unescaped credentials, which self-managed deployments hit more often
because the password is usually generated. pymongo reports those three different
ways and none of them mentions the password: '@', ':' and '%' raise an RFC 3986
complaint, '/' is read as the database separator and surfaces as Bad database name,
and an unescaped ':' looks like a bad port and comes back as a plain ValueError.
All three now point at the credentials. The ValueError branch's comment claimed it
fired on an unescaped '/', which pymongo actually reports as InvalidURI; corrected
to the port parse it really catches.
Verified against a self-managed mongod 8.0 with mongot, reached over plain
mongodb:// with no SRV and no TLS: 13 cases with live OpenAI embeddings, and 4
credential cases against an auth-enabled instance whose password holds % @ / and :.
list_search_indexes returns the same queryable and status fields there as on Atlas,
so the index-readiness check needed no change.
* fix(snowflake): normalize Cortex Claude request shapes
Co-authored-by: Kamron Javaherpour <kamron@kargo.com>
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
* style(snowflake): format Cortex request transformations
* fix(snowflake): annotate Cortex wire payloads
* fix(snowflake): route Cortex content through the shared Anthropic converters
* fix(snowflake): surface Cortex prompt-cache usage and thinking blocks
Parse Cortex's Anthropic-dialect responses and SSE with Anthropic's own parser so cache_creation/cache_read counts, thinking blocks and signatures reach the caller. Restore thinking for every Claude model: Cortex documents extended thinking broadly and only adaptive thinking is 4.6-gated.
* fix(snowflake): echo signed thinking blocks on every assistant turn
The reference converter extends signed thinking blocks on each assistant turn, not just tool-call turns, so a replayed thinking-plus-text response keeps its signed block. Content-less thinking turns send no empty text block.
* fix(snowflake): preserve thinking list content
---------
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
Keeps staging's response-id keying next to the pass-through failure-path helpers
and types the read-only raw_bytes parameters as Sequence[bytes] so the merged tree
stays inside the lint budgets
The block accepts output_cost_per_reasoning_token and cache_creation_input_token_cost. The generic
cost path and the DashScope calculator swap them in while a window is open, and unset keys keep the
standard rate. One shared TokenRates value replaces the DashScope-local copy, and
apply_off_peak_pricing takes and returns it.
