* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts
* refactor(anthropic): hold the chat completions document change for a later commit
* fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai
* fix(snowflake): turn file parts into document blocks for Cortex Claude
* fix(prompt_security): sanitize chat file parts that carry file_data
* fix: answer 400 or drop PDFs for models that cannot read them
Models whose model_info or cost map entry sets supports_pdf_input to
false, and Ollama, now get the PDF dropped under drop_params and a 400
from litellm without it. GigaChat uploads file parts, SAP keeps sending
them as image_url data URLs, azure_ai passes dict content through, and
annotations other than url_citation move to provider_specific_fields
instead of failing the response.
* fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys
Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and
litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider
annotations to provider_specific_fields, and split them the same way when a stream is rebuilt.
* fix(vertex_ai): send PDF file parts to Vertex partner models as image_url
Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages
reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as
unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL.
* fix(cost-map): leave native DeepSeek PDF support unset
/v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there
and would only change chat completions, which this PR does not otherwise touch.
* fix(main): drop a user turn that held only a PDF under drop_params
Dropping the PDF from a PDF-only user turn left an empty content list, which
Foundry and OpenAI-shaped providers reject with a 400, so drop_params still
failed for that request. The emptied user turn is now removed.
* fix: read OCI PDFs, gate deployment base models, and stop empty drops
OCI generic chat sends file parts as image data URLs again, so OCI Gemini
keeps reading PDFs from /v1/messages and chat file parts.
The PDF gate now reads model_info.base_model, drops PDFs under drop_params
for azure_ai deployments that never declared PDF support (the old flattening
behavior), and answers 400 when dropping the PDF leaves no user message.
The cost map marks the Vertex partner text models that a live probe showed
answering without reading the PDF.
* test(integration): cover the PDF input gate, messages PDF file parts and provider annotations
Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed
* test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells
* fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload
* fix(main): PDF check reads the deployment's top-level base_model
* fix(main): keep roles alternating after drop_params removes a PDF-only turn
When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other
* test: type the PDF and annotation tests the PR added
* test: type the GigaChat, SAP and Snowflake PDF tests
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
Providers like Mistral put every parallel tool call, with full arguments, into one
chat-completions chunk. The /v1/messages adapter opened one tool_use block for the
first call and concatenated every call's arguments into it, so clients got invalid
JSON like {...}{...}. _CombinedChunkSplitter now splits such a chunk into one chunk
per call. The first call streams live and calls opened behind it are held, gathering
their fragments by index, until a non-fragment chunk arrives or the stream ends
Fixes#44029
* fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary
AnthropicStreamWrapper.async_anthropic_sse_wrapper swallowed every upstream
exception and emitted its own Anthropic error frame, so a provider drop on a
streamed /v1/messages call never reached the proxy's streaming boundary — no
failure spend row, no failure callbacks.
Re-raise when the stream is proxy-managed (detached failure hook armed) so
async_streaming_data_generator runs post_call_failure_hook once and serializes
the error frame; when consumed standalone (SDK litellm.messages path), run the
logging object's async failure handler and keep the client-facing error frame.
Fixes#44742
* test(anthropic): satisfy PT012 single-statement rule in mid-stream error test
* fix(anthropic): dispatch both sync and async failure callbacks
* fix(anthropic): dispatch both sync and async failure callbacks
* test(anthropic): move the regression cases into the mapped test file
* refactor(anthropic): read the public detached-failure hook and drop the explanatory comments
* test(anthropic): drive the mid-stream failure tests through a real logging object
* fix(anthropic): re-raise the provider error from the chat wrapper envelope so the router keeps its pre-content retry
* test(integration): cover mid-stream failure bookkeeping of bridged /v1/messages streams
Nine integration cells for the chat bridge: raw httpx and the Anthropic SDK (sync and async) streams that fail after the first content delta now land exactly one failure SpendLogs row carrying the provider's error class, the non-streaming 500, the response-cache twin and a client that leaves mid-stream keep their bookkeeping, a 24-stream burst lands every call id once, and the standalone SDK stream reports the provider error to failure callbacks once.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): retry a /v1/messages stream the provider drops before the first content chunk
A /v1/messages stream that the upstream closed before any content reached the
client answered an error event after a single attempt, so the router's
num_retries never applied to that drop. The pre-content failure is now retried
within the model group before the fallback chain runs, with the budget resolved
the way a failure raised before the stream opened resolves it: a retry policy
that names the error class, then the request's num_retries, then the
deployment's, then the router's. A drop after content reached the client keeps
surfacing the provider's error after one attempt.
Fixes#44238
* fix(router): hand a retry's non-retriable error to the fallback chain and type the retry helpers
A retry that failed before its stream opened with an error no retry covers raised straight to the
client, skipping a fallback the first attempt would have used. assert_never now comes from
typing_extensions so the router imports on Python 3.10, and the retry helpers read their kwargs
through typed narrowing instead of Mapping[str, Any]
* fix(router): cast the untyped router fallback defaults the stream retry gate reads
The retry gate passed the router's fallback attributes, declared without element types, to the
typed request override helper, which basedpyright counted as new unknown-argument errors
* fix(router): consult context_window_fallbacks when a retried /v1/messages stream overflows
A retry attempt raising ContextWindowExceededError reached the fallback chain inside its
mid-stream envelope, so only the regular fallbacks list matched. The fallback attempt now
unwraps it the way it unwraps a content policy error. The new router helpers are covered for
the router code coverage check with two direct-call tests and named covering tests
* fix(router): retry a 408 raised by a /v1/messages retry and honor deployment num_retries before the stream opens
* fix(router): attribute a retried /v1/messages stream to the deployment that served it and bound the retry-policy hold
* fix(router): retry /v1/messages error frames under their retry-policy class and keep the first drop's committed budget
An `event: error` frame that arrives before the first content delta now raises the exception class the pre-stream mapping gives an HTTP answer with the same status (429 RateLimitError, 500 and 529 InternalServerError, 503 ServiceUnavailableError, 504 Timeout), so a retry policy's per-class budget governs it the way it governs the error before the stream opened. The status the client sees is unchanged
A retry that lands on a sibling deployment keeps the budget the first drop committed to, read back from the request's attempted_retries and max_retries, instead of recomputing it from the new deployment's num_retries, matching the pre-stream retry loop
* refactor(anthropic): keep the error-frame exception mapping under llms and type the retry test helper
The status-to-exception mapping an `event: error` frame gets before the retry policy is consulted now lives next to the Anthropic error status map in llms/anthropic/common_utils.py, with its own unit test, and the two-deployment retry test helper takes explicit typed parameters instead of a bare dict and untyped kwargs
* refactor(anthropic): map an error frame's status with explicit returns on every path
* fix(router): map stream error frames through the pre-stream exception mapping
An overloaded `event: error` frame on a /v1/messages stream now raises the InternalServerError a 529 answer maps to, built by exception_type from the frame's own body, so one retry policy class governs the error before and after the first byte; a failed fallback after such a frame answers 500 like every other litellm path instead of the frame map's 503
A model_group_retry_policy that does not parse (a non-integer budget, an entry that is not a mapping) no longer fails every healthy stream of that group before its first attempt: the stream runs with no policy and the plain num_retries budget, with a warning naming the group
* fix(router): forward an error frame nothing can take over for as the provider sent it
A pre-content error frame whose class the retry policy grants no retry, with no fallback configured, raised an HTTP error only on the first attempt while the same frame after exhausted retries reached the client verbatim. Both now pass through as sent, the way the merge base forwarded every frame.
* test(integration): audit /v1/messages pre-content retry across routes and budgets
Adds the /audit cells for the pre-content stream retry: the native Anthropic route
(drops and error frames before content, HTTP rejections before the stream opens, SDK
sync and async, after-content and non-retriable controls, budget exhaustion, cache
twin, spend row and headers), the chat and responses bridges, the generic routes
(responses, chat, vllm pass-through, Gemini generateContent, fine-tuning jobs list),
owned two-worker proxies for router-level budgets, retry policies and fallbacks, and
two chaos cells (a worker killed mid burst, an outage on every first attempt). Shared
helpers for scripted Anthropic SSE upstreams and OpenAI-compatible wire replies live
in tests/integration/_support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(hosted_vllm): keep reasoning_content on assistant messages in _transform_messages
vLLM accepts reasoning_content (200 on the wire) and qwen/deepseek/glm
chat templates consume it, so popping it made reasoning models lose
earlier reasoning across tool loops. thinking_blocks is still removed
for vLLM compatibility.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(hosted_vllm): forward replayed reasoning_content only when it is a string
* test(integration): cover hosted_vllm reasoning_content replay across endpoints
* test(integration): require the surviving worker to serve its held requests in the sigkill chaos cell
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* test(integration): group /v1/messages contracts under tests/integration/messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): make ci coverage census collect nested test dirs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): nest /v1/messages contracts under messages_endpoint/providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
