* test(e2e): cover Azure code_interpreter container files by native id with a service-account key
* test(e2e): require the code_interpreter tool, skip at collection, and scope the container call timeout
* fix(e2e): fail the containers suite when the Azure credentials are missing instead of skipping
* fix(proxy): record streamed /v1/responses container ownership before the response.completed frame
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): cover Azure code_interpreter container files by native id with a service-account key
* test(e2e): require the code_interpreter tool, skip at collection, and scope the container call timeout
* fix(e2e): fail the containers suite when the Azure credentials are missing instead of skipping
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): placement policy for mid-conversation system messages
Pure functions over the OpenAI-format message list: split off the leading
system run, keep later system messages as role=system at a placement Anthropic
accepts on models flagged supports_mid_conversation_system (after a user turn,
before an assistant turn or the end, never adjacent), and convert them to user
turns in place elsewhere, keeping tool_result first in a merged user turn.
* fix(anthropic): keep mid-conversation system out of the chat completions system prompt
translate_system_message hoisted every role=system message, at any index, into
the top-level system block. On a conversation carrying a mid-session reminder
that rewrites the cached prefix, so the provider re-bills the whole history at
cache-write pricing on every turn (#36559). #36968 fixed this on /v1/messages;
the chat completions path, shared by first-party Anthropic, Vertex, Azure AI
and Bedrock Invoke, still hoisted.
Only the leading system run becomes the system prompt now. Later system
messages go through the placement policy, and anthropic_messages_pt emits a
system message instead of rejecting the role. The caller's message list is no
longer mutated. Tests pin the two-turn prefix invariant across all four chat
configs and both flag states.
* refactor(anthropic): single-source the converted system note
The /v1/messages pass-through and the chat completions path must prefix a
converted system turn with the same operator note.
* test(e2e): prove the prompt cache survives a mid-conversation system reminder on chat completions
Same priming and assertions as the /v1/messages cases, through
/v1/chat/completions with OpenAI-format messages, for first-party Anthropic
and Bedrock Invoke on a flagged (Opus 4.8) and an unflagged (Haiku 4.5) model.
The reminder sits between the assistant turn and the next user turn, the shape
OpenAI-style agent frameworks send, which is the placement the chat path has
to translate.
* test(anthropic): cover the cache_control rebuild shapes and type the test helpers
Codecov flagged the 5m ttl branch and the empty-system path of the wire
builder; both now have a test. Greptile asked for full typing on the new
test helpers.
* refactor(anthropic): read the mid-conversation flag through a public supports_ helper
supports_mid_conversation_system joins the other supports_* helpers in
litellm.utils, so the chat transformation stops importing the private
_supports_factory.
* chore(typing): declare the mid-conversation type aliases with TypeAlias
The Final sweep tightened LIT010, which exempts TypeAlias declarations but
counts a bare alias assignment as an unannotated binding.
* fix(anthropic): let add_code_execution_tool take the pass-through message union
The translator now emits role=system inside messages for models that accept it,
so anthropic_messages_pt returns the pass-through union. add_code_execution_tool
still declared the narrower user/assistant union while only ever reading
content, so upstream's strip_advisor_blocks_from_messages call in between made
the mismatch visible to the type checker.
* fix(bedrock): keep mid-conversation system messages in place on converse path
* fix: ruff format + multi tool_result order + regression test
* fix: satisfy type-discipline gate + update osv ignore for mlflow PYSEC-2026-3865
* fix(bedrock): restore role narrowing in hoisted system loop for basedpyright budget
* test(bedrock): cover mid-conversation system conversion branches
- non-dict guard in _opens_with_tool_result
- in-place conversion without tool context
- str/list cache_control preservation in mid-conversation path
- drop unreachable non-system guard in hoisted loop
* Place type-discipline suppressions on the lines the gate scans
* Narrow hoisted loop to system role so basedpyright sees the right TypedDict
* fix(anthropic): place mid-conversation system runs by their neighbours only
A run after an assistant turn now slides behind the user turn that
immediately follows it, and a run that ends the array or precedes an
assistant turn becomes a user turn in place. No later message can move
an earlier run, so a client that replays the conversation with more
turns appended sends a byte-identical prefix and preserved thinking
blocks keep their binding
* refactor(bedrock): share the converted system note with the anthropic module
Converse imports CONVERTED_SYSTEM_NOTE instead of carrying its own copy
of the same text, and the reordering helpers lose their comments
* test: pin the replayed request prefix across preserved-thinking turns
One test per audited feature, through the real entrypoint: the chat
transformations for anthropic, bedrock invoke, vertex and converse, the
modify_params dummy tool result, dotprompt with unchanged variables, and
Presidio masking against an in-process fake. Each serializes system,
tools and the earlier messages of turn N and N+1 and asserts they match.
The e2e mid-conversation system test imports its content blocks from
models.py again and is marked provider_live
* fix(anthropic): move mid-conversation system placement into prompt_templates
The prompt factory imported the placement helper from the Anthropic provider
package, whose common_utils reads a factory constant at import time, so loading
the factory first raised ImportError. The module now sits next to
anthropic_messages_pt and every consumer imports core utils
A user turn with content [] or None puts no block on the wire, so a system run
anchored to it landed first in messages or behind an assistant turn. Such a run
now converts in place; empty strings and empty text blocks still anchor because
the factory fills them with a placeholder
* fix(anthropic): anchor system messages only on user turns that reach the wire
* fix(bedrock): type the converse system-message helpers over the message TypedDicts
* fix(anthropic): read replayed pydantic messages in the Converse helpers and convert a system run whose assistant follower sends nothing
A history that replays the previous turn as the litellm.Message object
was invisible to the Converse system-message helpers, so a mid-conversation
system stayed between a tool call and its result or reached Converse as
role: system. The helpers now read fields through the shared
message_field and parts_of accessors and drop the local role predicate.
Flagged placement anchored a system run on any assistant follower, but
anthropic_messages_pt drops an assistant turn that puts no block on the
wire (content None, an empty list, an unsigned thinking part), so the
system landed directly before the next user turn, which Anthropic
rejects. Such a run now converts in place. An empty or whitespace text
turn still anchors, since the converter pads it with a placeholder.
* fix(anthropic): treat bridged encrypted reasoning as a vanishing assistant turn for system placement
An assistant turn whose only blocks carry Responses API encrypted reasoning is
dropped by anthropic_messages_pt, so a mid-conversation system run anchored
before it landed directly before the next user turn. The unsignable-thinking
predicate now lives in common_utils and both the factory and the placement
policy consult it.
* fix(anthropic): let an inline thinking part hide separate thinking_blocks in system placement
anthropic_messages_pt skips an assistant turn's separate thinking_blocks as soon
as its content list carries an inline thinking or redacted_thinking part, so a
turn whose inline part is unsigned puts nothing on the wire even when the
separate block is signed. The placement policy now mirrors that rule.
---------
Co-authored-by: Shifat Islam Santo <shifatislamsanto764@gmail.com>
Co-authored-by: ege-arhan <egearhany@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(bedrock): route unmapped openai family model ids to converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): rename the e2e openai family backend constant
global.openai.gpt-6-sol has a cost-map row now, so the constant no longer
names an unmapped model
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): tolerate provider-side flakes on five full-suite cells
Mistral OCR retries a provider-relayed 429 with backoff, the Vertex vision
probe turns reasoning off so its 32 tokens go to the answer, the Vertex
cache cell spaces eight never-seen prefixes 15s apart around Google's
nondeterministic minimum-token rejection and prices the cached tokens
instead of prompt_tokens, and the Azure content-policy cell resends the
jailbreak prompt while Azure skips its filter
* test(e2e): shorten the new helper docstrings
* test(e2e): accept a relayed provider 429 on the rust OCR cells
The gateway already retries a provider 429 three times per call and the
Mistral key is shared across pipelines, so a throttle can hold across all
four attempts of the OCR cell. After the bounded retries the cell now
accepts the gateway's faithful relay of the provider's 429 (throttling_error,
code 429) as its second expected outcome; the gateway's own 429 and any
other error still fail the cell at once.
* test(e2e): drop the harness unit tests, the live cells cover the helpers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(e2e): route credential, cost map, and UI login calls to the control plane
* test(e2e): stop pinning the upstream model name in the credential-backed messages test
* test(e2e): drop the harness unit test, the split stack run proves the routing live
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): surface an upstream handshake refusal as an error event and policy close
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): tidy the handshake refusal e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): keep upstream exception text out of the Azure client error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(realtime): map handshake refusal close codes with a lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add conversational matrix across chat, messages and responses
Parameterizes one behavioral contract (reply, stream, cost log, tool call,
tool round trip) across /v1/chat/completions, /v1/messages and /v1/responses,
OpenAI and Anthropic models, and env-ref vs stored-credential auth, with
record/replay fixtures.
Adds general_settings.disable_model_info_refresh so the proxy fronting a
replay fixture does not poll every OpenAI-compatible deployment's /v1/models
in the background, which otherwise leaves unconsumed interactions in the
recorded bundle.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): force the weather tool on the first turn and rename Provider to Deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolve conflicts in the e2e Responses suite: main replaced the bespoke
EndpointsClient with provider SDK clients (76bf0cd579), so the four added
Vertex and Azure OpenAI /v1/responses tests are ported onto the OpenAI SDK
and tool_choice=required now comes from the SDK instead of a custom
request model.
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deleted wrapper put cache: {"no-cache": true} on every request body, so the
gateway's response cache never answered a re-sent prompt. The SDKs send nothing
of the sort, and the mid-conversation prompt-cache priming loop re-sends an
identical body until the provider reports a warm cache, which a cached reply
never does. NO_PROXY_CACHE in sdk_clients.py restores the field as extra_body
on every messages, responses, completions, and embeddings call.
The wrapper also returned a 4xx as a value where the SDKs raise. The Bedrock
safety_identifier test judges the captured Converse body, and Claude on Bedrock
rejects the forwarded field with a 400, so the /v1/responses leg now suppresses
openai.BadRequestError the way the chat leg carries the same 400 as a Result.
Resolves the six conflicts main accumulated against the SDK migration.
endpoints_client.py stays deleted (main only added the safety_identifier
kwarg to it), and its two new importers move onto the SDK clients: the
Bedrock safety_identifier forwarding test in test_responses_e2e.py now
sends its /v1/responses leg through the OpenAI SDK and its chat leg
through ProxyClient.chat, and test_messages_azure_foundry_e2e.py drops
the x-litellm-rust header check together with EXPECT_RUST, which main
removed from e2e_config in a84f68b6e3. proxy_client.py keeps both the
rerank and router-settings model imports. uv.lock is main's lock plus
the anthropic e2e-dev entry, and the pricing JSONs are taken verbatim
from main since this PR never meant to change them.
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
AGENTS.md allows a vendor-owned literal only when its source and date are cited
next to it. A live realtime test cannot avoid naming a model, so record how the
id was checked, and record that a retired id fails as a hang rather than an
error so the next reader does not start by suspecting litellm.
AWS retired amazon.nova-sonic-v1:0. GetFoundationModel now answers
ResourceNotFoundException "This model version has reached the end of its life",
and opening a bidirectional stream against it fails with ValidationException
"The provided model identifier is invalid". amazon.nova-2-sonic-v1:0 is the
active replacement.
The test passed on builds 219 (2026-09-16) and 246 (2026-09-17) and has failed
every run since, three attempts per build, with no litellm change to the
realtime path in between. The symptom was a clean websocket close: Bedrock ends
the stream rather than erroring, the forwarder treats a None receive as a normal
stream end and closes the client socket, so the client sees ConnectionClosedOK
and the test fails waiting for response.done.
litellm already carries both models in the cost map, with
"deprecation_date": "2026-09-14" on the old one, and the promptStart
transformation already sends the audioOutputConfiguration that nova-2-sonic
requires; only the test constant was left behind.
Verified against live Bedrock with the promptStart shape the transformation
builds: amazon.nova-sonic-v1:0 raises "The provided model identifier is
invalid", amazon.nova-2-sonic-v1:0 opens a session and returns a usageEvent.
The mocked handler and provider-cache tests keep the old id: it is only a label
there, no call reaches AWS.
The check spins up a hypercorn TLS peer and drives the SDK's own httpx
handlers at it, so it needs litellm importable, hypercorn installed and a
loopback socket. It lived under tests/e2e, whose Buildkite runner image
installs neither litellm nor hypercorn by design (the suite drives a
remote proxy over HTTP), so every scheduled e2e build since #230 failed
to import the module and pytest reported it as a collection error. The
unit tree bans sockets, so it does not belong there either
tests/integration is the CircleCI tier built for real TCP against local
protocol peers. This adds an sdk shard to it for cases that exercise the
SDK's clients with no gateway in the path, registers the two HTTP/2
nodes in the contracts manifest, and adds the shard to the CircleCI
matrix. The test now flips the feature through LITELLM_HTTP2 (the user
surface) instead of patching module attributes, and asserts the version
the peer observed on the wire next to the one the client reports
Each route now has litellm/rust_bridge/<route>/{entrypoints,callbacks}.py and a
public dispatch module (litellm/chat_completions/dispatch.py,
litellm/responses/dispatch.py, litellm/messages/dispatch.py) that binds the
public call to the legacy Python signature, builds a frozen request, and asks
the runtime to pick Rust or Python from the catalog. The legacy implementations
stay in litellm/main.py, litellm/responses/main.py and the anthropic messages
handler, and litellm/__init__.py re-exports the dispatch names over them the
same way it already does for ocr
The per-handler shims in rust_bridge/chat_completions/native.py and
rust_bridge/messages/native.py are removed along with their call sites in the
anthropic and bedrock chat handlers and the http handler. The exception
mapping that every callbacks module repeated moves to rust_bridge/failures.py
and the signature binding helpers to rust_bridge/public_call.py
The key normalizes a unique marker so two builds match, which is the whole
point, but it makes this test's identity collide with an earlier run's: it mints
a fresh receipt, sends it through a tool result, and asserts the model echoes it
back verbatim, so a stale recording matched and answered with the old receipt.
Build 223 is where that surfaced, once the corpus was full enough for the first
call to hit. A test that asserts a provider echoed this run's own unique value
belongs on the live path.
The first cache-enabled litellm-e2e build (211) showed three gaps in the shared provider cache:
Every OpenAI response carries Cloudflare bot-management Set-Cookie headers, and the capture rejected any response with Set-Cookie, so no OpenAI response was ever recorded (179 of 372 misses rejected). The edge already withholds Set-Cookie from the proxy, so drop it before validating and storing instead of rejecting.
The provider prompt-caching tests need fresh provider state: a replayed priming response reports cache creation rather than a cache read, and the TPM test then trips the key limit. Mark both modules provider_live.
TestApiBaseSeam::test_live_mode_returns_none ran inside the cache-enabled runner and saw the shared edge; isolate it from E2E_PROVIDER_CACHE.
Re-migrates the e2e tests that main extended through endpoints_client
since this branch was opened (Azure Foundry, mid-conversation system
messages, Bedrock web search, google native streaming) onto the
provider SDK clients, so no endpoints_client reference remains
* fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e
PUT /v1/mcp/toolset dumped its payload with exclude_none, so a field sent as
null looked exactly like one the caller left out and the stored value
survived. An admin could not clear a toolset's description: the save reported
success and the old text came straight back. It now dumps with exclude_unset,
so absent keeps and null clears, which is what PUT /v1/mcp/server already did.
A null tools list clears the selection to empty, and a null toolset_name is
ignored because a toolset always has a name.
Adds create, read, partial-update, clear and delete e2e coverage for MCP
servers and toolsets, with every read-back polled on every replica so an edit
that lands on one replica and not another fails the test, plus an enforcement
test proving a key granted a toolset lists exactly that toolset's tools
against the real Datadog upstream.
* fix(e2e): refuse a read-back that no replica serves
A read-back over an empty replica mapping satisfied every predicate and
returned as if it had converged, so it would have asserted nothing and
passed. No wiring can produce that today, since the replica list always
falls back to at least one URL, but a helper whose whole job is proving a
write reached every replica should not have a shape that passes vacuously.
* fix(mcp): keep a null tools list a no-op on toolset update
Treating a null tools list as a clear meant an existing client that sends
tools=null during a partial update, meaning "leave the selection alone",
silently lost every tool the toolset grants. That is a permission surface,
so the quiet version of it is the worst version.
A toolset always has a tool list, the same way it always has a name, so a
null on either is now a no-op. Emptying the selection is an explicit [],
which cannot be confused with a field the caller left out, and which is
what the dashboard already sends.
* fix(e2e): keep MCP admin routes on the data plane
/v1/mcp/* is a lazily mounted feature, so a gateway registers it on the first
matching request, which happens after the startup route trim that drops
management endpoints. Routing it to the control plane therefore sent every MCP
call to the one backend process: the new lifecycle read-backs proved a single
process rather than every replica, and mcp_client's await_registered barrier
waited on a registry that does not serve the tools/list call it guards, so the
existing MCP suites polled a gateway that had not synced yet until poll_timeout
Verified against a two-gateway split stack (backend on 4001, gateways on 4010
and 4011, one postgres): both gateways answer /v1/mcp/server and /v1/mcp/toolset,
and each served 6 server reads and 7 toolset reads over the run
* fix(e2e): grant the toolset by the tool's own name, not the wire name
tools/list serves a tool as <prefix><tool_name>, but a toolset grants by the
tool's own name: resolve_toolset_permissions reads toolset.tools[].tool_name
straight through, and the prefix is added on the way out. The test built the
toolset from the names tools/list reported, so the grant matched nothing, the
scoped key listed no tools, and await_tools ran out its whole poll_timeout
before failing
Measure the prefix off search_datadog_logs, whose own name is known, rather than
guessing it from the alias, since the proxy can be configured to prefix with a
short server id instead. The expectation compared against tools/list stays in
wire names; only what the toolset stores crosses back
* test(mcp): build immutable lifecycle updates and replica results
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
* test: honor Datadog search rate-limit reset headers
* test: share the Datadog read-back deadline across retries
* test: preserve captured MCP toolset update fields
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
Anthropic now returns the 64-token 'count to 20' reply in one to three content_block_delta events, measured directly against api.anthropic.com and through proxies at 7672399 and 49a1145 alike, so the incrementality assertion (at least two deltas) failed in litellm-e2e builds 125, 130 and the 278 rerun with no proxy change behind it. A 'count to 100' reply at max_tokens 400 arrived in five to fifty deltas across every measured run
