* feat(messages): resolve azure_ai keys and entra tokens like python on rust
The native azure_ai Messages config now authenticates in the order Python's
BaseAzureLLM._base_validate_azure_environment uses. The key comes from the
call, then litellm.api_key, litellm.azure_key, AZURE_OPENAI_API_KEY and
AZURE_API_KEY, each skipped only when empty, so a blank key is sent as given
as Python does. Without a key the call mints an Entra ID token through
litellm-auth-azure from azure_ad_token, AZURE_AD_TOKEN, tenant_id, client_id
and client_secret, the caller's azure_ad_token_provider, OIDC references and
litellm.enable_azure_ad_token_refresh, and sends it as a bearer. Without
either the request goes out unauthenticated and Azure's 401 is mapped as
Python maps it, attributed to azure_ai instead of anthropic.
The bridge reads litellm.api_key, litellm.azure_key and the token refresh
switch from the provider defaults snapshot, and calls the caller's token
provider on the blocking pool so its exception reaches the caller as Python
raises it.
Approved differences that stay: a caller x-api-key or non-blank bearer
replaces the operator key instead of being sent beside it.
Known differences: Python's azure_username/azure_password flow has no
native counterpart, and a token provider combined with an OIDC reference
uses the OIDC token where Python calls the provider.
* fix(messages): let a caller api-key replace the azure_ai operator key on rust
A caller api-key header on an azure_ai Messages call was forwarded next to
the operator's x-api-key, so Azure received both credentials. It now
replaces the operator key and is sent as x-api-key, as Python does for a
lowercase api-key.
Approved difference: Python only matches the lowercase spelling and sends
a mixed-case Api-Key beside the operator key. Rust matches any casing.
* fix(messages): order azure_ai entra id flows exactly as python on rust
Without a key the azure_ai config now walks Python's get_azure_ad_token
order itself instead of letting litellm-auth-azure pick. Every setting is
read from the call and then the secret lookup, as AZURE_API_KEY is.
A static token, the caller's azure_ad_token_provider and client secret run
natively. Username and password, OIDC references, os.environ/ references,
an uncallable provider and token refresh go to Python's own
get_azure_ad_token through the host, as Python does them, so a static token
with refresh on and a provider with an OIDC reference behave as in Python,
and a call Python leaves without a token goes out unauthenticated.
Every azure_ai call now asks the secret lookup for the Entra ID names up
front, where Python asks only when no key is set.
* fix(messages): read azure_ai entra settings where python does on rust
Python's get_azure_ad_token reads AZURE_AD_TOKEN through get_secret_str and
AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_USERNAME,
AZURE_PASSWORD and AZURE_SCOPE with os.getenv
(litellm/llms/azure/common_utils.py:580-588). The azure_ai config now does
the same, reading the process environment only when no key is set, and only
AZURE_AD_TOKEN stays in the secret names, so a value only a secret manager
holds is ignored where Python ignores it.
Approved by the owner, same class as A06-A08: any caller credential header
(x-api-key, api-key or Authorization other than a blank bearer, in any
casing) is the only credential sent, and the Entra ID path never runs for
it. That covers a caller bearer on an Entra-only deployment (A10), which
Python overwrites with the deployment's token, a lowercase authorization
Python joins with the Entra token, and a caller x-api-key Python sends
beside the Entra token.
* fix(messages): look up AZURE_AD_TOKEN only where python does on rust
Python reads AZURE_AD_TOKEN through get_secret_str only when the azure_ai
call has no key, no caller credential and no azure_ad_token param
(litellm/llms/azure/common_utils.py:580). Messages configs can now name
deferred secrets, resolved right before validate_environment and only when
the config names them for that call, so a keyed call or a call carrying its
own credential never asks the secret lookup for the Entra token.
Approved by the owner, same class as A06-A08: a caller Authorization other
than a bearer, such as Basic, replaces the azure_ai operator key instead of
being sent beside it as Python does.
* fix(messages): build azure_ai urls and resolve the base like python on rust
The native azure_ai Messages URL builder now ports Python's
azure_anthropic_messages_url. It cuts at /anthropic in the path only, so a
base whose userinfo starts with "anthropic" no longer sends the key to
http://anthropic/v1/messages (security regression fix). It keeps the base
untrimmed and moves the slash before a query as Python does.
The base resolves as Python's `api_base or AZURE_API_BASE`: a blank call base
is kept, an empty one falls back to the environment, and the environment
value is used as given. With neither, the call raises Python's bare
ValueError ("Missing Azure API Base ..."), flagged as raised by Python and as
a late rejection, so the failure log carries the resolved provider and model
parameters as Python's does.
* fix(messages): send bases the native parser misreads through python's client on rust
A Messages api_base with credentials, no host or a port Python cannot read
now goes through Python's client, as bases without an http(s) scheme or with
whitespace already did. Before, reqwest read such a URL differently from
httpx: for http:///d it sent the request, with the operator key, to host
`d`. This is a security fix for both azure_ai and anthropic; the
anthropic_base_empty_host parity case covers anthropic.
For azure_ai the choice reads the base Python's URL builder reads, falling
back to AZURE_API_BASE, so a padded environment base also goes through
Python's client. Python reads AZURE_API_BASE once: transport_handler returns
the value it read and the bridge seeds the call's secret source with it, so
the native route reuses it rather than reading the secret again. _sent_url
only rewrites anthropic URLs, since the azure_ai builder no longer trims.
Parity: 20 azure_ai and anthropic URL cases added; 65 aspects failed on the
previous extension, and the full messages parity suite now passes.
* fix(messages): look up AZURE_API_BASE only where python does on rust
Python reads AZURE_API_BASE only through the `or` fallback in
`api_base or get_secret_str("AZURE_API_BASE")`. The native azure_ai config
listed it among the secrets it always reads, so a call with its own base
still looked it up through the secret manager. It is now a deferred secret,
read only when the call's base is empty or absent. A blank base is kept
without a lookup, as in Python.
deferred_secret_names now takes the call's api_base, the same base the URL
builder reads. A lookup-counting test covers a call base (0 lookups), an
empty base (1) and no base (1); before this change the call-base case
looked it up once.
* fix(messages): filter azure_ai anthropic-beta through its row like python on rust
Python's handler runs update_headers_with_filtered_beta over the azure_ai row of
anthropic_beta_headers_config.json after the config adds its feature betas, so
unknown and null-mapped betas are dropped whatever the header casing, the
caller's and the auto-added ones alike. Rust forwarded the caller's betas and
every auto-added beta as sent.
A caller anthropic-beta replaces a deployment's extra_headers anthropic-beta
before either lane runs, as Python does, and a Router parity test pins it with
the filter applied to both. The advisor tool already matched, since the bridge
hands it to Python's interceptor and its beta is null in the row, so a parity
case pins it. extra_body, unknown top-level params and the -latest alias also
matched and get parity cases.
* fix(messages): strip beta-gated system fields under drop_params like python on rust
Python's transform runs strip_beta_gated_native_request with the final,
filtered headers, so under drop_params a mid-conversation system turn loses
the clear_at, output_config and tool_addition/tool_removal fields no sent beta
unlocks, and a turn left empty is dropped unless that would end the
conversation on an assistant turn. _normalize_system_role_messages strips a
turn it converts to a user turn again with no betas at all. Rust sent those
fields as they came.
azure_ai now picks its betas from the request as sent and runs the strip,
system-role normalization and the cache_control.scope strip in a new
finalize_request step, once the final header is known. Vertex gets the
conversion-time strip too, since it shares the normalization helper.
The strips and the speed drop log Python's warnings word for word. A
non-object output_format under drop_params is still popped after the headers
were built, so structured-outputs-2025-11-13 is sent without it, as Python
does.
* fix(messages): send the system prompt as text blocks like python on rust
Python's _normalize_system_role_messages always rebuilds the top-level system
as a list of content blocks, so a string system reaches azure_ai and Vertex as
one text block, and the success log, which only lists a string system as a
message, logs the messages as sent. Rust kept the string unless a leading
system turn was hoisted.
Python's billing strip in transform_anthropic_messages_request also pops a
system prompt it leaves empty, an empty string included, as Python does. The
shared helper now does the same for every config that strips billing
metadata: azure_ai, vertex_ai, deepseek, minimax, tencent and edenai, each
with its own test.
* fix(messages): keep non-claude azure_ai messages passthroughs on python
A deployment whose provider has no Messages config of its own, like a
non-Claude azure_ai model, but lists /v1/messages in
model_info.supported_endpoints is forwarded untranslated by Python's
OpenAILikeAnthropicMessagesConfig, with a bearer key to <base>/v1/messages.
Dispatch gave it the MESSAGES target, so a rule that serves azure_ai on Rust
would hand it to the native Anthropic config, which sends x-api-key to
/anthropic/v1/messages and breaks the success log.
Dispatch now targets MESSAGES only when the provider's own config serves
the call, and MESSAGES_PASSTHROUGH for the opt-in passthrough. The catalog
keeps that target on Python next to the chat and Responses bridges, so once
azure_ai is Rust-required these calls resolve to Python and the Rust-only
refusal does not fire for them. Proxy token counting treats both targets as
before.
The parity forced rule now serves a provider natively only for the MESSAGES
target, as a Rust-required rule below the target exclusions does. A pre-request
hook that sets a non-Claude model raises functools.partial's TypeError on both
lanes, as on main, and a parity case pins it.
* fix(messages): raise azure_ai transport and provider errors like python on rust
The bridge mapped a native failure with the whole call as exception_type's
extra_kwargs, so litellm_debug_info gained an API Base line from the named
api_base. Python's entry passes only its **kwargs there, so its debug info
names the model alone. The bridge now passes the same kwargs, for anthropic
and azure_ai alike, and a decoding failure on a stream carries the
deployment's debug info as Python's stream iterator attaches it.
A connect failure to an azure_ai base read from AZURE_API_BASE kept the
native "could not reach the provider" text, since the call had no api_base.
The host now hands Python's failure mapping the secrets it read to send the
call, so the aiohttp and httpx texts name the base the call went to, without
reading AZURE_API_BASE again.
A key or header value with a control character, like a deployment key with
a trailing newline, failed natively as APIConnectionError with the model
stripped of its provider. It is now the InternalServerError 500 Python's
transport raises, with aiohttp's "Forbidden control character detected in
headers" text and the model as called. On the httpx transport Python's text
quotes the header value, which may be the key, so Rust keeps its own text
there.
The parity outcome now compares litellm_debug_info. The caller client's
ReadTimeout from a stalled stream still has none on Rust and is recorded as
a gap.
* fix(messages): send a caller azure_ai api-key without an operator key like python on rust
With no deployment key, Rust renamed a caller api-key header in any casing
to x-api-key. Python's AzureAnthropicMessagesConfig renames only a header
spelled exactly api-key and sends any other casing, like API-Key, as given.
Rust now does the same when no operator key would be sent. With an operator
key the approved behaviour stays, and a caller api-key in any casing replaces
it as x-api-key.
* fix(auth-azure): retry entra token posts like python's azure-identity
A client secret or workload identity token endpoint that keeps answering
500 made the native path POST 9 times over about 80 s, because
azure_identity 1.0 uses the SDK's default exponential retry (8 retries,
200 ms doubling, 60 s cap). Python's azure-identity 1.25.2 posts 4 times
in about 5 s and then raises, so the Rust credential now carries a custom
retry policy with the same numbers.
Python sources in the wt-auth venv
- azure/identity/_internal/pipeline.py:67 builds the token pipeline with
azure-core's RetryPolicy(**kwargs) and no overrides
- azure/identity/_internal/msal_client.py:98 (client secret) and
aad_client.py:77 (client assertion, used by workload identity) run the
token POST with retry_on_methods=["POST"]
- azure/core/pipeline/policies/_retry.py (azure-core 1.41.0) lines 77-95
give retry_status=3, retry_backoff_factor=0.8, retry_backoff_max=120,
retried codes 408, 429, 500, 502, 503, 504 and 506-998. Lines 127-143
sleep 0 before the first retry and factor * 2^(n-1) after that, so
0 s, 1.6 s, 3.2 s. Lines 455-509 sleep exactly Retry-After (then
retry-after-ms, x-ms-retry-after-ms) instead of the backoff when it is
present and non-zero
azure_core only hands the policy a Retry-After value for 429 and 503, so
a Retry-After on another status still falls back to the backoff. Managed
identity keeps the SDK default since Python uses a separate policy there,
and client assertion keeps it because Python mints OIDC tokens through
litellm's own HTTP client. A 401 invalid_client is still one POST.
* fix(auth-azure): post an oidc token exchange once like python
Python exchanges an OIDC assertion through litellm's own HTTPHandler
(_exchange_oidc_token in litellm/llms/azure/common_utils.py), a single
client.post with no retry, so a 503 is one POST. The native client
assertion credential used azure_identity's default retry (9 POSTs over
about 80 s) and now runs with RetryOptions::none().
* fix(auth-azure): honour retry-after on entra token posts like python
azure_core runs a custom RetryPolicy through its own loop, which only
reads Retry-After for 429 and 503 and never retries a status outside the
policy's list. Python's azure-core RetryPolicy (1.41.0,
azure/core/pipeline/policies/_retry.py) retries any 4xx or 5xx that
carries Retry-After (is_retry, lines 210-242) and waits exactly the
Retry-After, retry-after-ms or x-ms-retry-after-ms value when it is
non-zero (get_retry_after in _utils.py:79-96, sleep at lines 455-509).
The client secret and workload identity credentials now run with the
SDK retry turned off and a per-call policy that follows Python's
loop. It keeps separate budgets of 3 for status, connect and read
retries (increment and is_exhausted, lines 265-330), backs off 0 s,
then 0.8 * 2^(n-1) capped at 120 s, and retries 408, 429, 500, 502,
503, 504 and 506-998 for the token POST. Client assertion still posts
once and managed identity keeps the SDK default.
A Retry-After that is neither a number nor an HTTP date now fails the
token request after the first POST with "Invalid HTTP date", as Python
does. azure-core's _parse_http_date (_utils.py:48-57) raises ValueError
from the sleep before the retry, azure-identity's wrap_exceptions
(decorators.py:63-80) turns it into ClientAuthenticationError
"Authentication failed: Invalid HTTP date", and get_azure_ad_token
(common_utils.py:663) raises RuntimeError "Failed to get Azure AD token:
Authentication failed: Invalid HTTP date".
* fix(messages): raise python's entra token errors for native azure_ai flows
On the Messages route every Entra token failure now reaches the caller as
Python's get_azure_ad_token and anthropic_messages raise it, with the same
class, status_code, llm_provider, message and log lines.
A native client secret flow keeps the token endpoint's final answer
(status, headers, body) in a typed EntraTokenFailure from the azure-identity
retry policy. The bridge hands it to the Messages host, which replays it
through azure-identity's own ClientSecretCredential over a transport that
returns that answer, so MSAL and azure-core read it exactly as they read it
off the wire. That covers a rejected secret (EntraRejectionError 401 with
the first AADSTS sentence and GUIDs redacted), an exhausted 5xx ("HTTP
Error: 500"), a malformed Retry-After ("Invalid HTTP date"), non-JSON and
odd JSON bodies, including the content-type dependent decode error. A
transport failure has no answer to replay and keeps Rust's own reason
after "Authentication failed:", so its text differs from requests' text.
The caller's azure_ad_token_provider goes through the helper
get_azure_ad_token now shares (azure_ad_token_failure), so a listed AADSTS
rejection is an AuthenticationError 401 instead of a RuntimeError, and a
non-string token raises the same TypeError. Both log Python's "Error
calling Azure AD token provider" and "returned non-string value" lines.
Every Python exception raised while getting the call's credential,
including the host-minted username/password, OIDC and workload identity
flows, now goes through Python's handler rule: a BaseLLMException is mapped
by exception_type (AuthenticationError 401, BadRequestError 422 with
llm_provider azure_ai), anything else reaches the caller as raised.
The Retry-After date parser stays RFC 7231 only. Python's
email.utils.parsedate_tz also accepts dates without a weekday, two-digit
years, numeric zones and missing seconds, which Rust rejects as an invalid
HTTP date and Python waits on. A NaN or infinite number of seconds now
falls back to the backoff, where Python sleeps 0 for NaN and fails on
infinity.
* fix(auth-azure): wait on retry-after and report connect failures like python
The Entra token retry policy now reads Retry-After the way azure-core
1.41.0 does. Seconds go through Python's float rules, including
underscores between digits, and anything else through a port of
email.utils.parsedate_tz, so a date without a weekday, a two-digit year,
a numeric or named zone, a zone glued to the time, missing seconds, a
dotted time, month before day and RFC 850 dates all wait as Python waits.
A date datetime() rejects (day out of range, an offset of a day or more)
still fails the request after one POST, as do unparseable values.
A wait that time.sleep cannot take (infinity, or 2^63 ns and past) now
fails after one POST as Python does. The host replays that answer with a
sleep that still calls time.sleep for such a wait, so the caller gets
"Authentication failed: timestamp out of range for platform time_t".
A transport failure keeps the token endpoint's host and port. A direct
connect that fails with an OS error, or a failed name lookup glibc words,
reaches Python as urllib3 2 words it after "Authentication failed:", for
example "HTTPSConnection(host='login.microsoftonline.com', port=443):
Failed to establish a new connection: [Errno 111] Connection refused".
Python hits it on MSAL's discovery GET to the same host. Failures through
a proxy, TLS failures, timeouts and resets while reading keep Rust's own
text, since urllib3's wording for them depends on details Rust does not
expose.
* fix(messages): treat a contentless system message as a system turn like python on rust
Python's _normalize_system_role_messages reads only the role, so a role
system message sent without content or with null content is still a system
turn. A leading one is hoisted and adds nothing to the top-level system, and
a later one on a model without mid-conversation system support becomes the
operator note user turn, placed after a tool result it sits before and
dropped under drop_params when its gated field is stripped. Rust failed to
parse the message and sent it raw, which also showed up in the logged
messages. A model that accepts the role still gets it as sent.
* fix(messages): let the provider filter a non-ascii anthropic-beta like python on rust
Python merges every anthropic-beta spelling into one header and the
provider config filters it before the HTTP client encodes it, so on azure_ai
an unknown non-ASCII beta is dropped and the call goes out, with or without
drop_params. Rust held the header back as unencodable before the beta filter
ran and failed the call with the ascii codec error. The beta header now
reaches the provider filter, and the wire check fails the call only when a
value Python's client cannot encode survives it. A value with a lone
surrogate still fails up front since it cannot reach Rust as text.
* fix(messages): log nothing for a stream the caller never read like python on rust
Python's Messages stream iterator only starts on the first read, so a
stream the caller closes or drops unread bills nothing and logs nothing.
Rust billed the empty stream on close and teardown, which logged "Unable to
build complete streaming response" and the read() error from the success
handler.
* fix(messages): log a traceback for failures python raises before the send on rust
Python raises its pre-flight errors, such as the missing azure_ai api_base
ValueError, the missing key AuthenticationError and the Entra token
provider failures, through Python frames, so the failure log and the OTel
span carry a stack trace. Rust built those exceptions without raising them,
so error_information.traceback was empty. The host now raises them through
a Python frame before handing them back.
* fix(messages): return python's streaming response types from native messages streams
Python's Messages handler returns an AnthropicMessagesStreamingResponse for
a stream, so isinstance checks, hidden_params and aclose behave the same for
every caller. The native async entrypoint returned the bare execution
AsyncStream. It now wraps the stream the way the deployment success hook
already did, sharing the stream's hidden params. The entrypoint is shared, so
this moves anthropic as well as azure_ai toward Python.
With a response cache configured Python hands back its caching handler's
types too. A streamed cache hit is now the CachedAnthropicMessagesStreamIterator
Python's handler builds from the cached events, stamped with the metadata
@client stamps, and the native execution closes unread so only that iterator
logs the hit, as Python does. A miss is wrapped in the
AnthropicMessagesStreamCacheWriter Python returns, marked persisted since Rust
stores the stream itself. The parity suite now compares the stream type for
every streamed case.
* fix(messages): read unparsed messages in beta detection and gated strips like python on rust
Python reads beta triggers and gated fields off any message mapping, while
Rust only looked at messages that parsed. A role system message sent without
content, or any message with a content block Rust cannot type, was skipped:
- a message carrying output_config now adds per-turn-control-2026-07-01
- a system message with a tool_reference addition or removal now adds
mid-conversation-tool-changes-2026-07-01
- a signed compaction block now adds compact-2026-09-04
- under drop_params an unparsed system message loses its gated clear_at,
output_config and tool changes, is dropped once emptied, and the
warning names the strip
- an unparsed assistant message counts as the last assistant turn when
deciding whether a strip would leave a prefill
- an unparsed message loses cache_control.scope on its content blocks
* fix(messages): rewrite and normalize unparsed messages and system prompts like python on rust
Python's sanitizer, advisor and encrypted-reasoning strips and system
normalization read any message mapping, while Rust skipped every message
or system block it could not type:
- an unparsed message now gets the sanitizer's rules: a text block whose
text is not a non-blank string and an empty unsigned thinking block go,
a message left with no blocks is dropped, tool ids are sanitized,
replayed web search results are flattened and provider_specific_fields
dropped; a message the rewrite leaves parseable is parsed again
- an unparsed assistant turn loses advisor exchanges, and an unparsed
message loses bridge-tagged reasoning blocks
- a kept redacted_thinking block gets Claude's own data back from a
Bedrock Converse base64 wrapper, parsed or not
- system is kept as sent when it is a list holding a block that does not
parse, or an object; billing blocks and cache_control.scope are still
stripped from such a list
- a leading system turn whose content does not parse is hoisted as sent,
and a later one becomes the operator note user turn as sent, placed
after a tool result turn that did not parse either
* fix(messages): keep anthropic's API Base debug line on rust failures
Mapping a native failure with only the Python entry's **kwargs dropped the
named api_base from exception_type's input. That matches azure_ai, whose
Python entry maps with **kwargs alone, but it removed the API Base line that
main's Rust-only anthropic route has always put in litellm_debug_info when the
caller passes api_base.
Python gives no rule that tells the two apart, because the legacy async entry
maps both providers the same way. The difference is that anthropic has no
Python path, so main's native route is its contract. The bridge now maps the
whole call for anthropic and the entry kwargs for every other provider.
The parity harness still uses the legacy Python entry as anthropic's
reference, so it now expects the API Base line main's anthropic route adds.
New cases cover an anthropic call on an azure_ai model failing with api_base,
unary and stream.
* fix(messages): keep anthropic stream types and header errors as main while azure_ai matches python
By the owner's ruling anthropic keeps main's behaviour for two things the azure_ai port had changed for every
native provider.
A native async stream returns Python's AnthropicMessagesStreamingResponse, and with a response cache the
CachedAnthropicMessagesStreamIterator or AnthropicMessagesStreamCacheWriter, only for azure_ai. The host asks
Python for those types only when the call resolves to azure_ai, so anthropic gets main's execution AsyncStream
again, for a cache hit or miss too.
A header value the native client cannot send, like a key with a control character, is Python's
InternalServerError with aiohttp's "Forbidden control character" text only for azure_ai. Anthropic raises
main's APIConnectionError "invalid request: failed to parse header value" with the bug report link again.
The parity stream_type aspect compares azure_ai with Python and expects main's AsyncStream for anthropic,
in the same way the outcome aspect already expects main's API Base debug line. New azure_ai cache hit and
miss stream cases check the cached types.
* fix(messages): keep anthropic as main for azure_ai parity side effects
By the owner's ruling anthropic, which has no Python path, keeps main's native
route; the azure_ai port's Python-parity changes now apply to azure_ai only.
- An unread stream: only azure_ai's starts on the first read and bills
nothing when closed or dropped unread. The logging adapter learns the
provider from the execution facts; anthropic bills it as main does.
- Failures before the send: the missing-key AuthenticationError and the
unmapped builtin errors are raised through a Python frame only for azure_ai;
anthropic hands them back unraised, without a traceback, as main does.
- A stream's decoding failure carries the deployment's debug info only for
azure_ai; anthropic raises it bare.
- A non-ASCII anthropic-beta is left to the provider filter only for
azure_ai. Anthropic holds it back before the beta merge as main does, so
the error names it ahead of any later unencodable header.
- Unparsed messages: the shared shaping reads messages and system prompts
Rust could not type only when the provider asks for it
(UnparsedMessages::Read), which only the azure_ai config does. Elsewhere
the sanitizer, advisor and encrypted-reasoning strips leave such messages
as sent, their beta triggers add no beta, a redacted_thinking Converse
wrapper is kept as sent, and a system prompt that does not parse is
main's "untagged enum SystemPrompt" request error from messages_body.
Each item has an anthropic test pinned to main's behaviour that fails on the
previous commit, beside the azure_ai cases comparing with Python.
* test(messages): pin the malformed api_base fix for anthropic and azure_ai
d1efdd2683 sends a base the native URL parser reads otherwise than httpx and
aiohttp through Python's client. Parity cases now cover, for anthropic and
azure_ai alike, the three shapes that sent the operator key to an unintended
host natively (an empty host, credentials in the URL, userinfo that reads like
a trusted host) and the two port shapes, asserting that nothing reaches the
upstream and that the caller gets Python's exception. They fail with the
check disabled.
The rust_bridge AGENTS.md states the rule.
* refactor(messages): serve azure_ai messages on rust only and delete its python path
Provider azure_ai on /v1/messages is now RUST_REQUIRED next to anthropic, so a
Claude deployment whose AzureAnthropicMessagesConfig serves the call runs on
Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python.
The PYTHON_ONLY targets rule still sits above it, so a non-Claude azure_ai
deployment that opts into /v1/messages (MESSAGES_PASSTHROUGH) and an azure_ai
call bridged to Chat Completions or Responses stay on Python. A raising
detector on a Claude azure_ai call gets the same UnsupportedParamsError as
anthropic. MESSAGES_PYTHON_FEATURES is empty, so no feature refuses today.
Deleted Python code that only served azure_ai Messages:
- AzureAnthropicMessagesConfig._remove_scope_from_cache_control, a copy of
the AnthropicMessagesConfig method it overrode. Its only caller is the
azure transform_anthropic_messages_request (grep "_remove_scope_from_cache_control"
litellm/: azure, databricks and vertex call self., the base defines it)
Kept, because the parity harness still runs every azure_ai case through the
Python handler (dispatch._PYTHON_AMESSAGES) as its baseline, and removing them
would send that baseline to api.anthropic.com with anthropic auth:
validate_anthropic_messages_environment, get_complete_url,
transform_anthropic_messages_request and should_strip_billing_metadata on
AzureAnthropicMessagesConfig (grep: only async_anthropic_messages_handler in
llm_http_handler.py calls them, reached from the Python fallback and the
harness). The Foundry beta-row branch was already removed with anthropic, and
resolve_beta_provider's Foundry row now serves only anthropic chat. Count
tokens has its own config and the Rust host reaches no deleted method.
The parity harness drops the rust_providers forcing, so azure_ai cases run
through the shipped rules and assert RUST_REQUIRED like anthropic. The azure
count tokens beta test sends its inference call through a caller client, as
Rust now serves it.
* test(messages): cover the azure_ai agentic follow-up deployment on rust
The Rust host builds an agentic follow-up's kwargs with _agentic_hook_kwargs,
so the follow-up must reach the azure_ai deployment's own api_base with its key.
This replaces the deleted LIT-5418 test, which drove the Python handler.
Dropping api_base from that helper fails it with Missing Azure API Base, and
dropping api_key sends the follow-up without x-api-key.
* fix(auth-azure): retry Entra ID token posts like Python only for azure_ai Messages
The azure-identity retry policy and the single OIDC post now apply only when a
consumer asks for them through AzureAuthInputs.token_retry. The azure_ai
Messages config asks for it, while Key Vault, Redis, blob cache, OCR and the
anthropic federation keep the SDK default as on main. The policy is part of
the credential cache key so consumers sharing a client never share a policy.
* fix(messages): keep azure_ai without a key as main when no Python host is wired in
Entra ID auth for an azure_ai call without a key now runs only when the route
has a Python host. The Rust gateway has none, so such a deployment fails with
the local MissingApiKey main raised instead of going out unauthenticated or
failing on an Azure AD token without a host, and a deployment string named
callable is never read as a caller token provider there.
* fix(messages): log the dropped speed warning only for azure_ai
The shared drop_unsupported_params helper logged Python's speed warning for
every Messages provider. Configs now opt in through logs_dropped_params, which
only azure_ai sets, so anthropic and the other providers log as main.
* test(messages): drop the azure_ai default headers structure test and date pinned error texts
default_headers_match_python only restated a constant. The MSAL, azure-core and
CPython texts the replay tests pin now carry the date they were checked.
* refactor(messages): name the anthropic error-mapping set for what it is
_RUST_ONLY_PROVIDERS held only anthropic, the provider that keeps main's native
error mapping, while azure_ai is Rust-only too. It is now
_MAIN_NATIVE_ERROR_PROVIDERS, and the docstrings no longer call azure_ai a
provider with a Python path.
* docs(messages): keep azure_ai's python request code as the live parity reference
azure_ai Messages stays Rust-required, but its Python request code is kept so
the parity suite runs the live Python leg, as anthropic, Vertex and Bedrock do.
* fix(messages): send azure_ai caller and operator credentials exactly as python does on rust
The owner reversed the earlier approved difference on 2026-11-13 (Maciej).
For azure_ai on /v1/messages every caller, operator and Entra ID credential
combination now sends the header set main's Python sends, quirks included.
Python builds a dict keyed by exact name, so a caller api-key spelled exactly
so stops the operator key and the Entra ID path, the operator key otherwise
goes in as api-key and the Entra ID token as Authorization beside or over the
caller's headers, api-key is renamed to x-api-key unless x-api-key is spelled
exactly so, and the HTTP client joins names that differ only in case.
Ledger rows that change, all previously "Rust sends only the caller credential".
A06 caller x-api-key now goes out beside the operator api-key.
A07 caller X-Api-Key is joined with the operator key in one x-api-key.
A08 caller bearer now goes out beside the operator x-api-key.
A09b caller API-Key (other casing) goes out as api-key beside the operator x-api-key.
A10 a caller Authorization bearer on an Entra-only deployment is overwritten
by the minted Entra ID token.
A10b a caller lowercase authorization is joined with the Entra ID bearer.
A10c a caller x-api-key goes out beside the Entra ID bearer.
A caller non-bearer Authorization (Basic) now goes out beside the operator key.
A09 (an exact api-key replaces the operator key as x-api-key) is unchanged.
AZURE_AD_TOKEN is now read wherever Python reads it, which is any call
without a key, an exact caller api-key or an azure_ad_token param. The parity
harness gaps for these rows are removed so they run as normal cases.
* fix(messages): log the azure_ai call flag on failures before the send
Python's Messages handler sets the aanthropic_messages flag before it gets the
Entra ID credential or refuses an unsupported param, so those failures log it.
The native host now flags azure_ai calls the same way when a token request
fails or the request is refused, and anthropic keeps main's unflagged failure.
* fix(auth-azure): fetch azure_ai entra tokens with python's proxy and ca settings
Python's azure-identity sends the client secret token request through
requests, which reads https_proxy or all_proxy unless its no_proxy matching
skips the host, trusts REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE or certifi's
Mozilla roots, never SSL_CERT_FILE, and times out after 300 seconds.
azure_core's reqwest client read HTTPS_PROXY before https_proxy, matched
NO_PROXY its own way, trusted the system store and SSL_CERT_FILE and timed out
after 20 and 60 seconds.
An explicit token_transport input on the auth request selects a transport
that applies requests' rules over a litellm-http client. Only azure_ai
Messages asks for it, so Key Vault, Azure Redis, blob cache and OCR keep
azure_core's transport.
* fix(messages): word azure_ai entra transport failures as python's requests does
Python's azure-identity reports a failed client secret token request with
requests' own text after "Authentication failed:", while the native flow kept
reqwest's text for anything but a direct connect or name lookup failure.
The requests transport now tags each failed send with the proxy it used, and
classifies what reqwest, hyper and rustls observed into the failures requests
words the same way every time: an unreachable or unresolvable proxy, a 407
answer to CONNECT, a proxy or server closing or resetting the connection, an
EOF in the TLS handshake, an expired, not yet valid or misnamed certificate,
and the 300 second read timeout. The host builds each text from the
urllib3, http.client and ssl exceptions Python raises for it, so build
details such as the _ssl.c line come from the running interpreter. A missing
REQUESTS_CA_BUNDLE path now raises requests' own text too.
* fix(messages): look up AZURE_AD_TOKEN for azure_ai only when a Python host is wired in
The Rust gateway has no Python host, so a key-less azure_ai Messages call
there fails with main's local MissingApiKey before any Entra ID flow. It no
longer asks the secret source for AZURE_AD_TOKEN first, which main never did.
Calls with a Python host read it where Python does, as before.
* fix(messages): forward a caller credential on a key-less azure_ai call without a Python host as main
Without a key or a Python host, as in the Rust gateway, main's key-only
azure_ai check forwarded a caller x-api-key or bearer and failed with
MissingApiKey otherwise. The branch made every such call a missing key, so
the gateway's entra_bearer_without_key case failed; it now forwards those
credentials again. Calls with a Python host are unchanged.
* fix(messages): time an azure_ai exchange through python's client once on rust
A call the Rust host sends through Python's client, such as a malformed or
padded api_base, is already timed by that client's track_llm_api_timing post.
The native logging adapter timed the same exchange again, so the Rust lane
reported two LLM service events and recorded the window twice where Python
reports one. The route host now tells the adapter when it sends through its
own client, and for azure_ai the adapter leaves the timing to it. Anthropic
keeps main's behaviour.
* fix(messages): look up advisor ids in a set like python on rust
* test(messages): cover the native cache, stream and host failure paths
* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* test(e2e): share the CLI version rule, env lockdown and rate limiter in tests/e2e/cli_harness
Moves the npm release rule (now taking the package name), the env allowlist plus fresh HOME, and the per-upstream rate limiter out of claude_code so the Codex suite can reuse them. Claude Code behavior is unchanged, its CI, cron and Dockerfile paths now call the shared resolver with @anthropic-ai/claude-code
* test(e2e): freeze the packument map and note the rate limiter self-test
* test(e2e): race real processes on one rate limiter bucket so a missing lock fails
Eight spawned processes start together and drain a bucket that never refills. With the flock they get exactly its burst between them, without it every attempt is granted
* test(e2e): keep the old Claude Code version resolver path for the Buildkite runner images
The runner Dockerfiles still run tests/e2e/claude_code/pr_gate_version_resolver.py and silently install @latest when it is missing, which drops the three day review window. The old path now forwards to cli_harness/npm_version_resolver.py
* test(integration): poll the partition lock witness off the event loop
The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting
* test(e2e): retry the reasoning turn until the model emits a reasoning item
Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion
* test(e2e): retry a Bedrock Converse model once when it reports no cache tokens
Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged
* test(e2e): poll until a deleted stored response stops being retrievable
Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx
* test(e2e): give the Together prompt cache five primed attempts
Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed
* test(e2e): allow 30s for the idle RSS reading at session start
A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged
* test(e2e): scope the MCP submission check to its own card
The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card
* test(router): wait for the primary success record before counting it
The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs
* test(proxy): give the spend-log cleanup run a 1s budget
A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves
* test(autorouter): release the slow token count at 2.05s instead of 2.3s
The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack
* test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning
grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting
* test(integration): give the two worker-kill chaos tests the owned-proxy time budget
Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests
* test(integration): compare regional image responses without their created timestamp
Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared
* test(integration): count only the S3 logger's flush task
The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels
* test(integration): send guardrail timeout probes eight at a time
All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits
* test(integration): count a killed Arize chaos worker as gone once it is a zombie
psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget
* test(integration): send the Typesafe connection test its real model and endpoint
The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly
* test(integration): keep DB-stored models out of the usage-routing Redis read test
The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off
* test(vertex_ai): prove the batch upload streams by laziness instead of peak memory
The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests
* test(integration): keep the alias test_connection call as a known bug
* test: type the counting mapper and the Bedrock rerun helper
* test: annotate the new test locals as Final and build them as tuples
* test(vertex_ai): prove pull-driven transforms with the garbage tail alone
* test(e2e): move the harness self-tests out of tests/e2e
The nightly Buildkite run copies tests/e2e into the runner image and runs
bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit
properties, the stack lock, the load aggregators, the Claude Code driver)
counted as e2e tests on the status page even though none of them reaches a
proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout,
and run in the GitHub Actions lint job and the CircleCI
provider_replay_harness job instead
* fix(ci): point the providers replay controls at tests/e2e_harness
The providers integration job still selected the four replay-control
tests under tests/e2e/test_provider_edge.py, so pytest exited before
they ran. The raw-HTTP check's file walk also drops to one loop per
comprehension
* style(tests): mark the raw-HTTP check's bindings Final
* test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata
* test(e2e): tag claude_code tests with Subject metadata and record CLI driver steps
* docs(e2e): name every markerless harness test file that carries no Subject
* test(e2e): keep the step discovery comprehensions to one for clause
* test(e2e): decorate run_claude directly so the label gate discovers its step
* feat(e2e): read management routes back from the control plane replicas
The suite's management read-backs (/key/info, /team/info and friends)
polled the same replica list as the data plane. On a componentized stack
whose LITELLM_PROXY_REPLICA_URLS names the gateway pods directly, that
list answers those routes 404, since a gateway pod trims the management
routes at startup. A new LITELLM_CONTROL_PLANE_REPLICA_URLS names the
addresses a management read-back polls instead: an exported list wins,
and when it is unset the old rule stands, the data-plane replicas while
the control plane shares the suite's base URL and the control-plane base
alone once it is split.
build_proxy_client takes the list as control_replica_urls and
read_back_everywhere picks its replicas per path, the way the rest of the
client already does.
* fix(e2e): derive the control replicas of a client built for another proxy
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
npm keeps an unpublished version's timestamp in the packument's time map but drops it from versions, so the resolver could hand npm install a version it refuses with ETARGET. Only versions still present in versions are candidates now.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(compat-matrix): resolve and install the Claude Code CLI per run
* chore(compat-matrix): drop the installer header comment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job
The daily Claude Code compatibility-matrix job ran as a systemd timer on
the litellm-compatibility-matrix-populator VM in the vertex-check GCP
project. Replace that with a Render Docker cron job built from a new
Dockerfile in tests/e2e/claude_code/cron_vm: pinned and checksummed
debian base, gh, uv, and Claude Code CLI, a non-root populator user, and
run_daily.sh as the entrypoint. run_daily.sh now clones a fresh blobless
checkout per run (Render cron disks are ephemeral), reads the publish PAT
from the github-token secret file under CREDENTIALS_DIRECTORY, and its
comments no longer describe systemd. The .service and .timer units are
gone; README.md and the env example describe the Render service, its
secret files, and the local docker build instead.
* docs(e2e): name the plan and trigger route Render's cron-job API accepts
Render answers a bare 404 for the legacy pro_max plan name on a cron job
(4c-16g is the same 4 CPU / 16 GB size) and the manual trigger route is
/v1/cron-jobs, not /v1/cronjobs.
* fix(e2e): install the published litellm wheel instead of building the tag from source
The tag builds a Rust extension through maturin, which needs a C and Rust
toolchain the cron image does not carry, so the first Render run failed at
uv sync with "linker cc not found". Sync the locked dependencies with
--no-install-project, install the PyPI wheel (what users run) with
--no-build, and pass --no-sync to every uv run so uv never puts the source
build back.
* fix(e2e): keep the SKIP_PUBLISH matrix where a Render run can read it
The validation run wrote the matrix into the image checkout, which nobody
can read once the container exits. Save it under HOME and print it at the
end of the log instead.
* fix(e2e): let the stale compat-matrix PR sweep see past the newest 100 docs PRs
The docs repo has a few hundred open PRs, so a 100-item list never
reached the week-old compat-matrix PR and the sweep left it open on
every run.
* docs(e2e): say the Render cron needs a manual deploy after each merge
Pushes never started a deploy during setup because Render only hears
about them through its GitHub app, which the org does not have, so the
README now carries the deploy command and the wait-for-live rule
* ci: build the compat-matrix cron image on pull requests
The CI coverage gate requires every Dockerfile to be built by a job, and
building this one on each PR that touches it also catches a broken pin
or checksum before Render does
* fix(e2e): shim the whole tests/e2e tree into the compat-matrix worktree
The five-file helper allowlist missed fixture_mode, which e2e_config now
imports, so the first Render run died at conftest load with
ModuleNotFoundError. Copy the image's whole tests/e2e tree instead and
keep pytest from loading the EKS-harness conftest with --confcutdir
* fix(e2e): scope the compat-matrix sweep to the publishing account's own PRs
The stale-PR sweep selected every open docs PR whose head branch starts
with compat-matrix/, so a contributor's fork PR under that name would
have been closed once a newer matrix PR existed. The sweep now resolves
the publishing login from the token and only closes same-repo PRs that
account opened
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
Build 232 took two compat cells red with a FileNotFoundError renaming
`.claude.json.197` onto `.claude.json`. `run_claude_models_parallel`
drives several models from one process, so a pid-suffixed staged name is
shared between threads: one thread renamed the file the other was still
writing, and the loser died on a path that no longer existed.
mkstemp in the same directory gives a name that is unique per thread as
well as per process, and the rename stays atomic.
The compat cells drove the CLI with a fresh HOME per invocation and the
pytest process's own working directory, and both reach the request body.
The system prompt names a memory directory built from
$CLAUDE_CONFIG_DIR/projects/<cwd slug>, so a per-invocation config
directory rewrote every body, and the CLI adds a git block for its
working directory, so inheriting the checkout rewrote every body once
per candidate. The device id churned for the same reason: the CLI mints
it once and persists it in .claude.json, which we threw away each call.
Nothing here was load-bearing. All three ride in metadata.user_id, whose
job is abuse detection, not quota, caching or continuity. So pin the
config directory and the working directory at fixed paths, seed the
device id, and pin the session id.
HOME stays fresh and empty per invocation, so the isolation is no weaker
than before, and the CLI's own state no longer outlives the pod either.
The working directory is deliberately not the checkout, so a
model-directed Read now sees an empty directory rather than the
repository.
A pinned session id needs --no-session-persistence beside it: the CLI
refuses a session id another live process holds, and the matrix runs its
cells across xdist workers. Without the flag, six of eight concurrent
invocations die on "Session ID is already in use".
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
The driver retries rate-limit-shaped failures twice with a 65s backoff, but
RATE_LIMIT_SHAPED_RE only matches 429 / rate limit / too many requests /
throttled / CLI timeout. A saturated provider does not always say any of
those: litellm-e2e-pr build 182 turned a green cell red on
status 503: litellm.ServiceUnavailableError: BedrockException -
{"message":"Bedrock is unable to process your request."}
which matched nothing and so was never retried.
Add TRANSIENT_UPSTREAM_SHAPED_RE for 503, 529, "service unavailable",
"overloaded" and Bedrock's "unable to process your request", and have the
retry loop ask is_retryable_shaped (either shape) instead of
is_rate_limit_shaped.
Kept as a second pattern rather than widened into the first on purpose. The
conftest feeds RATE_LIMIT_SHAPED_RE into the rate-limit summary, which the
binary-search helper reads to decide whether to lower a provider's request
rate. A 503 says the provider is out of capacity, not that we are asking too
often, so folding it in there would keep ratcheting our rate down against a
condition our rate never caused.
Retry-loop logging now names which shape fired, so a saturated upstream is
distinguishable from a throttled one in the run output.
Verified against the literal failure text from build 182: is_retryable_shaped
is True while is_rate_limit_shaped stays False.
Note this covers CLI-driven rows only. HTTP-probe rows (tool_search,
count_tokens) have no retry layer at all, so the specific cell that failed in
build 182 is still unprotected -- that is a separate change to http_probe.
The tool_search x bedrock_invoke cell only ever probed the first turn, so
nothing in the suite has sent a server_tool_use block back to a provider.
Every turn of a real Claude Code session after the first carries the
server_tool_use and tool_search_tool_result blocks the previous turn
produced, and that path was uncovered.
Adds probe_tool_search_multiturn, which takes the real assistant turn
back, answers any client-side tool_use with the id the model actually
emitted, and replays the whole thing as history with the tools still
declared. The assertion refuses to go green unless both server-tool
blocks made it into the replayed history, so a first turn truncated at
max_tokens reads as a failure instead of a vacuous pass.
The replay assertion's red paths never run in a green cell, so they get
markerless harness tests of their own alongside the existing
_builder_unit_tests tree.
No production code.
Claude Code drives Opus 4.7 with thinking {"type": "adaptive"} plus
output_config {"effort": "max"}. The anthropic-to-openai adapter
forwarded thinking verbatim for Claude models but dropped output_config,
and Bedrock Converse streams zero reasoningContent blocks for adaptive
thinking without an effort tier. Forward the effort subset of
output_config for Bedrock targets, accept it in the converse supported
params, and map it with the model's effort ceiling applied. Re-enable
the skipped e2e compat cell that catches this
Bedrock InvokeModel rejects tool_search_tool_* tool types unless the
request body carries the tool-search-tool-2025-10-19 beta. The model
allowlist gating that beta omitted Haiku 4.5 (and Opus 4.7, supported
since launch per live verification), so every tool-search request on
those models got a Bedrock 400. Add both to the allowlist and re-enable
the e2e compat cell that caught it.
The mateo-berri PAT now arrives via systemd LoadCredential as a file
instead of the EnvironmentFile, so pytest, the proxy, and the
model-driven claude CLI never inherit it and a same-UID /proc read
cannot lift it. run_daily.sh reads the credential when present, still
accepts an exported GITHUB_TOKEN for manual runs, and dies up front
when publishing is enabled with neither. Full CLI sandboxing is
tracked in LIT-5420
Ports the daily cron VM publisher from the unmerged tests/claude_code
checkout so the automation runs the e2e suite from litellm_internal_staging.
Adds find_regressions to matrix_builder for the green to red auto-merge
gate, pins the cron venv to Python 3.12, and ships the systemd units, env
template, and runbook alongside
* refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods
Migrate tests/e2e/claude_code/http_probe.py off its own httpx client onto the
shared transport, and promote count_tokens and native anthropic messages to
first-class Gateway methods (Gateway.count_tokens / Gateway.messages) with typed
request/response models in the shared models.py so other suites reuse them.
The probes now take an injected Gateway and issue their request through the
shared count_tokens/messages methods, reusing the split control/data-plane
routing, timeout, and typed Result handling the rest of tests/e2e uses. The wire
shape is preserved: the pydantic bodies serialize byte-for-byte to what the old
httpx probes sent, and the anthropic-version header is carried by a small
AnthropicHeaders model. httpx is gone from the module.
* test(e2e): drop unit-level probe harness test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
OpenAI GPT-5.6 Claude Code cells burn minutes on CLI timeouts under the
full stage suite; gate them behind COMPAT_OPENAI_GPT_CELLS=1 like Mantle.
Multi-window budget e2e used max_tokens=1 which gpt-5.5 rejects mid-message
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage
Point long_context_1m at 1M-capable models, harden complexity-smart-router
registration and spend-log assertions, fix key models dropdown selectors, and
add gateway/lifecycle/transport and claude_code unit tests
* test(e2e): harden remaining stage failures in harness
Register complexity-smart-router via create_model + callable probe, fix
create-key UI navigation race, retry management writes and budget ALB
502s, mark Vertex count_tokens N/A when unsupported, and tighten
tool_search model lists for Azure/Bedrock capability gaps
* test(e2e): drop claude_code and harness unit tests from this PR
Keep management, router, budget, and shared conftest harness fixes only
* test(e2e): restore E2E_RESULT pytest_runtest_makereport hook
Accidentally dropped in an earlier harness commit; Grafana status history
depends on these structured log lines
* test(e2e): drop management control-plane write retries
Transient 500 retries do not fix the underlying control plane failures
* test(e2e): skip stage-red claude_code cells; fix multi-window budget latency
Mark the twelve failing claude_code matrix cells skip until product/config
lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of
Claude so the reset wait stays under ALB target idle timeout rather than
masking awselb 502s
* test(e2e): require exactly one LLM-tier spend row for complexity router
Keep alias membership for compose vs stage model names, but assert
len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin
LIT-4521 skip and align LIT-4522/23/24 skip reasons
* test(e2e): harden router callable probe and multi-window budget exhaustion
_router_is_callable treated any non-success chat whose body lacked "Invalid
model name" as callable, so an unpropagated probe key (401), a generic 502, or
a connection reset let the session proceed and hit real "Invalid model name"
failures inside the tests. Require a Success outcome instead; the reload-race
400 and every infra/auth error now correctly read as not-callable.
The multi-window budget test capped the tight window at 3e-6, which gpt-5.5
exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the
20-call loop, turning a reset test into a spurious "window never enforced"
failure. Drop the tight cap to 1e-9 so the first billed call exhausts it
regardless of model price; the roomy 1m window stays at 1.0 and never blocks.
* test(e2e): use a tradeoff-decision prompt for the complexity router classifier
"Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so
gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which
made the test fail even though the classifier was running. The tier definitions
key on what the request demands, not how hard the answer is, and a short direct
question maps to SIMPLE regardless of subject.
Swap in "Should I pay off my mortgage early or invest the extra money instead?".
It carries none of the heuristic scorer's reasoning/technical/code keywords and
stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads
it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the
config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier
only has to avoid SIMPLE for the test to distinguish a real classifier run from the
heuristic fallback.
The openai and azure_openai columns have working credentials in every suite runner, so only the bedrock_mantle column still needs an opt-in flag while the AWS account waits on the Mantle allowlist; COMPAT_GPT_CELLS becomes COMPAT_MANTLE_CELLS. The six tool_use cells now resolve the proxy through claude_code._env like the basic_messaging cells instead of hardcoding LITELLM_PROXY_BASE_URL/LITELLM_PROXY_API_KEY.
* feat(e2e): emit structured E2E_RESULT lines for package status history
Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore
* test(e2e): drop unit tests from the live e2e tree
tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them
* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate
Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling
* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome
We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes
* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package
Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless
* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper
pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result
* fix(e2e): import e2e_result_reporter at module load
Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run
* test(e2e): restore coverage_registry/test_collector.py
Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
Adds a 'passthrough' feature row to the Claude Code compat matrix that
drives the real claude CLI in each cloud's native mode against
LiteLLM's passthrough routes (the LLM-gateway setup from
code.claude.com/docs/en/gateway) instead of the /v1/messages
translation layer:
- anthropic: ANTHROPIC_BASE_URL={proxy}/anthropic, forwarded verbatim
to api.anthropic.com
- bedrock_invoke: CLAUDE_CODE_USE_BEDROCK=1 against {proxy}/bedrock;
the router resolves the alias in /model/{alias}/invoke-with-response-stream
- vertex_ai: CLAUDE_CODE_USE_VERTEX=1 against {proxy}/vertex_ai/v1;
alias, project, location and credentials resolve from the deployment,
which now sets use_in_pass_through: true (and the canonical
vertex_project/vertex_location param names) in test_config.yaml
- azure: CLAUDE_CODE_USE_FOUNDRY=1 against {proxy}/azure via the
AZURE_API_BASE/AZURE_API_KEY fallback (documented in the cron env
example)
- bedrock_converse: not_applicable; Claude Code has no Converse-wire
client
The shared cell body lives in _passthrough.py with injectable runner
and env (no monkeypatching), unit-covered in
_driver_unit_tests/test_passthrough.py including pins on the per-mode
CLI env contracts captured from a real claude CLI (2.1.210) run
against a request-logging sink.
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restore the upload-coverage job dropped by mistake with the compat gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments
* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
