mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
29 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
maciej-berri
|
36edf9c4b9
|
feat(messages): serve azure ai foundry messages on rust only (#51721)
* feat(messages): resolve azure_ai keys and entra tokens like python on rust
The native azure_ai Messages config now authenticates in the order Python's
BaseAzureLLM._base_validate_azure_environment uses. The key comes from the
call, then litellm.api_key, litellm.azure_key, AZURE_OPENAI_API_KEY and
AZURE_API_KEY, each skipped only when empty, so a blank key is sent as given
as Python does. Without a key the call mints an Entra ID token through
litellm-auth-azure from azure_ad_token, AZURE_AD_TOKEN, tenant_id, client_id
and client_secret, the caller's azure_ad_token_provider, OIDC references and
litellm.enable_azure_ad_token_refresh, and sends it as a bearer. Without
either the request goes out unauthenticated and Azure's 401 is mapped as
Python maps it, attributed to azure_ai instead of anthropic.
The bridge reads litellm.api_key, litellm.azure_key and the token refresh
switch from the provider defaults snapshot, and calls the caller's token
provider on the blocking pool so its exception reaches the caller as Python
raises it.
Approved differences that stay: a caller x-api-key or non-blank bearer
replaces the operator key instead of being sent beside it.
Known differences: Python's azure_username/azure_password flow has no
native counterpart, and a token provider combined with an OIDC reference
uses the OIDC token where Python calls the provider.
* fix(messages): let a caller api-key replace the azure_ai operator key on rust
A caller api-key header on an azure_ai Messages call was forwarded next to
the operator's x-api-key, so Azure received both credentials. It now
replaces the operator key and is sent as x-api-key, as Python does for a
lowercase api-key.
Approved difference: Python only matches the lowercase spelling and sends
a mixed-case Api-Key beside the operator key. Rust matches any casing.
* fix(messages): order azure_ai entra id flows exactly as python on rust
Without a key the azure_ai config now walks Python's get_azure_ad_token
order itself instead of letting litellm-auth-azure pick. Every setting is
read from the call and then the secret lookup, as AZURE_API_KEY is.
A static token, the caller's azure_ad_token_provider and client secret run
natively. Username and password, OIDC references, os.environ/ references,
an uncallable provider and token refresh go to Python's own
get_azure_ad_token through the host, as Python does them, so a static token
with refresh on and a provider with an OIDC reference behave as in Python,
and a call Python leaves without a token goes out unauthenticated.
Every azure_ai call now asks the secret lookup for the Entra ID names up
front, where Python asks only when no key is set.
* fix(messages): read azure_ai entra settings where python does on rust
Python's get_azure_ad_token reads AZURE_AD_TOKEN through get_secret_str and
AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_USERNAME,
AZURE_PASSWORD and AZURE_SCOPE with os.getenv
(litellm/llms/azure/common_utils.py:580-588). The azure_ai config now does
the same, reading the process environment only when no key is set, and only
AZURE_AD_TOKEN stays in the secret names, so a value only a secret manager
holds is ignored where Python ignores it.
Approved by the owner, same class as A06-A08: any caller credential header
(x-api-key, api-key or Authorization other than a blank bearer, in any
casing) is the only credential sent, and the Entra ID path never runs for
it. That covers a caller bearer on an Entra-only deployment (A10), which
Python overwrites with the deployment's token, a lowercase authorization
Python joins with the Entra token, and a caller x-api-key Python sends
beside the Entra token.
* fix(messages): look up AZURE_AD_TOKEN only where python does on rust
Python reads AZURE_AD_TOKEN through get_secret_str only when the azure_ai
call has no key, no caller credential and no azure_ad_token param
(litellm/llms/azure/common_utils.py:580). Messages configs can now name
deferred secrets, resolved right before validate_environment and only when
the config names them for that call, so a keyed call or a call carrying its
own credential never asks the secret lookup for the Entra token.
Approved by the owner, same class as A06-A08: a caller Authorization other
than a bearer, such as Basic, replaces the azure_ai operator key instead of
being sent beside it as Python does.
* fix(messages): build azure_ai urls and resolve the base like python on rust
The native azure_ai Messages URL builder now ports Python's
azure_anthropic_messages_url. It cuts at /anthropic in the path only, so a
base whose userinfo starts with "anthropic" no longer sends the key to
http://anthropic/v1/messages (security regression fix). It keeps the base
untrimmed and moves the slash before a query as Python does.
The base resolves as Python's `api_base or AZURE_API_BASE`: a blank call base
is kept, an empty one falls back to the environment, and the environment
value is used as given. With neither, the call raises Python's bare
ValueError ("Missing Azure API Base ..."), flagged as raised by Python and as
a late rejection, so the failure log carries the resolved provider and model
parameters as Python's does.
* fix(messages): send bases the native parser misreads through python's client on rust
A Messages api_base with credentials, no host or a port Python cannot read
now goes through Python's client, as bases without an http(s) scheme or with
whitespace already did. Before, reqwest read such a URL differently from
httpx: for http:///d it sent the request, with the operator key, to host
`d`. This is a security fix for both azure_ai and anthropic; the
anthropic_base_empty_host parity case covers anthropic.
For azure_ai the choice reads the base Python's URL builder reads, falling
back to AZURE_API_BASE, so a padded environment base also goes through
Python's client. Python reads AZURE_API_BASE once: transport_handler returns
the value it read and the bridge seeds the call's secret source with it, so
the native route reuses it rather than reading the secret again. _sent_url
only rewrites anthropic URLs, since the azure_ai builder no longer trims.
Parity: 20 azure_ai and anthropic URL cases added; 65 aspects failed on the
previous extension, and the full messages parity suite now passes.
* fix(messages): look up AZURE_API_BASE only where python does on rust
Python reads AZURE_API_BASE only through the `or` fallback in
`api_base or get_secret_str("AZURE_API_BASE")`. The native azure_ai config
listed it among the secrets it always reads, so a call with its own base
still looked it up through the secret manager. It is now a deferred secret,
read only when the call's base is empty or absent. A blank base is kept
without a lookup, as in Python.
deferred_secret_names now takes the call's api_base, the same base the URL
builder reads. A lookup-counting test covers a call base (0 lookups), an
empty base (1) and no base (1); before this change the call-base case
looked it up once.
* fix(messages): filter azure_ai anthropic-beta through its row like python on rust
Python's handler runs update_headers_with_filtered_beta over the azure_ai row of
anthropic_beta_headers_config.json after the config adds its feature betas, so
unknown and null-mapped betas are dropped whatever the header casing, the
caller's and the auto-added ones alike. Rust forwarded the caller's betas and
every auto-added beta as sent.
A caller anthropic-beta replaces a deployment's extra_headers anthropic-beta
before either lane runs, as Python does, and a Router parity test pins it with
the filter applied to both. The advisor tool already matched, since the bridge
hands it to Python's interceptor and its beta is null in the row, so a parity
case pins it. extra_body, unknown top-level params and the -latest alias also
matched and get parity cases.
* fix(messages): strip beta-gated system fields under drop_params like python on rust
Python's transform runs strip_beta_gated_native_request with the final,
filtered headers, so under drop_params a mid-conversation system turn loses
the clear_at, output_config and tool_addition/tool_removal fields no sent beta
unlocks, and a turn left empty is dropped unless that would end the
conversation on an assistant turn. _normalize_system_role_messages strips a
turn it converts to a user turn again with no betas at all. Rust sent those
fields as they came.
azure_ai now picks its betas from the request as sent and runs the strip,
system-role normalization and the cache_control.scope strip in a new
finalize_request step, once the final header is known. Vertex gets the
conversion-time strip too, since it shares the normalization helper.
The strips and the speed drop log Python's warnings word for word. A
non-object output_format under drop_params is still popped after the headers
were built, so structured-outputs-2025-11-13 is sent without it, as Python
does.
* fix(messages): send the system prompt as text blocks like python on rust
Python's _normalize_system_role_messages always rebuilds the top-level system
as a list of content blocks, so a string system reaches azure_ai and Vertex as
one text block, and the success log, which only lists a string system as a
message, logs the messages as sent. Rust kept the string unless a leading
system turn was hoisted.
Python's billing strip in transform_anthropic_messages_request also pops a
system prompt it leaves empty, an empty string included, as Python does. The
shared helper now does the same for every config that strips billing
metadata: azure_ai, vertex_ai, deepseek, minimax, tencent and edenai, each
with its own test.
* fix(messages): keep non-claude azure_ai messages passthroughs on python
A deployment whose provider has no Messages config of its own, like a
non-Claude azure_ai model, but lists /v1/messages in
model_info.supported_endpoints is forwarded untranslated by Python's
OpenAILikeAnthropicMessagesConfig, with a bearer key to <base>/v1/messages.
Dispatch gave it the MESSAGES target, so a rule that serves azure_ai on Rust
would hand it to the native Anthropic config, which sends x-api-key to
/anthropic/v1/messages and breaks the success log.
Dispatch now targets MESSAGES only when the provider's own config serves
the call, and MESSAGES_PASSTHROUGH for the opt-in passthrough. The catalog
keeps that target on Python next to the chat and Responses bridges, so once
azure_ai is Rust-required these calls resolve to Python and the Rust-only
refusal does not fire for them. Proxy token counting treats both targets as
before.
The parity forced rule now serves a provider natively only for the MESSAGES
target, as a Rust-required rule below the target exclusions does. A pre-request
hook that sets a non-Claude model raises functools.partial's TypeError on both
lanes, as on main, and a parity case pins it.
* fix(messages): raise azure_ai transport and provider errors like python on rust
The bridge mapped a native failure with the whole call as exception_type's
extra_kwargs, so litellm_debug_info gained an API Base line from the named
api_base. Python's entry passes only its **kwargs there, so its debug info
names the model alone. The bridge now passes the same kwargs, for anthropic
and azure_ai alike, and a decoding failure on a stream carries the
deployment's debug info as Python's stream iterator attaches it.
A connect failure to an azure_ai base read from AZURE_API_BASE kept the
native "could not reach the provider" text, since the call had no api_base.
The host now hands Python's failure mapping the secrets it read to send the
call, so the aiohttp and httpx texts name the base the call went to, without
reading AZURE_API_BASE again.
A key or header value with a control character, like a deployment key with
a trailing newline, failed natively as APIConnectionError with the model
stripped of its provider. It is now the InternalServerError 500 Python's
transport raises, with aiohttp's "Forbidden control character detected in
headers" text and the model as called. On the httpx transport Python's text
quotes the header value, which may be the key, so Rust keeps its own text
there.
The parity outcome now compares litellm_debug_info. The caller client's
ReadTimeout from a stalled stream still has none on Rust and is recorded as
a gap.
* fix(messages): send a caller azure_ai api-key without an operator key like python on rust
With no deployment key, Rust renamed a caller api-key header in any casing
to x-api-key. Python's AzureAnthropicMessagesConfig renames only a header
spelled exactly api-key and sends any other casing, like API-Key, as given.
Rust now does the same when no operator key would be sent. With an operator
key the approved behaviour stays, and a caller api-key in any casing replaces
it as x-api-key.
* fix(auth-azure): retry entra token posts like python's azure-identity
A client secret or workload identity token endpoint that keeps answering
500 made the native path POST 9 times over about 80 s, because
azure_identity 1.0 uses the SDK's default exponential retry (8 retries,
200 ms doubling, 60 s cap). Python's azure-identity 1.25.2 posts 4 times
in about 5 s and then raises, so the Rust credential now carries a custom
retry policy with the same numbers.
Python sources in the wt-auth venv
- azure/identity/_internal/pipeline.py:67 builds the token pipeline with
azure-core's RetryPolicy(**kwargs) and no overrides
- azure/identity/_internal/msal_client.py:98 (client secret) and
aad_client.py:77 (client assertion, used by workload identity) run the
token POST with retry_on_methods=["POST"]
- azure/core/pipeline/policies/_retry.py (azure-core 1.41.0) lines 77-95
give retry_status=3, retry_backoff_factor=0.8, retry_backoff_max=120,
retried codes 408, 429, 500, 502, 503, 504 and 506-998. Lines 127-143
sleep 0 before the first retry and factor * 2^(n-1) after that, so
0 s, 1.6 s, 3.2 s. Lines 455-509 sleep exactly Retry-After (then
retry-after-ms, x-ms-retry-after-ms) instead of the backoff when it is
present and non-zero
azure_core only hands the policy a Retry-After value for 429 and 503, so
a Retry-After on another status still falls back to the backoff. Managed
identity keeps the SDK default since Python uses a separate policy there,
and client assertion keeps it because Python mints OIDC tokens through
litellm's own HTTP client. A 401 invalid_client is still one POST.
* fix(auth-azure): post an oidc token exchange once like python
Python exchanges an OIDC assertion through litellm's own HTTPHandler
(_exchange_oidc_token in litellm/llms/azure/common_utils.py), a single
client.post with no retry, so a 503 is one POST. The native client
assertion credential used azure_identity's default retry (9 POSTs over
about 80 s) and now runs with RetryOptions::none().
* fix(auth-azure): honour retry-after on entra token posts like python
azure_core runs a custom RetryPolicy through its own loop, which only
reads Retry-After for 429 and 503 and never retries a status outside the
policy's list. Python's azure-core RetryPolicy (1.41.0,
azure/core/pipeline/policies/_retry.py) retries any 4xx or 5xx that
carries Retry-After (is_retry, lines 210-242) and waits exactly the
Retry-After, retry-after-ms or x-ms-retry-after-ms value when it is
non-zero (get_retry_after in _utils.py:79-96, sleep at lines 455-509).
The client secret and workload identity credentials now run with the
SDK retry turned off and a per-call policy that follows Python's
loop. It keeps separate budgets of 3 for status, connect and read
retries (increment and is_exhausted, lines 265-330), backs off 0 s,
then 0.8 * 2^(n-1) capped at 120 s, and retries 408, 429, 500, 502,
503, 504 and 506-998 for the token POST. Client assertion still posts
once and managed identity keeps the SDK default.
A Retry-After that is neither a number nor an HTTP date now fails the
token request after the first POST with "Invalid HTTP date", as Python
does. azure-core's _parse_http_date (_utils.py:48-57) raises ValueError
from the sleep before the retry, azure-identity's wrap_exceptions
(decorators.py:63-80) turns it into ClientAuthenticationError
"Authentication failed: Invalid HTTP date", and get_azure_ad_token
(common_utils.py:663) raises RuntimeError "Failed to get Azure AD token:
Authentication failed: Invalid HTTP date".
* fix(messages): raise python's entra token errors for native azure_ai flows
On the Messages route every Entra token failure now reaches the caller as
Python's get_azure_ad_token and anthropic_messages raise it, with the same
class, status_code, llm_provider, message and log lines.
A native client secret flow keeps the token endpoint's final answer
(status, headers, body) in a typed EntraTokenFailure from the azure-identity
retry policy. The bridge hands it to the Messages host, which replays it
through azure-identity's own ClientSecretCredential over a transport that
returns that answer, so MSAL and azure-core read it exactly as they read it
off the wire. That covers a rejected secret (EntraRejectionError 401 with
the first AADSTS sentence and GUIDs redacted), an exhausted 5xx ("HTTP
Error: 500"), a malformed Retry-After ("Invalid HTTP date"), non-JSON and
odd JSON bodies, including the content-type dependent decode error. A
transport failure has no answer to replay and keeps Rust's own reason
after "Authentication failed:", so its text differs from requests' text.
The caller's azure_ad_token_provider goes through the helper
get_azure_ad_token now shares (azure_ad_token_failure), so a listed AADSTS
rejection is an AuthenticationError 401 instead of a RuntimeError, and a
non-string token raises the same TypeError. Both log Python's "Error
calling Azure AD token provider" and "returned non-string value" lines.
Every Python exception raised while getting the call's credential,
including the host-minted username/password, OIDC and workload identity
flows, now goes through Python's handler rule: a BaseLLMException is mapped
by exception_type (AuthenticationError 401, BadRequestError 422 with
llm_provider azure_ai), anything else reaches the caller as raised.
The Retry-After date parser stays RFC 7231 only. Python's
email.utils.parsedate_tz also accepts dates without a weekday, two-digit
years, numeric zones and missing seconds, which Rust rejects as an invalid
HTTP date and Python waits on. A NaN or infinite number of seconds now
falls back to the backoff, where Python sleeps 0 for NaN and fails on
infinity.
* fix(auth-azure): wait on retry-after and report connect failures like python
The Entra token retry policy now reads Retry-After the way azure-core
1.41.0 does. Seconds go through Python's float rules, including
underscores between digits, and anything else through a port of
email.utils.parsedate_tz, so a date without a weekday, a two-digit year,
a numeric or named zone, a zone glued to the time, missing seconds, a
dotted time, month before day and RFC 850 dates all wait as Python waits.
A date datetime() rejects (day out of range, an offset of a day or more)
still fails the request after one POST, as do unparseable values.
A wait that time.sleep cannot take (infinity, or 2^63 ns and past) now
fails after one POST as Python does. The host replays that answer with a
sleep that still calls time.sleep for such a wait, so the caller gets
"Authentication failed: timestamp out of range for platform time_t".
A transport failure keeps the token endpoint's host and port. A direct
connect that fails with an OS error, or a failed name lookup glibc words,
reaches Python as urllib3 2 words it after "Authentication failed:", for
example "HTTPSConnection(host='login.microsoftonline.com', port=443):
Failed to establish a new connection: [Errno 111] Connection refused".
Python hits it on MSAL's discovery GET to the same host. Failures through
a proxy, TLS failures, timeouts and resets while reading keep Rust's own
text, since urllib3's wording for them depends on details Rust does not
expose.
* fix(messages): treat a contentless system message as a system turn like python on rust
Python's _normalize_system_role_messages reads only the role, so a role
system message sent without content or with null content is still a system
turn. A leading one is hoisted and adds nothing to the top-level system, and
a later one on a model without mid-conversation system support becomes the
operator note user turn, placed after a tool result it sits before and
dropped under drop_params when its gated field is stripped. Rust failed to
parse the message and sent it raw, which also showed up in the logged
messages. A model that accepts the role still gets it as sent.
* fix(messages): let the provider filter a non-ascii anthropic-beta like python on rust
Python merges every anthropic-beta spelling into one header and the
provider config filters it before the HTTP client encodes it, so on azure_ai
an unknown non-ASCII beta is dropped and the call goes out, with or without
drop_params. Rust held the header back as unencodable before the beta filter
ran and failed the call with the ascii codec error. The beta header now
reaches the provider filter, and the wire check fails the call only when a
value Python's client cannot encode survives it. A value with a lone
surrogate still fails up front since it cannot reach Rust as text.
* fix(messages): log nothing for a stream the caller never read like python on rust
Python's Messages stream iterator only starts on the first read, so a
stream the caller closes or drops unread bills nothing and logs nothing.
Rust billed the empty stream on close and teardown, which logged "Unable to
build complete streaming response" and the read() error from the success
handler.
* fix(messages): log a traceback for failures python raises before the send on rust
Python raises its pre-flight errors, such as the missing azure_ai api_base
ValueError, the missing key AuthenticationError and the Entra token
provider failures, through Python frames, so the failure log and the OTel
span carry a stack trace. Rust built those exceptions without raising them,
so error_information.traceback was empty. The host now raises them through
a Python frame before handing them back.
* fix(messages): return python's streaming response types from native messages streams
Python's Messages handler returns an AnthropicMessagesStreamingResponse for
a stream, so isinstance checks, hidden_params and aclose behave the same for
every caller. The native async entrypoint returned the bare execution
AsyncStream. It now wraps the stream the way the deployment success hook
already did, sharing the stream's hidden params. The entrypoint is shared, so
this moves anthropic as well as azure_ai toward Python.
With a response cache configured Python hands back its caching handler's
types too. A streamed cache hit is now the CachedAnthropicMessagesStreamIterator
Python's handler builds from the cached events, stamped with the metadata
@client stamps, and the native execution closes unread so only that iterator
logs the hit, as Python does. A miss is wrapped in the
AnthropicMessagesStreamCacheWriter Python returns, marked persisted since Rust
stores the stream itself. The parity suite now compares the stream type for
every streamed case.
* fix(messages): read unparsed messages in beta detection and gated strips like python on rust
Python reads beta triggers and gated fields off any message mapping, while
Rust only looked at messages that parsed. A role system message sent without
content, or any message with a content block Rust cannot type, was skipped:
- a message carrying output_config now adds per-turn-control-2026-07-01
- a system message with a tool_reference addition or removal now adds
mid-conversation-tool-changes-2026-07-01
- a signed compaction block now adds compact-2026-09-04
- under drop_params an unparsed system message loses its gated clear_at,
output_config and tool changes, is dropped once emptied, and the
warning names the strip
- an unparsed assistant message counts as the last assistant turn when
deciding whether a strip would leave a prefill
- an unparsed message loses cache_control.scope on its content blocks
* fix(messages): rewrite and normalize unparsed messages and system prompts like python on rust
Python's sanitizer, advisor and encrypted-reasoning strips and system
normalization read any message mapping, while Rust skipped every message
or system block it could not type:
- an unparsed message now gets the sanitizer's rules: a text block whose
text is not a non-blank string and an empty unsigned thinking block go,
a message left with no blocks is dropped, tool ids are sanitized,
replayed web search results are flattened and provider_specific_fields
dropped; a message the rewrite leaves parseable is parsed again
- an unparsed assistant turn loses advisor exchanges, and an unparsed
message loses bridge-tagged reasoning blocks
- a kept redacted_thinking block gets Claude's own data back from a
Bedrock Converse base64 wrapper, parsed or not
- system is kept as sent when it is a list holding a block that does not
parse, or an object; billing blocks and cache_control.scope are still
stripped from such a list
- a leading system turn whose content does not parse is hoisted as sent,
and a later one becomes the operator note user turn as sent, placed
after a tool result turn that did not parse either
* fix(messages): keep anthropic's API Base debug line on rust failures
Mapping a native failure with only the Python entry's **kwargs dropped the
named api_base from exception_type's input. That matches azure_ai, whose
Python entry maps with **kwargs alone, but it removed the API Base line that
main's Rust-only anthropic route has always put in litellm_debug_info when the
caller passes api_base.
Python gives no rule that tells the two apart, because the legacy async entry
maps both providers the same way. The difference is that anthropic has no
Python path, so main's native route is its contract. The bridge now maps the
whole call for anthropic and the entry kwargs for every other provider.
The parity harness still uses the legacy Python entry as anthropic's
reference, so it now expects the API Base line main's anthropic route adds.
New cases cover an anthropic call on an azure_ai model failing with api_base,
unary and stream.
* fix(messages): keep anthropic stream types and header errors as main while azure_ai matches python
By the owner's ruling anthropic keeps main's behaviour for two things the azure_ai port had changed for every
native provider.
A native async stream returns Python's AnthropicMessagesStreamingResponse, and with a response cache the
CachedAnthropicMessagesStreamIterator or AnthropicMessagesStreamCacheWriter, only for azure_ai. The host asks
Python for those types only when the call resolves to azure_ai, so anthropic gets main's execution AsyncStream
again, for a cache hit or miss too.
A header value the native client cannot send, like a key with a control character, is Python's
InternalServerError with aiohttp's "Forbidden control character" text only for azure_ai. Anthropic raises
main's APIConnectionError "invalid request: failed to parse header value" with the bug report link again.
The parity stream_type aspect compares azure_ai with Python and expects main's AsyncStream for anthropic,
in the same way the outcome aspect already expects main's API Base debug line. New azure_ai cache hit and
miss stream cases check the cached types.
* fix(messages): keep anthropic as main for azure_ai parity side effects
By the owner's ruling anthropic, which has no Python path, keeps main's native
route; the azure_ai port's Python-parity changes now apply to azure_ai only.
- An unread stream: only azure_ai's starts on the first read and bills
nothing when closed or dropped unread. The logging adapter learns the
provider from the execution facts; anthropic bills it as main does.
- Failures before the send: the missing-key AuthenticationError and the
unmapped builtin errors are raised through a Python frame only for azure_ai;
anthropic hands them back unraised, without a traceback, as main does.
- A stream's decoding failure carries the deployment's debug info only for
azure_ai; anthropic raises it bare.
- A non-ASCII anthropic-beta is left to the provider filter only for
azure_ai. Anthropic holds it back before the beta merge as main does, so
the error names it ahead of any later unencodable header.
- Unparsed messages: the shared shaping reads messages and system prompts
Rust could not type only when the provider asks for it
(UnparsedMessages::Read), which only the azure_ai config does. Elsewhere
the sanitizer, advisor and encrypted-reasoning strips leave such messages
as sent, their beta triggers add no beta, a redacted_thinking Converse
wrapper is kept as sent, and a system prompt that does not parse is
main's "untagged enum SystemPrompt" request error from messages_body.
Each item has an anthropic test pinned to main's behaviour that fails on the
previous commit, beside the azure_ai cases comparing with Python.
* test(messages): pin the malformed api_base fix for anthropic and azure_ai
|
||
|
Yassin Kortam
|
db6dfb6bb6
|
chore: move workspace packages under litellm (#51000)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
59538ff1d9
|
feat(messages): serve anthropic messages on rust only (#51119)
* feat(messages): apply modify_params rewrites on rust * feat(messages): mint anthropic workload identity tokens on rust * feat(messages): apply prompt caching injection on rust * feat(messages): serve the easy python-only detectors on rust * feat(messages): retry rejected thinking signatures on rust * feat(messages): serve response cache reads on rust * feat(messages): retry rejected strict tools on rust * refactor(messages): delete the python path for anthropic messages Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a detector that raises) gets UnsupportedParamsError naming the feature, raised from one place in litellm/messages/dispatch.py. Other providers keep the Python fallback. A call whose arguments the handler signature rejects still gets Python's TypeError. Deleted Python code that only served anthropic Messages: - workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses were never eligible; the Rust host mints the token itself) - the anthropic Foundry beta-row and affinity-base branch in async_anthropic_messages_handler, and the "forward" mode of the beta filter that only that branch used The proxy MCP tools detector now uses the light tool precheck so a function tool never imports the gateway handler, since a raising detector now refuses. * feat(messages): forward message threads on rust The Rust path for anthropic /v1/messages dropped the thread body field, so a create or continue turn reached Anthropic without its thread or the message-threads-2026-08-12 beta. It now matches the Python handler from #50976: - the host forwards thread with the other body params - a non-null thread adds the message-threads beta - a well-formed continue turn keeps advisor blocks in history and derives no compaction, per-turn-control or tool-change beta from its messages - the recorded httpx_response carries the request that was sent, for JSON and streamed calls, so the proxy's thread ownership hook resends the create turn's betas on a continue as it does for Python Upstream error.details already reached the exception message unchanged, so thread_not_found relays and keeps the deployment out of cooldown on both paths. Parity cases cover create, continue, streaming, sync, retries, thread_not_found and the ownership hook round trip. * feat(messages): serve mock responses on rust * feat(messages): send caller clients and unsupported tls through python on rust * feat(messages): run pre-request hooks on rust * feat(messages): run agentic loop hooks on rust * feat(messages): run the safeguards classifier on rust * feat(messages): run registered interceptors on rust * fix(messages): match python stream cache gate, logging and interceptor order on rust Store a streamed Messages response in the cache only when Python would: a message_stop event line, no provider error frame and at least one frame with output. Leave dict results untouched by mark_rust_response so hidden params never reach result dicts, logging objects or hook digests. Ask interceptors whether they handle the call after the cache lookup, not at decode time, so a cache hit never calls can_handle. Emit the track_llm_api_timing service event for the provider exchange. Bill and log a stream superseded by a router fallback as the failure of its error frame, with the partial usage warning. Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the autorouter baseline cache sees retried requests. * fix(messages): forward python-only json values to anthropic as python does Accept any max_tokens, sampling scalar and messages collection that Python main accepts, and carry values serde_json cannot hold (ints beyond 64 bits, NaN and infinities, strings with lone surrogates) as validated raw JSON leaves written back verbatim on the wire, so the upstream body matches Python's json.dumps * fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust * fix(messages): log python's plain provider header names on rust streams * fix(messages): match python's request params, logged params and validation on rust Read a non-bool stream by Python truthiness and send it as given, log the caller's stream flag as Python's handler does, raise the raw Python errors main raises for malformed messages, tools, model_info and hook-set litellm_params or providers, read the tool custom-field drop from drops_unsupported_params, log the caller's nested fields that additional_drop_params strips and the Python values of raw JSON leaves, keep a missing stop_reason absent, and map provider-config rejections (missing max_tokens, unknown reasoning effort) as Python's HTTP handler maps them, with Python's text, effort order, model and provider * fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust * fix(messages): match python on hooked streams, empty cache writes and mcp cache hits * fix(messages): join already buffered stream reads and trim the env api key as python does on rust * fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust A stream cut by a transport error now hands the caller the bytes already buffered before raising, as aiohttp does for a caller that reads them before the event loop sees the cut, so the frames and their usage reach the caller and the failure log. Provider replies keep integers wider than 64 bits exactly and read numbers that overflow a float as infinity, through the response, the response cache and the logged result. The provider client follows up to 20 redirects as httpx does, and running out raises Python's text. A provider_specific_header value the ascii codec cannot encode fails before the send like a caller extra header. The deployment success hook gets the stream as AnthropicMessagesStreamingResponse, as Python's handler hands it. * feat(messages): serve proxy mcp tools on rust * test(messages): cover hook, agentic and interceptor edge cases in the parity suite * fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust * fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust A header value Python's client cannot encode as ASCII now fails before the send whichever header carries it, so a caller api_key with a non-ASCII character raises Python's InternalServerError instead of a raw UnicodeEncodeError after the reply. A caller api_base without an http(s) scheme, or with whitespace, control or non-ASCII characters, is sent through Python's client at the URL Python builds, so its relative-URL and invalid-URL errors match, and a base urlsplit cannot split raises Python's bare ValueError before the send. A provider_specific_header of the wrong shape is scoped through Python's ProviderSpecificHeaderUtils first, so a plain string raises Python's AttributeError. * fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust temperature and top_p now deserialize through recognized::exact_float, which keeps an integer where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent upstream and logged as the integer python forwards, and comparisons still go through scalar.rs cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python TypeError marked as a late rejection, and map_failure hands it to the caller unmapped * test(messages): format the non-string tool_result parity helper * fix(messages): flag and log the call before python's pre-send raises on rust The bridge now hands settings() and map_failure() whether the call is async. A provider_specific_header of the wrong shape sets the aanthropic_messages flag before re-raising, as Python's handler has already flagged the call. A base urlsplit cannot split also logs the resolved model, provider and optional params first, as Python raises it only after its logger is updated. The TypeError raised while capping the thinking budget for a string max_tokens sets the flag too * feat(messages): serve the web search short-circuit override on rust through the intercept host call * fix(messages): give the web search entry kwargs its mutable-ok reason * test(messages): drop the body hook gap the rust short-circuit closes * fix(messages): name the partially unknown handler types on the short-circuit host call |
||
|
Nate Armstrong
|
34314a99f0
|
fix(messages): make rust messages copy python messages behaviour (#49611)
Closes the Rust vs Python /v1/messages differences the parity suite marks as known gaps, copying Python exactly. |
||
|
Nate Armstrong
|
5e08cb0e63
|
test(messages): add rust vs python messages parity characterization suite (#49610)
Runs every case through the Python /v1/messages path and the Rust path against one recording upstream and compares dispatch, upstream request, outcome, hidden params, standard logging fields and callback kwargs. Known Rust differences are strict xfails tagged with their inventory ID. |
||
|
Nate Armstrong
|
bcd80b6aa1
|
feat(rust): encode LiteLLM response ids on native responses (#46572)
Native Responses now hands back the same managed ids as Python: the resp_ id carries the provider and deployment, encrypted-content items get encitem_ ids and wrapped content under affinity routing, and container ids become cntr_ ids. Encoded encrypted-content items in the input are restored before the provider request. The codec lives in inference-responses with goldens taken from litellm/responses/utils.py |
||
|
Nate Armstrong
|
4995d319ed
|
test(rust): add native vs Python parity harness for responses (#46567)
Each case runs one aresponses call through Python and through the native route against the same recorded OpenAI response and compares the dispatch, provider request, return value and success log. Known gaps are strict xfails so the PRs that close them must flip the marker |
||
|
Mateo Wang
|
3069a467d0
|
docs(agents): ban semicolons and colons in human-facing text (#45684)
* docs(agents): ban semicolons and colons in human-facing text Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "." * docs(agents): rewrite semicolons in nested AGENTS.md files * docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines * docs(agents): point at the PR template path that exists * docs(agents): restore the generated Next.js block in the dashboard AGENTS.md |
||
|
devin-ai-integration[bot]
|
d5173e3d70
|
test: move 76 live top-level tests to offline unit and integration coverage (#45352)
* test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
e2302be068
|
refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081)
* refactor(ocr): remove the Python OCR execution path and require the Rust route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fmt * refactor(ocr): tidy the native OCR passthrough binding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ocr): ruff format the azure passthrough transformation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ocr): resolve passthrough OCR costing in one Rust call Replace passthrough_url/passthrough_transform with passthrough_response, which matches the relayed endpoint against each Azure config's path segments instead of building a fake request to call get_complete_url. The binding drops the unused headers, status and api_base arguments. Catch the ValueError/RuntimeError the binding raises so a relayed body that is not OCR-shaped falls back to the passthrough object instead of failing logging, and cover the relay against the real binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ocr): drop the unused LlmProviders import from health check helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: drop the ocr_testing job now that tests/ocr_tests is gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): restore the live OCR matrix and the ocr_testing job The public litellm.ocr / aocr / Router interface is unchanged by the Rust migration, so the live provider matrix still applies. Drops the stale VCR skip list for the deleted test_rust_bridge.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ocr): import Final in the health check helper tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
kerry
|
8ecbf3dbc1 |
test: drop tests that pin provider-owned cost map values
The repo rule is that a test must only fail when litellm code changes, never when a vendor updates a price, renames a field, or drops a model. These tests asserted shipped catalog entries directly, comparing lookup results to literals copied from model_prices_and_context_window.json or requiring named entries to exist or be absent, so every cost map sync could break them without any litellm code changing Tests that exercise real litellm behavior with an injected local model_cost, invariants like backup parity, and assertions on non-lookup code paths are untouched Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yujong Lee
|
96baeb8b04 |
refactor(rust): remove gateway, config, router, realtime, and trace-parity infrastructure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yujong Lee
|
da839d4a11 | fix(harness): skip unavailable Rust traces | ||
|
Yujong Lee
|
27981c7d20 | fix: preserve rust setting in trace parity | ||
|
Yujong Lee
|
db378d8963 | fix(harness): preserve unmapped calls in execution traces | ||
|
Yujong Lee
|
209fc7afb0 | fix(harness): trace OCR Python and native dispatch paths | ||
|
Yujong Lee
|
d4869ba710 | refactor: print rust and python traces independently | ||
|
Yujong Lee
|
561533c596 | wip | ||
|
yujonglee
|
347b642bdd
|
refactor(ocr): complete native lifecycle and preserve Azure auth (#40734)
* refactor(ocr): extract call completion boundary * fix(ocr): release completion state after dispatch * test(ocr): prove wrapper completion handoff * test(ocr): narrow mapped failure assertion * fix(ocr): preserve wrapper invocation kwargs * fix(ocr): retain completion through finalization * fix(ocr): make completion ownership explicit * refactor(ocr): resolve logging executor explicitly * fix(callbacks): preserve completion lifecycle behavior * refactor(ocr): move public OCR into native lifecycle * refactor(ocr): remove unused rust bridge capability * wip * wip * refactor * wip * fix(ocr): preserve reducto native compatibility * wip * fix(ocr): document native callable casts * perf(ocr): bound responses and reduce native scheduling overhead * refactor(python-bridge): organize placeholder routes * refactor test * fix(ocr): normalize DeepSeek document content * perf(ocr): skip unused callback work and benchmark callback overhead * fix(ocr): align conversion contracts * test(ocr): cover official provider response shapes * fix(ocr): restore Python fallback and honor Rust opt-out * fixes and refactor * fix(ocr): preserve Azure Document Intelligence authentication * fix(rust): enforce OCR response limits and lint contracts * test(rust): align native OCR contract coverage * test(ocr): isolate Azure auth precedence coverage |
||
|
yujonglee
|
5f2b4d27d7
|
test(ocr): add SDK callback E2E parity (#40061)
* test(ocr): cover SDK callback parity * test(ocr): compare callback kwargs |
||
|
yujonglee
|
8da735410c
|
test(ocr): trace callback lifecycle parity (#40063) | ||
|
Yujong Lee
|
6675e1fd9c | fix: preserve Python 3.10 harness compatibility | ||
|
yujonglee
|
b75ac5cf52
|
feat(python): rename Rust rollout API (#39704) | ||
|
yujonglee
|
eb67e5402b
|
test(ocr): complete Rust unit test parity (#39689)
* feat(ocr): complete Rust unit test parity * test(ocr): keep parity changes harness-only * test(ocr): share azure DI native fixture across response tests * refactor(ocr): colocate gateway unit tests and extract lifecycle integration tests |
||
|
yujonglee
|
ee08c36fc0
|
refactor(tests): restructure rust python harness around strategy definitions (#39628)
* wip * refactor(tests): move sdk function tracing into rust python harness * dead code * fix: handle harness keyboard interrupts * refactor(tests): deduplicate rust python harness helpers * fix(harness): expose validated strategy choices * wip * refactor(harness): let strategies own parity reports * docs(harness): update strategy structure * refactor(harness): localize strategy report views * wip * fix(harness): satisfy mapping runner type checks * fix(harness): clarify trace parity output * wip * fix(harness): clarify unit mapping report * fix(harness): finalize trace parity contracts * refactor(harness): structure parity contracts * feat: derive unit test mapping from traces * feat(harness): map rstest test families * feat(ocr): port Azure document intelligence tests * feat(harness): enforce complete unit mappings * feat(ocr): add reducto core transforms * feat(harness): classify host-only unit tests * fix(ocr): complete Rust provider plumbing * fix(harness): reuse OCR parity workers |
||
|
Yujong Lee
|
8c180afd4b | test(ocr): record provider fixtures in the migration harness | ||
|
yujonglee
|
2c30fe16b0
|
Merge pull request #38765 from BerriAI/litellm_ocr_sdk_parity_tests
test(harness): add OCR parity with migration strategy runners |
||
|
ishaan-berri
|
e058aa68c4
|
test: add mistral ocr transformation parity coverage (#39482)
* test: cover mistral ocr transformation parity Co-Authored-By: Claude Code <noreply@anthropic.com> * test: map mistral ocr parity contracts Co-Authored-By: Claude Code <noreply@anthropic.com> --------- Co-authored-by: Claude Code <noreply@anthropic.com> |
||
|
ishaan-berri
|
a5639b8e2a
|
test: add OCR python-to-rust test parity ledger (WIP) (#39434)
* test: add OCR python-to-rust test parity ledger * feat: add ledger loader for OCR test parity data * feat: add drift audit for OCR test parity ledger (WIP, untested) * fix: correct drift in OCR test-parity ledger Two entries referenced a typo'd Python test name, four duplicated entries already tracked under TestProxySecurityGuard, five real Python tests in test_rust_bridge.py were untracked, and three real Rust custom_logger tests were missing from rust_only_tests. Found by running validate_ledger.py's audit against the live repo. * test: add regression coverage for the OCR ledger and audit script Covers schema validation, AST/regex test enumeration, drift detection on both the Python and Rust sides, LedgerDriftError content, and a live-repo clean-audit guard against future drift. * test: simplify ledger test to one drift-guard assertion Replace the ledger-internals unit tests with a single test that runs the real audit against the live repo and asserts every OCR test is accounted for (mapped, unmapped-with-reason, or rust_only), printing the exact diff on failure. * chore: move OCR test-parity ledger to core/ocr validate_sub_methods/ mixes strategy-catalog metadata with the ledger. Ledger data belongs under a per-function core/<function>/ path instead. * fix: point LEDGER_PATH at the new core/ocr location |