83 commits

Author	SHA1	Message	Date
maciej-berri	36edf9c4b9	feat(messages): serve azure ai foundry messages on rust only (#51721) ... * feat(messages): resolve azure_ai keys and entra tokens like python on rust The native azure_ai Messages config now authenticates in the order Python's BaseAzureLLM._base_validate_azure_environment uses. The key comes from the call, then litellm.api_key, litellm.azure_key, AZURE_OPENAI_API_KEY and AZURE_API_KEY, each skipped only when empty, so a blank key is sent as given as Python does. Without a key the call mints an Entra ID token through litellm-auth-azure from azure_ad_token, AZURE_AD_TOKEN, tenant_id, client_id and client_secret, the caller's azure_ad_token_provider, OIDC references and litellm.enable_azure_ad_token_refresh, and sends it as a bearer. Without either the request goes out unauthenticated and Azure's 401 is mapped as Python maps it, attributed to azure_ai instead of anthropic. The bridge reads litellm.api_key, litellm.azure_key and the token refresh switch from the provider defaults snapshot, and calls the caller's token provider on the blocking pool so its exception reaches the caller as Python raises it. Approved differences that stay: a caller x-api-key or non-blank bearer replaces the operator key instead of being sent beside it. Known differences: Python's azure_username/azure_password flow has no native counterpart, and a token provider combined with an OIDC reference uses the OIDC token where Python calls the provider. * fix(messages): let a caller api-key replace the azure_ai operator key on rust A caller api-key header on an azure_ai Messages call was forwarded next to the operator's x-api-key, so Azure received both credentials. It now replaces the operator key and is sent as x-api-key, as Python does for a lowercase api-key. Approved difference: Python only matches the lowercase spelling and sends a mixed-case Api-Key beside the operator key. Rust matches any casing. * fix(messages): order azure_ai entra id flows exactly as python on rust Without a key the azure_ai config now walks Python's get_azure_ad_token order itself instead of letting litellm-auth-azure pick. Every setting is read from the call and then the secret lookup, as AZURE_API_KEY is. A static token, the caller's azure_ad_token_provider and client secret run natively. Username and password, OIDC references, os.environ/ references, an uncallable provider and token refresh go to Python's own get_azure_ad_token through the host, as Python does them, so a static token with refresh on and a provider with an OIDC reference behave as in Python, and a call Python leaves without a token goes out unauthenticated. Every azure_ai call now asks the secret lookup for the Entra ID names up front, where Python asks only when no key is set. * fix(messages): read azure_ai entra settings where python does on rust Python's get_azure_ad_token reads AZURE_AD_TOKEN through get_secret_str and AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_USERNAME, AZURE_PASSWORD and AZURE_SCOPE with os.getenv (litellm/llms/azure/common_utils.py:580-588). The azure_ai config now does the same, reading the process environment only when no key is set, and only AZURE_AD_TOKEN stays in the secret names, so a value only a secret manager holds is ignored where Python ignores it. Approved by the owner, same class as A06-A08: any caller credential header (x-api-key, api-key or Authorization other than a blank bearer, in any casing) is the only credential sent, and the Entra ID path never runs for it. That covers a caller bearer on an Entra-only deployment (A10), which Python overwrites with the deployment's token, a lowercase authorization Python joins with the Entra token, and a caller x-api-key Python sends beside the Entra token. * fix(messages): look up AZURE_AD_TOKEN only where python does on rust Python reads AZURE_AD_TOKEN through get_secret_str only when the azure_ai call has no key, no caller credential and no azure_ad_token param (litellm/llms/azure/common_utils.py:580). Messages configs can now name deferred secrets, resolved right before validate_environment and only when the config names them for that call, so a keyed call or a call carrying its own credential never asks the secret lookup for the Entra token. Approved by the owner, same class as A06-A08: a caller Authorization other than a bearer, such as Basic, replaces the azure_ai operator key instead of being sent beside it as Python does. * fix(messages): build azure_ai urls and resolve the base like python on rust The native azure_ai Messages URL builder now ports Python's azure_anthropic_messages_url. It cuts at /anthropic in the path only, so a base whose userinfo starts with "anthropic" no longer sends the key to http://anthropic/v1/messages (security regression fix). It keeps the base untrimmed and moves the slash before a query as Python does. The base resolves as Python's `api_base or AZURE_API_BASE`: a blank call base is kept, an empty one falls back to the environment, and the environment value is used as given. With neither, the call raises Python's bare ValueError ("Missing Azure API Base ..."), flagged as raised by Python and as a late rejection, so the failure log carries the resolved provider and model parameters as Python's does. * fix(messages): send bases the native parser misreads through python's client on rust A Messages api_base with credentials, no host or a port Python cannot read now goes through Python's client, as bases without an http(s) scheme or with whitespace already did. Before, reqwest read such a URL differently from httpx: for http:///d it sent the request, with the operator key, to host `d`. This is a security fix for both azure_ai and anthropic; the anthropic_base_empty_host parity case covers anthropic. For azure_ai the choice reads the base Python's URL builder reads, falling back to AZURE_API_BASE, so a padded environment base also goes through Python's client. Python reads AZURE_API_BASE once: transport_handler returns the value it read and the bridge seeds the call's secret source with it, so the native route reuses it rather than reading the secret again. _sent_url only rewrites anthropic URLs, since the azure_ai builder no longer trims. Parity: 20 azure_ai and anthropic URL cases added; 65 aspects failed on the previous extension, and the full messages parity suite now passes. * fix(messages): look up AZURE_API_BASE only where python does on rust Python reads AZURE_API_BASE only through the `or` fallback in `api_base or get_secret_str("AZURE_API_BASE")`. The native azure_ai config listed it among the secrets it always reads, so a call with its own base still looked it up through the secret manager. It is now a deferred secret, read only when the call's base is empty or absent. A blank base is kept without a lookup, as in Python. deferred_secret_names now takes the call's api_base, the same base the URL builder reads. A lookup-counting test covers a call base (0 lookups), an empty base (1) and no base (1); before this change the call-base case looked it up once. * fix(messages): filter azure_ai anthropic-beta through its row like python on rust Python's handler runs update_headers_with_filtered_beta over the azure_ai row of anthropic_beta_headers_config.json after the config adds its feature betas, so unknown and null-mapped betas are dropped whatever the header casing, the caller's and the auto-added ones alike. Rust forwarded the caller's betas and every auto-added beta as sent. A caller anthropic-beta replaces a deployment's extra_headers anthropic-beta before either lane runs, as Python does, and a Router parity test pins it with the filter applied to both. The advisor tool already matched, since the bridge hands it to Python's interceptor and its beta is null in the row, so a parity case pins it. extra_body, unknown top-level params and the -latest alias also matched and get parity cases. * fix(messages): strip beta-gated system fields under drop_params like python on rust Python's transform runs strip_beta_gated_native_request with the final, filtered headers, so under drop_params a mid-conversation system turn loses the clear_at, output_config and tool_addition/tool_removal fields no sent beta unlocks, and a turn left empty is dropped unless that would end the conversation on an assistant turn. _normalize_system_role_messages strips a turn it converts to a user turn again with no betas at all. Rust sent those fields as they came. azure_ai now picks its betas from the request as sent and runs the strip, system-role normalization and the cache_control.scope strip in a new finalize_request step, once the final header is known. Vertex gets the conversion-time strip too, since it shares the normalization helper. The strips and the speed drop log Python's warnings word for word. A non-object output_format under drop_params is still popped after the headers were built, so structured-outputs-2025-11-13 is sent without it, as Python does. * fix(messages): send the system prompt as text blocks like python on rust Python's _normalize_system_role_messages always rebuilds the top-level system as a list of content blocks, so a string system reaches azure_ai and Vertex as one text block, and the success log, which only lists a string system as a message, logs the messages as sent. Rust kept the string unless a leading system turn was hoisted. Python's billing strip in transform_anthropic_messages_request also pops a system prompt it leaves empty, an empty string included, as Python does. The shared helper now does the same for every config that strips billing metadata: azure_ai, vertex_ai, deepseek, minimax, tencent and edenai, each with its own test. * fix(messages): keep non-claude azure_ai messages passthroughs on python A deployment whose provider has no Messages config of its own, like a non-Claude azure_ai model, but lists /v1/messages in model_info.supported_endpoints is forwarded untranslated by Python's OpenAILikeAnthropicMessagesConfig, with a bearer key to <base>/v1/messages. Dispatch gave it the MESSAGES target, so a rule that serves azure_ai on Rust would hand it to the native Anthropic config, which sends x-api-key to /anthropic/v1/messages and breaks the success log. Dispatch now targets MESSAGES only when the provider's own config serves the call, and MESSAGES_PASSTHROUGH for the opt-in passthrough. The catalog keeps that target on Python next to the chat and Responses bridges, so once azure_ai is Rust-required these calls resolve to Python and the Rust-only refusal does not fire for them. Proxy token counting treats both targets as before. The parity forced rule now serves a provider natively only for the MESSAGES target, as a Rust-required rule below the target exclusions does. A pre-request hook that sets a non-Claude model raises functools.partial's TypeError on both lanes, as on main, and a parity case pins it. * fix(messages): raise azure_ai transport and provider errors like python on rust The bridge mapped a native failure with the whole call as exception_type's extra_kwargs, so litellm_debug_info gained an API Base line from the named api_base. Python's entry passes only its **kwargs there, so its debug info names the model alone. The bridge now passes the same kwargs, for anthropic and azure_ai alike, and a decoding failure on a stream carries the deployment's debug info as Python's stream iterator attaches it. A connect failure to an azure_ai base read from AZURE_API_BASE kept the native "could not reach the provider" text, since the call had no api_base. The host now hands Python's failure mapping the secrets it read to send the call, so the aiohttp and httpx texts name the base the call went to, without reading AZURE_API_BASE again. A key or header value with a control character, like a deployment key with a trailing newline, failed natively as APIConnectionError with the model stripped of its provider. It is now the InternalServerError 500 Python's transport raises, with aiohttp's "Forbidden control character detected in headers" text and the model as called. On the httpx transport Python's text quotes the header value, which may be the key, so Rust keeps its own text there. The parity outcome now compares litellm_debug_info. The caller client's ReadTimeout from a stalled stream still has none on Rust and is recorded as a gap. * fix(messages): send a caller azure_ai api-key without an operator key like python on rust With no deployment key, Rust renamed a caller api-key header in any casing to x-api-key. Python's AzureAnthropicMessagesConfig renames only a header spelled exactly api-key and sends any other casing, like API-Key, as given. Rust now does the same when no operator key would be sent. With an operator key the approved behaviour stays, and a caller api-key in any casing replaces it as x-api-key. * fix(auth-azure): retry entra token posts like python's azure-identity A client secret or workload identity token endpoint that keeps answering 500 made the native path POST 9 times over about 80 s, because azure_identity 1.0 uses the SDK's default exponential retry (8 retries, 200 ms doubling, 60 s cap). Python's azure-identity 1.25.2 posts 4 times in about 5 s and then raises, so the Rust credential now carries a custom retry policy with the same numbers. Python sources in the wt-auth venv - azure/identity/_internal/pipeline.py:67 builds the token pipeline with azure-core's RetryPolicy(**kwargs) and no overrides - azure/identity/_internal/msal_client.py:98 (client secret) and aad_client.py:77 (client assertion, used by workload identity) run the token POST with retry_on_methods=["POST"] - azure/core/pipeline/policies/_retry.py (azure-core 1.41.0) lines 77-95 give retry_status=3, retry_backoff_factor=0.8, retry_backoff_max=120, retried codes 408, 429, 500, 502, 503, 504 and 506-998. Lines 127-143 sleep 0 before the first retry and factor * 2^(n-1) after that, so 0 s, 1.6 s, 3.2 s. Lines 455-509 sleep exactly Retry-After (then retry-after-ms, x-ms-retry-after-ms) instead of the backoff when it is present and non-zero azure_core only hands the policy a Retry-After value for 429 and 503, so a Retry-After on another status still falls back to the backoff. Managed identity keeps the SDK default since Python uses a separate policy there, and client assertion keeps it because Python mints OIDC tokens through litellm's own HTTP client. A 401 invalid_client is still one POST. * fix(auth-azure): post an oidc token exchange once like python Python exchanges an OIDC assertion through litellm's own HTTPHandler (_exchange_oidc_token in litellm/llms/azure/common_utils.py), a single client.post with no retry, so a 503 is one POST. The native client assertion credential used azure_identity's default retry (9 POSTs over about 80 s) and now runs with RetryOptions::none(). * fix(auth-azure): honour retry-after on entra token posts like python azure_core runs a custom RetryPolicy through its own loop, which only reads Retry-After for 429 and 503 and never retries a status outside the policy's list. Python's azure-core RetryPolicy (1.41.0, azure/core/pipeline/policies/_retry.py) retries any 4xx or 5xx that carries Retry-After (is_retry, lines 210-242) and waits exactly the Retry-After, retry-after-ms or x-ms-retry-after-ms value when it is non-zero (get_retry_after in _utils.py:79-96, sleep at lines 455-509). The client secret and workload identity credentials now run with the SDK retry turned off and a per-call policy that follows Python's loop. It keeps separate budgets of 3 for status, connect and read retries (increment and is_exhausted, lines 265-330), backs off 0 s, then 0.8 * 2^(n-1) capped at 120 s, and retries 408, 429, 500, 502, 503, 504 and 506-998 for the token POST. Client assertion still posts once and managed identity keeps the SDK default. A Retry-After that is neither a number nor an HTTP date now fails the token request after the first POST with "Invalid HTTP date", as Python does. azure-core's _parse_http_date (_utils.py:48-57) raises ValueError from the sleep before the retry, azure-identity's wrap_exceptions (decorators.py:63-80) turns it into ClientAuthenticationError "Authentication failed: Invalid HTTP date", and get_azure_ad_token (common_utils.py:663) raises RuntimeError "Failed to get Azure AD token: Authentication failed: Invalid HTTP date". * fix(messages): raise python's entra token errors for native azure_ai flows On the Messages route every Entra token failure now reaches the caller as Python's get_azure_ad_token and anthropic_messages raise it, with the same class, status_code, llm_provider, message and log lines. A native client secret flow keeps the token endpoint's final answer (status, headers, body) in a typed EntraTokenFailure from the azure-identity retry policy. The bridge hands it to the Messages host, which replays it through azure-identity's own ClientSecretCredential over a transport that returns that answer, so MSAL and azure-core read it exactly as they read it off the wire. That covers a rejected secret (EntraRejectionError 401 with the first AADSTS sentence and GUIDs redacted), an exhausted 5xx ("HTTP Error: 500"), a malformed Retry-After ("Invalid HTTP date"), non-JSON and odd JSON bodies, including the content-type dependent decode error. A transport failure has no answer to replay and keeps Rust's own reason after "Authentication failed:", so its text differs from requests' text. The caller's azure_ad_token_provider goes through the helper get_azure_ad_token now shares (azure_ad_token_failure), so a listed AADSTS rejection is an AuthenticationError 401 instead of a RuntimeError, and a non-string token raises the same TypeError. Both log Python's "Error calling Azure AD token provider" and "returned non-string value" lines. Every Python exception raised while getting the call's credential, including the host-minted username/password, OIDC and workload identity flows, now goes through Python's handler rule: a BaseLLMException is mapped by exception_type (AuthenticationError 401, BadRequestError 422 with llm_provider azure_ai), anything else reaches the caller as raised. The Retry-After date parser stays RFC 7231 only. Python's email.utils.parsedate_tz also accepts dates without a weekday, two-digit years, numeric zones and missing seconds, which Rust rejects as an invalid HTTP date and Python waits on. A NaN or infinite number of seconds now falls back to the backoff, where Python sleeps 0 for NaN and fails on infinity. * fix(auth-azure): wait on retry-after and report connect failures like python The Entra token retry policy now reads Retry-After the way azure-core 1.41.0 does. Seconds go through Python's float rules, including underscores between digits, and anything else through a port of email.utils.parsedate_tz, so a date without a weekday, a two-digit year, a numeric or named zone, a zone glued to the time, missing seconds, a dotted time, month before day and RFC 850 dates all wait as Python waits. A date datetime() rejects (day out of range, an offset of a day or more) still fails the request after one POST, as do unparseable values. A wait that time.sleep cannot take (infinity, or 2^63 ns and past) now fails after one POST as Python does. The host replays that answer with a sleep that still calls time.sleep for such a wait, so the caller gets "Authentication failed: timestamp out of range for platform time_t". A transport failure keeps the token endpoint's host and port. A direct connect that fails with an OS error, or a failed name lookup glibc words, reaches Python as urllib3 2 words it after "Authentication failed:", for example "HTTPSConnection(host='login.microsoftonline.com', port=443): Failed to establish a new connection: [Errno 111] Connection refused". Python hits it on MSAL's discovery GET to the same host. Failures through a proxy, TLS failures, timeouts and resets while reading keep Rust's own text, since urllib3's wording for them depends on details Rust does not expose. * fix(messages): treat a contentless system message as a system turn like python on rust Python's _normalize_system_role_messages reads only the role, so a role system message sent without content or with null content is still a system turn. A leading one is hoisted and adds nothing to the top-level system, and a later one on a model without mid-conversation system support becomes the operator note user turn, placed after a tool result it sits before and dropped under drop_params when its gated field is stripped. Rust failed to parse the message and sent it raw, which also showed up in the logged messages. A model that accepts the role still gets it as sent. * fix(messages): let the provider filter a non-ascii anthropic-beta like python on rust Python merges every anthropic-beta spelling into one header and the provider config filters it before the HTTP client encodes it, so on azure_ai an unknown non-ASCII beta is dropped and the call goes out, with or without drop_params. Rust held the header back as unencodable before the beta filter ran and failed the call with the ascii codec error. The beta header now reaches the provider filter, and the wire check fails the call only when a value Python's client cannot encode survives it. A value with a lone surrogate still fails up front since it cannot reach Rust as text. * fix(messages): log nothing for a stream the caller never read like python on rust Python's Messages stream iterator only starts on the first read, so a stream the caller closes or drops unread bills nothing and logs nothing. Rust billed the empty stream on close and teardown, which logged "Unable to build complete streaming response" and the read() error from the success handler. * fix(messages): log a traceback for failures python raises before the send on rust Python raises its pre-flight errors, such as the missing azure_ai api_base ValueError, the missing key AuthenticationError and the Entra token provider failures, through Python frames, so the failure log and the OTel span carry a stack trace. Rust built those exceptions without raising them, so error_information.traceback was empty. The host now raises them through a Python frame before handing them back. * fix(messages): return python's streaming response types from native messages streams Python's Messages handler returns an AnthropicMessagesStreamingResponse for a stream, so isinstance checks, hidden_params and aclose behave the same for every caller. The native async entrypoint returned the bare execution AsyncStream. It now wraps the stream the way the deployment success hook already did, sharing the stream's hidden params. The entrypoint is shared, so this moves anthropic as well as azure_ai toward Python. With a response cache configured Python hands back its caching handler's types too. A streamed cache hit is now the CachedAnthropicMessagesStreamIterator Python's handler builds from the cached events, stamped with the metadata @client stamps, and the native execution closes unread so only that iterator logs the hit, as Python does. A miss is wrapped in the AnthropicMessagesStreamCacheWriter Python returns, marked persisted since Rust stores the stream itself. The parity suite now compares the stream type for every streamed case. * fix(messages): read unparsed messages in beta detection and gated strips like python on rust Python reads beta triggers and gated fields off any message mapping, while Rust only looked at messages that parsed. A role system message sent without content, or any message with a content block Rust cannot type, was skipped: - a message carrying output_config now adds per-turn-control-2026-07-01 - a system message with a tool_reference addition or removal now adds mid-conversation-tool-changes-2026-07-01 - a signed compaction block now adds compact-2026-09-04 - under drop_params an unparsed system message loses its gated clear_at, output_config and tool changes, is dropped once emptied, and the warning names the strip - an unparsed assistant message counts as the last assistant turn when deciding whether a strip would leave a prefill - an unparsed message loses cache_control.scope on its content blocks * fix(messages): rewrite and normalize unparsed messages and system prompts like python on rust Python's sanitizer, advisor and encrypted-reasoning strips and system normalization read any message mapping, while Rust skipped every message or system block it could not type: - an unparsed message now gets the sanitizer's rules: a text block whose text is not a non-blank string and an empty unsigned thinking block go, a message left with no blocks is dropped, tool ids are sanitized, replayed web search results are flattened and provider_specific_fields dropped; a message the rewrite leaves parseable is parsed again - an unparsed assistant turn loses advisor exchanges, and an unparsed message loses bridge-tagged reasoning blocks - a kept redacted_thinking block gets Claude's own data back from a Bedrock Converse base64 wrapper, parsed or not - system is kept as sent when it is a list holding a block that does not parse, or an object; billing blocks and cache_control.scope are still stripped from such a list - a leading system turn whose content does not parse is hoisted as sent, and a later one becomes the operator note user turn as sent, placed after a tool result turn that did not parse either * fix(messages): keep anthropic's API Base debug line on rust failures Mapping a native failure with only the Python entry's **kwargs dropped the named api_base from exception_type's input. That matches azure_ai, whose Python entry maps with **kwargs alone, but it removed the API Base line that main's Rust-only anthropic route has always put in litellm_debug_info when the caller passes api_base. Python gives no rule that tells the two apart, because the legacy async entry maps both providers the same way. The difference is that anthropic has no Python path, so main's native route is its contract. The bridge now maps the whole call for anthropic and the entry kwargs for every other provider. The parity harness still uses the legacy Python entry as anthropic's reference, so it now expects the API Base line main's anthropic route adds. New cases cover an anthropic call on an azure_ai model failing with api_base, unary and stream. * fix(messages): keep anthropic stream types and header errors as main while azure_ai matches python By the owner's ruling anthropic keeps main's behaviour for two things the azure_ai port had changed for every native provider. A native async stream returns Python's AnthropicMessagesStreamingResponse, and with a response cache the CachedAnthropicMessagesStreamIterator or AnthropicMessagesStreamCacheWriter, only for azure_ai. The host asks Python for those types only when the call resolves to azure_ai, so anthropic gets main's execution AsyncStream again, for a cache hit or miss too. A header value the native client cannot send, like a key with a control character, is Python's InternalServerError with aiohttp's "Forbidden control character" text only for azure_ai. Anthropic raises main's APIConnectionError "invalid request: failed to parse header value" with the bug report link again. The parity stream_type aspect compares azure_ai with Python and expects main's AsyncStream for anthropic, in the same way the outcome aspect already expects main's API Base debug line. New azure_ai cache hit and miss stream cases check the cached types. * fix(messages): keep anthropic as main for azure_ai parity side effects By the owner's ruling anthropic, which has no Python path, keeps main's native route; the azure_ai port's Python-parity changes now apply to azure_ai only. - An unread stream: only azure_ai's starts on the first read and bills nothing when closed or dropped unread. The logging adapter learns the provider from the execution facts; anthropic bills it as main does. - Failures before the send: the missing-key AuthenticationError and the unmapped builtin errors are raised through a Python frame only for azure_ai; anthropic hands them back unraised, without a traceback, as main does. - A stream's decoding failure carries the deployment's debug info only for azure_ai; anthropic raises it bare. - A non-ASCII anthropic-beta is left to the provider filter only for azure_ai. Anthropic holds it back before the beta merge as main does, so the error names it ahead of any later unencodable header. - Unparsed messages: the shared shaping reads messages and system prompts Rust could not type only when the provider asks for it (UnparsedMessages::Read), which only the azure_ai config does. Elsewhere the sanitizer, advisor and encrypted-reasoning strips leave such messages as sent, their beta triggers add no beta, a redacted_thinking Converse wrapper is kept as sent, and a system prompt that does not parse is main's "untagged enum SystemPrompt" request error from messages_body. Each item has an anthropic test pinned to main's behaviour that fails on the previous commit, beside the azure_ai cases comparing with Python. * test(messages): pin the malformed api_base fix for anthropic and azure_ai d1efdd2683 sends a base the native URL parser reads otherwise than httpx and aiohttp through Python's client. Parity cases now cover, for anthropic and azure_ai alike, the three shapes that sent the operator key to an unintended host natively (an empty host, credentials in the URL, userinfo that reads like a trusted host) and the two port shapes, asserting that nothing reaches the upstream and that the caller gets Python's exception. They fail with the check disabled. The rust_bridge AGENTS.md states the rule. * refactor(messages): serve azure_ai messages on rust only and delete its python path Provider azure_ai on /v1/messages is now RUST_REQUIRED next to anthropic, so a Claude deployment whose AzureAnthropicMessagesConfig serves the call runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. The PYTHON_ONLY targets rule still sits above it, so a non-Claude azure_ai deployment that opts into /v1/messages (MESSAGES_PASSTHROUGH) and an azure_ai call bridged to Chat Completions or Responses stay on Python. A raising detector on a Claude azure_ai call gets the same UnsupportedParamsError as anthropic. MESSAGES_PYTHON_FEATURES is empty, so no feature refuses today. Deleted Python code that only served azure_ai Messages: - AzureAnthropicMessagesConfig._remove_scope_from_cache_control, a copy of the AnthropicMessagesConfig method it overrode. Its only caller is the azure transform_anthropic_messages_request (grep "_remove_scope_from_cache_control" litellm/: azure, databricks and vertex call self., the base defines it) Kept, because the parity harness still runs every azure_ai case through the Python handler (dispatch._PYTHON_AMESSAGES) as its baseline, and removing them would send that baseline to api.anthropic.com with anthropic auth: validate_anthropic_messages_environment, get_complete_url, transform_anthropic_messages_request and should_strip_billing_metadata on AzureAnthropicMessagesConfig (grep: only async_anthropic_messages_handler in llm_http_handler.py calls them, reached from the Python fallback and the harness). The Foundry beta-row branch was already removed with anthropic, and resolve_beta_provider's Foundry row now serves only anthropic chat. Count tokens has its own config and the Rust host reaches no deleted method. The parity harness drops the rust_providers forcing, so azure_ai cases run through the shipped rules and assert RUST_REQUIRED like anthropic. The azure count tokens beta test sends its inference call through a caller client, as Rust now serves it. * test(messages): cover the azure_ai agentic follow-up deployment on rust The Rust host builds an agentic follow-up's kwargs with _agentic_hook_kwargs, so the follow-up must reach the azure_ai deployment's own api_base with its key. This replaces the deleted LIT-5418 test, which drove the Python handler. Dropping api_base from that helper fails it with Missing Azure API Base, and dropping api_key sends the follow-up without x-api-key. * fix(auth-azure): retry Entra ID token posts like Python only for azure_ai Messages The azure-identity retry policy and the single OIDC post now apply only when a consumer asks for them through AzureAuthInputs.token_retry. The azure_ai Messages config asks for it, while Key Vault, Redis, blob cache, OCR and the anthropic federation keep the SDK default as on main. The policy is part of the credential cache key so consumers sharing a client never share a policy. * fix(messages): keep azure_ai without a key as main when no Python host is wired in Entra ID auth for an azure_ai call without a key now runs only when the route has a Python host. The Rust gateway has none, so such a deployment fails with the local MissingApiKey main raised instead of going out unauthenticated or failing on an Azure AD token without a host, and a deployment string named callable is never read as a caller token provider there. * fix(messages): log the dropped speed warning only for azure_ai The shared drop_unsupported_params helper logged Python's speed warning for every Messages provider. Configs now opt in through logs_dropped_params, which only azure_ai sets, so anthropic and the other providers log as main. * test(messages): drop the azure_ai default headers structure test and date pinned error texts default_headers_match_python only restated a constant. The MSAL, azure-core and CPython texts the replay tests pin now carry the date they were checked. * refactor(messages): name the anthropic error-mapping set for what it is _RUST_ONLY_PROVIDERS held only anthropic, the provider that keeps main's native error mapping, while azure_ai is Rust-only too. It is now _MAIN_NATIVE_ERROR_PROVIDERS, and the docstrings no longer call azure_ai a provider with a Python path. * docs(messages): keep azure_ai's python request code as the live parity reference azure_ai Messages stays Rust-required, but its Python request code is kept so the parity suite runs the live Python leg, as anthropic, Vertex and Bedrock do. * fix(messages): send azure_ai caller and operator credentials exactly as python does on rust The owner reversed the earlier approved difference on 2026-11-13 (Maciej). For azure_ai on /v1/messages every caller, operator and Entra ID credential combination now sends the header set main's Python sends, quirks included. Python builds a dict keyed by exact name, so a caller api-key spelled exactly so stops the operator key and the Entra ID path, the operator key otherwise goes in as api-key and the Entra ID token as Authorization beside or over the caller's headers, api-key is renamed to x-api-key unless x-api-key is spelled exactly so, and the HTTP client joins names that differ only in case. Ledger rows that change, all previously "Rust sends only the caller credential". A06 caller x-api-key now goes out beside the operator api-key. A07 caller X-Api-Key is joined with the operator key in one x-api-key. A08 caller bearer now goes out beside the operator x-api-key. A09b caller API-Key (other casing) goes out as api-key beside the operator x-api-key. A10 a caller Authorization bearer on an Entra-only deployment is overwritten by the minted Entra ID token. A10b a caller lowercase authorization is joined with the Entra ID bearer. A10c a caller x-api-key goes out beside the Entra ID bearer. A caller non-bearer Authorization (Basic) now goes out beside the operator key. A09 (an exact api-key replaces the operator key as x-api-key) is unchanged. AZURE_AD_TOKEN is now read wherever Python reads it, which is any call without a key, an exact caller api-key or an azure_ad_token param. The parity harness gaps for these rows are removed so they run as normal cases. * fix(messages): log the azure_ai call flag on failures before the send Python's Messages handler sets the aanthropic_messages flag before it gets the Entra ID credential or refuses an unsupported param, so those failures log it. The native host now flags azure_ai calls the same way when a token request fails or the request is refused, and anthropic keeps main's unflagged failure. * fix(auth-azure): fetch azure_ai entra tokens with python's proxy and ca settings Python's azure-identity sends the client secret token request through requests, which reads https_proxy or all_proxy unless its no_proxy matching skips the host, trusts REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE or certifi's Mozilla roots, never SSL_CERT_FILE, and times out after 300 seconds. azure_core's reqwest client read HTTPS_PROXY before https_proxy, matched NO_PROXY its own way, trusted the system store and SSL_CERT_FILE and timed out after 20 and 60 seconds. An explicit token_transport input on the auth request selects a transport that applies requests' rules over a litellm-http client. Only azure_ai Messages asks for it, so Key Vault, Azure Redis, blob cache and OCR keep azure_core's transport. * fix(messages): word azure_ai entra transport failures as python's requests does Python's azure-identity reports a failed client secret token request with requests' own text after "Authentication failed:", while the native flow kept reqwest's text for anything but a direct connect or name lookup failure. The requests transport now tags each failed send with the proxy it used, and classifies what reqwest, hyper and rustls observed into the failures requests words the same way every time: an unreachable or unresolvable proxy, a 407 answer to CONNECT, a proxy or server closing or resetting the connection, an EOF in the TLS handshake, an expired, not yet valid or misnamed certificate, and the 300 second read timeout. The host builds each text from the urllib3, http.client and ssl exceptions Python raises for it, so build details such as the _ssl.c line come from the running interpreter. A missing REQUESTS_CA_BUNDLE path now raises requests' own text too. * fix(messages): look up AZURE_AD_TOKEN for azure_ai only when a Python host is wired in The Rust gateway has no Python host, so a key-less azure_ai Messages call there fails with main's local MissingApiKey before any Entra ID flow. It no longer asks the secret source for AZURE_AD_TOKEN first, which main never did. Calls with a Python host read it where Python does, as before. * fix(messages): forward a caller credential on a key-less azure_ai call without a Python host as main Without a key or a Python host, as in the Rust gateway, main's key-only azure_ai check forwarded a caller x-api-key or bearer and failed with MissingApiKey otherwise. The branch made every such call a missing key, so the gateway's entra_bearer_without_key case failed; it now forwards those credentials again. Calls with a Python host are unchanged. * fix(messages): time an azure_ai exchange through python's client once on rust A call the Rust host sends through Python's client, such as a malformed or padded api_base, is already timed by that client's track_llm_api_timing post. The native logging adapter timed the same exchange again, so the Rust lane reported two LLM service events and recorded the window twice where Python reports one. The route host now tells the adapter when it sends through its own client, and for azure_ai the adapter leaves the timing to it. Anthropic keeps main's behaviour. * fix(messages): look up advisor ids in a set like python on rust * test(messages): cover the native cache, stream and host failure paths	2026-11-14 13:40:38 -08:00
tavren27	27c796c412	fix(proxy): count the request's tool_choice on the token count routes (#51133) ... * fix(proxy): count the request's tool_choice on the token count routes /utils/token_counter, /v1/messages/count_tokens and /v1/responses/input_tokens now pass tool_choice into the local estimate, and every DeepSeek model counts it as 0, as its providers bill * fix(token_counter): count a request with no model name again * fix(token_counter): count grok tool_choice the way xAI bills it xAI bills grok-4.3 and grok-4-1-fast-reasoning 21 tokens for "none", 11 for "required" and 12 plus the name for a named tool, where the OpenAI costs counted 1, 0 and 7 plus the name. The grok-4-1 and grok-4.3 chat template tool sections now carry those costs, so a grok count matches its bill for every tool_choice * fix(token_counter): count xAI's grok tool_choice costs only where xAI serves grok Azure AI Foundry bills grok-4-1's tool_choice differently, so grok on other hosts keeps counting it as 0. A tool_choice sent without tools counts nothing, since OpenAI and xAI reject it * fix(token_counter): key xAI's grok tool_choice costs on the provider that serves the deployment The count routes strip the provider prefix before counting, so Azure, Vertex AI and Perplexity grok matched the xAI name pattern. The routes now resolve the serving provider with get_llm_provider and drop xAI's costs when it is not xAI or OpenRouter's x-ai route. The costs move onto the grok tool sections so one pattern decides them * fix(token_counter): count no tool_choice for Gemini and none without tools in the Rust counter Gemini bills the same prompt tokens for every tool_choice, so it skips the OpenAI choice costs the way DeepSeek does. The Rust reservation counter now adds tool_choice tokens only when tools are present, matching the Python counter * fix(proxy): count no tool_choice a deployment drops, and treat an api.x.ai api_base as xAI A deployment that lists tool_choice in additional_drop_params never sends it, so the count routes no longer add its cost. An openai/ model whose api_base is on api.x.ai is served by xAI, so it gets xAI's tool_choice costs like xai/ grok * fix(proxy): count no tool_choice a request drops in its body /v1/messages/count_tokens takes additional_drop_params from the request body, and inference drops tool_choice when it is listed there, so the count no longer adds its cost either * fix(proxy): take additional_drop_params on /utils/token_counter and treat a malformed api_base as not xAI /utils/token_counter now passes a request body's additional_drop_params into the count, so a tool_choice the request drops counts as 0 there like on the other two count routes. An api_base that does not parse falls back to the deployment's provider instead of failing the count with a 500 * fix(proxy): take chatgpt and github_copilot providers from the model name in the count host check Resolving them through get_llm_provider read the saved login, logged a warning and could refresh a token on every count with a tool_choice * fix(proxy): skip a tool_choice or additional_drop_params too deeply nested to validate on the count routes A value nested 300 levels deep failed TokenCountRequest validation, so the count routes answered 422 or 400 where they used to count the request without it * fix(bedrock): count Kimi K3's forced tool_choice message on the exact count path Kimi K3's chat template adds a tool-choice system message when Bedrock gets toolChoice any or a named tool, and Bedrock bills it (170 input tokens unforced, 206 forced). The exact count path now passes tool_choice through the same Converse mapping inference uses and renders that message * test(proxy): type the Kimi K3 count test's tools parameter * perf(count_tokens): validate each kept count option once instead of twice * fix(dot_notation): parse drop paths in linear time when brackets never close * test(count_tokens): benchmark the unclosed-bracket drop path instead of timing it in a unit test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 07:53:27 -05:00
yujonglee	b88e4bff6b	refactor(router): count the whole router_usage family on the native store (#50862) ... * refactor(router): count the whole router_usage family on the native store usage-based-routing-v2, the model rate limit check (tpm, rpm, tpd, rpd) and the io-token check (itpm, otpm) count through the native router_usage store, next to the router core's global_router tpm and rpm. The pick's usage read rides the native pre-call batch and settles on the batch reply, keeping DualCache's per-key read throttle, and the success tpm increment rides the native post-call batch after counting in memory. io-token reservations stash their counter's fields instead of a Redis key. Redis keys and values are unchanged. A Redis service event is reported in a context named for whose it is, either the calling store op's, the request whose post-call batch its deadline sends, or nobody's for breaker transitions and unscoped calls. The deadline's tag is built when a post-call declaration arms it, so that flush parents under the request's span even though auth sets the span after the batch scope opens. An SDK call anchors the span its caller passed, so its usage reads and io-token refunds parent under it without a span argument threaded through the family. * fix(redis): keep an int counter past i64 above the limit instead of wrapping Number::add summed two ints with plain i64 addition. With Redis off and the OTPM counter at 1, a request with max_tokens=2**63-1 passed the MAX_INCREMENT check, and the memory count wrapped negative in a release build, so it passed admission and let other requests past the cap. Python's int never overflows, so main sees a count above the limit, rolls it back and answers 429 An int sum past i64 now becomes the nearest float, which compares above any limit the way Python's int does. The rollback after such a refusal can leave the count one token low, since the float cannot hold 2**63 exactly * fix(logging): run every callback under the span an SDK caller passed Native Redis service spans take their parent from the request span contextvar. The proxy sets it at auth and the SDK router scopes it around the routed call, but a streamed response is iterated after that scope closes, so its callbacks saw no span. Three callbacks reopened the scope from kwargs by hand, and the sync stream path had no wrap at all because its success handler runs on an executor thread The four Logging handlers now anchor the span from model_call_details around their dispatch, so every CustomLogger callback sees it, streamed or not, sync or async, with or without a Router. The per-callback wraps in lowest_tpm_rpm_v2 and model_rate_limit_check are gone. A span already anchored, as on the proxy, is kept * test(logging): cover a mid-stream failure, and drop the span wrap in Router._release The callback span test now also breaks a stream after its first chunk, sync and async, through a custom provider, so the failure handlers the stream wrapper starts on a bare thread are covered too A probe through the Router showed Router._release only runs inside the routed attempt, where the router has already anchored the span, and never after a stream returns. Its own wrap was a no-op, so it is gone * fix(router): refuse an io-token reservation past its limit before counting it The overflow fix let a reservation near i64::MAX become a float, so the count still read above the limit and the request got its 429. But the rollback subtracted the same amount from that float, and with OTPM usage at 40 the count came back as 0.0, so later requests reused capacity already spent A reservation above its limit now gets the 429 before the store is touched, in the async and sync paths. On main the count went up and straight back down for such a reservation, so the answer and the count after it are the same as main's. Only the usage named in the error message differs, which now shows the reservation instead of the count plus the reservation * test(redis): send the failed SDK call's refund through a real completion The refund span test called ModelRateLimitingCheck.async_log_failure_event directly, so it relied on the span wrap that callback used to carry. The Logging failure handlers now anchor the span for every callback, so the test sends a failing litellm.acompletion with the reservation in its metadata and checks the refund's Redis span still lands under the caller's span	2026-11-13 03:22:28 +00:00
maciej-berri	d6b3c549ee	perf(rust): stop re-serializing the Messages request body in Python logging callbacks (#51440) ... * perf(rust): store the sent wire bytes as the Messages request instead of re-serializing The Rust /v1/messages path handed record_response the body as a Python dict, and Python ran json.dumps on the whole body again to build httpx_response.request. Rust now keeps the bytes it serialized for the final attempt and passes them through, so the stored request is exactly what the provider received and Python does no extra full-body serialization. The hosted transport path sends the same Rust bytes instead of a second Python json.dumps. This also fixes the stored request after a retry with a wide int, NaN or Infinity, which previously carried the internal marked string instead of the real value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx * perf(rust): serialize the Messages logged input in Rust instead of json.dumps messages_input ran json.dumps on the whole pre_call body in Python for every Rust /v1/messages request. Rust now serializes it with the same raw_json writer it uses for the wire, at the same point (after caller substitution, before Logging.pre_call), so callback edits still reach the wire but not the logged input. Python only returns the top-level key order, with the handler signature read once, and top-level scalars keep the caller's own spelling. The logged input string becomes compact JSON with raw UTF-8. Top-level key order and values are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx * fix(rust): keep body keys missing from the order list in the Messages logged input The logged input only wrote keys named by Python's order tuple, so a key left out of that tuple would vanish from the log without an error. Keys the order does not name now follow in their sent order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bne5zRJx1T5sEYBcsGtvqx --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-12 17:13:55 -08:00
Nate Armstrong	7553c4f501	fix(messages): hand rust only the anthropic metadata user_id (#51355) ... A guardrail such as Pillar aliases the request's metadata to litellm_metadata, which holds the caller's UserAPIKeyAuth. Since #51119 the Rust host converts the whole metadata field to JSON before its own user_id filter runs, so the call failed with "unsupported type UserAPIKeyAuth". The host now passes only user_id, the one key the Rust validator and Python's AnthropicMetadata keep.	2026-11-12 19:39:53 +00:00
Nate Armstrong	2845677822	fix(messages): name the deployment on a rust stream's transport error (#51358) ... When the upstream connection drops or stalls mid-stream on the Rust Messages path, the caller gets the raw httpx error without litellm_debug_info, so the llm_exceptions alert names no API base, model group or deployment. Python's stream iterator attaches that info to the same error, and the Rust host now does too.	2026-11-12 17:29:37 +00:00
Yassin Kortam	adc4c01aa9	chore: move scripts and runtime sources under litellm (#51006) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:57 +00:00
Yassin Kortam	db6dfb6bb6	chore: move workspace packages under litellm (#51000) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:57 +00:00
Yassin Kortam	f6bee81c73	chore: move ui and vscode-extension under litellm and delete docs (#50992) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:57 +00:00
Yassin Kortam	6efbaf539c	chore: move deployment sources under litellm/deploy (#50981) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-12 04:19:56 +00:00
Nate Armstrong	59538ff1d9	feat(messages): serve anthropic messages on rust only (#51119) ... * feat(messages): apply modify_params rewrites on rust * feat(messages): mint anthropic workload identity tokens on rust * feat(messages): apply prompt caching injection on rust * feat(messages): serve the easy python-only detectors on rust * feat(messages): retry rejected thinking signatures on rust * feat(messages): serve response cache reads on rust * feat(messages): retry rejected strict tools on rust * refactor(messages): delete the python path for anthropic messages Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a detector that raises) gets UnsupportedParamsError naming the feature, raised from one place in litellm/messages/dispatch.py. Other providers keep the Python fallback. A call whose arguments the handler signature rejects still gets Python's TypeError. Deleted Python code that only served anthropic Messages: - workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses were never eligible; the Rust host mints the token itself) - the anthropic Foundry beta-row and affinity-base branch in async_anthropic_messages_handler, and the "forward" mode of the beta filter that only that branch used The proxy MCP tools detector now uses the light tool precheck so a function tool never imports the gateway handler, since a raising detector now refuses. * feat(messages): forward message threads on rust The Rust path for anthropic /v1/messages dropped the thread body field, so a create or continue turn reached Anthropic without its thread or the message-threads-2026-08-12 beta. It now matches the Python handler from #50976: - the host forwards thread with the other body params - a non-null thread adds the message-threads beta - a well-formed continue turn keeps advisor blocks in history and derives no compaction, per-turn-control or tool-change beta from its messages - the recorded httpx_response carries the request that was sent, for JSON and streamed calls, so the proxy's thread ownership hook resends the create turn's betas on a continue as it does for Python Upstream error.details already reached the exception message unchanged, so thread_not_found relays and keeps the deployment out of cooldown on both paths. Parity cases cover create, continue, streaming, sync, retries, thread_not_found and the ownership hook round trip. * feat(messages): serve mock responses on rust * feat(messages): send caller clients and unsupported tls through python on rust * feat(messages): run pre-request hooks on rust * feat(messages): run agentic loop hooks on rust * feat(messages): run the safeguards classifier on rust * feat(messages): run registered interceptors on rust * fix(messages): match python stream cache gate, logging and interceptor order on rust Store a streamed Messages response in the cache only when Python would: a message_stop event line, no provider error frame and at least one frame with output. Leave dict results untouched by mark_rust_response so hidden params never reach result dicts, logging objects or hook digests. Ask interceptors whether they handle the call after the cache lookup, not at decode time, so a cache hit never calls can_handle. Emit the track_llm_api_timing service event for the provider exchange. Bill and log a stream superseded by a router fallback as the failure of its error frame, with the partial usage warning. Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the autorouter baseline cache sees retried requests. * fix(messages): forward python-only json values to anthropic as python does Accept any max_tokens, sampling scalar and messages collection that Python main accepts, and carry values serde_json cannot hold (ints beyond 64 bits, NaN and infinities, strings with lone surrogates) as validated raw JSON leaves written back verbatim on the wire, so the upstream body matches Python's json.dumps * fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust * fix(messages): log python's plain provider header names on rust streams * fix(messages): match python's request params, logged params and validation on rust Read a non-bool stream by Python truthiness and send it as given, log the caller's stream flag as Python's handler does, raise the raw Python errors main raises for malformed messages, tools, model_info and hook-set litellm_params or providers, read the tool custom-field drop from drops_unsupported_params, log the caller's nested fields that additional_drop_params strips and the Python values of raw JSON leaves, keep a missing stop_reason absent, and map provider-config rejections (missing max_tokens, unknown reasoning effort) as Python's HTTP handler maps them, with Python's text, effort order, model and provider * fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust * fix(messages): match python on hooked streams, empty cache writes and mcp cache hits * fix(messages): join already buffered stream reads and trim the env api key as python does on rust * fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust A stream cut by a transport error now hands the caller the bytes already buffered before raising, as aiohttp does for a caller that reads them before the event loop sees the cut, so the frames and their usage reach the caller and the failure log. Provider replies keep integers wider than 64 bits exactly and read numbers that overflow a float as infinity, through the response, the response cache and the logged result. The provider client follows up to 20 redirects as httpx does, and running out raises Python's text. A provider_specific_header value the ascii codec cannot encode fails before the send like a caller extra header. The deployment success hook gets the stream as AnthropicMessagesStreamingResponse, as Python's handler hands it. * feat(messages): serve proxy mcp tools on rust * test(messages): cover hook, agentic and interceptor edge cases in the parity suite * fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust * fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust A header value Python's client cannot encode as ASCII now fails before the send whichever header carries it, so a caller api_key with a non-ASCII character raises Python's InternalServerError instead of a raw UnicodeEncodeError after the reply. A caller api_base without an http(s) scheme, or with whitespace, control or non-ASCII characters, is sent through Python's client at the URL Python builds, so its relative-URL and invalid-URL errors match, and a base urlsplit cannot split raises Python's bare ValueError before the send. A provider_specific_header of the wrong shape is scoped through Python's ProviderSpecificHeaderUtils first, so a plain string raises Python's AttributeError. * fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust temperature and top_p now deserialize through recognized::exact_float, which keeps an integer where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent upstream and logged as the integer python forwards, and comparisons still go through scalar.rs cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python TypeError marked as a late rejection, and map_failure hands it to the caller unmapped * test(messages): format the non-string tool_result parity helper * fix(messages): flag and log the call before python's pre-send raises on rust The bridge now hands settings() and map_failure() whether the call is async. A provider_specific_header of the wrong shape sets the aanthropic_messages flag before re-raising, as Python's handler has already flagged the call. A base urlsplit cannot split also logs the resolved model, provider and optional params first, as Python raises it only after its logger is updated. The TypeError raised while capping the thinking budget for a string max_tokens sets the flag too * feat(messages): serve the web search short-circuit override on rust through the intercept host call * fix(messages): give the web search entry kwargs its mutable-ok reason * test(messages): drop the body hook gap the rust short-circuit closes * fix(messages): name the partially unknown handler types on the short-circuit host call	2026-11-12 00:07:20 +00:00
yujonglee	b079bf5a51	refactor(proxy): let the gunicorn server take a private listener (#50871) ... Extract serving_port() and configure_database() as they were inline, turn the LITELLM_RUST parser into a reusable RustSwitch type, and reformat tests/unit/proxy/test_proxy_cli.py to the 120 column limit. run_gunicorn_server takes private_listener, an inherited socket or an assigned loopback port. With one, gunicorn binds it instead of host:port and trusts no peer's forwarding headers, since every peer on it is the local process in front. Both are gunicorn's own settings, bind and forwarded_allow_ips, so the uvicorn worker is unchanged. The reload options take factory. No caller passes either yet, so every entry point serves as before.	2026-11-11 19:40:00 +00:00
yujonglee	11af2eb22d	refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) ... * refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 01:57:54 +00:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
tavren27	cd9f98d095	fix(anthropic-messages): apply additional_drop_params before the unsupported param checks (#50753) ... * fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates * fix(anthropic-messages): apply additional_drop_params before the unsupported param checks Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request * fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body * test(anthropic-messages): annotate the new test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 01:35:35 +00:00
Nate Armstrong	13c1c2a364	fix(messages): write the native unary cache entry after the router stamps it like python (#50426) ... Python stores a unary messages response from a background cache write task that runs after the router stamps _hidden_params, so the entry carries model_id and the router headers and a later hit returns x-litellm-cache-key. The native path stored the entry inside the native call, before the router saw the response. The async unary store is now deferred until the route encodes its public response and is handed to Python's own cache write task	2026-11-10 00:24:21 +00:00
Nate Armstrong	6a4aab41f0	fix(messages): bill a native stream the client abandoned like python (#50425) ... When the client disconnects mid-stream, the native adapter now logs success for the chunks it delivered before teardown, so spend, SpendLogs and the key's budget and tpm reservations settle the same way they do on the Python path The proxy records a client disconnect, with its 499 error information, in the stream generator's cleanup. The native teardown billed the stream before that cleanup, so the row had empty error information. A proxy stream torn down by cancellation now bills from the task's done callback, after the disconnect is recorded	2026-11-10 00:24:21 +00:00
Nate Armstrong	ee34a3c2d4	fix(messages): raise native messages failures as python's errors (#50424) ... A stream whose upstream body ends mid-frame reached proxy callers as a mapped APIConnectionError with status 500. Python's transport raises an unmapped httpx error at that point. The native body read now records a body cut and the host raises the same exception the active transport would Python wraps a connect, DNS, TLS or reset failure in a synthesized 500 AnthropicError, which maps to InternalServerError. The native path surfaced these as APIConnectionError with the bug-report footer, which also changed cooldown decisions The proxy bills the estimated prompt of a failed call only when the Timeout was caused by an httpx ReadTimeout. The native timeout now chains the same httpx timeout Python's client raises, so its failure row bills the prompt Python rebuilds an upstream error response from the decoded body without content-encoding and content-length. The native failure now drops those wire headers, and a failure raised without an HTTP response states no headers at all, like Python	2026-11-10 00:24:20 +00:00
Nate Armstrong	1467b13fd2	fix(rust): run native async calls through python's async wrapper steps (#50423) ... Python's async client wrapper stamps the effective timeout and num_retries on every failure, which the proxy reads for x-litellm-timeout and the router reads for retries. The native async paths skipped that wrapper, so error responses from Rust-served async calls had no x-litellm-timeout header Python's async chat completions, responses, embeddings and messages entries hand the call to run_in_executor before they build the provider body, so a during_call guardrail started next to the call masks the request first. The native paths built their request without yielding, so masked text never reached the provider The shared native_acall_hook now yields once and stamps the failure, and every async dispatch passes it as its native hook. OCR passes native_acall_hook_on_loop, which only stamps the failure, since its Python entry went through the client wrapper but built its request on the loop	2026-11-10 00:24:20 +00:00
Nate Armstrong	d631f3cf05	fix(messages): log native messages request fields like python (#50421) ... Python logs the requested optional params, taken before the provider transform, as model_parameters. The native handler logged the transformed body, so a reasoning_effort request was logged as thinking and output_config. The request context now carries the requested params Python updates model_call_details with the transformed request body before pre_call, so the logged messages are what was sent. The native adapter kept the caller's messages and compared params as JSON values, so an int 0 logged as 0.0 Python only records custom_llm_provider in model_call_details. The native adapter also wrote it into litellm_params, and the proxy reuses one logger across router hops, so a fallback hop's pre_call log showed the provider where Python shows null Python only stamps cache_hit when a result came from the cache. The native adapter wrote False on every provider result Python calls pre_call with the request body as a json user message, and a stream's spend-log cache_key is computed over that input. The native adapter logged the string Messages, so a streamed call's cache_key differed	2026-11-10 00:24:19 +00:00
Nate Armstrong	1ddc913002	fix(messages): log native messages success once, after the proxy releases it (#50420) ... * fix(messages): log native messages success once, after the proxy releases it The proxy defers non-stream success logging past post-call guardrails, but the native host parked its log in its own slot and the release only ran for OCR, so a unary /v1/messages call served by Rust logged no success event, no SpendLogs row and no key, team, user or end-user spend. A deferred native success now parks in enqueue_deferred_logging with deferred_success_result, the same slots Python uses, so the proxy releases it like any other call and its guardrail step pruning and cost preparation see it too Python runs sync success callbacks inside the deferred logging closure, so a post-call hook that blocks the response drops them. A parked native success now carries its sync callbacks and runs them on release A refused first attempt and its fallback share one logger. Python logs the async success once, for the first attempt, so the native adapter now claims the async success log once across fallback attempts. Like Python, an attempt that loses the claim runs its sync callbacks only when logging is not deferred, and an internal call logs nothing * test(messages): cover the deferred native success park and release	2026-11-09 16:06:53 -08:00
Nate Armstrong	c6794ac08b	fix(messages): let proxy /v1/messages requests run on the rust path (#50335) ... The proxy attaches its shared aiohttp session to every request and a default stream_timeout to every stream, and the messages detectors treated both as Python-only features, so no proxy request reached Rust. A shared session is only a connection-reuse hint and no native client reads it, so the common caller_client detector now fires only for an explicit client. The native messages call now takes the one timeout Python's messages handler resolves from timeout, request_timeout, stream_timeout and an httpx.Timeout, so no timeout shape keeps a messages call on Python	2026-11-09 16:06:52 -08:00
Nate Armstrong	eae60c0bc9	perf(router): skip unused deployment reads in the callback filter step and cache anthropic capability probes (#50641)	2026-11-08 17:59:55 -08:00
Kerry Lu	07c7229895	feat(proxy): per-request header opts a stream into usage.cost without the global include_cost_in_streaming_usage flag (#50574) ... * feat(proxy): let one streaming request opt into usage.cost with x-litellm-include-cost-in-streaming-usage The global include_cost_in_streaming_usage flag moved every stream off the fast path. A request can now send x-litellm-include-cost-in-streaming-usage: true to get usage.cost on its usage-carrying stream event (/v1/messages and the Anthropic and OpenAI pass-throughs) while requests without the header keep the fast path. The global flag keeps working unchanged. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: type the new stream cost header tests * fix(types): keep reportUnknownArgumentType at its merge-base count * fix(proxy): the stream cost header only accepts true --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 16:57:20 -08:00
ryan-crabbe-berri	02d610f1a9	fix(anthropic): disable optional default thinking for reasoning effort none (#50541)	2026-11-07 12:51:37 -08:00
Kerry Lu	9ec3b5fd8b	feat(cost-map): move -latest model aliases into the cost map (#50491) ... * feat(cost-map): move -latest model aliases into the cost map Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: load the finalized local cost map in the moonshot alias test and assert alias pricing by behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: compare alias cost map rows to their targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(cost-map): give each -latest alias its own alias_of row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): keep alias_of rows out of alias lists and add them to the registry schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 91e6cf2c27) * fix(cost-map): let alias lists win over alias_of rows in both loaders and never serve a malformed pointer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit a54c42218a) * feat(cost-map): add -latest alias rows for the tier 1 and 2 hosting providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost-map): derive hosted alias targets from the cost map Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): skip alias rows when checking cache-write rates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 11:56:47 -08:00
moe-berri	b02bb99b08	feat: add OpenRouter-style -latest model aliases for first-party providers (#50315) ... * Add first-party floating model aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Align Moonshot alias parameter test with its target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias proxy budgets and rate limits with targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove target-only team budgets count and block floating aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias quotas with project and organization targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Coalesce colliding floating alias rate limit counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix latest alias Rust fallback and failure accounting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: adapt latest alias checks to typed rate-limit store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-06 18:22:38 +08:00
yujonglee	b1b236826a	feat(redis): serve the proxy's common Redis call with Rust stores (#50170) ... * feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 06:01:11 +00:00
yujonglee	a1b87bcca8	feat(redis): extend the native Redis foundation for the store families (#50169) ... Reworks the redis-client, connect, resolve and testing crates, adds python-compat json loads and TTL seconds, replaces the redis-store memory tier with a DualCache plus a shared failure log, and gives the Python side the Redis timeout log throttle and a request OTEL parent span for service events Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:30:07 +09:00
Nate Armstrong	34314a99f0	fix(messages): make rust messages copy python messages behaviour (#49611) ... Closes the Rust vs Python /v1/messages differences the parity suite marks as known gaps, copying Python exactly.	2026-11-02 21:02:52 -08:00
mika-lt	183212e8be	feat(anthropic): pick the beta allowlist row per deployment and surface input_transformations (#49689) ... * feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): type the provider_specific_fields source dict and beta row test helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): input_transformations reaches chat completions and /v1/messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): read the Anthropic response as a Mapping when building provider_specific_fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drop the duplicated beta tuple and mark filter results Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row A Foundry-hosted anthropic deployment now filters through the azure_ai row, which dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools, oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and without compact-2026-01-12 a compaction request answers 400, so the row now allows all seven on both the Python map and the Rust table * fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row An openai/ deployment that passes /v1/messages through to a Foundry host lost every beta, and a request without betas read the beta table from GitHub. Only an anthropic deployment on a Foundry host now takes the azure_ai row, and the check reads the provider and host before the table. * test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment * style(rust): wrap the azure_ai beta header test case the way rustfmt does * fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did. cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has no sync /v1/messages handler. The json_mode chat response keeps input_transformations, and the first /v1/messages call with betas loads the beta table off the event loop. * fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor loop the same way an azure_ai deployment does * test(anthropic): drive the Foundry advisor loop test through recorded HTTP The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta * fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests Reverts 083990f6d7, since review found it reaches a pre-existing gap that is now with security engineering. The beta row tests drive a real HTTP transport, the beta table fetch test checks the fetch thread instead of a 2 second deadline, and the touched test files keep main's formatting * test(anthropic): give the Foundry beta row logging fixture a fixed datetime start time * test(anthropic): check that chat completions pick the beta row by api_base --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:55:26 +00:00
Nate Armstrong	87d91d78b7	refactor(router): route the router family through RoutingContext and close the outside surface (#49118) ... Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points	2026-10-30 19:18:54 -07:00
Mateo Wang	23ad8870b4	fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill (#47554) ... * test(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill * fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding * fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing Detect raster images from the page content stream, including inline images on pages with empty resources, and walk each form once. Strip only openai/, azure/ and Azure data-zone prefixes, bound the model family scan, read Azure-only chat models from the cost map, and keep cl100k_base for listed non-chat OpenAI models. * fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources Prices a PDF by the model name's last path segment, the same name the proxy counts with, keeps listed non-chat models on Claude pricing, walks each form once per inherited resources dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing only when a message carries a PDF * fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes count like their base model * fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models * fix(token_counter): count reservation and project ITPM with the serving deployment's model Budget reservation and project ITPM counted with the requested group name, so a GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count with each deployment's model minus its provider prefix, the name count_tokens uses, and take the largest. A custom OpenAI tokenizer with no model counts with cl100k again instead of raising, and listed Azure gpt- chat entries price as OpenAI. * perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator pypdf parsed every operator of a page's content stream to find the images it draws, so a compressed page of a million operators took about 30 seconds to price for an OpenAI model. A linear scan of the decoded stream for name Do and inline images finds the same draws. * fix(token-counter): count each team deployment by its PDF pricing and honor base_model Budget reservation and the TPM limiter now look up the deployments that serve the caller's team, by base_model when set. A deployment that reads PDF pages and prices them unlike the group name counts with its own model, any other with the group name. A comment between an XObject name and Do now reads as whitespace, and the timing tests moved to CodSpeed benchmarks. * fix(benchmarks): install pypdf in the benchmark environment The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path. * fix(proxy): count PDF pages for deployments served under multi-segment model names * fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge * fix(proxy): count routing-group and alias members with the caller's team, as routing serves them	2026-10-29 18:24:04 -07:00
zoveki	68b5a737a1	fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params (#49102) ... * fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params The native /v1/messages path sent tools with `strict` straight through, so when Claude's strict-tool limits rejected them with a 400 the caller got that 400 even with drop_params set. Under drop_params, a 400 that matches Claude's strict-limit phrases on a request with a strict tool is now sent once more with `strict` removed from every tool. Without drop_params nothing changes. The Claude strict-limit matcher moves from bedrock/common_utils.py to anthropic/common_utils.py so Bedrock converse and the base messages config share it, and the Rust bridge routes strict tools under drop_params to Python. * fix(anthropic): retry without strict only when a tool sends strict true A tool with only strict false no longer triggers a resend of an identical body, the warning no longer blames tool schemas when the rejected schema may be the output format, and requests with only additional_drop_params stay on the Rust messages route since the Python retry ignores them * fix(anthropic): keep sync messages calls with strict tools on Rust Python has no sync /v1/messages handler, so routing a sync call with a strict tool to Python raised ValueError where Rust served it before. The strict-tools detector now applies to async calls only. The retry warning also names the strict tool count without a plural. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 17:18:59 -07:00
zoveki	310c96162b	fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) ... * fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:36:20 -07:00
rj-cairn	5eb1042bbf	fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728) ... * fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): move the o200k provenance note into the assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust_bridge): azure/gpt-4o now resolves to o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know * fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer * fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model. * fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model * fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model * fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted. * test: type the new token-counting tests' parameters * fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it * fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test * perf(token-counter): classify each distinct served model once when walking a model group Budgeted requests walk every deployment in their model group, since the router's cached copy is outside the router surface the proxy may use. Classifying each distinct served model once instead of once per deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms with the same result. --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:08:12 -07:00
David Koh	57f05e56f6	fix(bedrock): omit system prompt from audio transcription and report its token usage (#41565) ... * fix(bedrock): omit system prompt from audio transcription * fix(bedrock): report Converse token usage on audio transcription Bedrock transcriptions returned usage null and were billed at $0. Map Converse inputTokens, outputTokens and totalTokens into the transcription usage so spend uses the model's per-token price. input_token_details is optional because Converse does not split audio and text tokens. Also switch the updated request-shape test to rstest. * fix(bedrock): default transcription temperature to 0 and price ap-southeast-2 Voxtral regionally Voxtral wrapped some transcripts in quotes at its default sampling temperature. In direct Converse calls in ap-southeast-2, 0 of 48 calls at temperature 0 were framed against 7 of 72 at the default, and instruction wording made no clear difference, so the request now sends temperature 0 when the caller sets none, matching OpenAI's transcription default. Calls in ap-southeast-2 billed at the us-east-1 price. The cost map gains bedrock/ap-southeast-2 Voxtral rows with all four service tiers, the Bedrock transcription response carries the deployment region in its hidden params, and the transcription token cost uses the region-resolved model key. The bridge now builds the response with model_copy instead of assigning usage after construction, and the native wheel test checks the usage the compiled route returns. * fix(bedrock): forward transcription language, reject unsupported formats, and bill env and Tokyo/London regions get_optional_params_transcription had no Bedrock branch, so language, response_format and timestamp_granularities were dropped before the Rust transform ever saw them. Bedrock now forwards language, prompt and temperature, accepts response_format json or text (the only shapes it can return), and answers 400 for srt, vtt, verbose_json or timestamp_granularities unless drop_params is set. The billing region now falls back to AWS_REGION_NAME and then AWS_REGION when aws_region_name is unset, the same order the Rust resolver uses to pick the endpoint, so an env-configured region gets its regional price. Adds ap-northeast-1 and eu-west-2 Voxtral rows (standard, batch, flex and priority) from the AWS Bedrock offer files, the other two regions AWS serves Voxtral audio in. * fix(bedrock): send transcription temperature as a number and name the rejected param The proxy keeps multipart form values as strings, so -F temperature=0.2 reached Converse as "0.2" and Bedrock answered 400. The Bedrock mapper now converts temperature to a float and answers 400 (or drops it under drop_params) when it is not a finite number. get_optional_params_transcription copied prompt and temperature back from the raw call after the Bedrock branch ran, which undid that conversion and any drop. The Bedrock branch now owns every OpenAI transcription param. An unsupported param such as timestamp_granularities now gets a 400 that names it instead of the generic "user/encoding format" message. * refactor(bedrock): return unsupported transcription params as a value and raise at the param step * fix(bedrock): drop the retired max_tokens field from the regional voxtral price rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 19:23:15 -07:00
yujonglee	1f68cc5649	refactor(rust-bridge): require the native extension (#48852) ... * refactor(rust-bridge): require the native extension _native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): build Rust bridge in Windows smoke tests --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 13:16:56 -07:00
Mateo Wang	14193435b8	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) ... * refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 10:08:25 -07:00
moe-berri	9fab92672a	fix(opencode): forward stable sessions automatically across inference routes (#48121) ... Forward caller-supplied stable session metadata to recognized OpenCode destinations across Chat Completions, Responses, and native Messages. Preserve explicit headers, configured affinity headers, and existing transports. Fixes #39503	2026-10-23 17:38:47 -07:00
yucheng-berri	dd511a2b22	fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) ... * fix(logging): fire one CustomLogger hook per async call outside the async flag list aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are missing from is_sync_litellm_request's flag list, so the sync success and failure handlers called a CustomLogger's sync hook on top of the async hook the async handler had already called for the same request. The sync handlers now skip a CustomLogger's log_success_event and log_failure_event when that run is paired with an async handler run (for_async_request) and the async handler gives the logger its own async hook. The async client wrapper, dispatch_failure_handlers, the proxy-only failure dispatch and the native bridge's sync failure step for an async call mark their sync failure run as paired. Resolves LIT-9486 * fix(logging): keep the sync CustomLogger hook when no async success runs The async client wrapper skips async success logging for calls carrying a fallbacks kwarg, and the native bridge skips it for internal and fallbacks calls, yet the paired sync success run still dropped a both-hooks logger's sync hook, so that logger got no success event at all. The sync success run now suppresses the sync hook only when the async success handler was actually scheduled for the call: the client wrapper passes async_handler_runs=not is_completion_with_fallbacks, and the native host records per call whether it enqueued or deferred the async success. Tests that pinned the doubled failure hook on native aocr, messages and responses calls and on proxy-rejected requests now expect the async hook alone. The integration test adds /v1/messages and /v1/responses cells and runs the key-auth rejection on every route. * fix(logging): type dispatch_success_handlers kwargs The new async_handler_runs keyword on success_handler made the untyped **kwargs that dispatch_success_handlers forwards an unknown argument for it, which grew the reportUnknownArgumentType count past its merge-base ceiling. Typing the forwarded kwargs as Any, as _success_handler_body already does, removes those diagnostics without changing what is forwarded. * test(integration): cover CustomLogger hook counts across modes, chaos and router strategies Moves the hook count rig into tests/integration/_support/hook_counts.py and extends the cells it drives: - streamed chat, Messages and Responses requests, success and failure - loggers the async handler never reaches (sync-only overrides, sync event lists only) keep one sync hook per request on every surface outside the async flag list, key-auth rejections included - a concurrent burst while the logger's destination stalls and then fails, and a burst while a worker is killed, never give a request two hooks - least-busy keeps counting a request in flight while async speech requests finish around it, and usage-based routing v2 counts an async image request's tokens once * test(integration): Datadog and LangSmith deliver one record per async request Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local sink and checks every async surface outside the flag list: one Datadog log entry and one LangSmith run per request (two on the merge base), and never two OpenTelemetry spans, which its own emit-once guard already ensured. * fix(logging): give a both-hooks logger its sync hook when the async run never reaches it The sync hook was skipped whenever the logger sat in the async callback list, so a logger the async run never reached got no hook at all: a sibling stalled past the logging worker timeout, a batch poll that skips default logging, or an async failure run cut off. The sync run now holds the hook and the async run hands it back unless it reached the logger * fix(logging): run the held sync success hooks when the proxy drops a blocked response's success A post-call guardrail block drops the parked async success, which left the sync success hook held for it unrun. Base gave a both-hooks logger that hook on these call types, so the drop now ends the pairing and the held hook runs as before * fix(logging): run held sync hooks only after every sync run of the request has closed A held hook released by the async run could run on another thread while the sync run was still walking its other callbacks. The pairing now waits until the async run has ended and every sync run has closed, and the last sync run to close runs the held hooks itself * fix(logging): end the async run when its coroutine is dropped before it starts A logging queue that drops the async success coroutine, or a deferred success that fails to schedule, left the sync hooks held for it unrun. The enqueued async run now ends when its coroutine is discarded, and the proxy ends it when firing the deferred success raises * fix(logging): leave a still-queued async run to the exit flush The discard finalizer ran at interpreter exit before the logging worker's exit flush, so a queued request could get its held sync hook and then its async hook * test: keep the router and file-stream tests in step with main Main gives each owned proxy a private Redis by default, so the router cells share the worker's Redis to read its counters, and the file-stream stub takes the hook pairing the stream now passes * fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up A during_call guardrail that raised after the LLM call parked its deferred success skipped the proxy's flush, and async runs the exit flush never started were never ended, so a both-hooks logger lost its sync hook in both cases * fix(logging): skip the sync failure hook when the async failure run logs to the logger An async call outside is_sync_litellm_request's flag list ran both failure handlers, so a CustomLogger with both failure hooks got both. The sync run of an async request now skips log_failure_event for a logger that overrides async_log_failure_event and is registered for async failures. The four sync runs of async failures (dispatch_failure_handlers, the @client async wrapper, the proxy-only failure thread and the Rust host) pass for_async_request=True. Drops the HookPairing fallback. Successes are left to #47192, which already gives such a logger only async_log_success_event. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-20 15:19:14 -07:00
yujonglee	730db4d228	refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines (#46714) ... * refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines * refactor(rust-bridge): split and document route-specific detectors --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 15:06:25 -07:00
yujonglee	0ccae7ab35	refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates (#46750) ... * refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates * refactor(rust-bridge): simplify catalog compatibility checks --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 15:06:24 -07:00
yujonglee	a1c9fa951a	test(rust-bridge): relocate dispatch tests before routing changes (#46926) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 12:29:33 -07:00
yujonglee	d66d8ac127	refactor(rust-bridge): let each route host pick the body params (#46787) ... * refactor(rust-bridge): let each route host pick the body params The native routes projected request body params from a name list, hard-coded in Rust for Messages and read from `PARAMETERS` for Chat Completions and Responses. Each route host now answers `body_params(request)` over the keywords the call would send, so the Python side owns which params reach the provider. No behavior change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(rust-host): use read-only body parameter types Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 18:27:06 -07:00
yujonglee	55c8ec4647	refactor(rust-bridge): move the route host modules under host/routes (#46786) ... * refactor(rust-bridge): move the route host modules under host/routes Each native route's Python half lived in its own `<route>/route_host.py`, apart from the shared host modules Rust also imports by name. They now sit together in `litellm/rust_bridge/host/routes/<route>.py`, with their unit tests moved to the mirrored path. No behavior change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(rust-host): reference messages route alias Co-Authored-By: yujonglee.dev@gmail.com <yujong@berri.ai> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 18:27:06 -07:00
yujonglee	ebac947907	fix(rust-bridge): honor the configured request_timeout on native Messages (#46749) ... The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 16:50:02 -07:00
yujonglee	c0d7264e8b	feat(rust): add native OpenRouter Responses (#46000) ... The Responses route now resolves openrouter alongside openai. The OpenRouter config reuses the OpenAI request and response transforms and owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and base URL (OPENROUTER_API_BASE). Wire model ids follow Python's get_llm_provider, including the openrouter/auto native id with an explicit provider. The Python route host no longer hands litellm.openai_key to OpenRouter. The OpenRouter live test lives in tests/live/openrouter.rs and returns early when the provider key is not exported, so cargo test stays offline by default instead of hiding it behind --ignored. Co-authored-by: nate-berri <nate@berri.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-15 11:02:03 -07:00
yujonglee	dd0c023338	refactor(rust-bridge): remove the LoggerRule concept from the catalog (#46711) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 17:49:51 +00:00
yujonglee	7d79ccb7cc	test(rust-bridge): select native routes per test without LITELLM_RUST (#46620) ... * feat(rust-bridge): scope catalog rules with a contextvar Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): select native integration routes with scoped rules Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): run the Rust extension suite without LITELLM_RUST Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): use scoped catalog rules in dispatch tests Co-Authored-By: Yujong Lee <yujong@berri.ai> * refactor(rust-bridge): keep catalog rules a static global and override them once per test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 09:12:08 -07:00