56587 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
yassin	93d1c74c24	feat(control_plane): authenticated management passthrough and audit log ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 22:11:28 +00:00
yassin	b09331b394	feat(control_plane): fan-out primitive and global usage and fleet aggregates ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 22:09:01 +00:00
yassin	693c677b23	feat(control_plane): dataplane registry, credential strategies and client ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 22:06:33 +00:00
yassin	2bedefa567	feat(control_plane): service skeleton, settings, license gate, JWT auth and scope RBAC ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 21:58:52 +00:00
srik-straiker	ff997e6970	feat(guardrails): straiker stream_mode, per-phase logging and prompt-block replies (#51401) ... A Straiker verdict that blocks the prompt now comes back as a reply (HTTP 200, finish_reason content_filter) on chat completions, text completions, messages and responses, the way an answer-side block already does, instead of an HTTP 400 that some clients show as a malformed request. Routes that cannot send a reply, such as embeddings and /guardrails/apply_guardrail, keep the HTTP 400. A blocked answer on those routes is now an HTTP 400 too, where it was a 500. A backend failure still fails closed with an error. HTTP 408, 429 and 500 from Straiker now follow unreachable_fallback once retries are spent, and an attempt that times out is no longer retried. The new stream_mode option picks how a streamed response reaches the client. "buffered", the default, holds it until Straiker has checked the whole response. "streaming" sends chunks as they arrive and checks the assembled response at the end. Each v3 phase now writes its own guardrail log entry with the session, action, turn id and controls, in place of the generic entry that read "allow" even when the guardrail failed open. A claude-cli/ User-Agent is named Claude only for Claude Code entrypoints, so other clients that share the prefix are no longer misnamed.	2026-11-14 13:50:05 -08:00
mrinal-berri	b0c580d391	feat: email key owners before opted-in virtual keys expire (#51482) ... * feat: email owners before opted-in virtual keys expire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: batch expiry alert queries and use branded email template Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: cast expiry alert query bounds as UTC timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: satisfy expiry alert CI contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat: single-day key expiry alerts with global default and daily check - metadata.key_expiry_alert_days is one integer (1-365) instead of a list - general_settings.key_expiry_alert_days sets a default for every key with an expiry and owner email; key metadata overrides it - LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS defaults to 24 hours * fix: isolate key expiry alert setup and check every 6 hours - build the expiry alert manager in its own startup step that logs and returns None on bad config or setup errors, so later background jobs still register - default LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS to 6 hours * refactor: import litellm_enterprise email helpers directly in key expiry alerts Replace the import_module + Protocol module validation with plain imports, matching the key rotation hook and check_batch_cost. litellm_enterprise is absent from the type-check venv, so the imported names carry the same reasoned pyright ignore used in response_api_endpoints. * fix: apply global key expiry default to keys stored with null metadata Keys created without metadata (for example via /user/new) store JSON null, which failed metadata parsing and skipped the global key_expiry_alert_days default. Read null metadata as empty. * fix: guard litellm_enterprise import in key expiry alert job check_unsafe_enterprise_import requires enterprise imports outside the enterprise package to sit in try/except ImportError. Skip the run when the package is absent, matching the key rotation email hook. * feat: run key expiry alert scan on one pod per cycle, every 12 hours Take the shared PodLockManager lock (litellm_key_expiry_alert_job) like key rotation and expired UI session cleanup, so a single pod scans LiteLLM_VerificationToken each cycle; the per-key Redis claim still guards each email. Default LITELLM_KEY_EXPIRY_ALERT_CHECK_INTERVAL_SECONDS to 12 hours. * fix: scan for key expiry alerts when the job lock cannot be read from Redis acquire_lock returns False both when another pod holds the lock and when Redis errors. Skip the cycle only when another pod's id is on the lock key; otherwise scan without the lock, keeping the per-key claim as the duplicate guard, so a Redis blip cannot drop warnings for a 12-hour cycle. Replace the startup wiring assertion with a behaviour test against the proxy's pod lock. * fix: hold the key expiry alert lock for the whole cycle Each pod's timer fires at its own boot time plus jitter, so releasing the lock after a millisecond scan let every pod run each cycle. Take it non-reentrant for nearly the full interval and never release it, like the weekly spend report, so the first pod to fire runs the cycle and the rest skip. * fix: key the key expiry alert lock by clock window Pods' interval timers fire at their own boot time plus jitter, so a fixed lock name with a TTL shorter than the interval let a late-booted pod run twice in one interval, and a TTL equal to the interval made a pod skip its own next run. Name the lock after the clock window (int(now // interval) * interval, as file_usage_caps buckets its counters) with a one-interval TTL: the first pod in each window runs and the rest skip. Release the window if the scan raises so another pod can retry, and run anyway when the Redis circuit breaker is open. * test: keep key expiry lock tests in-process with a fixed clock Simulate the Redis outage with a disconnected in-process fakeredis server (wired like the router_redis fixture) instead of a socket to a closed port, and give the startup-built job the test clock instead of the wall clock, per tests/unit/AGENTS.md. * fix: declare the key expiry alerts service family on process_alerts process_alerts reads the window lock holder from Redis directly, so test_every_function_that_reaches_redis_directly_declares_its_family requires it to carry its family, as _process_key already does. * refactor: align key expiry alerts with existing litellm surfaces - read general_settings.key_expiry_alert_days on every run via _current_general_settings(), like the performance alert and spend capture jobs, so a UI or API edit applies without a restart; an invalid value is logged and ignored - share the event name as VIRTUAL_KEY_EXPIRING_EMAIL_EVENT in litellm/types, like MODEL_DEPRECATION_EMAIL_EVENT - default the Virtual Key Expiring email event off, like the other scheduled emails - name the Redis family for the claim data (key_expiry_alert_claims) and read the window lock under POD_LOCK_TARGET, mirroring reset_budget_job._lease_is_held - give the per-key check its own validator, run with the logging check through raise_on_invalid_key_metadata * chore: drop two comments that restate the code --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 13:47:16 -08:00
maciej-berri	36edf9c4b9	feat(messages): serve azure ai foundry messages on rust only (#51721) ... * feat(messages): resolve azure_ai keys and entra tokens like python on rust The native azure_ai Messages config now authenticates in the order Python's BaseAzureLLM._base_validate_azure_environment uses. The key comes from the call, then litellm.api_key, litellm.azure_key, AZURE_OPENAI_API_KEY and AZURE_API_KEY, each skipped only when empty, so a blank key is sent as given as Python does. Without a key the call mints an Entra ID token through litellm-auth-azure from azure_ad_token, AZURE_AD_TOKEN, tenant_id, client_id and client_secret, the caller's azure_ad_token_provider, OIDC references and litellm.enable_azure_ad_token_refresh, and sends it as a bearer. Without either the request goes out unauthenticated and Azure's 401 is mapped as Python maps it, attributed to azure_ai instead of anthropic. The bridge reads litellm.api_key, litellm.azure_key and the token refresh switch from the provider defaults snapshot, and calls the caller's token provider on the blocking pool so its exception reaches the caller as Python raises it. Approved differences that stay: a caller x-api-key or non-blank bearer replaces the operator key instead of being sent beside it. Known differences: Python's azure_username/azure_password flow has no native counterpart, and a token provider combined with an OIDC reference uses the OIDC token where Python calls the provider. * fix(messages): let a caller api-key replace the azure_ai operator key on rust A caller api-key header on an azure_ai Messages call was forwarded next to the operator's x-api-key, so Azure received both credentials. It now replaces the operator key and is sent as x-api-key, as Python does for a lowercase api-key. Approved difference: Python only matches the lowercase spelling and sends a mixed-case Api-Key beside the operator key. Rust matches any casing. * fix(messages): order azure_ai entra id flows exactly as python on rust Without a key the azure_ai config now walks Python's get_azure_ad_token order itself instead of letting litellm-auth-azure pick. Every setting is read from the call and then the secret lookup, as AZURE_API_KEY is. A static token, the caller's azure_ad_token_provider and client secret run natively. Username and password, OIDC references, os.environ/ references, an uncallable provider and token refresh go to Python's own get_azure_ad_token through the host, as Python does them, so a static token with refresh on and a provider with an OIDC reference behave as in Python, and a call Python leaves without a token goes out unauthenticated. Every azure_ai call now asks the secret lookup for the Entra ID names up front, where Python asks only when no key is set. * fix(messages): read azure_ai entra settings where python does on rust Python's get_azure_ad_token reads AZURE_AD_TOKEN through get_secret_str and AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_USERNAME, AZURE_PASSWORD and AZURE_SCOPE with os.getenv (litellm/llms/azure/common_utils.py:580-588). The azure_ai config now does the same, reading the process environment only when no key is set, and only AZURE_AD_TOKEN stays in the secret names, so a value only a secret manager holds is ignored where Python ignores it. Approved by the owner, same class as A06-A08: any caller credential header (x-api-key, api-key or Authorization other than a blank bearer, in any casing) is the only credential sent, and the Entra ID path never runs for it. That covers a caller bearer on an Entra-only deployment (A10), which Python overwrites with the deployment's token, a lowercase authorization Python joins with the Entra token, and a caller x-api-key Python sends beside the Entra token. * fix(messages): look up AZURE_AD_TOKEN only where python does on rust Python reads AZURE_AD_TOKEN through get_secret_str only when the azure_ai call has no key, no caller credential and no azure_ad_token param (litellm/llms/azure/common_utils.py:580). Messages configs can now name deferred secrets, resolved right before validate_environment and only when the config names them for that call, so a keyed call or a call carrying its own credential never asks the secret lookup for the Entra token. Approved by the owner, same class as A06-A08: a caller Authorization other than a bearer, such as Basic, replaces the azure_ai operator key instead of being sent beside it as Python does. * fix(messages): build azure_ai urls and resolve the base like python on rust The native azure_ai Messages URL builder now ports Python's azure_anthropic_messages_url. It cuts at /anthropic in the path only, so a base whose userinfo starts with "anthropic" no longer sends the key to http://anthropic/v1/messages (security regression fix). It keeps the base untrimmed and moves the slash before a query as Python does. The base resolves as Python's `api_base or AZURE_API_BASE`: a blank call base is kept, an empty one falls back to the environment, and the environment value is used as given. With neither, the call raises Python's bare ValueError ("Missing Azure API Base ..."), flagged as raised by Python and as a late rejection, so the failure log carries the resolved provider and model parameters as Python's does. * fix(messages): send bases the native parser misreads through python's client on rust A Messages api_base with credentials, no host or a port Python cannot read now goes through Python's client, as bases without an http(s) scheme or with whitespace already did. Before, reqwest read such a URL differently from httpx: for http:///d it sent the request, with the operator key, to host `d`. This is a security fix for both azure_ai and anthropic; the anthropic_base_empty_host parity case covers anthropic. For azure_ai the choice reads the base Python's URL builder reads, falling back to AZURE_API_BASE, so a padded environment base also goes through Python's client. Python reads AZURE_API_BASE once: transport_handler returns the value it read and the bridge seeds the call's secret source with it, so the native route reuses it rather than reading the secret again. _sent_url only rewrites anthropic URLs, since the azure_ai builder no longer trims. Parity: 20 azure_ai and anthropic URL cases added; 65 aspects failed on the previous extension, and the full messages parity suite now passes. * fix(messages): look up AZURE_API_BASE only where python does on rust Python reads AZURE_API_BASE only through the `or` fallback in `api_base or get_secret_str("AZURE_API_BASE")`. The native azure_ai config listed it among the secrets it always reads, so a call with its own base still looked it up through the secret manager. It is now a deferred secret, read only when the call's base is empty or absent. A blank base is kept without a lookup, as in Python. deferred_secret_names now takes the call's api_base, the same base the URL builder reads. A lookup-counting test covers a call base (0 lookups), an empty base (1) and no base (1); before this change the call-base case looked it up once. * fix(messages): filter azure_ai anthropic-beta through its row like python on rust Python's handler runs update_headers_with_filtered_beta over the azure_ai row of anthropic_beta_headers_config.json after the config adds its feature betas, so unknown and null-mapped betas are dropped whatever the header casing, the caller's and the auto-added ones alike. Rust forwarded the caller's betas and every auto-added beta as sent. A caller anthropic-beta replaces a deployment's extra_headers anthropic-beta before either lane runs, as Python does, and a Router parity test pins it with the filter applied to both. The advisor tool already matched, since the bridge hands it to Python's interceptor and its beta is null in the row, so a parity case pins it. extra_body, unknown top-level params and the -latest alias also matched and get parity cases. * fix(messages): strip beta-gated system fields under drop_params like python on rust Python's transform runs strip_beta_gated_native_request with the final, filtered headers, so under drop_params a mid-conversation system turn loses the clear_at, output_config and tool_addition/tool_removal fields no sent beta unlocks, and a turn left empty is dropped unless that would end the conversation on an assistant turn. _normalize_system_role_messages strips a turn it converts to a user turn again with no betas at all. Rust sent those fields as they came. azure_ai now picks its betas from the request as sent and runs the strip, system-role normalization and the cache_control.scope strip in a new finalize_request step, once the final header is known. Vertex gets the conversion-time strip too, since it shares the normalization helper. The strips and the speed drop log Python's warnings word for word. A non-object output_format under drop_params is still popped after the headers were built, so structured-outputs-2025-11-13 is sent without it, as Python does. * fix(messages): send the system prompt as text blocks like python on rust Python's _normalize_system_role_messages always rebuilds the top-level system as a list of content blocks, so a string system reaches azure_ai and Vertex as one text block, and the success log, which only lists a string system as a message, logs the messages as sent. Rust kept the string unless a leading system turn was hoisted. Python's billing strip in transform_anthropic_messages_request also pops a system prompt it leaves empty, an empty string included, as Python does. The shared helper now does the same for every config that strips billing metadata: azure_ai, vertex_ai, deepseek, minimax, tencent and edenai, each with its own test. * fix(messages): keep non-claude azure_ai messages passthroughs on python A deployment whose provider has no Messages config of its own, like a non-Claude azure_ai model, but lists /v1/messages in model_info.supported_endpoints is forwarded untranslated by Python's OpenAILikeAnthropicMessagesConfig, with a bearer key to <base>/v1/messages. Dispatch gave it the MESSAGES target, so a rule that serves azure_ai on Rust would hand it to the native Anthropic config, which sends x-api-key to /anthropic/v1/messages and breaks the success log. Dispatch now targets MESSAGES only when the provider's own config serves the call, and MESSAGES_PASSTHROUGH for the opt-in passthrough. The catalog keeps that target on Python next to the chat and Responses bridges, so once azure_ai is Rust-required these calls resolve to Python and the Rust-only refusal does not fire for them. Proxy token counting treats both targets as before. The parity forced rule now serves a provider natively only for the MESSAGES target, as a Rust-required rule below the target exclusions does. A pre-request hook that sets a non-Claude model raises functools.partial's TypeError on both lanes, as on main, and a parity case pins it. * fix(messages): raise azure_ai transport and provider errors like python on rust The bridge mapped a native failure with the whole call as exception_type's extra_kwargs, so litellm_debug_info gained an API Base line from the named api_base. Python's entry passes only its **kwargs there, so its debug info names the model alone. The bridge now passes the same kwargs, for anthropic and azure_ai alike, and a decoding failure on a stream carries the deployment's debug info as Python's stream iterator attaches it. A connect failure to an azure_ai base read from AZURE_API_BASE kept the native "could not reach the provider" text, since the call had no api_base. The host now hands Python's failure mapping the secrets it read to send the call, so the aiohttp and httpx texts name the base the call went to, without reading AZURE_API_BASE again. A key or header value with a control character, like a deployment key with a trailing newline, failed natively as APIConnectionError with the model stripped of its provider. It is now the InternalServerError 500 Python's transport raises, with aiohttp's "Forbidden control character detected in headers" text and the model as called. On the httpx transport Python's text quotes the header value, which may be the key, so Rust keeps its own text there. The parity outcome now compares litellm_debug_info. The caller client's ReadTimeout from a stalled stream still has none on Rust and is recorded as a gap. * fix(messages): send a caller azure_ai api-key without an operator key like python on rust With no deployment key, Rust renamed a caller api-key header in any casing to x-api-key. Python's AzureAnthropicMessagesConfig renames only a header spelled exactly api-key and sends any other casing, like API-Key, as given. Rust now does the same when no operator key would be sent. With an operator key the approved behaviour stays, and a caller api-key in any casing replaces it as x-api-key. * fix(auth-azure): retry entra token posts like python's azure-identity A client secret or workload identity token endpoint that keeps answering 500 made the native path POST 9 times over about 80 s, because azure_identity 1.0 uses the SDK's default exponential retry (8 retries, 200 ms doubling, 60 s cap). Python's azure-identity 1.25.2 posts 4 times in about 5 s and then raises, so the Rust credential now carries a custom retry policy with the same numbers. Python sources in the wt-auth venv - azure/identity/_internal/pipeline.py:67 builds the token pipeline with azure-core's RetryPolicy(**kwargs) and no overrides - azure/identity/_internal/msal_client.py:98 (client secret) and aad_client.py:77 (client assertion, used by workload identity) run the token POST with retry_on_methods=["POST"] - azure/core/pipeline/policies/_retry.py (azure-core 1.41.0) lines 77-95 give retry_status=3, retry_backoff_factor=0.8, retry_backoff_max=120, retried codes 408, 429, 500, 502, 503, 504 and 506-998. Lines 127-143 sleep 0 before the first retry and factor * 2^(n-1) after that, so 0 s, 1.6 s, 3.2 s. Lines 455-509 sleep exactly Retry-After (then retry-after-ms, x-ms-retry-after-ms) instead of the backoff when it is present and non-zero azure_core only hands the policy a Retry-After value for 429 and 503, so a Retry-After on another status still falls back to the backoff. Managed identity keeps the SDK default since Python uses a separate policy there, and client assertion keeps it because Python mints OIDC tokens through litellm's own HTTP client. A 401 invalid_client is still one POST. * fix(auth-azure): post an oidc token exchange once like python Python exchanges an OIDC assertion through litellm's own HTTPHandler (_exchange_oidc_token in litellm/llms/azure/common_utils.py), a single client.post with no retry, so a 503 is one POST. The native client assertion credential used azure_identity's default retry (9 POSTs over about 80 s) and now runs with RetryOptions::none(). * fix(auth-azure): honour retry-after on entra token posts like python azure_core runs a custom RetryPolicy through its own loop, which only reads Retry-After for 429 and 503 and never retries a status outside the policy's list. Python's azure-core RetryPolicy (1.41.0, azure/core/pipeline/policies/_retry.py) retries any 4xx or 5xx that carries Retry-After (is_retry, lines 210-242) and waits exactly the Retry-After, retry-after-ms or x-ms-retry-after-ms value when it is non-zero (get_retry_after in _utils.py:79-96, sleep at lines 455-509). The client secret and workload identity credentials now run with the SDK retry turned off and a per-call policy that follows Python's loop. It keeps separate budgets of 3 for status, connect and read retries (increment and is_exhausted, lines 265-330), backs off 0 s, then 0.8 * 2^(n-1) capped at 120 s, and retries 408, 429, 500, 502, 503, 504 and 506-998 for the token POST. Client assertion still posts once and managed identity keeps the SDK default. A Retry-After that is neither a number nor an HTTP date now fails the token request after the first POST with "Invalid HTTP date", as Python does. azure-core's _parse_http_date (_utils.py:48-57) raises ValueError from the sleep before the retry, azure-identity's wrap_exceptions (decorators.py:63-80) turns it into ClientAuthenticationError "Authentication failed: Invalid HTTP date", and get_azure_ad_token (common_utils.py:663) raises RuntimeError "Failed to get Azure AD token: Authentication failed: Invalid HTTP date". * fix(messages): raise python's entra token errors for native azure_ai flows On the Messages route every Entra token failure now reaches the caller as Python's get_azure_ad_token and anthropic_messages raise it, with the same class, status_code, llm_provider, message and log lines. A native client secret flow keeps the token endpoint's final answer (status, headers, body) in a typed EntraTokenFailure from the azure-identity retry policy. The bridge hands it to the Messages host, which replays it through azure-identity's own ClientSecretCredential over a transport that returns that answer, so MSAL and azure-core read it exactly as they read it off the wire. That covers a rejected secret (EntraRejectionError 401 with the first AADSTS sentence and GUIDs redacted), an exhausted 5xx ("HTTP Error: 500"), a malformed Retry-After ("Invalid HTTP date"), non-JSON and odd JSON bodies, including the content-type dependent decode error. A transport failure has no answer to replay and keeps Rust's own reason after "Authentication failed:", so its text differs from requests' text. The caller's azure_ad_token_provider goes through the helper get_azure_ad_token now shares (azure_ad_token_failure), so a listed AADSTS rejection is an AuthenticationError 401 instead of a RuntimeError, and a non-string token raises the same TypeError. Both log Python's "Error calling Azure AD token provider" and "returned non-string value" lines. Every Python exception raised while getting the call's credential, including the host-minted username/password, OIDC and workload identity flows, now goes through Python's handler rule: a BaseLLMException is mapped by exception_type (AuthenticationError 401, BadRequestError 422 with llm_provider azure_ai), anything else reaches the caller as raised. The Retry-After date parser stays RFC 7231 only. Python's email.utils.parsedate_tz also accepts dates without a weekday, two-digit years, numeric zones and missing seconds, which Rust rejects as an invalid HTTP date and Python waits on. A NaN or infinite number of seconds now falls back to the backoff, where Python sleeps 0 for NaN and fails on infinity. * fix(auth-azure): wait on retry-after and report connect failures like python The Entra token retry policy now reads Retry-After the way azure-core 1.41.0 does. Seconds go through Python's float rules, including underscores between digits, and anything else through a port of email.utils.parsedate_tz, so a date without a weekday, a two-digit year, a numeric or named zone, a zone glued to the time, missing seconds, a dotted time, month before day and RFC 850 dates all wait as Python waits. A date datetime() rejects (day out of range, an offset of a day or more) still fails the request after one POST, as do unparseable values. A wait that time.sleep cannot take (infinity, or 2^63 ns and past) now fails after one POST as Python does. The host replays that answer with a sleep that still calls time.sleep for such a wait, so the caller gets "Authentication failed: timestamp out of range for platform time_t". A transport failure keeps the token endpoint's host and port. A direct connect that fails with an OS error, or a failed name lookup glibc words, reaches Python as urllib3 2 words it after "Authentication failed:", for example "HTTPSConnection(host='login.microsoftonline.com', port=443): Failed to establish a new connection: [Errno 111] Connection refused". Python hits it on MSAL's discovery GET to the same host. Failures through a proxy, TLS failures, timeouts and resets while reading keep Rust's own text, since urllib3's wording for them depends on details Rust does not expose. * fix(messages): treat a contentless system message as a system turn like python on rust Python's _normalize_system_role_messages reads only the role, so a role system message sent without content or with null content is still a system turn. A leading one is hoisted and adds nothing to the top-level system, and a later one on a model without mid-conversation system support becomes the operator note user turn, placed after a tool result it sits before and dropped under drop_params when its gated field is stripped. Rust failed to parse the message and sent it raw, which also showed up in the logged messages. A model that accepts the role still gets it as sent. * fix(messages): let the provider filter a non-ascii anthropic-beta like python on rust Python merges every anthropic-beta spelling into one header and the provider config filters it before the HTTP client encodes it, so on azure_ai an unknown non-ASCII beta is dropped and the call goes out, with or without drop_params. Rust held the header back as unencodable before the beta filter ran and failed the call with the ascii codec error. The beta header now reaches the provider filter, and the wire check fails the call only when a value Python's client cannot encode survives it. A value with a lone surrogate still fails up front since it cannot reach Rust as text. * fix(messages): log nothing for a stream the caller never read like python on rust Python's Messages stream iterator only starts on the first read, so a stream the caller closes or drops unread bills nothing and logs nothing. Rust billed the empty stream on close and teardown, which logged "Unable to build complete streaming response" and the read() error from the success handler. * fix(messages): log a traceback for failures python raises before the send on rust Python raises its pre-flight errors, such as the missing azure_ai api_base ValueError, the missing key AuthenticationError and the Entra token provider failures, through Python frames, so the failure log and the OTel span carry a stack trace. Rust built those exceptions without raising them, so error_information.traceback was empty. The host now raises them through a Python frame before handing them back. * fix(messages): return python's streaming response types from native messages streams Python's Messages handler returns an AnthropicMessagesStreamingResponse for a stream, so isinstance checks, hidden_params and aclose behave the same for every caller. The native async entrypoint returned the bare execution AsyncStream. It now wraps the stream the way the deployment success hook already did, sharing the stream's hidden params. The entrypoint is shared, so this moves anthropic as well as azure_ai toward Python. With a response cache configured Python hands back its caching handler's types too. A streamed cache hit is now the CachedAnthropicMessagesStreamIterator Python's handler builds from the cached events, stamped with the metadata @client stamps, and the native execution closes unread so only that iterator logs the hit, as Python does. A miss is wrapped in the AnthropicMessagesStreamCacheWriter Python returns, marked persisted since Rust stores the stream itself. The parity suite now compares the stream type for every streamed case. * fix(messages): read unparsed messages in beta detection and gated strips like python on rust Python reads beta triggers and gated fields off any message mapping, while Rust only looked at messages that parsed. A role system message sent without content, or any message with a content block Rust cannot type, was skipped: - a message carrying output_config now adds per-turn-control-2026-07-01 - a system message with a tool_reference addition or removal now adds mid-conversation-tool-changes-2026-07-01 - a signed compaction block now adds compact-2026-09-04 - under drop_params an unparsed system message loses its gated clear_at, output_config and tool changes, is dropped once emptied, and the warning names the strip - an unparsed assistant message counts as the last assistant turn when deciding whether a strip would leave a prefill - an unparsed message loses cache_control.scope on its content blocks * fix(messages): rewrite and normalize unparsed messages and system prompts like python on rust Python's sanitizer, advisor and encrypted-reasoning strips and system normalization read any message mapping, while Rust skipped every message or system block it could not type: - an unparsed message now gets the sanitizer's rules: a text block whose text is not a non-blank string and an empty unsigned thinking block go, a message left with no blocks is dropped, tool ids are sanitized, replayed web search results are flattened and provider_specific_fields dropped; a message the rewrite leaves parseable is parsed again - an unparsed assistant turn loses advisor exchanges, and an unparsed message loses bridge-tagged reasoning blocks - a kept redacted_thinking block gets Claude's own data back from a Bedrock Converse base64 wrapper, parsed or not - system is kept as sent when it is a list holding a block that does not parse, or an object; billing blocks and cache_control.scope are still stripped from such a list - a leading system turn whose content does not parse is hoisted as sent, and a later one becomes the operator note user turn as sent, placed after a tool result turn that did not parse either * fix(messages): keep anthropic's API Base debug line on rust failures Mapping a native failure with only the Python entry's **kwargs dropped the named api_base from exception_type's input. That matches azure_ai, whose Python entry maps with **kwargs alone, but it removed the API Base line that main's Rust-only anthropic route has always put in litellm_debug_info when the caller passes api_base. Python gives no rule that tells the two apart, because the legacy async entry maps both providers the same way. The difference is that anthropic has no Python path, so main's native route is its contract. The bridge now maps the whole call for anthropic and the entry kwargs for every other provider. The parity harness still uses the legacy Python entry as anthropic's reference, so it now expects the API Base line main's anthropic route adds. New cases cover an anthropic call on an azure_ai model failing with api_base, unary and stream. * fix(messages): keep anthropic stream types and header errors as main while azure_ai matches python By the owner's ruling anthropic keeps main's behaviour for two things the azure_ai port had changed for every native provider. A native async stream returns Python's AnthropicMessagesStreamingResponse, and with a response cache the CachedAnthropicMessagesStreamIterator or AnthropicMessagesStreamCacheWriter, only for azure_ai. The host asks Python for those types only when the call resolves to azure_ai, so anthropic gets main's execution AsyncStream again, for a cache hit or miss too. A header value the native client cannot send, like a key with a control character, is Python's InternalServerError with aiohttp's "Forbidden control character" text only for azure_ai. Anthropic raises main's APIConnectionError "invalid request: failed to parse header value" with the bug report link again. The parity stream_type aspect compares azure_ai with Python and expects main's AsyncStream for anthropic, in the same way the outcome aspect already expects main's API Base debug line. New azure_ai cache hit and miss stream cases check the cached types. * fix(messages): keep anthropic as main for azure_ai parity side effects By the owner's ruling anthropic, which has no Python path, keeps main's native route; the azure_ai port's Python-parity changes now apply to azure_ai only. - An unread stream: only azure_ai's starts on the first read and bills nothing when closed or dropped unread. The logging adapter learns the provider from the execution facts; anthropic bills it as main does. - Failures before the send: the missing-key AuthenticationError and the unmapped builtin errors are raised through a Python frame only for azure_ai; anthropic hands them back unraised, without a traceback, as main does. - A stream's decoding failure carries the deployment's debug info only for azure_ai; anthropic raises it bare. - A non-ASCII anthropic-beta is left to the provider filter only for azure_ai. Anthropic holds it back before the beta merge as main does, so the error names it ahead of any later unencodable header. - Unparsed messages: the shared shaping reads messages and system prompts Rust could not type only when the provider asks for it (UnparsedMessages::Read), which only the azure_ai config does. Elsewhere the sanitizer, advisor and encrypted-reasoning strips leave such messages as sent, their beta triggers add no beta, a redacted_thinking Converse wrapper is kept as sent, and a system prompt that does not parse is main's "untagged enum SystemPrompt" request error from messages_body. Each item has an anthropic test pinned to main's behaviour that fails on the previous commit, beside the azure_ai cases comparing with Python. * test(messages): pin the malformed api_base fix for anthropic and azure_ai d1efdd2683 sends a base the native URL parser reads otherwise than httpx and aiohttp through Python's client. Parity cases now cover, for anthropic and azure_ai alike, the three shapes that sent the operator key to an unintended host natively (an empty host, credentials in the URL, userinfo that reads like a trusted host) and the two port shapes, asserting that nothing reaches the upstream and that the caller gets Python's exception. They fail with the check disabled. The rust_bridge AGENTS.md states the rule. * refactor(messages): serve azure_ai messages on rust only and delete its python path Provider azure_ai on /v1/messages is now RUST_REQUIRED next to anthropic, so a Claude deployment whose AzureAnthropicMessagesConfig serves the call runs on Rust without LITELLM_RUST or litellm.rust(True) and never falls back to Python. The PYTHON_ONLY targets rule still sits above it, so a non-Claude azure_ai deployment that opts into /v1/messages (MESSAGES_PASSTHROUGH) and an azure_ai call bridged to Chat Completions or Responses stay on Python. A raising detector on a Claude azure_ai call gets the same UnsupportedParamsError as anthropic. MESSAGES_PYTHON_FEATURES is empty, so no feature refuses today. Deleted Python code that only served azure_ai Messages: - AzureAnthropicMessagesConfig._remove_scope_from_cache_control, a copy of the AnthropicMessagesConfig method it overrode. Its only caller is the azure transform_anthropic_messages_request (grep "_remove_scope_from_cache_control" litellm/: azure, databricks and vertex call self., the base defines it) Kept, because the parity harness still runs every azure_ai case through the Python handler (dispatch._PYTHON_AMESSAGES) as its baseline, and removing them would send that baseline to api.anthropic.com with anthropic auth: validate_anthropic_messages_environment, get_complete_url, transform_anthropic_messages_request and should_strip_billing_metadata on AzureAnthropicMessagesConfig (grep: only async_anthropic_messages_handler in llm_http_handler.py calls them, reached from the Python fallback and the harness). The Foundry beta-row branch was already removed with anthropic, and resolve_beta_provider's Foundry row now serves only anthropic chat. Count tokens has its own config and the Rust host reaches no deleted method. The parity harness drops the rust_providers forcing, so azure_ai cases run through the shipped rules and assert RUST_REQUIRED like anthropic. The azure count tokens beta test sends its inference call through a caller client, as Rust now serves it. * test(messages): cover the azure_ai agentic follow-up deployment on rust The Rust host builds an agentic follow-up's kwargs with _agentic_hook_kwargs, so the follow-up must reach the azure_ai deployment's own api_base with its key. This replaces the deleted LIT-5418 test, which drove the Python handler. Dropping api_base from that helper fails it with Missing Azure API Base, and dropping api_key sends the follow-up without x-api-key. * fix(auth-azure): retry Entra ID token posts like Python only for azure_ai Messages The azure-identity retry policy and the single OIDC post now apply only when a consumer asks for them through AzureAuthInputs.token_retry. The azure_ai Messages config asks for it, while Key Vault, Redis, blob cache, OCR and the anthropic federation keep the SDK default as on main. The policy is part of the credential cache key so consumers sharing a client never share a policy. * fix(messages): keep azure_ai without a key as main when no Python host is wired in Entra ID auth for an azure_ai call without a key now runs only when the route has a Python host. The Rust gateway has none, so such a deployment fails with the local MissingApiKey main raised instead of going out unauthenticated or failing on an Azure AD token without a host, and a deployment string named callable is never read as a caller token provider there. * fix(messages): log the dropped speed warning only for azure_ai The shared drop_unsupported_params helper logged Python's speed warning for every Messages provider. Configs now opt in through logs_dropped_params, which only azure_ai sets, so anthropic and the other providers log as main. * test(messages): drop the azure_ai default headers structure test and date pinned error texts default_headers_match_python only restated a constant. The MSAL, azure-core and CPython texts the replay tests pin now carry the date they were checked. * refactor(messages): name the anthropic error-mapping set for what it is _RUST_ONLY_PROVIDERS held only anthropic, the provider that keeps main's native error mapping, while azure_ai is Rust-only too. It is now _MAIN_NATIVE_ERROR_PROVIDERS, and the docstrings no longer call azure_ai a provider with a Python path. * docs(messages): keep azure_ai's python request code as the live parity reference azure_ai Messages stays Rust-required, but its Python request code is kept so the parity suite runs the live Python leg, as anthropic, Vertex and Bedrock do. * fix(messages): send azure_ai caller and operator credentials exactly as python does on rust The owner reversed the earlier approved difference on 2026-11-13 (Maciej). For azure_ai on /v1/messages every caller, operator and Entra ID credential combination now sends the header set main's Python sends, quirks included. Python builds a dict keyed by exact name, so a caller api-key spelled exactly so stops the operator key and the Entra ID path, the operator key otherwise goes in as api-key and the Entra ID token as Authorization beside or over the caller's headers, api-key is renamed to x-api-key unless x-api-key is spelled exactly so, and the HTTP client joins names that differ only in case. Ledger rows that change, all previously "Rust sends only the caller credential". A06 caller x-api-key now goes out beside the operator api-key. A07 caller X-Api-Key is joined with the operator key in one x-api-key. A08 caller bearer now goes out beside the operator x-api-key. A09b caller API-Key (other casing) goes out as api-key beside the operator x-api-key. A10 a caller Authorization bearer on an Entra-only deployment is overwritten by the minted Entra ID token. A10b a caller lowercase authorization is joined with the Entra ID bearer. A10c a caller x-api-key goes out beside the Entra ID bearer. A caller non-bearer Authorization (Basic) now goes out beside the operator key. A09 (an exact api-key replaces the operator key as x-api-key) is unchanged. AZURE_AD_TOKEN is now read wherever Python reads it, which is any call without a key, an exact caller api-key or an azure_ad_token param. The parity harness gaps for these rows are removed so they run as normal cases. * fix(messages): log the azure_ai call flag on failures before the send Python's Messages handler sets the aanthropic_messages flag before it gets the Entra ID credential or refuses an unsupported param, so those failures log it. The native host now flags azure_ai calls the same way when a token request fails or the request is refused, and anthropic keeps main's unflagged failure. * fix(auth-azure): fetch azure_ai entra tokens with python's proxy and ca settings Python's azure-identity sends the client secret token request through requests, which reads https_proxy or all_proxy unless its no_proxy matching skips the host, trusts REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE or certifi's Mozilla roots, never SSL_CERT_FILE, and times out after 300 seconds. azure_core's reqwest client read HTTPS_PROXY before https_proxy, matched NO_PROXY its own way, trusted the system store and SSL_CERT_FILE and timed out after 20 and 60 seconds. An explicit token_transport input on the auth request selects a transport that applies requests' rules over a litellm-http client. Only azure_ai Messages asks for it, so Key Vault, Azure Redis, blob cache and OCR keep azure_core's transport. * fix(messages): word azure_ai entra transport failures as python's requests does Python's azure-identity reports a failed client secret token request with requests' own text after "Authentication failed:", while the native flow kept reqwest's text for anything but a direct connect or name lookup failure. The requests transport now tags each failed send with the proxy it used, and classifies what reqwest, hyper and rustls observed into the failures requests words the same way every time: an unreachable or unresolvable proxy, a 407 answer to CONNECT, a proxy or server closing or resetting the connection, an EOF in the TLS handshake, an expired, not yet valid or misnamed certificate, and the 300 second read timeout. The host builds each text from the urllib3, http.client and ssl exceptions Python raises for it, so build details such as the _ssl.c line come from the running interpreter. A missing REQUESTS_CA_BUNDLE path now raises requests' own text too. * fix(messages): look up AZURE_AD_TOKEN for azure_ai only when a Python host is wired in The Rust gateway has no Python host, so a key-less azure_ai Messages call there fails with main's local MissingApiKey before any Entra ID flow. It no longer asks the secret source for AZURE_AD_TOKEN first, which main never did. Calls with a Python host read it where Python does, as before. * fix(messages): forward a caller credential on a key-less azure_ai call without a Python host as main Without a key or a Python host, as in the Rust gateway, main's key-only azure_ai check forwarded a caller x-api-key or bearer and failed with MissingApiKey otherwise. The branch made every such call a missing key, so the gateway's entra_bearer_without_key case failed; it now forwards those credentials again. Calls with a Python host are unchanged. * fix(messages): time an azure_ai exchange through python's client once on rust A call the Rust host sends through Python's client, such as a malformed or padded api_base, is already timed by that client's track_llm_api_timing post. The native logging adapter timed the same exchange again, so the Rust lane reported two LLM service events and recorded the window twice where Python reports one. The route host now tells the adapter when it sends through its own client, and for azure_ai the adapter leaves the timing to it. Anthropic keeps main's behaviour. * fix(messages): look up advisor ids in a set like python on rust * test(messages): cover the native cache, stream and host failure paths	2026-11-14 13:40:38 -08:00
tavren27	9063ba6929	fix(proxy): stop wildcard aliases over unknown orgs listing every catalog id (#51805) ... * fix(proxy): stop listing catalog ids under an org the catalog lacks for every provider whose path is its model id * fix(proxy): keep listing Fireworks router and Azure model_router wildcards The catalog cannot price the names these routes call, so the strict org check dropped callable ids. They now list as before, and every exempt provider has a test row that fails if it leaves the list * fix(proxy): check unknown orgs beneath router routes and keep firerouter pools * fix(proxy): keep listing Fireworks routers/firerouter pools and Azure AI speech wildcards * fix(proxy): check unknown orgs under OpenAI and Cohere and drop unreachable Fireworks router ids * fix(proxy): keep the full wildcard listing for deployments behind a custom api_base * fix(proxy): count a provider base URL set in the env or litellm.api_base as a custom endpoint * fix(proxy): match a gateway env var on any part of the provider name * fix(proxy): count a provider's off-pattern base URL env vars as a gateway * test(proxy): cover every off-pattern base URL env var in the gateway table --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 16:34:59 -05:00
norkelbyte	fd6e4fb6a5	fix(sdk): stop global model_fallbacks from recursing on every completion call (#51817) ... * fix(sdk): stop global model_fallbacks from recursing on every completion call With litellm.model_fallbacks set, acompletion and completion read the global chain on every call, including the calls the fallback runner itself makes for each attempt. Each attempt started a new fallback run, so the calls grew into a tree until Python hit RecursionError and no provider was ever reached. The fallback runner now marks every attempt with an internal _litellm_ kwarg, and one resolver in fallback_utils decides the chain for a call. A marked attempt gets no chain. Otherwise a per-call fallbacks value wins, including an empty list to opt out, and the global list is used only when the call sets none. An empty chain skips the runner. The resolved chain is what the runner gets, so completion's sync path passes it explicitly. The marker is dropped from provider params and cache keys like every other _litellm_ kwarg * fix(fallbacks): keep each provider's credentials, typed errors, sync streams and one success log A fallback to another provider drops the original request's credentials and endpoint, and a fallback to another model drops per-call custom pricing. When every attempt fails, the original model's typed error is re-raised. The sync runner calls litellm.completion directly so sync streams work, and the sync wrapper leaves the success log to the attempt inside the runner. Health checks never fall back, and every attempt keeps the caller's litellm_call_id. * fix(fallbacks): set up fallbacks only when tried and keep health checks off the global list in every mode Each fallback is now built only when the runner reaches it, using the provider lookup that never starts a ChatGPT or GitHub Copilot device login, so an unused fallback cannot block a healthy call Health checks run inside a scope that turns the global model_fallbacks off, which also covers modes bridged to chat. The per-site fallbacks=() overrides are removed The sync logging test now forces the inner and outer success handlers to overlap with a barrier instead of sleeping * fix(fallbacks): keep guardrails on sync attempts and keep the global list out of Router calls The sync fallback runner now runs the deployment pre and post call hooks around every attempt, so guardrails a sync call names still run on each model, and it skips the post hook for streams like the async wrapper does Router deployment calls run with the global list turned off, so on the proxy the Router owns fallbacks and a fallback answer is never priced or logged as the failed deployment A fallback to another provider now also drops the request's headers and litellm_credential_name, and an original model whose provider cannot be looked up counts as a switch When every model fails the caller gets main's documented "All fallback attempts failed" error again * fix(fallbacks): keep custom provider settings for unprefixed fallbacks and never cache a rejected answer An unprefixed fallback whose name resolves to no provider on its own now inherits the request's custom_llm_provider, so a fallback on the same OpenAI-compatible server keeps its api_base and key as it did on main Sync fallback attempts no longer write to the cache, since their guardrail runs after the inner call returns. The outer call still caches the answer that passed, so a rejected answer never lands in the cache * fix(fallbacks): run a sync attempt's output guardrail before it is cached or logged The sync runner ran the post-call guardrail after litellm.completion returned, so the attempt's success log already held the raw answer and a rejected answer was logged as a success. The sync wrapper now runs the guardrail for a fallback attempt at the same point the async wrapper does, before the cache write and the success log, which also replaces the no-store workaround * fix(fallbacks): keep search grants, deployment hooks and logs intact Router deployment calls wrap their handler to turn the global list off, which hid asearch from the search fallback grant check. The check now unwraps the handler down to asearch A failed sync fallback attempt runs the post-call failure deployment hook again, as main did through its async detour atext_completion through the global list uses the async runner so it logs one success, and check_valid_key turns the global list off so a fallback answer never makes a bad key look valid * fix(proxy): keep the global fallback list off on direct SDK calls Pass-through models and routes the Router does not serve reach litellm directly, so the global list ran there with no check that the caller's key may use the fallback model. Those calls now turn the global list off, as Router deployment calls already do * fix(fallbacks): price a fallback by its own model and skip the proxy global list A fallback to another model kept the original call's base_model, so its answer was billed at the original model's rates, as on main. base_model now drops with the other per-call pricing params The proxy turns the global fallback list off on every model call, so litellm_settings.model_fallbacks now logs a startup warning that points to router_settings.fallbacks and is not applied --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 16:28:38 -05:00
devin-ai-integration[bot]	84f975d128	test(anthropic): expect adaptive thinking temperature dropping (#51859) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 21:20:11 +00:00
norkelbyte	85a3e566f3	fix(router)!: name the request size when usage-based routing refuses an over-tpm request (#51838) ... * fix(router)!: name the request size when usage-based routing refuses an over-tpm request usage-based-routing-v2 answered "No deployments available" with no usage shown when one request alone was over every deployment's tpm, and a single-deployment group held the caller for 3 x 60 seconds on retries that could never clear it. The router now raises RequestOverDeploymentTpmError, a RateLimitError that names the estimated tokens and the largest tpm limit, carries no retry-after, skips retries, and still reaches group fallbacks. * fix(router): count cooling deployments before refusing an over-tpm request, keep the Prometheus label The over-tpm refusal now also reads every deployment in the model group, so a request that fits a cooling deployment keeps the old retryable 429. RequestOverDeploymentTpmError keeps the RateLimitError exception_class label in Prometheus. * fix(router): count only cooling deployments the request can still reach, and keep the logged error class The over-tpm refusal now reads cooling deployments from the caller's pool, kept to the ones team, region, and tag routing would still send the request to, so team models and tag-filtered requests get the right answer. Logs, OTEL, and Datadog keep naming the refusal RateLimitError like Prometheus does, and a tpm written as a number string counts as that number * fix(router): keep the sync over-tpm refusal a ValueError and count every deployment the caller may use The sync Router raises SyncRequestOverDeploymentTpmError, which is still a ValueError, so existing except ValueError handlers keep catching it. The never-retryable refusal now counts every deployment in the caller's pool that region and tag routing allow, so a deployment left out of this pick by a budget, health check, order, or pin keeps the retryable 429 * fix(router): keep retry records and trace events naming the over-tpm refusal RateLimitError Fallback records in previous_models and the litellm.routing.retry trace event now read the error's logged class name, as the logging payload already did, so they keep RateLimitError for the over-tpm refusal as they did before this change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 16:16:21 -05:00
yucheng-berri	580203cce6	fix(otel): send the Langfuse environment, release and version on OTel v2 (#51385) ... * fix(otel): send the Langfuse environment, release and version on OTel v2 * fix(otel): keep the Langfuse environment on the Langfuse exporter only * fix(otel): annotate the exporter resource attributes as a read-only mapping * fix(otel): shed the Langfuse release and version first at the attribute limit * fix(otel): keep base attributes when the Langfuse release and version do not fit - the model span sheds release and version before the identity fields - the request root takes release and version only when they fit under its attribute limit - an exporter routed to a tenant's credentials drops the proxy's Langfuse environment - a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT * test(otel): annotate the new attribute-limit and routing tests	2026-11-14 11:40:35 -08:00
kirti22jain	73e17cba74	feat(proxy): add per-model project batch enqueued token limits (#51608) ... * feat(proxy): add per-model project batch enqueued token limits Enqueued batch token limits only read key and team metadata, so a deployment that gives each application its own project under a shared team cannot cap one application's queued batch tokens for a given model. The team limit is shared by every project in it, and JWT or custom-auth callers get a new key scope with every token. Read model_batch_enqueued_token_limit from project metadata, keyed by model like model_tpm_limit. Each limited model group the batch runs is reserved against only its own rows' tokens, rows that name no model count toward the routing model, and the reservation records each scope's charge so the terminal-state refund releases exactly that. Only proxy admins may set it, on project create and update as well as on keys and teams, so a team admin cannot lift their own project's cap. * fix(proxy): charge the routing model and keep daily caps for project batch limits Rows can name any model while the batch runs on the routing deployment, for example LiteLLM-executed batches replace every row's model with the run model, so charging only row models let a batch skip a capped deployment's per-model limit. Charge the routing model the whole batch as well as each other limited row model its own rows' tokens. An enqueued-token reservation replaced every batch counter, so a project limit also skipped the key, team and end-user daily token caps. Keep checking those daily caps after reserving, and refund the reservation if one rejects the batch. * fix(proxy): keep per-minute checks for batches a project allowance does not cover A per-model project allowance replaced every per-minute batch check as soon as any row's model had one. A batch routed to an unlimited model could add a single tiny row naming a limited model and skip the routing model's RPM/TPM and project ITPM/OTPM quotas, although every row runs on the routing model. Waive those checks only when the reservation covers the whole batch, through a key or team allowance or a project allowance on the routing model; otherwise reserve the covered models and run the normal checks too. * test(proxy): cover the project per-model batch cap through custom auth on a live proxy * fix(proxy): keep key and team batch allowances authoritative and harden project limits Key and team enqueued allowances again replace the other batch checks. A project allowance on the routing model keeps only the daily caps. Team admins can only be blocked from the project key, invalid project maps answer 400, the edit modal keeps stored limits, aliases share their tightest limit, file-bound batches charge only the routing model, deployment ids resolve to their model group, and reservation records keep project scopes apart so older pods still refund key and team scopes. * fix(proxy): charge row models for model-bound batch uploads Only a managed-file upload rewrites rows to the deployment model. A model-bound upload keeps each row's model, so its rows still count toward the project caps of the models they name. * fix(proxy): skip per-minute batch limits only when the project caps every model in the batch A project cap on the routing model alone used to switch off the project's per-minute input and output limits for the other models a batch file names. The skip now needs a project cap on every one of those models. -latest names are charged under the cap of the model they point to, the limit map only takes JSON integers, and a /project/update that sends only top-level fields keeps the stored metadata instead of replacing it. * fix(proxy): lift per-minute limits off managed batches under a project cap and edit the cap in the Admin UI * fix(ui): keep the project batch cap read-only for team admins and require a whole number above 0 * fix(ui): narrow project metadata directly when keeping the stored batch cap * fix(proxy): charge dated snapshot rows and header or query routed batches to project caps --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 14:36:53 -05:00
paveln83	9fba44d300	fix(openai): cap and bill web search rounds on Responses-pinned chat completions (#51802) ... * fix(openai): cap web search agentic loops with max_agentic_loops on OpenAI chat completions The OpenAI chat dispatcher ran every custom logger's chat agentic loop without a depth check, and the web search follow-up never carried the loop depth, so max_agentic_loops was ignored on openai/ deployments. The dispatcher now skips loggers the shared chat completion loop in litellm.acompletion already runs, so those go through its depth, cap, and repeated-call checks once. Loggers that only implement the chat-specific hooks still run in the OpenAI dispatcher * fix(openai): honor max_agentic_loops when chat completions bridge to Responses A gpt-5.4+ chat completion with function tools on api.openai.com is sent to /v1/responses by the Responses bridge. The Responses handler then ran its own agentic loop on that call, which never saw the request's max_agentic_loops or depth, and the shared chat completion loop in litellm.acompletion ran again on what it returned, so every chat follow-up restarted the Responses loop at depth 0. With max_agentic_loops=1, web search made 8 Responses calls and 7 searches The async bridge now marks its Responses call as part of a chat completion turn, and the Responses handler skips its own loop for such calls. The shared chat loop owns the whole turn, so its depth, cap, and repeated-call checks run once * fix(responses): keep search follow-ups on the Responses API for responses/ pinned models When an async chat completion crosses the Responses bridge, the shared chat loop now owns the turn and builds each web search follow-up from the model recorded on the logging object. responses() overwrote that record with the stripped model, so a model pinned with the responses/ prefix (for example openai/responses/<model> on a Responses-only gateway) sent its follow-up to chat completions, and the turn ended on the unexecuted tool call responses() now leaves the record alone when the chat loop owns the turn, the same way completion() skips its own record when the reverse bridge owns it * fix(agentic-loop): bill every round of a non-streamed Responses-bridged chat loop once A bridged chat request logs its first round before the chat loop runs, so a follow-up run as an internal sub-call was never billed. Each follow-up of a non-streamed bridged parent now writes its own spend row, and a failed follow-up no longer re-bills rounds that already wrote one * fix(agentic-loop): keep streamed Responses-bridged web search from billing its final round twice * fix(openai): bill every bridged search round in one row after guardrails On the proxy a non-streamed Responses-pinned chat request now parks one spend row that sums every round the loop served, so the row and the x-litellm-response-cost header show the whole request and nothing is written before post-call guardrails pass. A streamed request keeps round one's row and folds every later round into the replayed final row. The bridge now skips only Responses hooks that run their own chat loop, so hooks that only handle the Responses API still run. * fix(openai): price each bridged search round at its own size Each round of a bridged chat request is now priced on its own usage and the combined proxy row adds those prices, so summed usage never crosses a long-context price tier. A streamed replay keeps that total and its cost breakdown * fix(agentic-loop): run the chat hook of bridged callbacks that define both agentic hooks The bridge skipped the Responses hook of a callback that runs its own chat loop, and the shared loop only asked the general gate, so a custom callback with both hooks stopped intercepting bridged chat. Ask its chat gate after the shared loop. Price tier tests now use test-owned gpt-5.6 prices. * fix(agentic-loop): route bridged chat to the chat hook only for callbacks that opt in A custom callback that defined both agentic hooks lost its Responses hook on chat completions bridged to the Responses API, so a chat hook that declined left the request unhandled, and one that claimed it changed which hook answered compared with main. Callbacks now opt in with wants_chat_completion_hooks_on_bridged_chat. Web search interception sets it, so its loop still runs once on the chat side. Every other callback keeps main's routing through its Responses hook. * fix(websearch_interception): keep main's Responses loop for a bridged web_search_options call --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 14:36:10 -05:00
devin-ai-integration[bot]	bba6f497f6	test(configuration): restore isolated gateway launcher imports (#51848) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 19:14:36 +00:00
devin-ai-integration[bot]	826c35bff2	test(spend): align queued baggage mapping fixtures (#51847) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 19:13:36 +00:00
devin-ai-integration[bot]	969386fb73	test(streaming): assert queued first-error JSON response (#51849) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 19:12:33 +00:00
tavren27	8fc6789295	chore: move LiteLLM's configs and e2e tests off gpt-4o and gpt-4.1 before their 2027-01-14 shutdown (#51800) ... * chore: move LiteLLM's configs, cookbooks and e2e tests off gpt-4o and gpt-4.1 before their 2027-01-14 shutdown OpenAI shuts down gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini on 2027-01-14. The example proxy configs, cookbook guides and e2e suites now call gpt-5.6, gpt-5.6-luna or gpt-5.4-mini, and the predicted-outputs e2e test is removed because no current OpenAI model accepts prediction * docs(cookbook): name gpt-5.6 instead of gpt-4o as a tool-capable model in the gollem troubleshooting tips * chore(ci): delete the observatory proxy config, which nothing has loaded since its workflow was removed * chore: send max_completion_tokens in batch lines and move README and deploy examples to gpt-5.6 gpt-5.6-luna rejects max_tokens on OpenAI's batch lines, which reach OpenAI unchanged, so the batch JSONL builders send max_completion_tokens. The README quick start, the adaptive router README and the Helm and Terraform example configs move off gpt-4o with the configs * chore: pin output prices next to input prices in the adaptive router example The adaptive router reads only input_cost_per_token from config, and a deployment priced by input alone bills output at $0, so the example and its README pin both at the cost map's current prices * chore: move the configs and e2e tests to gpt-6-luna and gpt-6.1-sol The cost map names gpt-6-luna as the successor of gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini. Cheap roles move to gpt-6-luna and full-size and vision roles to gpt-6.1-sol, the pair main's other configs use. The adaptive router example pins the gpt-6 prices, and the installer quick start and two READMEs drop gpt-4o --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 10:50:59 -08:00
zoveki	da67a9c8c3	fix(bedrock): map OpenAI vector store search filters to AWS RetrievalFilter (#51291) ... * fix(bedrock): map OpenAI vector store search filters to AWS RetrievalFilter Bedrock Knowledge Base search forwarded OpenAI's type-keyed filters ({"type": "eq", ...}, {"type": "and", "filters": [...]}) to AWS unchanged, and AWS answered 400 for every one. Comparison and compound filters now map to AWS's equals/notEquals/greaterThan/.../andAll/orAll keys, a one-member group is unwrapped, and AWS-native filters pass through. A malformed OpenAI filter answers 400 before any AWS call, or is dropped under drop_params, which vector store search now reads from the request or the global setting instead of hard-coding it off. * fix(bedrock): reject booleans in OpenAI in/nin filter lists and map older filter shapes first A list value now holds only strings and numbers, as in OpenAI's schema, so pydantic no longer turns true/false into 1/0 before the filter reaches AWS. The older operator and and/or shapes are checked before the type key, so an older filter that also carries a type key maps as it did before. * fix(bedrock): pass AWS-native vector store filters through when they also carry a type key * test(bedrock): check every AWS RetrievalFilter key passes through with a type key * fix(bedrock): stop validating a vector store filter at its first bad item A malformed OpenAI filter listed every pydantic error in its 400, so one 30 KB filter built a 1.4 MB message that stalled the worker while it was redacted. Validation now stops at the first bad list item or group member, the reason is cut at 2000 characters, and the 400 no longer chains the pydantic error into the logged traceback. * fix(bedrock): report a bad vector store filter member without calling its group empty An and/or group's non-empty check ran as min_length on the partly validated tuple after fail_fast stopped at a bad member, so every enclosing group also reported that it had no members. It now runs as an after-validator, which only sees a group whose members all validated, so one bad member gives one error at any depth. A reason over 2000 characters now keeps its first and last 1000 characters, so a deeply nested path still ends on the bad field and its message. * fix(bedrock): keep the vector store filter types importable on Python 3.10 Python 3.10's typing.Union hashes every member, and the FailFast() marker in the filter value's list member is an unhashable dataclass, so importing litellm raised TypeError: unhashable type: 'FailFast'. Field(fail_fast=True) sets the same fail-fast flag on the tuple schema and is hashable, so the value and the and/or group use it instead. * fix(bedrock): check OpenAI vector store filter values per operator and send numeric in/nin as AWS accepts them AWS in and notIn take only a non-empty list of strings, so an in or nin list holding a number now goes out as orAll of equals or andAll of notEquals, and a one-item list as a single equals or notEquals Each OpenAI filter now refuses keys it does not define and checks its value by operator: eq and ne take a scalar, gt, gte, lt and lte take a number, in and nin take a non-empty list. An empty filters object is invalid too. Each of these answers 400 before any AWS call, or is dropped under drop_params, where before it reached AWS and failed there or, for a stray key, was silently ignored _aws_retrieval_filter joins the recursion detector's ignore list, since it walks only a validated filter that pydantic-core stops at 255 levels * fix(bedrock): join same-operator filter groups so OpenAI filters stay within the AWS nesting limit AWS rejects a RetrievalFilter nested more than 2 levels deep, and the orAll or andAll a numeric in or nin turns into used to add one, so and(eq, or(in [2023, 2024], eq)) failed at AWS. A group whose member is the same kind of group, written by the caller or made from a numeric in or nin, now takes that member's items, so it goes out as andAll(equals, orAll(equals, equals, equals)) The mapping is unchanged otherwise, and a mix of and and or nested deeper than AWS allows still fails at AWS _aws_group_members takes _aws_retrieval_filter's place in the recursion detector's ignore list, with the same bound --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 13:39:43 -05:00
Prince Jain	ec48e2491c	fix(aiohttp): reset the upstream connection when a stream closes early, so aborted Bedrock Invoke streams stop (#50657) ... * fix(bedrock): close httpx response stream when invoke stream is aborted * test(bedrock): use Final state holders and fixed datetime in invoke abort tests * fix(bedrock): call response.aiter_bytes eagerly and await aclose safely * refactor(bedrock): share one stream close helper module across routes Move SupportsAclose, aclose_if_supported, and aiter_bytes_then_close into litellm/llms/custom_httpx/stream_close.py so the Bedrock invoke paths, the generic HTTP handler, and the Anthropic messages iterator use one copy, and drop the mock-tolerant close in production code * fix(aiohttp): drop the provider connection when a stream closes before its body ends Closing an unfinished aiohttp response on uvloop starts a graceful TLS shutdown that keeps reading the provider's body for up to 30 s. Aborting the transport first ends the upstream read at once. * test(bedrock): cover each invoke stream layer closing its source * test(aiohttp): pin the TLS test certificate dates and run it through uvloop.run * test(bedrock): assert the converse stream closes its response when an exception event ends it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 12:08:03 -05:00
mika-lt	cad4570bbe	fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user (#51829) ... * fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user A group the IdP pushes before LiteLLM has a team for it made the roster write fail, which answered 404 before the user row was written, so the user stayed active and their keys kept serving. #37700 made that failure fatal. Groups with no team row are now logged and left out of the roster write, the persisted teams, and the role resolution. Group ops after a replace now apply in order, as RFC 7644 section 3.5.2 says. * fix(scim): drop the redundant unchanged-groups return and cover the warning and PATCH admin role * fix(scim): strip line breaks from the teamless-group warning * fix(scim): read the PUT role from every requested group, as POST does A PUT naming only a group with no team no longer demotes an admin whose admin group has no team. The new tests also get typed fixtures and Final locals * fix(scim): look up the requested teams on the primary database, as the roster write does --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 12:06:52 -05:00
norkelbyte	8140fdf54f	fix(gateway)!: apply a deployment's Messages params and fill model capabilities from the cost map (#51834) ... * fix(gateway): apply a deployment's sampling params on /v1/messages The gateway ignored temperature, top_p, top_k, max_tokens, and stop_sequences set in a deployment's litellm_params, while the Python proxy sends them upstream. Merge them under the client body, so a value the client sets, null included, still wins. * fix(gateway): apply a deployment's non-sampling litellm_params on /v1/messages The gateway dropped thinking, tools, tool_choice, system, and the other Anthropic Messages params a deployment sets in litellm_params, while the Python proxy sends them upstream. Every key Python forwards now applies as a default under the client body, so a client value, null included, still wins A deployment's tools go first and the client's tools follow, as the Python router merges them. Client tools that are not an array stay as sent, and extra_body or any key outside that set still never reaches the provider * test(gateway): cover a deployment tools value that is not a list on /v1/messages * fix(gateway): fill model capabilities from the cost map so deployment params are shaped per model Ports the capability fill from #50780 onto the moved litellm/rust crates, so a deployment's thinking, reasoning_effort, speed, and temperature are shaped for the model instead of reaching Claude 5.5 models untranslated and failing with a 400. Also forwards a deployment's thread param, which Python's AnthropicMessagesRequestOptionalParams lists * fix(gateway): keep MCP server tokens out of the debug request log A deployment's mcp_servers now reach the provider body, so the debug provider-request log would print each server's authorization_token. The logged copy replaces it with REDACTED while the provider still receives the real token * fix(gateway): resolve os.environ/ references and honor additional_drop_params in deployment params A deployment's untyped litellm_params now resolve os.environ/NAME at config load the way the Python proxy's config loader does, including nested objects and true/false words, so a system prompt or MCP server token kept in the environment reaches the provider resolved. A deployment's additional_drop_params now reach the Messages shaping, as on the Python proxy --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 11:51:52 -05:00
mika-lt	1f6e401a68	fix(together_ai): stop listing retired Together ids (#51827) ... * fix(together_ai): stop listing retired Together ids and duplicate bare names together_ai/* wildcard listing and get_valid_models("together_ai") returned 20 bare ids hardcoded in 2023 that Together no longer serves, next to the cost map's together_ai/ rows, so the same model could appear twice and callers got 400 or 503 on the retired ones. together_ai_models is now built only from the cost map, like the other cost-map-built provider sets, and validate_environment looks up a bare Together id under its together_ai/ key so it still reports TOGETHERAI_API_KEY. The cost map also marks DeepSeek-V3, Kimi-K2-Instruct and CodeLlama-34b-Instruct with the deprecation dates from Together's deprecations page, so the existing deprecation filter drops them from listings * test(together_ai): read retired Together rows through is_hidden_as_retired after main moved the helper * fix(together_ai): use the 2026-10-30 retirement date the other Together cost map sweep uses for the three shared rows * fix(together_ai): accept every Together key name in the key check * fix(together_ai): keep other providers' key checks for shared bare ids and date rnj-1-instruct * fix(together_ai): keep the constants import and stop the W&B test depending on cost map dates --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 11:15:23 -05:00
giladk1221	9d827093c5	feat(github_copilot): support GitHub App authentication with PEM upload (#51427) ... * feat(github_copilot): support GitHub App installation authentication * Add GitHub App authentication to the Copilot credential form * Explain secure private-key setup beside Copilot App fields * Store uploaded Copilot App keys through encrypted named credentials * fix(github_copilot): handle App validation errors and credential replacement * fix(ui): make GitHub App field help accessible * fix(github_copilot): preserve saved App key references on edit * fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health * fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers. App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares. A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change. App deployments are deployment-owned, so they share the response cache like shared device-login deployments. App credentials are validated even when credential_info names no provider. * fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future * fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again * fix(github_copilot): report a deployment naming an unloaded credential as unhealthy The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login * fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base A Copilot deployment that names an unloaded credential uses the shared device login again, as on base. A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call. get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores. * fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure * fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403 Only an App deployment reports its App host as api_base before a configured api_base. Per-user OAuth sessions report the same api_base they did on base. The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication. The token-dir fixture in the App api_base test is applied with usefixtures. * fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete * test(github_copilot): type the incomplete App settings test parameter --------- Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 10:33:20 -05:00
paveln83	05518975a5	test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816) ... * test(e2e): move audio and realtime tests off OpenAI model ids with shutdown dates * test(e2e): move langfuse completion, image, and vertex realtime ids off dated models * test(e2e): register the weather tool in the first realtime session update and read Langfuse v2 flattened attributes * test(e2e): expect the OCR upload's file summary in the Langfuse generation input The OCR upload route has sent the uploaded file, not a data URL, since the native OCR route, and the generation input summarizes that form as the mime type and file name. The broken Langfuse lookup hid the stale data URL expectation * test(e2e): move the image generation test off gpt-image-1-mini gpt-image-1-mini has a 2026-12-01 deprecation_date, so the image generation e2e would start failing that day. gpt-image-2.5-flare has no date and already serves the Langfuse image tests * test(e2e): use gpt-image-2.6-flare and check every n=2 choice reaches Langfuse OpenAI names gpt-image-2.6-flare as the successor of gpt-image-1-mini, so the image tests use it. The n=2 completions test now needs one logged message per choice, since both choices repeat the same text and a single logged message used to pass --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 09:51:53 -05:00
Michael Bennett	fd6c00200c	feat(decisions): support Vercel AI Gateway decision models (#50642) ... * feat(decisions): support Vercel AI Gateway decision models * test(decisions): cover Vercel AI Gateway dispatch through the proxy * feat(decisions): add Vercel AI Gateway cost map rows for Jev * fix(decisions): bill Vercel AI Gateway decisions at the charge Vercel reports Vercel returns the charge for every call under provider_metadata.gateway.cost, already net of cache discounts, so unpriced evaluation models and luna cache hits now bill what Vercel charged instead of zero or the full input rate. Jev 1.14.0 and the jev alias also take box questions, as Vercel serves them * fix(decisions): bill Vercel gateway charges with margin and BYOK, forward providerOptions Decisions now records a provider's reported charge as upstream_usage_cost, so the operator's cost discount and margin apply to it like any other provider-billed cost. Vercel calls served on the caller's own provider key bill marketCost on top of the gateway fee, the cache split comes from the upstream usage Vercel nests under provider_metadata, providerOptions reach the gateway, remote images are inlined, and typesafe-ai/jev, which reads text only, refuses or drops images * fix(decisions): stop forwarding caller providerOptions to Vercel and send Jev images Decisions no longer sends a caller's providerOptions to Vercel. Gateway routing and provider keys are set on the Vercel team instead. Vercel serves typesafe-ai/jev images at 768x768 (the earlier 400 came from a 96x96 image TypeSafe rejects for its size), so the Jev rows read images and the per-model image refusal is removed. * fix(decisions): report provider charges on the cost header and read Vercel's cache split from the serving provider Decisions goes back to reporting a provider's own charge through the response cost header, the way main does for OpenRouter. Moving it onto upstream_usage_cost dropped every cost header on free calls, priced base_model deployments from the cost map, and put the margin in the response header but not in the SystemOne body cost. Vercel's cache split now comes from the provider_metadata entry of the provider that served the call, and only when that entry reports input_tokens_details. Any credentialType value is accepted. The Jev 1.13.0 and 1.14.0 rows record their image probes. * test(decisions): type the async image fetch fixture in the Vercel inline image test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 09:43:08 -05:00
norkelbyte	ac1cc5111c	fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions (#51753) ... * fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions * fix(proxy): check the key's max_budget on /queue/chat/completions * fix(proxy): run budget, tag, and request policy checks on queued chat calls * fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls A queued stream now hands its logging object to the stream generator, so a client that leaves mid-stream is billed for the reply it already got, and a stream left before the first model chunk releases its budget reservation. A team's require_trace_id and the failure logging for rejected calls now use the same route check as the other inference checks. * fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses * fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls * fix(proxy): send the call ID on a queued call the provider rejects before streaming * fix(proxy): release a dropped stream's budget hold only when nothing bills it A guardrail that buffers the stream can pull model chunks the client never saw. The early-drop release settled the hold at the input cost first, then the partial bill for those chunks added its full cost on top. The release now runs after partial billing and only when no bill fired * fix(proxy): fold the caller's litellm_metadata into a queued call's metadata A queued call that sent a litellm_metadata bucket, even an empty one, made tag routing and logging read that bucket, so the key and team tags the handler merged into metadata no longer narrowed deployments. The queue handler now folds the bucket into metadata and drops it, as chat does, after stripping the proxy-owned user_api_key_* and internal control fields from both buckets, so a caller can no longer set the queued spend row's organization ID either * fix(proxy): release a queued call's budget hold when the call is cancelled before it answers * fix(proxy): refund a cancelled queued call's hold in full until the model call claims it * fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 08:54:54 -05:00
tavren27	27c796c412	fix(proxy): count the request's tool_choice on the token count routes (#51133) ... * fix(proxy): count the request's tool_choice on the token count routes /utils/token_counter, /v1/messages/count_tokens and /v1/responses/input_tokens now pass tool_choice into the local estimate, and every DeepSeek model counts it as 0, as its providers bill * fix(token_counter): count a request with no model name again * fix(token_counter): count grok tool_choice the way xAI bills it xAI bills grok-4.3 and grok-4-1-fast-reasoning 21 tokens for "none", 11 for "required" and 12 plus the name for a named tool, where the OpenAI costs counted 1, 0 and 7 plus the name. The grok-4-1 and grok-4.3 chat template tool sections now carry those costs, so a grok count matches its bill for every tool_choice * fix(token_counter): count xAI's grok tool_choice costs only where xAI serves grok Azure AI Foundry bills grok-4-1's tool_choice differently, so grok on other hosts keeps counting it as 0. A tool_choice sent without tools counts nothing, since OpenAI and xAI reject it * fix(token_counter): key xAI's grok tool_choice costs on the provider that serves the deployment The count routes strip the provider prefix before counting, so Azure, Vertex AI and Perplexity grok matched the xAI name pattern. The routes now resolve the serving provider with get_llm_provider and drop xAI's costs when it is not xAI or OpenRouter's x-ai route. The costs move onto the grok tool sections so one pattern decides them * fix(token_counter): count no tool_choice for Gemini and none without tools in the Rust counter Gemini bills the same prompt tokens for every tool_choice, so it skips the OpenAI choice costs the way DeepSeek does. The Rust reservation counter now adds tool_choice tokens only when tools are present, matching the Python counter * fix(proxy): count no tool_choice a deployment drops, and treat an api.x.ai api_base as xAI A deployment that lists tool_choice in additional_drop_params never sends it, so the count routes no longer add its cost. An openai/ model whose api_base is on api.x.ai is served by xAI, so it gets xAI's tool_choice costs like xai/ grok * fix(proxy): count no tool_choice a request drops in its body /v1/messages/count_tokens takes additional_drop_params from the request body, and inference drops tool_choice when it is listed there, so the count no longer adds its cost either * fix(proxy): take additional_drop_params on /utils/token_counter and treat a malformed api_base as not xAI /utils/token_counter now passes a request body's additional_drop_params into the count, so a tool_choice the request drops counts as 0 there like on the other two count routes. An api_base that does not parse falls back to the deployment's provider instead of failing the count with a 500 * fix(proxy): take chatgpt and github_copilot providers from the model name in the count host check Resolving them through get_llm_provider read the saved login, logged a warning and could refresh a token on every count with a tool_choice * fix(proxy): skip a tool_choice or additional_drop_params too deeply nested to validate on the count routes A value nested 300 levels deep failed TokenCountRequest validation, so the count routes answered 422 or 400 where they used to count the request without it * fix(bedrock): count Kimi K3's forced tool_choice message on the exact count path Kimi K3's chat template adds a tool-choice system message when Bedrock gets toolChoice any or a named tool, and Bedrock bills it (170 input tokens unforced, 206 forced). The exact count path now passes tool_choice through the same Converse mapping inference uses and renders that message * test(proxy): type the Kimi K3 count test's tools parameter * perf(count_tokens): validate each kept count option once instead of twice * fix(dot_notation): parse drop paths in linear time when brackets never close * test(count_tokens): benchmark the unclosed-bracket drop path instead of timing it in a unit test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 07:53:27 -05:00
KVR Vishnu	8b28818fa2	fix(openrouter): suppress empty post-finish choice before usage trailer (#50210) ... * fix(openrouter): drop empty post-finish choice from usage trailer OpenRouter's accounting chunk after finish_reason repeats an empty assistant delta (role + finish) alongside usage. After usage was retained for cost tracking, that leftover choice was still emitted between the terminal finish and the final usage chunk (#50156). Clear choices on post-finish usage trailers so clients only see content, finish, then usage. Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com> * test(openrouter): cover sync and no-stream_options post-finish usage trailers * test(openrouter): type the post-finish trailer test and cover async over a sync stream * fix(streaming): skip the usage trailer in async iteration of a sync stream when usage is off The async-over-sync loop returned a usage chunk unchanged when the caller did not ask for usage, so the post-finish trailer arrived as a chunk with no choices. It now records the usage and skips that chunk, as the native async loop and the sync loop already do. * fix(streaming): keep a post-finish choice that carries a payload when dropping the empty trailer choice The usage trailer copy cleared every choice, so images, audio, thinking blocks or logprobs that a provider put on that chunk never reached the caller. It now drops only a choice whose delta holds nothing beyond its role and empty content and that has no logprobs. * test(streaming): read the payload test's streamed choices through one comprehension clause * fix(streaming): set the final chunk's hidden usage when async iteration reads a sync stream * style(streaming): drop the comment that restates the post-finish usage branch * fix(streaming): keep the final chunk's hidden usage when include_usage is false Dropping the empty post-finish choice also dropped the chunk that carried OpenRouter's usage to callers who set include_usage false, so stream_chunk_builder fell back to a token guess with no cost. The final chunk's hidden usage and its post-loop correction now apply whenever no visible usage chunk is sent. * refactor(streaming): bind the async-over-sync returned chunk with a walrus --------- Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 07:48:43 -05:00
quietjules	007d1c1b5e	feat(proxy): support budget_limits windows on internal users (#51763) ... * feat(proxy): support internal user budget windows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): make user budget migration repeatable and sync API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bound lock wait for user budget migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): omit unset JSON windows when creating users Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): retain personal budget windows for proxy admins Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): give user budget windows their own counters and keep them on the user User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path * fix(proxy): keep windows on team auto keys and write all users' windows in one transaction * fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets. JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do * fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits * test(proxy): check default window resets against the reset helper instead of the wall clock * fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes * fix(proxy): fail an all-users bulk update whole when its transaction fails The per-user fallback after a failed all-users transaction wrote the newest user and then crashed on a missing user_email attribute, so the call answered 500 with one user changed. Racing window bulk updates past the 5s transaction limit reached it. Answer 500 with no user changed instead --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 07:44:35 -05:00
norkelbyte	1f2b2734e5	fix(logging): send one async success log per sync bridged Responses stream (#51787) ... * fix(logging): send one async success log per sync bridged Responses stream A sync litellm.completion(stream=True) on an openai/responses/ model called each CustomLogger's async_log_success_event three or four times for one call. The outer CustomStreamWrapper runs a per-chunk async_success_handler on the logging thread pool while the inner Responses iterator logs the completed response on the caller's thread. A per-chunk run that passed the early-return check before the completed response was stored re-read the shared model_call_details at callback time, found the key the other run had stored, and logged a second success instead of a stream event. Each run now decides once whether it delivers the final stream log, from whether it stored the assembled (or pass-through) response itself, and every stream branch of the delivery loop (CustomLogger, ShadowEvalLogger, custom functions, openmeter, dynamodb) uses that decision * fix(logging): gate v2 stream delivery on the run's own final log Chunk logs on a sync stream run on executor threads, so the shared async_complete_streaming_response key can appear while a chunk run is between its early return and the v2 gate. include_v2 now uses the same per-run delivers_final_stream_log flag as the legacy delivery. Also types the race test callbacks. * fix(logging): gate sync string-callback stream logs on the run's own final log The sync success handler still read the shared complete_streaming_response key to decide whether supabase, logfire, lunary, helicone, s3 and athina get the final stream log. A sync stream chunk run that read the key after the final run filled it sent those integrations the finished response again, so one call wrote the same s3 log 3 to 6 times. Each sync run now uses its own assembled response, like the async handler * fix(logging): send sync stream callback functions and chunk-logging callbacks only the final log A sync stream ran success callback functions and the promptlayer, wandb, athina, and traceloop callbacks once per chunk log, so a cost tracking callback summed one call's cost several times. They now skip a run whose result is one streamed chunk, which keeps the single log of a sync file content stream * fix(logging): decide the sync stream chunk gate before message redaction With turn_off_message_logging on, the sync handler redacted a stream chunk to a dict before checking whether the run carried a chunk, so callback functions and the promptlayer, wandb, athina, and traceloop callbacks still logged every chunk. The run now records that before redaction * fix(logging): send a sync stream's end log once even when its response builder fails A sync stream whose stream_chunk_builder raised ended with a best-effort usage chunk, which the chunk gate skipped, so success callback functions and the promptlayer, wandb, athina and traceloop callbacks got no call. The stream wrapper now sends its end log once through stream_end_success_handler, which the gate lets through, and skips the second submit after an include_usage chunk --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 07:25:03 -05:00
mika-lt	d81c55f4cc	feat(custom_llm): let custom providers serve native generateContent (#51083) ... * feat: support native generateContent in CustomLLM Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve custom generateContent accounting and built-in priority Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward custom generateContent success callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve custom generateContent failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: retain async callbacks across generateContent fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(google_genai): scope custom generateContent hooks to native fields and stream only with a stream hook * fix(google_genai): log custom adapter calls on the caller's logging object and reject untranslatable config Custom adapter calls now share the caller's logging object, so the proxy's deferred logging covers them and one request logs one row under the caller's call id and call type. Native hook responses log from any pydantic, SDK, or dict shape, per-request prices apply to custom handlers, and a missing stream hook or a generationConfig key the adapter cannot send answers 400 * fix(google_genai): return custom hook objects as their camelCase dict so the router tags the deployment * fix(google_genai): keep tool args, drop SDK-only fields, and let routers retry custom legs that cannot stream Tool call args and function responses keep the caller's keys and null values. SDK objects no longer send sdkHttpResponse, automaticFunctionCallingHistory or parsed to the client or the logs. A custom leg without a stream hook answers 500 like CustomLLM's other unimplemented hooks, so a mixed router group retries the request on another leg. The wire-value walk is depth capped for the recursive function check * fix(google_genai): keep router legs serving chat after a custom stream without a hook, and accept config the adapter already meets * fix(google_genai): reject JSON mode a custom leg's completion adapter cannot send The custom-leg generationConfig check now accepts responseMimeType and response schema keys only when the adapter sends them as response_format, or when they ask for plain text. JSON mode without an object schema answers 400 unless drop_params is set * fix(google_genai): send a custom hook's dict back without SDK-only fields A hook that returns a dumped google-genai response as a dict sent its snake_case keys, parsed, sdk_http_response and automatic_function_calling_history to the client, past the guardrail that rewrites only candidate text. Dicts now go back as the same filtered camelCase body as every other hook return, and the router tags that copy instead of writing into the hook's own dict --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 05:57:58 -05:00
tavren27	6fc368a15c	fix(azure): send no api-version on Foundry project /openai/v1 Responses bases (#51767) ... * fix(azure): send no api-version on Foundry project /openai/v1 Responses bases Azure answers 400 "api-version query parameter is not allowed when using /v1 path" for every api-version on a Foundry project's /api/projects/<p>/openai/v1/responses route, so drop it there and keep it everywhere else * fix(azure): send no api-version on Foundry project v1 chat and WebSocket, bill routed models Chat completions on a Foundry project /openai/v1 base now use the v1 OpenAI client, so they send no api-version, which that path rejects. The Responses WebSocket URL builder strips a bare /openai/v1 base before adding /openai/v1/responses, so it no longer dials a doubled path. An azure/model-router call that routes to a model priced only under azure_ai/ is billed from that row plus the router fee once * fix(azure): keep the Azure client off chat-only paths, honor api_version none, drop api-version from v1 base queries Only chat completions switch to the v1 OpenAI client on a Foundry project /openai/v1 base, so transcription and azure_text keep the Azure client and no longer fail inside LiteLLM. A deployment with api_version none sends no api-version on Responses and chat, for gateways that hide the project path. The v1 client now moves an api_base query into default_query without api-version instead of gluing the route after it * fix(azure): pick the v1 client for o-series chat, keep api_version none off a doubled /openai/responses path * fix(azure): send no api-version on embeddings under api_version none, test async and streaming chat on v1 bases * fix(azure): join openai/v1 once onto a v1 client base that ends in /openai Chat and embeddings under api_version none, and chat under api_version v1, built /openai/openai/v1 from an api_base ending in /openai, while Responses on the same base already built /openai/v1. The v1 client now joins the route the same overlap-aware way * fix(azure): keep a percent-encoded gateway path as typed on the v1 client base Join openai/v1 by string instead of add_path_to_api_base, which decodes %2F, %25, %3F and %23 in the base path. Type the chat test helper's mode as a Literal with assert_never. * test(azure): import assert_never from typing_extensions so the Azure tests load on Python 3.10 typing.assert_never needs Python 3.11, and the package supports 3.10 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 05:24:12 -05:00
paveln83	3ad164382b	fix(responses): send cache_control as a prompt_cache_breakpoint on the chat-to-Responses bridge (#51783) ... * fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode caching never wrote. The bridge now renames the marks before converting the messages when the served deployment reads breakpoints, using the same gate the Responses config uses for native /v1/responses input. Deployments that do not read breakpoints keep today's behavior. The rename also leaves unmarked content parts untouched instead of validating them, so a part with non-string keys no longer raises on either path. * fix(responses): keep leading system messages in client order once one goes to input A marked first system string became a list and moved to input while the next unmarked string stayed in instructions, so the model read them in reverse. The leading run that joins instructions now stops at the first system message whose content is not a string * fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge Bedrock Mantle already caches the growing prefix on its own, so turning cache_control into breakpoints there added write charges. The client-order rule for leading system messages now applies only when a mark turns a system string into a list, so unmarked chats send the same request as before. * test(responses): cover a marked user turn in the unmarked system split test A cache_control mark on the user message must not switch unmarked leading system messages to the client-order rule. The split test now runs with the user turn plain and marked, so dropping the role check from the system string test fails it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 05:00:49 -05:00
paveln83	7d4c0ab91a	fix(proxy): retrieve and delete a managed file when the request also names a model (#51780) ... * fix(proxy): retrieve and delete a managed file when the request also names a model GET and DELETE /v1/files/{id} sent a proxy-managed file id to the provider unchanged whenever the x-litellm-model header or ?model= named a model, so the provider answered 404 and DELETE left the file in place. A managed file id now skips model-based routing and resolves to each deployment's provider file id, as the content route already did. * fix(proxy): keep forwarding a managed file id by the model hint on a proxy without a database A proxy with no database loads no managed files hook, so it cannot resolve a managed file id itself and must keep sending it to the named model, which may be another LiteLLM proxy that owns the file. Skip model routing for a managed id only when the managed files hook is loaded. The tests now run the real managed files hook over a stored row with only the OpenAI file routes faked, and cover the no-database proxy. * test(proxy): type the managed file table helper's delete filter * test(proxy): keep a raw provider file id routed by the model hint when managed files are on A proxy with a database loads the managed files hook, which now skips model routing for its own file ids. This test makes sure a raw provider file id sent with a model hint on that proxy still goes to the hinted deployment with its credentials, so the skip cannot widen to every id unnoticed * test(proxy): cover a model hint that names a model the managed file is not on A hint naming another deployment must still resolve the managed id through the managed files hook, so these cases fail if routing is skipped only when the hint matches the file's own model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 04:55:36 -05:00
tavren27	4e29aea11c	fix(responses): accept a string or tuple allowed_openai_params on Responses models instead of answering 500 (#51751) ... * fix(responses): accept any sequence for allowed_openai_params and give UnsupportedParamsError a response A deployment whose allowed_openai_params is a string (or an SDK caller passing a tuple) made every Responses-mode request answer 500 "can only concatenate list". The Responses path now builds the allowlist from any sequence, matching the chat path. UnsupportedParamsError now runs BadRequestError's constructor, so it carries response, body and request like every other 400 instead of raising AttributeError on .response. * fix(params): read a bare-string allowed_openai_params as one param name on every path A deployment or request with allowed_openai_params: "presence_penalty" now allowlists presence_penalty on chat completions, embeddings, the Responses API, the chat-to-Responses bridge, the router's tier and precall filters, and Mistral's prompt_cache_key check. Before, chat and Responses read the string letter by letter, so it allowlisted nothing, and the router ignored it. UnsupportedParamsError's args now hold its own message instead of the BadRequestError one. * fix(params): read a dict or comma-separated allowed_openai_params the way chat and embeddings did A dict allowlist is read for its keys and a string is split on commas, so configs that worked on chat or embeddings before keep working, and the router reads them the same way. The fallback errors header names an error by its class when its type is empty, so UnsupportedParamsError still shows up there by name. * fix(params): read every param name in a string allowed_openai_params A string allowlist now yields every name in it, whether the names are split by commas or spaces or written as a list inside the string, so embeddings configs like "dimensions encoding_format" keep working. A request with no allowlist returns early instead of raising and catching a validation error. * fix(fallbacks): keep the fallback errors header type for errors other than UnsupportedParamsError The header named an error by its class whenever its type was empty, which changed "None" to the class name for timeouts and other errors built without a provider body. Only UnsupportedParamsError needs its class name now that it inherits an empty type, so every other error keeps the type it carries. * perf(params): read a string allowed_openai_params with one regex scan The string reader walked every character in Python and ran twice per request, so a 1 MB allowlist added about 300 ms of event-loop time. `\w+` matches exactly the alphanumeric-or-underscore runs the old loop kept, checked over every Unicode code point, and runs in C. * test(params): cover an int allowed_openai_params and an admin request for a model two teams share An allowlist that is no string, mapping, or sequence now allowlists nothing instead of raising TypeError, and an admin request with no team for a model name two teams share answers the multiple-teams 400 with the model name --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 04:32:38 -05:00
zhengqiuyang	ad24ca9f31	fix(cohere): honor COHERE_API_BASE for embeddings (#50249) ... * fix(cohere): honor COHERE_API_BASE for embeddings The cohere embedding branch in the dispatch passed the caller's api_base straight through with no env fallback, so COHERE_API_BASE was silently ignored for embeddings — the request went to the hardcoded api.cohere.ai/v2/embed host. The rerank path (rerank_api/main.py) already reads the same variable, and the databricks/hosted_vllm/openrouter embedding branches follow the same convention; this branch was the asymmetric one (same family as voyage's #49411). Adds two respx regression tests: the env-routed gateway URL is used when the variable is set, and the hardcoded default still applies when it is not. Signed-off-by: 郑润泽 <506166760@qq.com> * fix(cohere): route COHERE_API_BASE through ensure_api_route and pin the url semantics Review follow-up on #50249: COHERE_API_BASE means different things across cohere's own paths (chat reads it as the full /v2/chat endpoint, rerank as a base URL) and the PR initially used the verbatim reading — which breaks a bare-host gateway silently (the request would land on the gateway root). get_complete_url now runs whichever value it receives through ensure_api_route(..., "v2/embed"): a bare host joins the route, a full /v2/embed endpoint passes verbatim, and the no-env default is unchanged. The default domain (api.cohere.ai vs the api.cohere.com the siblings use) is left alone on purpose — flagging it here for a maintainer call instead of bundling a behavior change into this fix. Tests: three get_complete_url unit pins (bare host joins, full endpoint verbatim, default unchanged) plus the env-routed respx test reshaped to the bare-host form. 76 passed in tests/unit/llms/cohere/embed/. Signed-off-by: 郑润泽 <506166760@qq.com> * fix(cohere): fail fast when COHERE_API_BASE is a chat endpoint; pin verbatim-path semantics Review of #50249 surfaced two gaps: 1. Chat reads COHERE_API_BASE as a complete URL (the value may end in /v2/chat or /v1/chat), while embeddings now treat it as a base to join /v2/embed onto. A chat-configured deployment upgrading would get embeddings at <chat-url>/v2/embed and 404. get_complete_url now fails fast with an actionable error when the value is a chat endpoint, so the user configures embeddings separately instead of silently losing them on upgrade. 2. The added tests only covered a bare gateway host — the one shape where join and verbatim agree. Adds a /v1/embed verbatim pin (the issue's original shape) and a chat-endpoint rejection test. The query-string corruption in ensure_api_route (pre-existing, locked by test_url_utils.py) is real but out of scope here — it predates this PR and needs its own change plus a decision on the locked assertion. Release-note-worthy: chat-only deployments that set COHERE_API_BASE and then configure embeddings will now see the fail-fast error instead of a silent 404. Signed-off-by: 郑润泽 <506166760@qq.com> * fix(cohere): resolve the embed api base without rebinding the parameter AGENTS.md asks for no variable mutation and no comments that restate the code. The base is now a Final local next to cohere_key, and the three-line comment is gone. py_compile passes; the local pytest run is blocked by an unreleased native symbol (NativeRedisClient), so CI verifies the suite. * fix(cohere): keep an explicit embedding endpoint verbatim The route was appended to any path that did not already end in /v2/embed, so a gateway exposing /v1/embed or /embeddings received /v1/embed/v2/embed. An explicit embedding endpoint is now used as given, and a chat endpoint still fails fast. * fix(cohere): keep the configured URL out of the chat-endpoint error The message reaches proxy error responses, and a server-configured URL can carry credentials in its userinfo or query string. * fix(cohere): raise BadRequestError for a chat-endpoint base and fix the verbatim test Review on #50249: aembedding wrapped the guard ValueError into APIConnectionError, so the proxy answered a network error instead of 400. Raise litellm.BadRequestError and assert it. The explicit-endpoint test compared against api_base.rstrip(/) while the code returns the base untouched; compare against api_base. Also drop a corrupted character in the message. Signed-off-by: 郑润泽 <506166760@qq.com> * fix(cohere): send embeddings for a chat-endpoint COHERE_API_BASE to the same host's embed route * fix(cohere): keep an explicit embed api_base as is and derive COHERE_API_BASE safely An explicit api_base is the full request URL again, as before this PR. Only COHERE_API_BASE is turned into an embed URL: a trailing /vN, /vN/chat or /vN/rerank is dropped, /v2/embed is joined on the path so a query string survives, and only a path ending in /embed is used as is. litellm.api_base is no longer consulted for Cohere embeddings. * test(cohere): drop an extra blank line in the embed URL tests * fix(cohere): read COHERE_API_BASE for embeddings from the environment only A proxy with a secret manager no longer makes an extra lookup on every Cohere embedding call, and a bare /chat or /rerank path is now covered by tests. --------- Signed-off-by: 郑润泽 <506166760@qq.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 04:30:11 -05:00
Mateo Wang	1a8ed27237	refactor: clear fresh debt in customer hierarchy and v2 callback modules (#51804) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-14 01:14:06 -08:00
jesus-berri	3988ce2865	fix(proxy): honor cancel_on_disconnect on the rerank endpoints (#51252) ... * fix(proxy): honor cancel_on_disconnect on the rerank endpoints * test(proxy): cover a connected rerank succeeding with cancel_on_disconnect on * test(proxy): cover a connected rerank answer through the cancel watcher in the integration rig * fix(proxy): read cancel_on_disconnect on rerank the same way the chat path does * fix(proxy): set up the rerank request logger before routing so a hang-up after dispatch logs the query's tokens like chat * fix(proxy): keep flag-off rerank logging unchanged and price a cut-off rerank from its query and documents The rerank endpoint now builds its request logger only under cancel_on_disconnect, so a flag-off rerank logs one unbilled 408 row as on main. The failure estimate for rerank counts the query and every document sent, prices the deployment the router picked, and bills one query on a model priced per query. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 03:59:22 -05:00
Yogeshwaran Ravichandran	35b9df1f40	feat(mistral): add native upstream token counting support (#50652) (#50653) ... * feat(mistral): add native upstream token counting support * test(mistral): standardize on mistral-large-4 and add tool calls tests * fix(mistral): unify credential resolution and guard Azure endpoint boundary * fix(mistral): resolve api base from provider env vars and route responses count natively * fix(test): use groq for bridged proxy tests and increase mistral coverage Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov. * fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter * fix(mistral): suppress basedpyright warnings on untyped httpx handler * fix(mistral): keep chat credential resolution and count with the key chat resolves The token counter now resolves its Mistral key and base through the same helper chat uses, so a count goes to the endpoint chat would call. A deployment that would route to Azure, or an Azure-only env key, falls back to the local estimate instead of sending that key to Mistral. Chat key and base precedence is unchanged from main. * fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route * fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge, which restores previous_response_id history. The count now takes the same bridge instead of forwarding previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone. The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them. Tests now go through the real handler with respx, and the integration Mistral peer serves the count route. * fix(mistral): leave empty assistant turns out of the count, as Mistral chat does Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400 through previous_response_id, so the count now skips the same turns chat skips. * fix(mistral): count what chat sends and keep the bundled Mistral model list A user turn that mixed file_id files with an image or an inline file lost the image or inline file in the count, since the converted content arrives as a tuple and only lists were read. Both content readers now take either shape A /v1/responses count with a json_schema text.format now sends that format to Mistral, so the count matches what the bridged chat call bills Counts for Mistral models that are not chat models (embeddings, OCR, transcription) now fall back to the local estimate instead of returning Mistral's 400 Count calls now use the deployment's timeout when one is set, and a count response without input_tokens falls back to the local estimate MistralConfig.get_models goes back to the bundled model list, so turning on check_provider_endpoint does not change what /v1/models shows for mistral/* The previous_response_id count test now fakes the spend logs table instead of the history lookup, so the real lookup and its query run * fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral * fix(mistral): answer 408 on a count timeout and use the shared deployment timeout Main's native counters answer a count timeout with 408 and a connection error with 500. The Mistral counter fell back to a local estimate on both, so a hung Mistral endpoint looked like a successful count. It now raises the same count errors, and it reads the deployment timeout with the shared deployment_timeout helper instead of its own copy * fix(mistral): join count text blocks the way Mistral chat does * fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat * fix(mistral): count the thinking chunks Mistral chat replays in assistant history The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is skipped the way chat skips it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 03:19:57 -05:00
quietjules	9ba216d521	fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239) ... * fix(vertex_ai): reuse one Google token across vector store searches * fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst * test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy * test(vertex_ai): type the fixtures and token callback in the concurrent search test * fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects A shared RAG Engine config made one store's slow token fetch hold up every other store, and kept a rotated key file's deleted key until a restart. First loads and expired refreshes now take a lock per credentials and project, refreshes take a lock per credentials object, and any RefreshError reloads the credentials once. * fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls * test(vertex_ai): check async callers sharing an expired credential refresh it once * fix(vertex_ai): reload Google credentials only when a refresh is rejected for good A retryable refresh error such as a 503 now raises without reloading the key, so an outage does not double google-auth's retries. Credentials that cannot be weak-referenced share the config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase and holds the token response until the other callers have started * fix(vertex_ai): reload Google credentials only when Google answers invalid_grant google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did * fix(vertex_ai): pick up a swapped Google key file on the next call The credential cache key now carries the key file's inode, mtime and size, for a vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of after Google rejects the old key or the proxy restarts * fix(vertex_ai): keep the earlier key while a replaced key file does not load A key file caught halfway through a non-atomic write, or replaced by one that does not load, now keeps the credentials loaded from its earlier version in use and logs a warning, as the cache did before key files were re-read. The next complete file is loaded on the next call. The pub/sub test no longer writes a key file it only reaches through the seeded cache * fix(vertex_ai): try a replaced key file again when its first token fetch fails The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one * fix(vertex): raise when an async reload keeps an earlier key Google rejected --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 03:12:09 -05:00
tavren27	c4c16c7b3f	fix(azure): drop the traceback from stream-cut errors sent to clients and alerts (#51760) ... * fix(azure): drop the traceback from stream-cut errors sent to clients and alerts An Azure chat or text completion stream cut mid-response raised an APIConnectionError whose message carried the full Python traceback, so the in-stream error frame, the /v1/completions 500 body, and every llm_exceptions alert showed server file paths. The Azure no-status branch now keeps the error line alone, and route_failure_proxy_exception redacts the message it answers clients with, the way the chat route already does * fix(proxy): keep route error messages as raised and test the Azure stream cut through the proxy routes Drop the route-level client message redaction, which rewrote valid guardrail and hook 400 messages, and cover the Azure chat and text stream cut end to end through respx * fix(azure): name the error type when a connection error has no message An httpx reset raises ReadError with an empty message, which left the client line with no cause once the traceback was gone * test(proxy): drop the docstring from the Azure stream cut fixture * test(proxy): stop pinning the MidStreamFallbackError prefix in the Azure text stream cut test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 08:00:58 +00:00
quietjules	93f8d86c89	fix(gateway): return Claude thinking blocks on chat completions instead of a 500 (#51774) ... * fix(gateway): return Claude thinking blocks on chat completions instead of a 500 The gateway's Anthropic chat transform answered 500 on any thinking or redacted_thinking block. It now maps them into reasoning_content and thinking_blocks the way the Python proxy does. * fix(rust): replay Claude thinking blocks a chat client sends back on its next turn A client that sends a thinking reply back now gets it replayed to Anthropic the way Python does it, dropping blocks Anthropic could not have signed and unwrapping Bedrock-wrapped redacted data, where the gateway used to decline with 501 * fix(rust): replay a cut-off Claude thinking reply and send a repeated thinking block once A reply that runs out of max_tokens mid-thought comes back with content null and thinking_blocks. Sending it back now replays its thinking blocks alone, the way Python does, instead of answering 501 A thinking block that several merged assistant messages carry goes to Anthropic once, where it first appears, like Python's _without_repeated_thinking New and changed tests use rstest, as litellm/rust/AGENTS.md asks * perf(rust): drop repeated thinking blocks in one pass * fix(rust): send a thinking reply on to OpenAI-compatible models like Python Python passes chat messages to OpenAI-compatible endpoints verbatim, so a Claude reply carrying thinking_blocks and reasoning_content reaches the model as sent. The gateway declined it with 501 instead * fix(rust): drop null message fields before an OpenAI-compatible call like Python Python's cleanup_none_field_in_message removes None fields before the call, since some endpoints reject them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 02:44:23 -05:00
paveln83	dcba0cf6f0	fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content (#51747) ... * fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content Foundry's Fireworks-hosted Kimi K3 takes reasoning_effort as a level string, but azure_ai refused it (400 on /v1/messages with thinking) or dropped it. Models whose cost-map row declares reasoning_effort_levels now get the param, rounded up to a declared level, with deployment-declared levels taking precedence. A dict effort sends its level, and one without a level answers 400 or drops under drop_params. FW- models also keep reasoning_content on replayed assistant turns, which Kimi reads back across tool calls * fix(azure_ai): read declared reasoning levels from the loaded catalog The router copies a deployment's model_info, reasoning_effort_levels included, onto the shared azure_ai/<model> key in litellm.model_cost, so reading that live map let one deployment's levels open reasoning_effort for an undeclared sibling. Reading the catalog as loaded keeps the gate on what the cost map itself declares, while deployment levels still round the request through the deployment path * fix(azure_ai): pass Foundry reasoning levels through and match the catalog row regardless of case Foundry Kimi K3 accepts every reasoning_effort level, so the level is sent as the caller wrote it instead of rounded onto the catalog row. "default" leaves the level to the model. A value outside the level names answers 400, or is dropped with a warning under drop_params. An effort object sends its level and warns that its other fields are dropped. The catalog row lookup now also matches a key that differs only in case * fix(azure_ai): read base_model for Foundry reasoning effort and replay, accept auto * test(azure_ai): cover global drop_params for a Foundry effort that is not a level * fix(azure_ai): keep replayed reasoning_content for Foundry kimi- models Foundry's MoonshotAI-format Kimi deployments (kimi-k2.6, Kimi-K3.1-Pro) read replayed reasoning_content back, so it is no longer stripped for them. The effort-object warning now names only the fields it drops. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 02:08:11 -05:00
tavren27	d76cdcbb8a	fix(proxy): list key and team aliases on /model_group/info and apply listing callbacks to them (#51754) ... * fix(proxy): list key and team model aliases on /model_group/info * fix(proxy): keep listing-callback and pause filters on model alias rows A name a legacy or V2 listing callback hides stays hidden on /v1/models, /v1/models/{model_id} and /model_group/info even when it is also an alias. V2 Hook.list_models subscribers now limit which target an alias row describes, /model_group/info describes no alias by a paused model, and ?model_group=<name> offers listing callbacks only the names that row reads. * test(proxy): annotate the new alias test locals as Final * fix(proxy): keep a caller alias listed when a router alias of the same name points at a hidden model A key or team alias always wins over a router model_group_alias of the same name, so alias rows are now dropped only for names a listing callback hid itself, not for router aliases whose target it hid * fix(proxy): describe an alias by an unlisted target only when listing callbacks keep it Listing callbacks are now also offered the unlisted model each listed key or team alias sends requests to, and the model a router alias of that name resolves to. The alias row is described by that model only when the callback keeps every one of those names, and a name the callback hides there hides no listed name. Before, any installed listing callback, V2 included, left such an alias with no limits and a 404 lookup * fix(proxy): describe an alias to a team model's public name when the key lists its internal name /model_group/info lists team models by their internal routing name, so an alias to the public name found no listed target and got no row while /v1/models listed it. Alias resolution now reads the listed names by their public names. The listing callback docstring also names litellm.model_alias_map among the alias sources. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 01:10:32 -05:00
zoveki	8973e2dc4f	feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project (#50732) ... * feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project Adds the opt-in litellm.vertex_request_label_fields setting, the Vertex twin of bedrock_request_metadata_fields. When set, Gemini, embeddings, Imagen and Gemini image generation, rerank, and Claude on Vertex (chat and /v1/messages) send the listed proxy identity fields and spend_logs_metadata as request labels, identity first, with the user_api_key_ prefix reserved so a caller cannot forge or evict them. Identity text is rewritten to fit Google's label rules. Gemini image generation now forwards labels at all. * feat(vertex_ai): send the opt-in identity labels on multimodal embeddings, the Google GenAI route, and batches With litellm_settings.vertex_request_label_fields on, the multimodal embedding predict body, the native generateContent and streamGenerateContent body, and the BatchPredictionJob now carry the key's identity labels the same way chat does Caller labels on GenAI are resolved after the extra_body merge, so neither a top-level labels field nor extra_body.labels can write the reserved prefix. Batch labels come only from the proxy's litellm_metadata, never the caller's OpenAI batch metadata. With the setting off every route sends the same body as before * fix(vertex_ai): keep identity labels when a Gemini caller sends extra_body labels * fix(vertex_ai): label Interactions, image edits, Mistral, and the /vertex_ai pass-through, and gate Gemini image labels on the setting * fix(vertex_ai): move body labels into the labels header on Vertex rawPredict pass-through calls * fix(vertex_ai): send Mistral extra_body labels in the labels header instead of the body Labels a caller nests in extra_body now merge over top-level labels before the X-Vertex-AI-Labels header is built, the way the Gemini body merges them, and leave the outbound body. The token-fetch stubs in the Mistral and multimodal embedding tests are scoped to their instance so later tests see the real method * fix(vertex_ai): limit pass-through labels to POST and to publishers that take the labels header rawPredict to custom endpoints and publishers other than Anthropic and Mistral, and PATCH or PUT bodies, now reach Vertex as the caller sent them. The pass-through drops a caller's labels header where it writes body labels, a single field written as text turns the setting on for that field, and an undecodable Claude labels header falls back to the caller's labels * fix(vertex): label Veo video, Gemini transcription, and pass-through predictLongRunning requests * fix(vertex): read the pass-through publisher from the end of the upstream path A project id named publishers shifted the first-match lookup, so a forged X-Vertex-AI-Labels header reached Claude or Mistral untouched * fix(vertex): drop a caller labels header on body-labelled Vertex routes while identity labels are on * test(vertex): script the v1beta1 GenAI target in the labels header integration test * test(vertex): type the transcription test's access token override --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-14 00:39:01 -05:00
Mateo Wang	c6aa2a3b2a	test(logging): scope bridged call recorder to its request (#51339) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 21:34:56 -08:00
Mateo Wang	e87d7bd714	fix(anthropic): restore sync messages streams for OpenAI and Azure (#51305) ... * fix(anthropic): return a sync stream from sync messages.create on openai and azure models * fix(anthropic): keep the sync messages stream readable with async for and end it on close * fix(responses): close the chat completions stream when a sync messages stream closes * fix(anthropic): release sync streams on async close Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 21:34:21 -08:00
Mateo Wang	ccb7f753c9	fix(mistral): send Magistral reasoning as supported native levels (#51329) ... * fix(mistral): send reasoning_effort and thinking to magistral models as a level Mistral accepts magistral-small-latest and magistral-medium-latest now take reasoning_effort (none or high) as their reasoning switch, but LiteLLM dropped reasoning_effort and thinking for any magistral model and injected Magistral's deprecated reasoning system prompt instead, so streamed calls returned no reasoning and "none" still reasoned. Forward reasoning_effort rounded to the cost map's declared levels, map thinking onto it (disabled becomes none), drop the system prompt, and declare none and high on both magistral rows. * fix(mistral): send adaptive thinking to magistral as a reasoning level * fix(mistral): reject reasoning_effort strings that are not a level, and leave "default" to Mistral A string such as "HIGH" or "auto" went to Mistral as sent and Mistral answered 422, even with drop_params. It now takes the same path as a number: dropped with a warning under drop_params, a 400 without it. "default", which litellm.completion lists as a value, sends no reasoning_effort so Mistral applies its own default. * fix(mistral): preserve thinking when unsupported effort is dropped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mistral): raise reasoning failures at the public boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mistral): share fixed capabilities with warning regressions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 21:33:54 -08:00
norkelbyte	8decbe4801	fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources (#50577) ... * fix(bedrock): send s3:// image, document and video URLs to Converse as S3 sources An s3:// URL in an image_url or file part now becomes a Converse s3Location source instead of failing with "Unsupported image type". The format comes from the part's format or the key's extension, an optional bucket_owner is sent as bucketOwner, and an unknown or unsupported format answers 400. An image in an assistant turn no longer crashes the async path. * fix(proxy): limit non-admin s3:// media to allowed buckets and refuse managed prefixes A non-admin key may only send an s3:// media URL whose bucket is listed in general_settings.allowed_client_side_s3_buckets (or when client-side credentials are allowed), and never one under litellm's managed prefixes. Proxy admins are not limited. A bucket_owner on an s3:// URL that is not a non-empty string is a 400, the Responses bridge forwards the owner only for s3:// URLs, audio on S3 is a 400, and video in a tool result is sent to Bedrock. * fix(proxy): check s3:// media in every request field and refuse it in Bedrock batch files The s3:// media check now reads the whole request body instead of a fixed list of fields, and it also reads Gemini file_uri and fileUri, so a /v1beta generateContent request with an s3:// file part gets the same bucket rules as chat, Responses, and Anthropic requests. A Bedrock batch input line with an s3:// media URL is a 400 before upload, since Bedrock batch files do not take s3:// media URLs yet. * fix(bedrock): say that LiteLLM refuses s3:// media in Bedrock batch files The 400 for a Bedrock batch input line with an s3:// media URL now says "LiteLLM does not accept s3:// media URLs in Bedrock batch files", since Bedrock batch takes S3 locations and it is LiteLLM that refuses the line. * fix(proxy): check s3:// URLs only inside media parts and refuse them only on Converse batch lines The s3:// bucket check now reads URL fields only inside the content parts that a Bedrock transform turns into an S3 source, and it uses the same s3:// test as the prompt factory. Values in metadata, annotations, tool arguments and other fields pass as before. Each URL is named once in the 403 and 400 messages Bedrock batch files refuse s3:// media only on the lines that go through the Converse transform, so passthrough-provider lines upload unchanged * test(proxy): check that a media part's non-URI fields never count as s3 media * fix(proxy): check s3 URIs inside encoded file ids and on queued chat completions The s3 media check now decodes model-embedded and unified file ids, since completion() turns them back into the raw s3 URI they carry. The /queue/chat/completions endpoint skips common request processing, so it now runs the same check. The media walk goes level by level instead of recursing, which the recursive_detector code-quality step requires. * fix(proxy): let a key send managed file ids it may read and refuse s3 sources hidden in model_file_id_mapping The s3 media gate refused every LiteLLM managed file id whose upload sits under the managed prefix, even when the key's own user or team created it. It now asks the managed files hook which unified ids the key may read and skips only the objects those ids embed. A forged or unowned id, or a proxy without the hook, still gets a 403. The gate also reads the provider file ids in a client-supplied model_file_id_mapping, so an alias can no longer point a non-admin request at an unlisted or managed object. Markdown documents (.md, .markdown) now map to Bedrock's md document format, so an s3 Markdown source without an explicit format no longer fails with an unknown media type. * fix(proxy): check s3 media on every Responses WebSocket frame and keep inline tool-result video dropped Every response.create frame on WS /v1/responses now runs the same s3 media check as HTTP, so a non-admin key can no longer send an unlisted or managed s3 object over a WebSocket A tool result forwards a video to Bedrock only when it is an s3Location. An inline base64 video is dropped with a warning, as it was before this PR, since Converse rejects video in a toolResult block A non-admin key that sends an AWS identity its deployment admits through configurable_clientside_auth_params (aws_role_name, aws_web_identity_token or aws_profile_name) skips the bucket list, because Bedrock then reads the object with that identity. The managed-prefix 403 still applies. A WebSocket frame's own AWS identity fields never open the bucket list, since the managed WebSocket handler does not forward them and Bedrock reads with the deployment's identity The check now skips the body walk when no string starts with s3:// and no file_id or model_file_id_mapping key is present, which keeps large image bodies cheap * fix(proxy): let a client AWS identity skip the s3 bucket list only when Bedrock signs with it An admitted aws_role_name, aws_web_identity_token or aws_profile_name now skips the bucket list only when it decides the credentials get_credentials picks, on every deployment of the model group. A web identity token without a role and a session name does not, since get_credentials then ignores it. A profile does not when the deployment or the environment sets a role, since the role branch runs first. In those cases Bedrock still reads with the proxy's credentials, so the bucket list applies BaseAWSLLM.identity_choosing_params mirrors the branch order of get_credentials, and both read the same AWS_* environment fallbacks through one helper * fix(proxy): keep the s3 bucket list for non-admin keys that send a client AWS identity A client AWS identity field does not prove that the caller can read the bucket. A role or profile name picks credentials the proxy holds, and Bedrock reads the object with them. So the bucket list now applies to every non-admin s3 media part, whatever AWS identity fields the request sends. allow_client_side_credentials: true still turns it off This removes BaseAWSLLM.identity_choosing_params and the router lookup that only the skip needed * fix(proxy): look up managed file ids only when they can change the s3 refusal * fix(bedrock): check every file id field, skip MCP tool calls, accept registered S3 media types * fix(proxy): refuse unlisted s3 media on the rag query and adapter routes * style(proxy): format the merged collections.abc import * fix(proxy): walk tuples when checking s3 media and skip plain number lists * fix(proxy): skip numbers when walking a request body for s3 media A body with large token-id lists and any s3:// string or file_id key made the full walk build a node per number, taking seconds per request. The walk and its prefilter now skip int, float, bool and None values. * test(proxy): check the s3 refusal through the real request setup * test(proxy): exercise real setup for allowed s3 media Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-armor): fail closed on remote s3 documents Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): suppress cyclic fallback payload inference Limit the suppression to reportGeneralTypeIssues on the legacy payload assignment. The fallback payload is constructed as a dict and the runtime AST is unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-13 21:33:21 -08:00