51 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
yuneng-jiang	44d99475a5	test(integration): stop GET /__observations from draining the upstream log (#47256) ... * test(integration): stop GET /__observations from draining the upstream log GET /__observations used to empty the queue, so a second read in a test saw nothing and a "never reached upstream" check could pass whether or not the request was forwarded. GET now returns every request since the last reset and DELETE /__observations is the reset. An autouse fixture resets the worker upstream before each test, which also covers tests that read without the gateway fixture. Discarded reads used as resets become DELETEs, helpers that accumulated drained reads now read one snapshot, and tests with several phases reset between them. The three vacuous negative checks (rag allowlist, guardrail opt-out chaos, xAI batch output) now assert the absence on the same read that shows a request which did reach the upstream. * style(integration): space the observations unit test like its neighbours	2026-10-18 09:58:05 +00:00
yuneng-jiang	fc27dfdc7d	test(integration): give each owned proxy a private database and Redis by default (#47213) ... * test(integration): give each owned proxy a private database and Redis by default owned_proxy and owned_proxy_process now boot on a scratch database and a per-proxy redis-server unless the caller opts into the worker's state with state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips DATABASE_URL. Under the replica profile the private database gets the writer and reader roles so the reader path stays covered. Gateway carries the database_url its proxy writes to, and the Scenario cleanup checks and delete_key_if_present read from it, so they keep checking something on a private database. The 189 test files with owned proxy sites that depend on the worker's database or Redis opt in with a shares_worker_state module marker, which a module-scoped conftest fixture turns into the shared state. The 7 helper modules with such sites opt in per call, since conftest fixtures and other test files reach them. * test(integration): reach session fixtures with the shared marker and keep the worker retry default The module-scoped fixture that applied shares_worker_state ran after session-scoped fixtures, so proxies those booted went private in marked files. The marker is now applied around each marked item's setup, call and teardown, which covers fixtures of every scope. Owned proxies on the worker database inherited the worker gateway's router_settings.num_retries of 0 from it. A private proxy now gets the same default in its config copy unless the config sets num_retries itself. Five providers files read the owned proxy's spend rows or response cache through the worker's database or Redis and opt into sharing.	2026-10-18 03:37:41 +00:00
Mateo Wang	897dda7920	fix(router)!: stop a chat-mode Mantle deployment from rewriting the shared Responses routing row (#46263) ... * fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row A deployment with an explicit custom_llm_provider and model_info.mode: chat used to overwrite the shared litellm.model_cost row, moving every other deployment and plain SDK calls onto Chat Completions until restart, because the downgrade guard read a double-prefixed key and never fired. The guard now reads the real key, and a deployment's own mode decides its routing in responses_api_bridge_check, except on chatgpt and github_copilot where Chat Completions is dead for responses-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): keep the bridge-check mode tests in the sharded bridge flag file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the downgrade guard's mode from the row registration writes A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit responses/ prefix now also outranks a deployment's chat pin. * fix(router): let an explicit responses/ prefix outrank a registered chat row The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the lookup found mode chat and the request went to Chat Completions as responses/<model>. * fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions. * fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them. * fix(router): skip a non-string model_info.mode when keeping the registered mode * fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API A Mantle row that also serves the Responses API now follows the GPT-5 tool and reasoning summary bridge rules, since Mantle Chat Completions rejects function tools on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat gets a dict reasoning_effort as its effort level. The opt-in auto summary asks each Responses config for its level, auto on Mantle's OpenAI path, so a client's own detailed summary follows drop_params as at base. No mode replaces a registered responses mode, and chatgpt and github_copilot rows take any pin until they are responses, which restores the Copilot chat opt-out. * fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort * test(integration): run the Mantle region-prefixed v1 base case again This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip * fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request * fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request * fix(router): keep a non-string deployment mode off the shared routing row A deployment whose model_info.mode was an int, a list, or an object still reached the shared cost-map row, so an unpinned sibling on the same backend stopped routing to the Responses API. The shared registration now drops a mode that is not a string, and the deployment keeps its own raw value. Adds integration cells for per-deployment routing across the OpenAI and Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream outage, config changes under traffic, and a worker kill plus restart. * test(integration): find each routing cell's spend row by the response it logged A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts * fix(bedrock_mantle): run extra_body fields through the chat checks A chat-pinned Mantle request that nests n or reasoning_effort in extra_body now gets the same 400, drop, and effort-level handling as the top-level fields. The bridge test injects a MockTransport client instead of patching HTTPHandler.post. * test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels * fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort * fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request * fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode * fix(router): drop the debug log that wrote a request's model name from keep_registered_mode * test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort * fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down * test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks * test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them * fix(router): route a fine-tune the same whichever deployment registers first Main's register_model now seeds an unmapped fine-tune's row from its base model's built-in row, mode included, on the first registration. The mode guard read only the live row, so a responses pin marked the shared row only when it registered before its siblings. The guard now reads the row the registration merges onto, the live row or else the row it would seed. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 21:39:14 +00:00
Mateo Wang	4a58784fa2	fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037) ... * fix(test-harness): preserve sibling auth modules in isolated worker configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect servers in MCP connect-flow descriptions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect ptu_hours in daily activity metrics after #43043 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): kill every serving worker once the SCIM push blocks on the row lock * fix(cost): price exact cost-map names before price-free capability rule matches * test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price an explicit base_model under its own provider after candidate reordering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): retain priced generalized candidates and skip explicit-pricing lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): assert custom pricing makes no remote model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): keep the per-candidate pricing provider final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments * fix(cost): sort cost candidates from the static cost map and rules, without provider lookups * test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-17 06:46:52 +00:00
yuneng-jiang	910c30381b	test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) ... * test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries	2026-10-16 22:45:29 +00:00
Mateo Wang	b39eadcb82	feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966) ... * feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>	2026-10-16 22:25:15 +00:00
ryan-crabbe-berri	9cf09c18fe	fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377) ... * fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version * fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider * fix(utils): refresh provider model sets after register_model * fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row * test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version * fix(utils): keep router deployment ids out of provider model sets register_model now refreshes the provider sets, so the per-request and router registrations keyed by a deployment id put those opaque ids into wildcard model listings. Those callers now pass add_to_provider_model_sets=False. * fix(azure): carry responses_api_version through the chat to Responses bridge completion() dropped responses_api_version from the litellm_params it hands the Responses bridge, so azure/responses/<model> on a dated chat api_version still built the 404ing /openai/responses URL. Also cover the auto-router marker never forwarding the new param. * fix(azure): stop listing a wrong set of reasoning efforts when none is rejected The message said only low, medium and high are supported, which is wrong for models that also take xhigh such as gpt-6.1-sol. * test(azure): use the latest grok and gpt-5.4 rows in the new azure tests * fix(proxy): list each wildcard-expanded model once A provider set can hold a model both bare and provider-prefixed, which now happens after any priced request registers its openai/<model> key, so the openai/* listing showed that model twice. * fix(azure): route Responses past dated GA api_versions by default A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before * chore(azure_ai): mark the grok chunk_parser dict as an override contract * fix(azure): rewrite only the route's /openai segment when moving a call to v1 The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses. Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too, so the rewrite now only touches the last /openai path segment. The Responses version resolver no longer duplicates the builder's default for an unset api_version. Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite now has one test for the dated GA default and one where only responses_api_version can make the call work * test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params * fix(utils): list ai21 chat and vertex code models under their provider wildcard register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too. * fix(main): keep caller-chosen model names out of provider wildcard listings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai>	2026-10-16 15:23:46 -07:00
武本龍門	43be1c2ae7	fix(router)!: keep RPM, region, and param checks when token counting fails (#44544) ... * fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 09:20:34 +00:00
Michelangelo Setaro	c288dc7da0	fix(router): let an exact base_model widen a deployment's context window (#46197) ... * fix(router): honor explicit bedrock base model capacity * fix(router): honor an exact base_model's capacity for every non-Azure provider Extends the Bedrock-only check to every non-Azure provider and only uses the base_model when it has its own exact cost-map entry, so an unmapped or wildcard base_model falls back to the route id instead of raising in the router. * test(router): call _maps_to_exact_model_info directly in its cost-map tests * fix(router): let an exact base_model only widen a deployment's context window A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model * test(router): count the base_model window helpers as covered through get_router_model_info * fix(router): widen only from a model_info base_model naming the same model A stand-in base_model for a different, larger model widened a smaller deployment's window, so pre-call checks sent it prompts the provider rejects with 400. The window now widens only when model_info.base_model names the same provider model id as the route, which is also the field /model/info reads. * fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers * fix(router): read litellm_params.base_model too and look the base row up as written * test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 06:37:40 +00:00
yuneng-jiang	b602209ca6	test(integration): hand every harness server a port from one per-worker block (#46761) ... * test(integration): hand every harness server a port from one per-worker block About 45 places in the integration harness picked a port by binding port 0, reading the number back, closing the socket and starting a server on that number later. Another process could take the port in between, which surfaced as "Address already in use" flakes. Earlier fixes retried around the race in single call sites without removing it. tests/integration/_support/ports.py now owns every port the harness picks ahead of a server. Each pytest process claims a 200-port block between 10000 and 32767 by taking an exclusive flock on a per-block lock file in the system temp directory, so parallel runs on one machine never share a block and the lock goes away with the process. The range is above the fixed service ports and below the Linux and macOS ephemeral ranges. Ports go out in order, are never reused in a run, skip anything an outside process already holds, and a process that exhausts its block claims the next free one. The three _free_port copies, worker_endpoints' port-0 binds, free_loopback_port, the owned proxy's lost-port-race retry and the owned Redis retry loop are gone. Tests that restart a server on the same port now allocate that port up front instead of reading it back from the first server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read back with getsockname anywhere under tests/integration. * ci(integration): record listening sockets and the ephemeral port range * Revert "ci(integration): record listening sockets and the ephemeral port range" This reverts commit d3fafc3da2.	2026-10-15 16:56:40 -07:00
yuneng-jiang	46913be92b	test(integration): wait on the right signal in five flaky integration contracts (#46737) ... * test(integration): wait on the right signal in five flaky integration contracts The Prometheus shared-label tests waited for the latency metric and then read the request counter, which the logger writes after an await. They now wait for the counter The auto-router session poll returned on the first 200 while the caller expected two turns, so it now waits for the expected turn count The S3 queue cap test let its failing sink recover on a 15s clock that could expire mid-flush. The sink now stays down until a flush fails after every event is queued K3 normally takes 168-175s against a 180s limit, so the limit is now 300s The MCP env var browser spec reused an idle keep-alive socket that the proxy closes after 5s, so its API calls now send Connection: close * test(integration): add until_asserted so a wait polls the exact asserts it guards eventually() takes a predicate separate from the asserts that follow it, which lets a test wait on one value and then assert on another written later. until_asserted reruns a block of asserts until they all pass and re-raises the last failure on timeout The Prometheus shared-label tests and the auto-router session check now use it	2026-10-15 13:48:51 -07:00
devin-ai-integration[bot]	e344dbd8db	feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) ... * feat(proxy): split Azure PTU deployments across teams by share A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens. * fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings. * fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations * fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id * fix(ptu): refuse ptu_shares naming a team that does not exist * refactor(ptu): keep Azure detection under llms/azure and check share teams with one query * fix(proxy): drop the reverted TeamIdSearchFilter import after merging main Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally * fix(proxy): chunk the ptu_shares team lookup under the IN-list bound * fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling * fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling * fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour * fix(ui): seed total_ptu_hours in the empty daily activity metadata * fix(ptu): size usage rows by the name the request used * fix(ptu): size a team's usage rows by the deployment it is served from * fix(ptu): never size a team's usage row by another team's reservation * fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length * fix(ptu): read a share map only through the terms registration requires A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and ptu_effective_from, or one that does not add up to ptu_count, is refused at registration but was enforced once loaded: every other team got the 400 and the share holder a ceiling while no flat cost accrued. _deployment_shares now reads the shares through ptu_terms, so such a row reserves nothing, the same as before the map was read. * fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off 6fb24d1d read ptu_shares through the pricing terms so a termless map reserved nothing, but with the flag off the router skips ptu_config_error for config.yaml entries, so such an entry loaded and served every team, against the rule that a declared split restricts access whether or not cost attribution is on. _deployment_shares reads the declared map again, and the router now raises ptu_config_error for a config-sourced entry declaring ptu_shares regardless of the flag, so a malformed split is refused at boot instead of being honoured or ignored silently. * fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling A name served by a wildcard route now resolves to the shared deployment behind it, so the limiter's ceiling and the team daily activity sizing both find it. Pass-through responses settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a reservation taken at admission is settled even when the success metadata lost the team. * fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids. * test(router): pin the wildcard route precedence that PTU sizing and the router share * fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths. * fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move * fix(proxy): write each PTU deployment day's team charges in one transaction The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team. * fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes * test(router): list the cached PTU ceiling helpers in the router coverage check * fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force * fix(proxy): bill a rerun PTU day to the current share holders alone * fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list * fix(proxy): read the PTU flag from the secret manager first, like get_secret * fix(router): scope retry health lookups to the caller only for PTU share holders * fix(router): move the share-holder retry check into ptu_shares Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them * test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 18:44:30 +00:00
Mateo Wang	18831b818c	fix: keep model capability flags out of provider requests (#46647) ... * fix: keep model capability flags out of provider requests A supports_* flag or another model map field set on a deployment's litellm_params was forwarded to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on chat traffic and on the background health check alike. The shared owned-kwarg predicate now treats every ProviderSpecificModelInfo field as litellm-owned. Builds on the health-check-only approach in #40952 by leilei3167. * fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params * fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments * fix(router): strip model capability flags from the router's deployments instead of every request * fix(router): keep capability flags out of health probes and read them from the loaded catalog * perf(router): read the catalog's supports_ field set once per cost map load Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags	2026-10-15 14:30:10 +00:00
Wang Yunzhuo	a2f792634e	fix: preserve explicit provider for async image, audio, and realtime routes (#36346) ... * fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 05:21:43 -07:00
devin-ai-integration[bot]	e136e91273	fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) ... * fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 21:10:09 -07:00
Kerry Lu	7d7e28829f	test(integration): delete flaky team model TPM concurrency test (#46541) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 16:24:08 -07:00
yuneng-jiang	f95e466bb7	test(integration): fix three more CircleCI integration reds from test setup (#46510) ... OpenRouter basic translation expected the pre-#44788 response shape. Read refusal, logprobs and system_fingerprint from the fixture at their new positions. The team model TPM test required exactly one of three concurrent requests to be served. The pipelined limiter can reject all of them, which is an accepted trade-off, so assert no over-admission and that a lone retry is served when the burst got nothing. The Voyage host tests count every host their proxy dials. The responses cost poller could dial api.openai.com for background-response rows that other tests left in the shared worker database, so turn polling off in those proxies.	2026-10-14 15:23:14 -07:00
devin-ai-integration[bot]	2d4472efc1	fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) ... * fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 12:40:48 -07:00
yuneng-jiang	0915691d32	test(integration): remove six setup causes of CircleCI integration reds (#46490) ... Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind Start a peer proxy for the accounting group, which has a test that needs one	2026-10-14 12:00:35 -07:00
Rick	4f3cb48dc1	feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) ... * feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model Studio's async task API: create hits /api/v1/services/aigc/video-generation/video-synthesis and status and content poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the qwencloud and qwen_ai_platform brand aliases, following the same get_dashscope_family_*_config pattern image generation uses. Unlike the Sora-shaped providers the request body is nested, so prompt and media go under input and everything else goes under parameters. X-DashScope-Async is always sent, because DashScope rejects the call outright without it OpenAI params translate to DashScope's contract. seconds becomes duration, and size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus 720P), since sending only the ratio leaves the request on the default 1080P and silently bills at that rate. input_reference becomes a single media entry, base64 encoded into a data URI for file inputs. Which media type a model takes (first_frame or, on the reference-to-video models, reference_image) is read from provider_specific_entry.dashscope_video_reference_input via get_model_info rather than hardcoded; models without it predate the media array and get the flat img_url field. A caller-supplied media array always wins, as that is the only way to express multi-role combinations The create call is the only billed call, so it bills what DashScope will deliver: the requested duration and tier, or the documented 5 second and 1080P defaults when they are omitted. Smart duration (duration -1) lets the service pick the length after that call, so it is rejected rather than recorded at zero, with a 400. Task creation and task lookup both report failures as a 200 carrying top-level code and message with no output, so that shape is raised instead of surfacing as a queued video with an empty id. Status responses re-encode the deployment model from the polled id, so follow-up polls and downloads route to the same deployment. Timestamps parse as UTC+8. DashScope publishes no list, delete or remix endpoint, so those raise rather than guess at a URL The end-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the JSON body httpx actually encoded. That is what catches a mapper that drops input_reference before the request is built, or a body fragment json cannot serialize, neither of which a test calling the transform directly would see Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v, wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models, for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output second by resolution tier from Model Studio's USD rate cards. dashscope and qwen_ai_platform default to the Beijing host and use its rate card; qwencloud defaults to the international host and uses Singapore's. The three *_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920 Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): fail loudly on video responses that cannot be read Addresses the review on the video generation PR. A body that is not the documented task shape, an output block missing task_id or task_status, an undocumented task_status and a missing output all raise DashScopeVideoError now, instead of defaulting into a queued video with an empty id that callers poll forever. A JSONDecodeError from a gateway response no longer escapes as a generic connection error. task_id and task_status are required on _TaskOutput, and a missing output raises even when the body carries no code or message. Download failures move into the 4xx range, since a 5xx has the proxy retry them: 409 while the task is still running, 400 for a task that failed, 404 for an id that expired. The status a provider reported is kept when it reported one, so an in-body failure that arrives on a 200 is billed as the 400 it is rather than as a retryable connection error. characters and image are only dropped when drop_params is set; the params that are rebuilt into the body or consumed by the caller (model, prompt, user, extra_*) stay silently skipped, since the proxy stamps user from the caller's end-user header. Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): clear the lint gates after merging main * fix(dashscope): accept file tuples and paths as video input_reference video_generation() types input_reference as FileTypes, which includes (filename, content[, content_type[, headers]]) tuples and pathlib paths, but the DashScope encoder only read bytes and file objects, so those image-to-video calls raised before reaching DashScope. Bare strings stay URLs and are never opened as paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dashscope): snap size to a supported ratio and read file tuples and paths * test(dashscope): assert whole mapped values and bill against hand-written rates The tier pricing test read its expected rate from the same map the cost calculator uses, so a wrong rate passed. Billing is now checked against fixed rates driven through the status response usage, while the map test only asserts every supported tier has its own rate. Size mapping tests compare the full mapped dict, and the single-test wiring class is a plain function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(videos): download content through the caller's client and stop retrying dead tasks Two Cursor Bugbot findings on the DashScope video PR. The DashScope artifact lives on OSS, so transform_video_content_response makes a second request. It built its own cached client, discarding an injected client and its ssl_verify. Thread the client through the content response the way transform_video_status_retrieve_response already does, and use it in the three providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The remaining providers accept it and read the response body, matching how they already accept an unused client on status retrieve. A FAILED or CANCELED task reported without a code or message fell past the in-flight check into the 502 branch, so the proxy retried a task that can never produce a video. Raise 400 for a terminal status instead. * fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes A download that lands while the task is still PENDING or RUNNING answered 409. That reads as a client error, which was the intent, but litellm's retry predicate counts 409 as retryable, so the proxy polled DashScope twice more before giving up and then counted the caller's own early download against the model's error rate 404 is what the OpenAI videos API documents for content that is not ready, and it is the one status in that family the retry predicate leaves alone. The rest of the mapping is unchanged: 400 for a task the provider reported as failed, 404 for an expired task id, 502 when a succeeded task carries no artifact The download tests now drive avideo_content and assert the caller sees NotFoundError rather than an internal server error * fix(dashscope): route a video poll back to the model group that created the task The create response encoded the upstream model name into the returned id. Both the dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts, so a proxy with a deployment per region had deployments that only the model group told apart. Encoding the bare model name resolved a poll to whichever deployment the router lists first, which sent it to the other region's host with that region's key and a task id that region never issued. That is the reroute the earlier proof records as the dashscope and qwen_ai_platform poll ids coming back tagged qwencloud The id now carries the model group the create was routed to, which is what the proxy resolves a poll back to, and falls back to the model name for a direct SDK call that has no router The regression test drives a real router with one deployment per region, under a different prefix and under the same one, and asserts the poll carries the key of the region that created the task * fix(dashscope): read the routed model group from litellm_metadata The router stamps the model group it picked into litellm_metadata. metadata only carries a copy when the caller sent no metadata of its own, and every proxy request sends some, so a proxied create still encoded the bare model name and the poll resolved to whichever region's deployment the router lists first The regression test now also creates the video with caller metadata, which fails without this change * fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients * fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body * fix(dashscope): check video request values before the billed create and honor per-call drop_params Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task, pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and refuse to map size onto models litellm has no price for. * fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll A key scoped to an alias or to a name a wildcard serves could create a video and then got a 403 on the poll and the download, because the id carried the group that served the call (wan3, qwencloud/*) and the key was never given that name. The id now carries the name the router was asked for on that hop, and the router resolves an alias, routing group or wildcard name in an id back to itself when no deployment name or provider model matches it first * fix(dashscope): keep a routing group's video polls on the member that created the task A routing group spans several model groups, so an id carrying the group's name load-balanced its polls, and some left for another region's host and key with a task id that region never issued. The id now carries the member that served the task when the request named a routing group, and keeps the requested name for a model group, an alias, or a wildcard match. * fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region * fix(proxy): pin a video's served group only after key and team aliases resolve * test(integration): audit DashScope video create, poll, download, routing and chaos paths * fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group * test(routing): expect a forged served group to be dropped from the polled video id * test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 10:10:07 -07:00
devin-ai-integration[bot]	e226016aee	test(integration): cover health_routing endpoint contracts (#44744) ... * test(integration): cover health_routing endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten health_routing contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format observed routing fallback cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing final review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): resolve health_routing review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 06:34:22 +00:00
yuneng-jiang	ec58b889cc	test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) ... Each test below passed or failed on the same commit depending on test order or port timing in the CircleCI integration workflow - readiness probe only polls a port the launched proxy owns, so a stolen port no longer sends GET /health/integration-readiness to another test's wire server - team update tests seed fresh member users, since a proxy admin relocation adds team members to the destination org and leaked org_b membership into the shared INTERNAL_USER - realtime whisper tests wait until every proxy worker serves only the new gpt-realtime-whisper deployment, via a per-worker deployment id route - configured fallbacks test runs on its own proxy and database, because /config/update persists router settings and cannot unset fallbacks, which left [] and changed the no-fallback error text other tests assert - otel excluded services tests each get a private Redis so a sibling proxy cannot warm the budget registry cache the tenant span depends on - Langfuse audit upstream uses unique response ids instead of len(targets), which could repeat under concurrency and drop SpendLogs rows	2026-10-13 09:55:11 -07:00
devin-ai-integration[bot]	7efce7f8e0	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) ... * test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-12 05:27:33 -07:00
devin-ai-integration[bot]	ca0ff24726	fix(databricks): call the ai_decide AI Function route for decisions and the auto-router (#45722) ... * fix(databricks): call the ai_decide AI Function route for decisions and the auto-router Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not on a serving endpoint's /invocations route. The decisions provider and the auto-router classifier now accept only the ai_decide model, post to that route without a model field, unwrap the response envelope, and map a noul answer's probability to the System One noul field. * test(databricks): bound the megabyte model name test by scaling, not wall time * test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound * test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route * fix(databricks): keep serving endpoint decisions beside the ai_decide route * test(databricks): assert the whole classifier request body in the test routing case * test(databricks): correct System One spend call type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 23:58:16 -07:00
Shivam Rawat	15154a40a8	fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back (#45459) ... * fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): sync per-model rate limit API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type per-model fallback setting access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the per-model hard cap inside the fallback loop and test it through the real limiter * fix(proxy): list the per-model hard cap setting on the settings page and cover every per-model descriptor * fix(proxy): read disable_fallbacks_on_per_model_rate_limits as a boolean value and cover the team-wide per-model cap * test(proxy): cover per-model rate limit 429s skipping fallbacks on a live proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-09 23:10:23 -07:00
Mateo Wang	883e3210ee	feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider (#45200) ... * feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider * fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/ * fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor * fix(ui): name every OSS classifier provider in the classifier radio copy * feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry * fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry * fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router A /model/new for an auto router lands on one replica. A request routed to that auto router on a sibling replica missed it, the model read-through loaded only the auto router's own row, and the pre-routing hook then picked a tier the sibling had not loaded, so the caller got a 400 "no healthy deployments for <tier>" until the next periodic reload. An auto router miss now runs the full add_deployment reconcile, tiers included. Found by the audit's two-worker rig on this PR's routing cells; CircleCI's one-worker proxy never opens the window. * test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing cells for the complexity router's databricks classifier on all three endpoints with the Postgres rows read back, management cells for the classifier's spend writes and for the evaluation-mode health check of a Databricks serving endpoint.	2026-10-09 16:19:23 -07:00
yuneng-jiang	cddc7cde97	test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) ... * test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone	2026-10-09 11:40:14 -07:00
yuneng-jiang	72ab736863	test(routing): wait on deployment registration instead of boot model-info traffic (#45267)	2026-10-08 21:31:53 -07:00
devin-ai-integration[bot]	50ab9a3211	fix(token_counter): price a base64 PDF document per page instead of as one image (#45301) ... * fix(token_counter): price a base64 PDF document per page instead of as one image A `document` or `file` block carrying inline PDF bytes was priced like a single image (85 tokens), so when a provider's count-tokens endpoint rejected the model (Bedrock Opus) the local fallback answered 116 for a 12-page PDF the provider then billed at 35941 input tokens. The counter now reads the PDF with pypdf and prices each page as its extracted text plus the image Anthropic renders it to (1568 px long edge, 1.15 MP, 750 pixels per token), falling back to the old image pricing when pypdf is missing or the bytes are not a readable PDF. * refactor(token_counter): count PDF pages as they are read Sum each page's text and rendered-image tokens straight from the pypdf reader instead of materializing a page list first, keep the fallback to image pricing atomic when a page cannot be read, and annotate the new tests' locals as Final * test(integration): audit cells for page-priced PDF documents in count_tokens, pre-call checks and spend * test(integration): release the held peer when the concurrent budget wait times out --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 21:20:46 -07:00
devin-ai-integration[bot]	8a176c0f0a	fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393) ... * fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt An order-based or configured fallback hop replayed the failed provider's encrypted reasoning items to the next deployment, which answered 400 (Bedrock Mantle: invalid encrypted reasoning; OpenAI: invalid_encrypted_content), so every multi-turn Responses fallback for Codex-style clients failed. The hop now drops the encrypted reasoning its target cannot decrypt and keeps each item's readable summary. With encrypted_content_affinity on, the pin narrows to the hop's target order instead of emptying it, so the hop reaches the next order instead of failing with no deployments available. * fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt Unmarked encrypted reasoning on a hop is attributed to the deployment that just failed, read from the retry breadcrumb, so a hop to a deployment on the same api_base and api_key keeps it and a cross-provider hop still drops it. The hop tests script the upstream at the httpx boundary instead of doubling the handler, and the router coverage script lists the two hop helpers with their tests * fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot * test(router): use a real Router without the origin deployment in the hop strip test * test(router): cover the hop strip when no failed deployment is known * fix(proxy): drop the router's fallback hop state keys from the client body * fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys * test(integration): audit cells for the fallback hop encrypted reasoning strip Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 17:30:01 -07:00
devin-ai-integration[bot]	d5173e3d70	test: move 76 live top-level tests to offline unit and integration coverage (#45352) ... * test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 10:33:35 -07:00
devin-ai-integration[bot]	9df3d4c96b	test(integration): boot owned proxies with a readiness-sized worker healthcheck budget (#45292) ... * test(integration): boot owned proxies with a readiness-sized worker healthcheck budget * test(integration): keep owned_proxy_process's extra_arguments for cells that pass their own flags --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 23:40:26 -07:00
Mateo Wang	a7ee038592	fix(router): keep include_fallback_errors off the provider call on the sync Router path (#45218) ... * fix(router): keep include_fallback_errors off the provider call on the sync Router path Router.completion spread include_fallback_errors into litellm.completion, so OpenAI answered 400 Unknown parameter and a fallback walk lost the backup's reply. Only Router._acompletion popped the router-only flags. Both paths now build the provider call through one shared helper that drops them, and a wire-level respx test covers each path. * test(router): pin include_fallback_errors off the wire across the sync, async, and proxy paths	2026-10-07 22:25:12 -07:00
RachelHuangZW	bf9f59d76b	fix(scheduler): remove a request's queue entry once it stops waiting (#43061) ... * fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 17:34:38 -07:00
devin-ai-integration[bot]	e98bbc2f8e	fix(exceptions): map upstream 402 to PaymentRequiredError and cool down 402 deployments (#44879) ... * fix(exceptions): keep upstream 402 status and cool down 402 deployments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): map 402 to PaymentRequiredError subclass of BadRequestError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): annotate PaymentRequiredError methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): skip 402 cooldown on single-deployment model groups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): single prefix and 402 fallback response for PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): map billing_error to PaymentRequiredError regardless of status Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): honor explicit allowed-fails policy for single-deployment 402s Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(anthropic): drop billing_error body mapping to PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): type the PaymentRequiredError constructor parameters * test(integration): cover 402 PaymentRequiredError mapping and cooldown --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-06 20:35:13 -07:00
devin-ai-integration[bot]	68d9b8bbb8	fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) ... * fix(router): retry a /v1/messages stream the provider drops before the first content chunk A /v1/messages stream that the upstream closed before any content reached the client answered an error event after a single attempt, so the router's num_retries never applied to that drop. The pre-content failure is now retried within the model group before the fallback chain runs, with the budget resolved the way a failure raised before the stream opened resolves it: a retry policy that names the error class, then the request's num_retries, then the deployment's, then the router's. A drop after content reached the client keeps surfacing the provider's error after one attempt. Fixes #44238 * fix(router): hand a retry's non-retriable error to the fallback chain and type the retry helpers A retry that failed before its stream opened with an error no retry covers raised straight to the client, skipping a fallback the first attempt would have used. assert_never now comes from typing_extensions so the router imports on Python 3.10, and the retry helpers read their kwargs through typed narrowing instead of Mapping[str, Any] * fix(router): cast the untyped router fallback defaults the stream retry gate reads The retry gate passed the router's fallback attributes, declared without element types, to the typed request override helper, which basedpyright counted as new unknown-argument errors * fix(router): consult context_window_fallbacks when a retried /v1/messages stream overflows A retry attempt raising ContextWindowExceededError reached the fallback chain inside its mid-stream envelope, so only the regular fallbacks list matched. The fallback attempt now unwraps it the way it unwraps a content policy error. The new router helpers are covered for the router code coverage check with two direct-call tests and named covering tests * fix(router): retry a 408 raised by a /v1/messages retry and honor deployment num_retries before the stream opens * fix(router): attribute a retried /v1/messages stream to the deployment that served it and bound the retry-policy hold * fix(router): retry /v1/messages error frames under their retry-policy class and keep the first drop's committed budget An `event: error` frame that arrives before the first content delta now raises the exception class the pre-stream mapping gives an HTTP answer with the same status (429 RateLimitError, 500 and 529 InternalServerError, 503 ServiceUnavailableError, 504 Timeout), so a retry policy's per-class budget governs it the way it governs the error before the stream opened. The status the client sees is unchanged A retry that lands on a sibling deployment keeps the budget the first drop committed to, read back from the request's attempted_retries and max_retries, instead of recomputing it from the new deployment's num_retries, matching the pre-stream retry loop * refactor(anthropic): keep the error-frame exception mapping under llms and type the retry test helper The status-to-exception mapping an `event: error` frame gets before the retry policy is consulted now lives next to the Anthropic error status map in llms/anthropic/common_utils.py, with its own unit test, and the two-deployment retry test helper takes explicit typed parameters instead of a bare dict and untyped kwargs * refactor(anthropic): map an error frame's status with explicit returns on every path * fix(router): map stream error frames through the pre-stream exception mapping An overloaded `event: error` frame on a /v1/messages stream now raises the InternalServerError a 529 answer maps to, built by exception_type from the frame's own body, so one retry policy class governs the error before and after the first byte; a failed fallback after such a frame answers 500 like every other litellm path instead of the frame map's 503 A model_group_retry_policy that does not parse (a non-integer budget, an entry that is not a mapping) no longer fails every healthy stream of that group before its first attempt: the stream runs with no policy and the plain num_retries budget, with a warning naming the group * fix(router): forward an error frame nothing can take over for as the provider sent it A pre-content error frame whose class the retry policy grants no retry, with no fallback configured, raised an HTTP error only on the first attempt while the same frame after exhausted retries reached the client verbatim. Both now pass through as sent, the way the merge base forwarded every frame. * test(integration): audit /v1/messages pre-content retry across routes and budgets Adds the /audit cells for the pre-content stream retry: the native Anthropic route (drops and error frames before content, HTTP rejections before the stream opens, SDK sync and async, after-content and non-retriable controls, budget exhaustion, cache twin, spend row and headers), the chat and responses bridges, the generic routes (responses, chat, vllm pass-through, Gemini generateContent, fine-tuning jobs list), owned two-worker proxies for router-level budgets, retry policies and fallbacks, and two chaos cells (a worker killed mid burst, an outage on every first attempt). Shared helpers for scripted Anthropic SSE upstreams and OpenAI-compatible wire replies live in tests/integration/_support --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 22:53:39 +00:00
devin-ai-integration[bot]	3286782dea	fix(caching): key response cache by router model group in litellm_metadata (#44542) ... * test: align integration fixtures with current behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the spend flush before asserting its trace placement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): key response-cache entries by model group from litellm_metadata /v1/responses routes through the router with model_group in litellm_metadata, which the cache key ignored, so identical requests to different model groups sharing one underlying model hit each other's cached responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-05 17:06:21 +00:00
devin-ai-integration[bot]	9d16412341	fix(caching): count tool_call cache_control marks in the injection census (#43556) ... * fix(caching): count tool_call cache_control marks in the injection census Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): remove cache census casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): only skip injection on message or content marks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): skip injection on messages whose tool calls carry marks Reverts 9f08d8aef8. A default 5m mark injected on assistant text lands before the client's 1h tool_use mark, which Anthropic rejects with a 400 because a 1h breakpoint must not follow a 5m one. Keeping the full census in the skip check leaves the client's tool_call breakpoint as the only one on that message. * fix(caching): count every client tool_call cache mark in the breakpoint census The census gated tool_call marks on type function and dict shape, so a client mark on a call without a type or with a string cache_control slipped past the count and injection overflowed the 4 breakpoint cap. Count any non-None tool_call mark, keep the server tool exclusion, and add integration cells for the capped surfaces, the yaml stand-down, Bedrock and Gemini, and router affinity * test(integration): hold the upstream so the worker kill lands mid-burst * refactor(caching): reuse the transform's server tool lookup in the breakpoint census The census now calls the same helper the Anthropic transform uses to decide whether a marked tool call becomes a server tool block, so the two cannot drift apart. The owned-proxy burst test waits up to 90 seconds for the burst to reach the wire before it kills a worker * refactor(anthropic): move the server tool rebuild check under llms/anthropic * test(integration): audit the tool call mark census across chat, messages, responses, and chaos Adds the /audit cells for the breakpoint census on assistant tool_calls marks: the Responses stream bridge, the OpenAI and Anthropic SDK clients, in-process Pydantic messages, response cache twins, request-level points, a provider 401 on a capped request, malformed provider_specific_fields and tool_call ids, null or empty points, a points update mid-burst, a proxy restart mid-burst, and a worker SIGKILL that picks the worker holding the burst's upstream connections * test(integration): close the SDK clients the cache census cells open --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-03 17:33:48 -07:00
devin-ai-integration[bot]	e9bf2cfd01	refactor(types): replace Any with proven types in 9 files (#44389) ... * refactor(types): replace Any with proven types in 13 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep provider error paths for malformed prefetch and poll JSON Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's login body parsing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's Copilot auth and budget alert typing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): audit cells for Any sweep 20261003_2 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): load audit video deployments at proxy start Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's video-edit prefetch handling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fix audit chaos Responses cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): probe every route after audit worker kill Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 21:03:16 +00:00
devin-ai-integration[bot]	fe9b6fd603	refactor(types): replace Any with proven types in 7 files (#43844) ... * refactor(types): replace Any with proven types in 9 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin xecguard-adjacent guardrail retry, mcp mixed tools, jwt routing and complexity router paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep only live-provable Any removals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): type cache_hit as bool | None on the sync success path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-02 02:47:47 -07:00
yuneng-jiang	a93c396a5c	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) ... * test(integration): move legacy proxy, router and Redis tests into tests/integration Port 39 legacy tests to the integration tier that owns them, running against the scripted upstream, local Postgres and Redis, test-owned wire peers and owned proxies. Delete 5 legacy tests whose contract is already owned by an existing integration test, and remove the legacy functions, files and helpers left unused. * test(integration): cover recovery of a spent key after its budget is raised	2026-10-01 23:00:54 -07:00
yuneng-jiang	65a316fc92	ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) ... * ci(circleci): test Redis behavior against local Redis and print short tracebacks The redis_caching_unit_tests job ran three legacy files against the shared remote Redis. The DualCache and batch-read logic that never needed a server now lives in tests/unit/caching/test_dual_cache.py with a mocked RedisCache, and the behavior that does need one (the increment-with-floor Lua script, read-through, deletes, batch reads) moved to tests/integration, which starts a local Redis. test_returned_settings only read REDIS_PORT and is replaced by a unit test of Router.get_settings CircleCI pytest runs now use --tb=short so failure output stays readable in the test results tab * ci(integration): print short tracebacks from run.py and allow Redis in the sdk shard	2026-10-01 13:00:54 -07:00
yuneng-jiang	bba85f0b6c	chore(lint): remove the LIT002 mutable-construction rule (#43971) ... * chore(lint): remove the LIT002 mutable-construction rule Drop LIT002 from scripts/check_type_discipline.py along with its helpers, its budget entry, its unit tests, and the AGENTS.md and gate docstring mentions. `# mutable-ok` now only suppresses LIT001, so the markers that only existed to silence LIT002 became LIT013 stale suppressions and are removed. The files whose layout depended on those trailing comments are reformatted with ruff format. Every other LIT rule count is unchanged and the ASTs of all touched litellm/ files match main apart from one docstring. * chore(lint): keep the leftover mutable-ok markers for a follow-up Restore the ~1.4k `# mutable-ok` markers stripped in the previous commit so this PR only touches the checker, its tests, the budget, and docs. Those markers no longer suppress anything, so `# mutable-ok` is exempt from LIT013 until a follow-up strips them. * Revert "chore(lint): keep the leftover mutable-ok markers for a follow-up" This reverts commit c35bc0b84e. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-01 12:24:02 -07:00
devin-ai-integration[bot]	6bc17f98d7	refactor: clean up fresh tech debt from 2026-09-29 (#43830) ... * refactor: clean up fresh tech debt from 2026-09-29 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(routing): pin usage-based routing Redis reads through the proxy and SDK Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-30 02:40:37 -07:00
devin-ai-integration[bot]	e450f2d54c	fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015) ... * test(integration): reproduce encrypted_content_affinity 503 when origin has no boundary peer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep encrypted content affinity turn one out of the response cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): degrade encrypted_content_affinity when the origin has no encryption-boundary peer A routed-group candidate that is currently unavailable and shares its (api_base, api_key) with no healthy deployment used to raise a proxy-level 503/429 from _unavailable_origin_error, even though healthy siblings in the same model group could still serve the turn. Strip the encrypted reasoning and dispatch to the healthy pool instead, matching the existing cross-group behavior, and log a warning naming the origin model_id and routed group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): bound the degraded-affinity log marker and assert the strip on the wire Address review findings: restore num_retries alongside optional_pre_call_checks in the integration test teardown, record scenario request bodies on the scripted upstream so the tests can assert no encrypted reasoning reaches the sibling, and truncate the client-supplied model_id in the degraded-dispatch warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): only record JSON bodies on the scripted upstream Multipart uploads to scripted POST routes have no JSON body, so gate the observation recording on the request content-type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): make encrypted_content_affinity runtime-toggleable so /config/update can turn it off --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-09-24 17:31:43 -07:00
devin-ai-integration[bot]	b72a030501	test: take keys out of the legacy proxy, enterprise and mcp unit tests before moving them (#42901) ... * ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 14:56:52 -07:00
devin-ai-integration[bot]	3fb6f8740b	test(integration): add read-replica routing harness to the CircleCI integration suite (#42692) ... * test(integration): add read-replica routing harness * refactor(integration): hoist the maintenance url imports * fix(integration): keep per-test databases and the witness sequence readable under replica roles * fix(integration): opt bespoke database and pool tests out of the injected read replica * test(integration): commit recorded replica routing expectations * fix(integration): judge routing by role containment so shrinking role sets do not fail * fix(integration): run the pool-limit shutdown choreography on the superuser database url * ci(integration): add the mcp group to the replica matrix * fix(integration): judge routing by exact role sets with a named either-role allowlist * test(integration): drop containment-era routing expectations for re-recording * chore(integration): drop docstrings from the replica harness scripts * docs(integration): describe exact routing matching and the either-role list * test(integration): record exact replica routing expectations * test(integration): allow the SELECT 1 health probe on either role * test(integration): replace committed routing expectations with an on-demand base-vs-head parity run * test(integration): fix parity env scope, readme wording, and seed-deterministic serialization test * test(integration): make the sorted-role serialization test deterministic in-process * test(integration): swap all product code in parity runs and pin role gains * ci(integration): force tracked-file removal before parity checkout --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 00:25:03 -07:00
devin-ai-integration[bot]	a286ebf42e	test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) ... * test(integration): optional Anthropic tool properties stay optional on the OpenAI Responses wire (Pylon #6619) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock InvokeModel count-suffixed cache usage fields are reported and charged (Pylon #6708) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): anthropic messages honors the deployment request timeout (Pylon #6505) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop client_metadata before the Bedrock Converse body reaches the provider (Pylon #6645) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): repeat Bedrock requests under one session name assume the role once (Pylon #6681) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): messages stream keeps include_usage off the Responses wire with always_include_stream_usage (Pylon #6466) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): clamp sub-16 max_tokens to the Responses API floor instead of 400 (Pylon #6539) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): forwarded client x- headers reach the provider on /v1/responses (Pylon #6565) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Anthropic messages stop_sequences reach OpenAI-compatible providers as stop (Pylon #6536) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex namespace tools reach a chat upstream flattened and round-trip through /v1/responses (Pylon #6409) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep Claude 4.6 legacy thinking budget_tokens on /v1/messages (Pylon #6727) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): nvidia nim ranking keeps image passages and applies top_n without sending top_k (Pylon #6401) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tpm-only model rejects priority traffic once recorded tokens reach the model tpm (Pylon #6344) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): reasoning-only chunks open an Anthropic thinking block at index zero on /v1/messages streams (Pylon #6337) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): file content streams to the client before the upstream finishes sending (Pylon #6315) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): agent whose card lives only at agentCard/v1.0 is reached with bearer auth (Pylon #6249) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): vertex batch create returns a batch when outputInfo is null (Pylon #6374) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fireworks session id is sent as x-session-affinity and cached tokens land in spend log metadata (Pylon #6220) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor sub-call failure does not cool down the executor deployment (Pylon #6212) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Gemini /v1/messages cache_control creates cachedContent with Anthropic ttl (Pylon #6221) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock_mantle max_output_tokens below 16 is clamped before reaching Mantle (Pylon #6262) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): missing thinking signature 400 on /v1/messages retries without thinking blocks (Pylon #6222) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): format Gemini messages cache_control wire test (Pylon #6221) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): rebuilt shared aiohttp session keeps the configured keepalive timeout (Pylon #6387) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): sagemaker_chat signs the inference component header and sends hf_model_name as the body model (Pylon #6187) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Converse DeepSeek drops Anthropic thinking and sends V3 reasoning_effort raw (Pylon #6149) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): concurrent team model TPM requests are reserved before the provider call (Pylon #6075) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): /v1/messages honors the configured timeout against a stalled upstream (Pylon #6025) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex additional_tools input items reach Bedrock Mantle as top-level tools (Pylon #6012) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor api_base without api_key never sends the proxy Anthropic key to the caller host (Pylon #6226) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): rerank responses carry call id, latency and cost headers (Pylon #5981) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): parse the outbound Anthropic body with the typed JSON adapter (Pylon #6025) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Knowledge Base search forwards userContext to the Retrieve body (Pylon #5991) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Marengo 3.0 text embeddings reach Bedrock nested under inputType (Pylon #5949) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): sub-16 max_tokens over a responses deployment reaches OpenAI as 16 (Pylon #6008) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): midturn system correction reaches the OpenAI Responses wire via /v1/messages (Pylon #6449) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): concurrent requests over a key tpm limit are rejected before reaching the provider (Pylon #5737) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): chat to responses bridge keeps deployment AWS credentials for Bedrock Mantle SigV4 (Pylon #5870) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): vertex gemini stream split across many fragments completes without stalling the proxy (Pylon #5838) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock mantle /v1/messages stream keeps stream true and relays SSE events (Pylon #5596) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): large chat payloads are released from worker memory after the request ends (Pylon #5920) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streaming success logs v3 rate limit remaining values for callbacks (Pylon #5767) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): a database created search tool backs Anthropic web search interception (Pylon #5669) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): register july provider regression contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): make july provider regression tests deterministic under cache and worker sharing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop order-fragile worker memory probe pending a real retention regression check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): apply ruff import sorting and formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop stale contract entry and pass question to advisor executor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): use tiktoken-backed executor model in advisor tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-23 09:51:06 -07:00
devin-ai-integration[bot]	5c0b374f0a	test(integration): regression tests for August provider translation and streaming bugs (#42621) ... * test(integration): Bedrock batch files upload completions and responses records as user messages (Pylon #6882) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): client Anthropic OAuth token never replaces Bedrock SigV4 authorization (Pylon #6888) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bridge /v1/messages and /v1/responses streams through empty-choices chunks (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prepend azure content-filter metadata chunk to the messages stream (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fireworks routers/ slug reaches the provider as accounts/fireworks/routers/<id> (Pylon #7030) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock hidden thinking tokens are not reported as text tokens (Pylon #7067) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai FLUX.2-flex image generation targets the flex provider path with the BFL body (Pylon #7092) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep Databricks streaming usage and cache reads in the client stream and spend log (Pylon #7094) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai-compatible image edits forward provider-specific form fields to the backend (Pylon #7122) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): replayed intercepted web search turn reaches Bedrock as text through /v1/messages (Pylon #7181) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed web search turn capped by max_agentic_loops ends the turn with snippets and ordered blocks (Pylon #7230) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock rerank keeps forwarded client headers out of the SigV4 signature (Pylon #7284) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai rerank authenticates with an Entra token when no api key is set (Pylon #7303) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): perplexity stream with cost breakdown object completes and bills total_cost (Pylon #7331) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai strips Anthropic message fields before the Foundry request (Pylon #7336) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): capped intercepted web search ends the turn without an internal tool_use block (Pylon #7378) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock passthrough converse-stream keeps event-stream content-type (Pylon #7482) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): v1/messages success exposes v3 priority rate limit headers (Pylon #7532) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): config deployment dropped by a stale boot cost map is restored after reload (Pylon #7564) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prefixed Opus 4.8 reasoning_effort reaches Bedrock as adaptive thinking (Pylon #7586) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): dashscope chat forwards reasoning_effort to the provider (Pylon #7606) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): failing stream logging callback still releases the max_parallel_requests slot (Pylon #7608) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gen 5 Claude Bedrock Invoke tool search sends the Bedrock beta field (Pylon #7642) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): databricks ai gateway api_base requests OAuth token from workspace origin (Pylon #7724) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): deepseek vision image content list reaches the provider unchanged (Pylon #7729) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle context overflow surfaces as 400 prompt is too long (Pylon #7732) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex history items reach Bedrock Mantle as supported Responses input types (Pylon #7783) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): chat over responses deployment returns finish_reason length when output tokens run out (Pylon #7784) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock_mantle rewrites Codex history items before the Responses wire (Pylon #7812) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor sub-call on /v1/messages uses the configured advisor deployment (Pylon #7828) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tencent thinking reaches the provider body instead of failing the request (Pylon #7834) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): xAI chat web search reaches /v1/responses with instructions and nested filters (Pylon #7835) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send Bedrock Converse config blocks once at top level (Pylon #7839) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock converse sends gpt-5 reasoning_effort as reasoning.effort (Pylon #7850) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock cohere.embed-english-v3 embeddings accept encoding_format and dimensions (Pylon #7963) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed chat completions emit SSE keepalive pings while the upstream is silent before its first token (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format the TTFT keepalive regression test (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai chat drops tool_choice when the request has no tools (Pylon #8022) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): stream whose first chunk has no choices falls back and bills the fallback (Pylon #8006) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583)" This reverts commit 864b65811f. * test(integration): reconcile contracts manifest for bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): isolate proxy config writes in bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review feedback on keepalive, cost map reload and websearch order Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-22 20:52:00 -07:00